Unit testing Cocoa user interfaces: Cocoa Bindings

About a year ago, I wrote about unit testing target-action connections for Cocoa user interfaces. That covers the traditional mechanism by which user interfaces have typically been constructed in Cocoa since the NeXTstep days. However, with the release of Mac OS X 10.3 Panther we’ve had a newer interface technology available — Cocoa bindings — which has presented some interesting application design and testing challenges.

Among other hurdles, to properly use Cocoa bindings in your own applications, you need to ensure that the code you write properly supports key-value coding and key-value observing. However, since the release of Mac OS X 10.4 Tiger, the necessary APIs have been available to easily do test-driven development of your application’s use of Cocoa bindings, following a trust, but verify approach. (It’s also been quite easy from the start to test your support for key-value coding and key-value observing, to ensure that your code meets the necessary prerequisites for supporting bindings. I can write more on this topic in another post if anyone is interested.)

The key to writing unit tests for Cocoa bindings is the -infoForBinding: method in AppKit’s NSKeyValueBindingCreation informal protocol. Using this simple method, you can interrogate any object that has a binding for all of the information about that binding! It simply returns a dictionary with three keys:

  1. NSObservedObjectKey, which is the object that the binding is bound to;
  2. NSObservedKeyPathKey, which is the key path that is bound — in Interface Builder terms, this is the controller key path combined with the model key path, with a dot in between them; and
  3. NSOptionsKey, which is a dictionary of additional binding options unique to the binding. These are all of those additional checkboxes and pop-ups in the Interface Builder bindings inspector for setting things like a value transformer.

By specifying what this dictionary should contain for a particular binding, you can describe the binding itself and thus start doing test-driven development of your Cocoa bindings-based user interface. Note that all of the system-supported binding names — as well as the binding option names — are specified in <AppKit/NSKeyValueBinding.h> and are documented, too!

Let’s take a simple example, like the one in last year’s target-action example, of a window controller whose window has a static text field in it. The field should have its value bound to the name of a person through an object controller for that person. Assume that I’ve already created the test case and set up some internal methods on my window controller to refer to the contents of the window via outlets, and to load the window (without displaying it) in -setUp just like in the target-action example.

First, to see that my text field has a value binding, I might write something like this:

- (void)testPersonNameFieldHasValueBinding {
    NSTextField *personNameField = [_windowController personNameField];

    NSDictionary *valueBindingInfo = [personNameField infoForBinding:NSValueBinding];
    STAssertNotNil(valueBindingInfo,
        @"The person name field's value should be bound.");
}

Of course, this tells us nothing about how the binding should be configured, so it needs some fleshing out…

Let’s check the object and key path for the binding.

- (void)testPersonNameFieldHasValueBinding {
    NSTextField *personNameField = [_windowController personNameField];

    NSDictionary *valueBindingInfo = [personNameField infoForBinding:NSValueBinding];
    STAssertNotNil(valueBindingInfo,
        @"The person name field's value should be bound.");

    NSObjectController *personController = [_windowController personController];
    STAssertEquals([valueBindingInfo objectForKey:NSObservedObjectKey], personController,
        @"The person name field should be bound to the person controller.");

    STAssertEqualObjects([valueBindingInfo objectForKey:NSObservedKeyPathKey], @"name",
        @"The person name field's value should be bound to the 'name' key.");
}

Not very exciting, and a little verbose, but it'll easily lead us through what needs to be set up in Interface Builder for this binding to work. If you want to cut down the verbosity, you can of course extract a method to do the basic checking...

- (BOOL)object:(id)object shouldHaveBinding:(NSString *)binding
            to:(id)boundObject throughKeyPath:(NSString *)keyPath
{
    NSDictionary *info = [object infoForBinding:binding];

    return ([info objectForKey:NSObservedObjectKey] == boundObject)
            && [[info objectForKey:NSObservedKeyPathKey] isEqualToString:keyPath];
}

- (void)testPersonNameFieldHasValueBinding {
    NSTextField *personNameField = [_windowController personNameField];
    NSObjectController *personController = [_windowController personController];
    
    STAssertTrue([self object:personNameField shouldHaveBinding:NSValue
                           to:personController throughKeyPath:@"name"],
    @"Bind person name field's value to the person controller's 'name' key path.");
}

If you're writing code that needs, say, a value transformer, it's a simple matter to extend this model to also check that the correct value transformer class name is specified for the NSValueTransformerNameBindingOption key in the binding options dictionary returned for NSOptionsKey.

You can even extract these kinds of checks into your own subclass of SenTestCase that you use as the basis for all of your application test cases. This will let you write very concise specifications for how your user interface should be wired to the rest of the code, that you can use to just walk through Interface Builder and connect things together — as well as use to ensure that you don't break it accidentally by making changes to other items in Interface Builder.

This is the real power of test-driven development when combined with Cocoa: Because you can trust that the framework will do the right thing as long as it's set up right, you simply need to write tests that specify how your application's interface should be set up. You don't need to figure out how to create events manually, push them through the run loop or through the window's -sendEvent: method, how to deal with showing or not showing the window during tests, or anything like that. Just ensure that your user interface is wired up correctly and Cocoa will take care of the rest.

Easily speed up your Xcode builds

A lot of developers don’t heavily modify the projects they instantiate from Xcode’s templates. This is a shame, because not only is it a great way to learn Xcode in depth, it’s also a great way to ensure your projects build as fast as possible!

To that end, then, here are two simple tips that you can apply to your projects right now that should get them building faster.

Normalize your build settings!

Projects and targets in Xcode have configurations — collections of build settings that influence how a target is built. The Debug configuration, for example, will typically specify the compiler optimization build setting with as low a value as possible. The Release configuration, on the other hand, will typically specify a relatively high value for this build setting. This begs the question: The Debug and Release configuration of what?

When you create a new project from one of Xcode’s templates, the principal target in that project will typically have a number of build settings customized in its configurations. The project itself, on the other hand, won’t have very may customized build settings. For the one-target-per-project case this doesn’t matter much. However, if you create a project with multiple targets, you can wind up with a number of targets that specify the same (or even subtly different!) information.

Instead of leaving your project this way, you can normalize your build settings such that build settings you want to specify for every target in the project — for example, that compiler optimization should be off for the Debug configuration — are specified at the project level rather than at the target level. This takes advantage of the fact that build settings are inherited in Xcode; if a build setting isn’t customized in a target, the value specified in the project is used.

What does this buy you? It ensures that you have a single, consistent set of settings that are passed to the compilers, linkers, and other tools that are used to build your software. That way you won’t wind up with subtle bugs like code built with C++ RTTI turned on calling a plug-in built with C++ RTTI turned off. But just as importantly, it’s enables the next trick, which can have a significant impact on the speed of large, multi-target builds.

Share your precompiled prefix!

Xcode, like many other IDEs, supports prefix files — header files that are included implicitly by every one of your source files when they’re built. Normally, as I described above, these are specified in target configurations. The text in the prefix file that is copied out of Xcode’s templates even mentions that it’s for a specific target.

Don’t be fooled!

Prefix files get precompiled for performance, so they can simply be loaded by the compiler rather than recomputed for every file you build. Precompiling a prefix file takes quite a bit of time though, especially if you use multiple languages in your project (such as C, Objective-C, and C++) because each language needs a separate precompiled prefix due to differences in how they’ll treat “the same” code.

However, just because precompiled prefix files can’t be shared between languages doesn’t mean they can’t be shared between targets. In fact, for performance, Xcode automates this sharing for you — if you meet it halfway. The critical thing that you need to do is to ensure that your prefix files are:

  1. Named the same; and,
  2. Built using the same compiler-related build settings.

That’s where the normalization I talked about above comes in. You can even promote your prefix file-related build settings to your project instead of your targets, so you can be certain that they’re shared.

In fact, if they meet the criteria that Xcode uses to determine whether precompiled prefix files should be shared, even multiple projects will wind up sharing them!

The pause between builds of a target’s dependent targets to generate a new precompiled prefix file is like a pipeline stall: An unwelcome hiccup that holds everything else up until it’s over. If Xcode can precompile a single set of prefix files at the start of your build and then re-use them for the entire rest of your build, it will stream past as quickly as it possibly can with only the occasional pause for linking. For large projects with a lot of dependent targets, this can make a big difference.

Separate your preprocessor macros!

“But Chris,” you say, “I have target-specific preprocessor macros that I can’t get rid of!” That’s OK. As long as you don’t need them in your prefix files, you can set them in the special Preprocessor Macros Not Used in Precompiled Headers build setting. These will be passed to the compiler when your sources are compiled, just like the regular Preprocess Macros will, but they won’t be passed when precompiling a prefix file. So you can have your cake and eat it, too.

Of course, there are macros that you do want to set in your precompiled prefix headers, because they change the behavior. Macros like NDEBUG to turn off C assert or NS_BLOCK_ASSERTIONS to turn off Foundation’s NSAssert are important to specify for your precompiled prefix files. Fortunately these types of macros typically differ only by configuration, and also remain consistent across targets and projects, allowing you to specify them at the project level rather than the target level.

Just these three small changes have the potential to make a dramatic difference in how quickly Xcode builds your project:

  1. Normalizing your build settings, so common settings across all your targets are specified at the project level;
  2. Increasing sharing of your precompiled prefix files by naming them consistently and using consistent compiler-related build settings; and
  3. Specifying separate preprocessor macros for your prefix files than for your targets,

Try it out and see for yourself!

Unit testing Cocoa user interfaces: Target-Action

It’s really great to see that a lot of people are adopting unit testing for their projects and dramatically improving their quality. Test-driven development and agile development methodologies built around it are really taking off. However, a lot of people still feel that their user interface is difficult to test through code, and either requires a capture-playback tool or requires a different design approach based heavily on interfaces/protocols to get right.

In last year’s post Trust, but verify. I tried to dispel some of the mystery of testing your application’s user interface when using the Cocoa frameworks. However, I’ve still had a lot of (entirely well-justified!) requests for examples of how to put it into practice. So here’s a simple example of what I’d do to write a unit test for a button in a window that’s supposed to perform some action.

First, when implementing my window, I’d follow the standard Cocoa pattern of having a custom NSWindowController subclass to manage my window. This window controller will have an outlet connected to each of the views in the window, and will also wind up with a private accessor method — used only within the class and any subclasses, and in testing — for getting the value of each of its outlets. This design flows naturally from the test which I would write to specify that the window should contain a button. First, here’s the skeleton into which I’d put tests:

// TestMyWindow.h

#import <SenTestingKit/SenTestingKit.h>

@class MyWindowController;

@interface TestMyWindow : SenTestCase {
    MyWindowController *_windowController;
    NSWindow *_window;
}
@end

// TestMyWindow.m

#import "TestMyWindow.h"
#import "MyWindowController_Private.h"

@implementation TestMyWindow

- (void)setUp {
    // MyWindowController knows its nib name and
    // invokes -initWithWindowNibName: in -init
    _windowController = [[MyWindowController alloc] init];

    // Load the window, but don't show it.
    _window = [_windowController window];
}

- (void)tearDown {
    [_windowController release];
    _window = nil; // owned by _windowController
}

@end

That’s the infrastructure into which I’d put my other test methods for this window. For example, I’ll want to specify the nib name for the window controller and ensure that it actually knows its window:

- (void)testNibName {
    STAssertEqualObjects([_windowController windowNibName], @"MyWindow",
      @"The nib for this window should be MyWindow.nib");
}

- (void)testWindowLoading {
    STAssertNotNil(_window,
      @"The window should be connected to the window controller.");
}

Now let’s check that I have a “Do Something” button in the window, and that it sends an action directly to the window controller.

- (void)testDoSomethingButton {
    // _doSomethingButton is a private method that returns the button
    // conected to the doSomethingButton outlet
    NSButton *doSomethingButton = [_windowController _doSomethingButton];
    
    STAssertNotNil(doSomethingButton,
      @"The window should have a 'Do something' button.");
    
    STAssertEqualObjects([doSomethingButton title], @"Do Something",
      @"The button should be titled accordingly.");

    STAssertEquals([doSomethingButton action], @selector(doSomething:),
      @"The button should send -doSomething: to its target.");

    STAssertEquals([doSomethingButton target], _windowController,
      @"The button should send its action to the window controller.");
}

You’ll notice something I’m not doing in the above: I’m not simulating interaction with the interface. This is the core of the trust, but verify approach to unit testing of your user interface.

I can trust that as long as I verify everything is hooked up properly that Cocoa will cause the button to send its action message to its target — whether it’s a specific object or, if the target is nil, the responder chain — whenever the button is clicked while it’s enabled and not hidden. I don’t need to simulate a user event, and I don’t even need to display the interface while running the unit tests. All I need to do is inspect, through code, that everything is wired up correctly.

Note that I can do way more than the above in testing my interface design, too. For example, I can ensure that the control layout is correct according to what my interface designer has specified, by checking bounding rectangles for example. But testing only the functionality of my interface has significant advantages, too. For example, it doesn’t matter if I wind up using a custom kind of button to achieve exactly the kind of look and feel or behavior I need. It doesn’t matter if I wind up changing the layout in response to feedback. No matter what I do, I’ll know that functionality won’t accidentally break while I’m messing around in Interface Builder — even if I completely rip out my interface and replace it with a new one!

This approach can also be used for testing Cocoa bindings using the -infoForBinding: method that was introduced in Mac OS X 10.4 Tiger. I hope to write up a post soon on how to approach Cocoa bindings using these same techniques, but it should be fairly straightforward given the above and the above documentation.

Update: I’ve struck through the check of the button’s title above, because you may or may not want to do that. For example, if you’re primarily running your unit tests against your development localization, you may want to put it in. But if you want to run your unit tests against a localized build of your application, you’ll probably want to avoid checking a localized title against an English string. A “have your cake and eat it too” strategy might be to keep a variable somewhere in your application that can be used to selectively disable checks of only localized strings.

Update July 7, 2007: I’ve finally written a post, Unit testing Cocoa user interfaces: Cocoa bindings, on how to write tests for Cocoa bindings. Now there’s no excuse for not doing test-driven development of your Cocoa user interfaces!

“Enterprise” thought leadership?

David Heinemeier Hansson, creator of Rails at 37signals, takes James McGovern — some Java/J2EE author — to task for his über-lame rant against Ruby in the Enterprise in a great post titled Boy, is James McGovern enterprise or what!

> So by Enterprise, Architect, and Enterprise Architect standards, this gent must be the top of the pop. Thus, allow me to make this perfectly clear: I would be as happy as a clam never to write a single line of software that guys like James McGovern found worthy of The Enterprise.

> If Ruby, Rails, and the rest of the dynamic gang we’re lumped together to represent, is not now, nor ever, McGovern Enterprise Readyâ„¢, I say hallelujah! Heck, I’ll repeat that in slow motion just to underscore my excitement: HAL-LE-LU-JAH!

> With that out of the way, we’re faced with a more serious problem. How do we fork the word enterprise? The capitalized version has obviously been hijacked by McGovern and his like-minded to mean something that is synonymous with hurt and pain and torment.

Indeed, McGovern’s rant reads more like a parody of a rant than the real thing:

> 13\. Lets say there is a sixteen week project and the productivity stuff was true and Ruby could save me an entire three weeks which would be significant. Since Ruby is a new vendor and not represented by existing vendors I already do business with, do you think that I will spend more than three weeks in just negotiating the contract?

Yes, because there is some vendor out there named “Ruby that you need to sign a contract with before you can begin a project.

Despite his claims to be agile, McGovern obviously doesn’t know the first thing about agile development. People come first, sure, but agile development doesn’t say that tools aren’t important. Not using good tools makes it harder for good people to do good work.

That’s why I love developing software for Mac OS X and why I love helping people develop software on Mac OS X: We have great tools like Cocoa, Core Data, Interface Builder, OCUnit, WebObjects, and Xcode, and these can be used by great developers to do great things.

Xcode: Working With Xcode Build Settings

The Apple Developer Connection just posted Working with Xcode Build Settings. It builds on Understanding Xcode Projects:

There is a lot of flexibility in the Xcode build system for customization, so you can choose to keep your settings simple or make them as complex as you like. But for most purposes, a basic understanding of how build settings and build configurations work will make this topic easy to master—once you have these concepts clear, you can choose the settings that work best for you.

If you’re developing Mac OS X software with Xcode, read Understanding Xcode Projects and Apple Developer Connection posted Working with Xcode Build Settings

Xcode: Understanding Xcode Projects

Anyone interested in Mac OS X development should check out the latest Xcode article on the Apple Developer Connection, Understanding Xcode Projects:

To become truly proficient with Xcode, whether you’ve been using it for a while or are just starting to make the transition, an understanding of the Xcode user interface and how it is used to organize a software project is required. This understanding will take you a long way towards understanding the philosophy behind Xcode and how it works, and will help to make it work better for you.

In short, the article introduces what we call the Xcode project model, which defines how Xcode projects are organized, how Xcode generates products from targets, and how projects and targets relate to one another.

It doesn’t cover the details of the build settings that are used by targets or how they’re collected in target and project configurations. Both topics are covered in depth in the Xcode User Guide. I also wrote about configurations when they were introduced in Xcode 2.1 as a replacement for build styles.

Update: The article Working with Xcode Build Settings was recently posted covering configurations and build settings.

Xcode: Debugging Cocoa application unit tests

A couple weeks ago as part of my Unit Testing Series I talked about how to use Xcode to write unit tests for Cocoa frameworks, debug unit tests in Cocoa frameworks, and write unit tests for Cocoa applications. However, I haven’t yet described how to debug your unit tests in Objective-C Cocoa applications. I’ll take care of that tonight.

After you’ve set up unit testing in your Cocoa application, debugging your unit tests is similar to debugging them in a Cocoa framework. All you have to do is adjust the arguments and environment variables your application’s Executable is configured to use in Xcode. You don’t even have to create a new executable.

To start, bring up the Info window for your application’s executable (which is its entry in Xcode’s Executable smart group). In the Arguments tab, add the argument -SenTest All. This tells the unit testing infrastructure that you want to run all of the unit tests, not just the ones that are built in to the executable. (After all, you don’t have any unit tests in your executable itself.)

Now we’ll need to engage in a little bit of environment variable magic. When you test an application, what you’re really doing is injecting your unit test bundle into the application and telling it to run its tests at its first opportunity. This is accomplished through by telling dyld to insert a private framework, DevToolsBundleInjection.framework, into your application on launch, and telling that framework via an environment variable, XCInjectBundle, the path of the test bundle to inject.

You also have to tell the injection framework itself the full path to the application executable you want to inject the bundle into, via the XCInjectBundleInto environment variable. This is needed to avoid injecting your test bundle into other executables that are run by the application you’re testing, or that are run as a side-effect of running the application you’re testing. (For example, gdb generally wants to run applications from within a shell, so that environment variables are expanded in its environment and command-line parameters.)

In the Arguments tab of your application executable, first add an environment variable named DYLD_INSERT_LIBRARIES. Set its value to the path to the DevToolsBundleInjection.framework/DevToolsBundleInjection library in the framework of the same name that’s included in the developer tools. Prior to Xcode 2.5, this was in $(SYSTEM_LIBRARY_DIR)/PrivateFrameworks but as of Xcode 2.5 and Xcode 3.0, it has been moved to $(DEVELOPER_LIBRARY_DIR)/PrivateFrameworks

Then add a second environment variable, XCInjectBundle. Set its value to $(BUILT_PRODUCTS_DIR)/MyTestBundle.octest.

Add a third environment variable, XCInjectBundleInto. Set its value to the full path to your application’s executable — not just the application bundle — e.g. $(BUILT_PRODUCTS_DIR)/MyApplication.app/Contents/MacOS/MyApplication. This is the debugging equivalent of the Test Host build setting you used to tell Xcode what executable to inject your tests into when running them.

For Xcode 3.0 and later, add a final environment variable, DYLD_FALLBACK_FRAMEWORK_PATH to your executable. Set its value to $(DEVELOPER_LIBRARY_DIR)/Frameworks.

Why do you need to do this? In order to support moving and renaming the Developer folder, all of the frameworks within it — including OCUnit — use runpath search paths. This means that the internal name of the framework, including the one copied into your test bundle, will start with @rpath rather than an absolute path starting with /Developer/Library/Frameworks. Unfortunately this means that your unit tests won’t find SenTestingKit.framework without some extra help. That’s what DYLD_FALLBACK_FRAMEWORK_PATH does: It tells dyld to try an additional set of directories in place of @rpath when it can’t be resolved. (More information on runpath-relative install names can be found in the ld(1) man page.)

Make sure the check marks next to all three of these environment variables — and your -SenTest All argument, of course — are set.

Troubleshooting note: Troubleshooting note: If this doesn’t work — that is, if your test bundle isn’t found and run — change the executable’s working directory (in the General tab) to Built Products Directory and remove $(BUILT_PRODUCTS_DIR) above. Generally this is caused by $(BUILT_PRODUCTS_DIR) not being expanded to a full path, but rather to a partial path relative to your project directory.

Now if you choose Run Executable from the Debug menu, your application should launch, you should see the results of executing your unit tests in the Run Log, and as soon as your unit tests are complete your application should quit!

To debug a failing test, build your tests and set a breakpoint on the line where the failure occurs. Now choose Debug Executable from the Debug menu. As with a Cocoa framework, do not choose Build and Debug from the Build menu. You need to use Debug Executable because your build will fail due to the failing test. Debug Executable will work as long as your executable itself is actually present.

Having done all this, you should be stopped at the breakpoint!

Just as any other time you use OCUnit, instead of -SenTest All you can specify -SenTest MyTestCaseClassName to run just the tests in the test case class MyTestCaseClassName, or -SenTest MyTestCaseClassName/testMethodName to run just a single test.

Update July 7, 2007: Added the troubleshooting note about removing $(BUILT_PRODUCTS_DIR) if you get errors about not being able to load the bundle.

Update March 17, 2008: I’ve updated this a bit to handle some changes in the process introduced with Xcode 3.0.

Update September 5, 2008: I’ve updated this again to cover the changes that were made to bundle injection for Xcode 3.1; the change is the introduction of the XCInjectBundleInto environment variable.

Xcode: Unit Testing Cocoa Applications

Yesterday, I talked about how to add unit tests to Cocoa frameworks using Xcode. There’s only a little more set-up required to add tests to Objective-C Cocoa applications.

First, turn off ZeroLink for the application target you want to test. Just as with a framework, your unit tests will be built as a test bundle containing only the tests, completely separate from the code being tested. The test bundle will access the code in your application by linking against it, and you can’t link against something built with ZeroLink. Note: You only need to do this for Xcode 2.1 through Xcode 2.5. Xcode 3.0 removed support for ZeroLink, since the linker is now sufficiently fast as to obviate it.

Next, add a new Cocoa Unit Test Bundle target to your application project. This is the target that will actually contain your tests. When you add the test bundle target, Xcode shows its Info window. In the General tab of this window, press the + button and choose your application target from the sheet that appears. This will make your test bundle target depend on your application target, so you can just build your test bundle all the time and have your application automatically rebuild first.

Now you actually need to make your test bundle link against your application. You do this using the Bundle Loader property under Linker Options in your test bundle target’s configurations. You need to set the value of this option to the path of your application executable and not just its wrapper: $(BUILT_PRODUCTS_DIR)/MyApplication.app/Contents/MacOS/MyApplication. The $(BUILT_PRODUCTS_DIR) variable is expanded at build time to be the path to the build products for the currently-selected configuration, which lets you avoid using an absolute path.

The final step is to tell the unit testing infrastructure how to run your tests. For a framework, the dynamic loader does the heavy lifting; when the test rig loads your test bundle, the loader will automatically load the frameworks it depends on including the one you’re testing. However, for an application to be tested the application must be launched and the test bundle injected into it. You can specify that this should happen using the Test Host property under Unit Testing in your test bundle target’s configurations. Just as with the Bundle Loader property, you need to set the value to the full path of your application executable. Since you’ve already done that once, you can just re-use the value of the Bundle Loader setting by specifying $(BUNDLE_LOADER) — this is the underlying variable that the property is associated with.

Now just like with frameworks you can add test cases to your test bundle just by adding new files based on the Cocoa Objective-C test case class template. Your application code can remain in your application target and your unit testing code can remain in your test bundle target. Whenever you build your test bundle target, once everything is built your application will launch, its tests will be run, and it will quit — and the test results will be reported via the Build Results window just like compiler and linker errors.

Xcode: Debugging Cocoa framework unit tests

So you’ve set up unit testing for your Objective-C Cocoa framework and it’s been working great. But now you’ve written a test and it fails, and you can’t figure out why. It’s time to break out the debugger, but how, exactly, do you do that? Your unit tests are built as a bundle, and you can’t run a bundle.

It’s simple. All you have to do is set up an appropriate Executable in Xcode to run the test rig that runs your bundle, and then debug that. To get started, choose the Project > New Custom Executable menu item. For its name, specify otest — this is the name of the test rig used by OCUnit. Specify /Developer/Tools/otest as the path to your tool.

When you add the custom executable, Xcode will bring up its info window. Switch to the Arguments tab. Here you’ll need to enter some command-line arguments to pass to the test rig and some environment variables to affect how it’s run.

First, add the argument -SenTest Self to your executable. This tells otest to run all of the tests in your bundle. (It’s actually a pair of arguments, but you can add it as one as a convenience.) Then for your second argument specify $(BUILT_PRODUCTS_DIR)/MyTestBundle.octest where MyTestBundle is the name of your test bundle. This tells otest the path of the test bundle to load; $(BUILT_PRODUCTS_DIR) will be expanded to the appropriate build products directory for your project at run time. (If you get the order wrong, just drag the arguments around in the table.)

Troubleshooting note: If this doesn’t work — that is, if otest complains it can’t find your test bundle — change the executable’s working directory (in the General tab) to Built Products Directory and remove $(BUILT_PRODUCTS_DIR) above. Generally this is caused by $(BUILT_PRODUCTS_DIR) not being expanded to a full path, but rather to a partial path relative to your project directory.

Next add a pair of environment variables named DYLD_FRAMEWORK_PATH and DYLD_LIBRARY_PATH to your executable. These will tell the loader to check your build products directory first for frameworks and libraries whenever the executable is run from within Xcode. Specify $(BUILT_PRODUCTS_DIR) for the value of each variable.

Finally, from the Project > Set Active Executable menu choose your new otest executable. This way any time you run or debug within Xcode, it will run otest with the arguments and environment you’ve specified.

To actually debug a failing test, build your tests and set a breakpoint on the line where the failure occurs. Now choose Debug Executable from the Debug menu. Do not choose Build and Debug from the Build menu. You need to use Debug Executable because your build will fail due to the failing test; Debug Executable doesn’t try to build first, it only cares whether an executable is present.

You should successfully stop at your breakpoint!

If your tests take a long time to run — they shouldn’t, they’re unit tests after all, but it can still happen — you may want to just run the tests for one test case, or just one individual test. This is easy too. Rather than specifying -SenTest Self in your arguments to otest, you can specify -SenTest MyTestCaseClassName to run the all the tests in the specified test case. To run just a single test, use -SenTest MyTestCaseClassName/testMethodName instead.

Update July 7, 2007: Added the troubleshooting note about removing $(BUILT_PRODUCTS_DIR) if you get errors about not being able to load the bundle.