Xcode: Unit Testing Cocoa Applications

Yesterday, I talked about how to add unit tests to Cocoa frameworks using Xcode. There’s only a little more set-up required to add tests to Objective-C Cocoa applications.

First, turn off ZeroLink for the application target you want to test. Just as with a framework, your unit tests will be built as a test bundle containing only the tests, completely separate from the code being tested. The test bundle will access the code in your application by linking against it, and you can’t link against something built with ZeroLink. Note: You only need to do this for Xcode 2.1 through Xcode 2.5. Xcode 3.0 removed support for ZeroLink, since the linker is now sufficiently fast as to obviate it.

Next, add a new Cocoa Unit Test Bundle target to your application project. This is the target that will actually contain your tests. When you add the test bundle target, Xcode shows its Info window. In the General tab of this window, press the + button and choose your application target from the sheet that appears. This will make your test bundle target depend on your application target, so you can just build your test bundle all the time and have your application automatically rebuild first.

Now you actually need to make your test bundle link against your application. You do this using the Bundle Loader property under Linker Options in your test bundle target’s configurations. You need to set the value of this option to the path of your application executable and not just its wrapper: $(BUILT_PRODUCTS_DIR)/MyApplication.app/Contents/MacOS/MyApplication. The $(BUILT_PRODUCTS_DIR) variable is expanded at build time to be the path to the build products for the currently-selected configuration, which lets you avoid using an absolute path.

The final step is to tell the unit testing infrastructure how to run your tests. For a framework, the dynamic loader does the heavy lifting; when the test rig loads your test bundle, the loader will automatically load the frameworks it depends on including the one you’re testing. However, for an application to be tested the application must be launched and the test bundle injected into it. You can specify that this should happen using the Test Host property under Unit Testing in your test bundle target’s configurations. Just as with the Bundle Loader property, you need to set the value to the full path of your application executable. Since you’ve already done that once, you can just re-use the value of the Bundle Loader setting by specifying $(BUNDLE_LOADER) — this is the underlying variable that the property is associated with.

Now just like with frameworks you can add test cases to your test bundle just by adding new files based on the Cocoa Objective-C test case class template. Your application code can remain in your application target and your unit testing code can remain in your test bundle target. Whenever you build your test bundle target, once everything is built your application will launch, its tests will be run, and it will quit — and the test results will be reported via the Build Results window just like compiler and linker errors.

Xcode: Debugging Cocoa framework unit tests

So you’ve set up unit testing for your Objective-C Cocoa framework and it’s been working great. But now you’ve written a test and it fails, and you can’t figure out why. It’s time to break out the debugger, but how, exactly, do you do that? Your unit tests are built as a bundle, and you can’t run a bundle.

It’s simple. All you have to do is set up an appropriate Executable in Xcode to run the test rig that runs your bundle, and then debug that. To get started, choose the Project > New Custom Executable menu item. For its name, specify otest — this is the name of the test rig used by OCUnit. Specify /Developer/Tools/otest as the path to your tool.

When you add the custom executable, Xcode will bring up its info window. Switch to the Arguments tab. Here you’ll need to enter some command-line arguments to pass to the test rig and some environment variables to affect how it’s run.

First, add the argument -SenTest Self to your executable. This tells otest to run all of the tests in your bundle. (It’s actually a pair of arguments, but you can add it as one as a convenience.) Then for your second argument specify $(BUILT_PRODUCTS_DIR)/MyTestBundle.octest where MyTestBundle is the name of your test bundle. This tells otest the path of the test bundle to load; $(BUILT_PRODUCTS_DIR) will be expanded to the appropriate build products directory for your project at run time. (If you get the order wrong, just drag the arguments around in the table.)

Troubleshooting note: If this doesn’t work — that is, if otest complains it can’t find your test bundle — change the executable’s working directory (in the General tab) to Built Products Directory and remove $(BUILT_PRODUCTS_DIR) above. Generally this is caused by $(BUILT_PRODUCTS_DIR) not being expanded to a full path, but rather to a partial path relative to your project directory.

Next add a pair of environment variables named DYLD_FRAMEWORK_PATH and DYLD_LIBRARY_PATH to your executable. These will tell the loader to check your build products directory first for frameworks and libraries whenever the executable is run from within Xcode. Specify $(BUILT_PRODUCTS_DIR) for the value of each variable.

Finally, from the Project > Set Active Executable menu choose your new otest executable. This way any time you run or debug within Xcode, it will run otest with the arguments and environment you’ve specified.

To actually debug a failing test, build your tests and set a breakpoint on the line where the failure occurs. Now choose Debug Executable from the Debug menu. Do not choose Build and Debug from the Build menu. You need to use Debug Executable because your build will fail due to the failing test; Debug Executable doesn’t try to build first, it only cares whether an executable is present.

You should successfully stop at your breakpoint!

If your tests take a long time to run — they shouldn’t, they’re unit tests after all, but it can still happen — you may want to just run the tests for one test case, or just one individual test. This is easy too. Rather than specifying -SenTest Self in your arguments to otest, you can specify -SenTest MyTestCaseClassName to run the all the tests in the specified test case. To run just a single test, use -SenTest MyTestCaseClassName/testMethodName instead.

Update July 7, 2007: Added the troubleshooting note about removing $(BUILT_PRODUCTS_DIR) if you get errors about not being able to load the bundle.

Xcode: Unit Testing Cocoa frameworks

It’s straightforward to write unit tests for Objective-C Cocoa frameworks with Xcode 2.1 and later.

First, turn off ZeroLink when building your framework. ZeroLink is a great technology, but you can’t link against something that’s built with ZeroLink, and that’s exactly what your unit tests are going to do. Note: You only need to do this for Xcode 2.1 through Xcode 2.5. Xcode 3.0 removed support for ZeroLink, since the linker is now sufficiently fast as to obviate it.

Next, add a new Cocoa Unit Test Bundle target to your framework project. This is the target that will actually contain your tests.

When you add the new test bundle target Xcode should bring up its Info window. Switch to its General tab and press the + button in the lower-left to add a new dependency. In the resulting sheet, choose your framework target. This tells the build system that your framework target needs to be built before your test bundle target.

Now that you’ve declared the dependency between the targets, you need to make sure your test bundle actually links against your framework. Make your test bundle target the active target by choosing it from the Project > Set Active Target submenu. Then highlight the Products group in the Groups & Files view at the left of the Xcode project window. In the detail view to the right, you’ll see all of the products that are built by the various targets in your project; one of the rows will be for your framework. Click the checkbox at the very right of its row, in the Target Membership column, to make your test bundle link against your framework.

Now you can add test cases to your test bundle by just adding new files based on the Cocoa Objective-C test case class template. You don’t need to add any of your framework code to your test bundle, and you don’t need to add any testing code to your framework. And you’re not restricted to using only header files with public visibility in tests, either; you should be able to use anything from your framework target, so long as it’s exported at link time.

Xcode: Unit Testing

Xcode 2.1 introduced integrated unit testing to the Xcode IDE. Xcode includes two unit testing frameworks, target templates for setting up test bundle targets, and infrastructure for running unit tests every time you build your project and reporting their results in the Build Results window just like compilers and linkers do.

With Xcode unit testing, you group your test cases into test bundles which are built by separate targets. This means that you don’t need to build two versions of your software — one with tests and one without tests — and can run your tests against the actual software you want to deliver.

Xcode 2.1 and later include OCUnit for unit testing of Objective-C Cocoa software, and it includes a new framework called CPlusTest for unit testing of C++ software. Using either you should be able to test C code with relative ease. Corresponding target templates are included for creating unit test bundles appropriate for Cocoa and Carbon applications and frameworks, and file templates are included for creating OCUnit and CPlusTest test case classes.

The test bundle target templates have a shell script build phase at the very end that invokes /Developer/Tools/RunUnitTests. RunUnitTests looks at the build settings it’s passed via its environment and determines from that information how to run the tests in your test bundle.

If you’re testing a framework, RunUnitTests will run the appropriate test rig and tell it to load and run the tests in your bundle. Since your test bundle should link against your framework, your framework will be loaded when the test rig loads your bundle.

If you’re testing an application, you need to specify the application as the Test Host and Bundle Loader for your test bundle in its configuration’s build settings. The Bundle Loader setting tells the linker to link your bundle against the application that’s loading it as if the application were a framework, allowing you to refer to classes and other symbols within the application from your bundle without actually including them in the bundle. The Test Host setting tells RunUnitTests to launch the specified application and inject your test bundle into it in order to run its tests.

There’s even support in RunUnitTests for invoking a test rig of your own, rather than the test rig for one of the supplied frameworks. You just need to specify the Test Rig build setting for your test bundle; this should be the path to a tool to run. It will be passed the path to your test bundle as its first argument, and if it needs to generate failure information it can just generate it in a gcc/compiler-like format on stderr:

FailingTest.c: 10: error: (1 == 0) failed

This will cause it to show up in the Build Results window as an error, just like a compiler or linker error. You can see the RunUnitTests manpage for more information on the environment in which your test rig will be run.

There’s a great Unit Testing Guide on the Apple Developer Connection web site that has lots of information on getting started with Xcode unit testing. Check it out!

Code Width

There’s a question on wrapping code at 80 columns in the Joel on Software Forum. Most of the time, when I see 80 columns as a standard it’s to enable easy printing of the code. However, in this day and age, why are you printing code? I print code occasionally when I need to read it in-depth, but other than that I do most of my reading on-screen. It’s certainly not a regular occurrence.

I also use mostly large or widescreen displays for coding. The smallest display I use regularly is my laptop, which is 1152 by 768. Most of the time I use my 19-inch monitor, which I run at 1600 by 1200. As a result, if I want to write really long lines, I can. But I don’t.

I generally limit the length of lines I write to 80 to 120 characters. Most of the time this is easy, because Objective-C’s syntax lends itself to very readable multi-line expressions:

NSDictionary *d = [NSDictionary dictionaryWithObject:@"Chris"
                                              forKey:@"name"];

The main reason I do it, though, is that I use multiple windows while I code. I usually have at least 6 windows open in Xcode while I’m coding: The Xcode project itself, the source and header file for the class I’m working on, the source and header file for the unit test I’m working on, and the build results window (which also shows my unit test results). Often it’s much more, with more class and unit test sources & headers open. And it’s much easier to fit these windows all on a screen when they don’t need to be as wide as the screen to see their contents…

Platform Futures

On Windows, many developers seem to want to run as fast as possible away from Microsoft Visual C++ and embrace Microsoft’s C# and .NET platform for new development. Most Windows developers that I’ve seen seem downright enthusiastic about these technologies. It’s disconcerting; I’m not used to seeing Windows developers (or users) be enthusiastic about their platform.

On the Mac, many developers are trying to hold onto C++ and Carbon for as long as they can, even for new development. A new Mac developer on the Carbon list actually said he wished Apple had a C++ framework that used MFC-like “message maps” for Mac OS X-only Carbon development “to make it easier to build software fast!” (Paraphrased.) And Metrowerks is spending money & time building a next-generation C++ PowerPlant framework for Mac OS X-only Carbon development! And some developers keep on Apple’s case to try and maintain feature parity between Carbon and Cocoa.

Fortunately, Apple isn’t giving in to them as much as they might think. For instance, WebKit has a Carbon wrapper, but it’s just a wrapper; WebView is really a Cocoa framework and if you want to extend it you’re going to have to use Cocoa. The Cocoa Controller layer is only really possible to do with a rich dynamic runtime; it’ll never make it to Carbon. You can only build screen savers using Cocoa and Objective-C. You can only build preference panes using Cocoa and Objective-C. Virtually all new applications coming out of Apple are built using Cocoa and Objective-C.

(Keynote, SoundTrack, LiveType, iCal, iPhoto, iSync, iChat AV, Safari… Final Cut and Logic don’t count, since they ware originally developed for the traditional Mac OS and thus aren’t new. Neither does Shake, since it was originally developed for Irix and X11 — though it wouldn’t surprise me at all to see it rearchitected as a Cocoa application in the next couple of years.)

The future of development on Windows is C# and .NET. This has been clear since Microsoft first released .NET, and it’s especially clear in light of the latest PDC and Longhorn.

The future of development on the Mac is Objective-C and Cocoa. This has been clear ever since Apple bought NeXT, and it’s especially clear in light of the latest WWDC and Panther.

Deal with it.

Joel Climbs Into the Trunk

Joel is smoking the good stuff! Or is he drinking the purple stuff?

Joel Spolsky of Joel on Software is actually claiming “.NET appears so far to be one of the most brilliant and productive development environments ever created.” He goes on to say “ASP.NET is as big a jump in productivity over ASP as Java is to C. Wow.”

Living inside Microsoft’s locked trunk for too long obviously ruins your sense of perspective. They’ve been steadily improving their tools over the years, but they started in such a truly awful state that what they have now seems like absolute luxury to Windows developers. If these people would bother to look outside their cramped quarters occasionally, they’d see that they really, honestly don’t have it very well off at all.

Here’s a choice quote:

I love the fact that you can use an ASP.NET calendar widget, which generates HTML that selects a date on a monthly calendar, and be confident that the “date” class you get back from that thing (System.DateTime, I believe) will be the exact same date class that the SQL Server classes expect. In the past you would not believe how much time we wasted doing things like reformatting dates for SQL statements, or converting COleDateTimes to someOtherKindOfDateTime.

I’d believe it because I’ve seen the kind of crap you have to go through to get anything done with Microsoft’s development tools and programming interfaces. In fact, I’ve written ported Macintosh software to Windows with them, and even tried to build Macintosh software with them (using their terrible Macintosh MFC SDK). It’s amazing Windows developers are ever able to get anything done.

Contrast that to OpenStep, WebObjects, and the Enterprise Objects Framework circa late 1995. There is one date class, NSGregorianDate. You didn’t have to write low-level code to access databases, so you didn’t have to worry about turning dates into strings you could embed in SQL statements. Instead, your Enterprise Objects – objects transparently backed by your database could just have attributes containing NSGregorianDate objects and The Right Thing would happen automatically.

As I read through this, I find more and more absolutely laughable comments. For instance:

First, they had the world’s best language designer, the man who was responsible for 90% of the productivity gains in software development in the last 20 years: Anders Hejlsberg, who gave us Turbo Pascal (thank you!), Delphi (thank you!), WFC (nice try!) and now .NET (smacked the ball outta the park).

Yes, the creator of Turbo Pascal is responsible for all of our productivity gains in software development. Especially since, even though Joel credits him as a “language designer” he didn’t really “design” Pascal or Object Pascal. Not, say, Alan Kay, leader of the team that invented true Object-Oriented Programming in the 1970s at Xerox and popularized it in 1980 with Smalltalk-80. Then again, that’s 22 years ago – maybe it doesn’t count (since Joel said “20 years”), even though industrywide movement to OOP didn’t really have critical mass until the 1990s…

Perhaps with .NET, Microsoft is only 5 years behind where NeXT was in 1995 with OpenStep, WebObjects, and the Enterprise Objects Framework. But they’re still 5 years behind where NeXT was, and now that NeXT has funding – in the form of millions of Apple Macintosh users running Mac OS X – they’re starting to move ahead quickly. If they can bring back the Objective-C version of EOF for Cocoa, they’ll be way ahead of the game.