Automated Testing with GHUnit and KIF
Testing in iOS is something that we as developers don’t really think about, or do. Writing tests can help us write code that lasts for years to come. But, this post isn’t about why you should write tests, it’s about how to use GHUnit and KIF with Jenkins CI.
There are two main types of testing frameworks, Unit and UI. OCUnit, which ships with Xcode, and GHUnit are both Unit testing frameworks. UIAutomation, Frank, and KIF are examples are UI testing frameworks.
GHUnit
Unit tests are used to test vertical, isolated slices of functionality. Your app won’t actually be running when these tests are run. Instead, you’ll only import and create objects you want to test directly.
Setup
GHUnit has some fantastic setup steps. The most important thing to note is that GHUnit has its own target, which is not a duplicate target of your app. This means that your app will not actually be running when the tests run.
Writing Tests
GHTestCase
All of your tests will be subclasses of GHTestCase. And you can group these together by creating a GHTestGroup. If you’ve used OCUnit before, this should all sound familar.
Each test case will determine sucess or failure by using assert macros (you can see a full list of them on this page).
Automation
GHUnit also happens to have a great guide for setting up tests using CI. Note that if you’re running Xcode 4.5 or newer there is a bug where the old way of running tests does not work. This is being actively tracked in GitHub Issues for the project, as well as on openradar.
GHUnit also will output JUnit test results, which Jenkins can track over time. This is great for keeping track of code stability, and for finding trends in how well your tests are doing.
Gotchas
Arguably the hardest part of automated testing is the fact that it’s been broken since Developer Preview 3 of Xcode 4.5. This means that you have to work around the command line tools in order to get your tests to run. It’s something that is actively being tracked, and there are solutions, but they are far from ideal.
Since your app won’t be running during these tests, everything you test is very isolated. That might be great for your app, but not all apps can be tested thoroughly with only small pieces running at a time.
Also, since your app is not actually running, running UI tests with GHUnit is not ideal.
KIF
KIF is a framework created by the folks at Square. It uses the <UIAccessibility> protocol to interact with your app, which is a private API. When adding KIF to your project, you have the choice of using git submodules, or Cocoapods.
Setup
KIF setup is fairly simple, and is documented in the project’s README. The main difference between KIF and GHUnit is that KIF duplicates your application target, so all of your source code is added to KIF.
KIFTestController
The Test Controller determines at runtime what tests to run, if you don’t want to run all of them (which you won’t). You can choose to run different sets of tests based on just about anything, the iOS version of the device, the device idiom, etc.
KIFTestScenario & KIFTestStep
The bulk of your test writing will be creating scenarios, like “user logs in” or “user visits cart”, which are comprised of steps. To create reuseable steps, you subclass KIFTestStep, and then can define your step using blocks. Likewise, you can subclass KIFTestScenario.
Automation
In the KIF README there is a section on Automation that provides instructions, and a bash script, for getting it running. However, I’ve found that ios-sim works a bit better, but both work.
KIF doesn’t automatically output test results in any useable format. There’s currently a pull request open to get that in. Or, you could get a patch from an older pull request, which is the one I’ve been using for months. Either way, you need to fork KIF.
Here’s a simple command to run KIF with an .app file stored in $APPFILE and then save the JUnit xml:
/usr/local/bin/ios-sim launch $APPFILE --family ipad > /tmp/KIF-ipad-$$.out 2>&1
cp "`grep "JUNIT XML RESULTS AT " /tmp/KIF-ipad-$$.out | sed 's/.*JUNIT XML RESULTS AT //'`" 'test-reports/KIF-ipad-results.xml'
Gotchas
KIF is fast
KIF runs at processer speed. It doesn’t have a “reaction time” like people do, so KIF will try to find that view on the screen before your navigation push animation finishes. To compensate for this, you’ll end up doing a lot of:
[scenario addStep:[KIFTestStep stepToWaitForTimeInterval:1 description:@"wait"]];
[scenario addStep:[KIFTestStep stepToWaitForViewWithAccessibilityLabel:@"Table"]];
Now, this isn’t necessarily a bad thing. But it will take time to get in the habit of thinking to add waits into your tests. And tests will take longer to run, of course.
KIF can’t recover
KIF has a handy way of adding a step, or set of steps, to be run before every scenario:
[KIFTestScenario setDefaultStepsToSetUp:[KIFTestStep setupSteps]];
However, in these default setup steps you need to be able to recover from a failure at any point in your app. If you don’t recover, then there’ll be a domino-effect of failed tests. This will mean having methods available to esentially reset the application state.
You’re code coverage will not be evenly distributed
If your app requires a login for example, it’s likely you’ll have something like this in many scenarios:
[scenario addStepsFromArray:[LoginTestStep stepsToLoginWithEmail:TEST_EMAIL password:TEST_PASSWORD]];
Because you want to “reset the application state” between scenarios, you’ll end up testing the parts of your code that deal with common actions more frequently than other parts of your app. This is both a pro and a con; the parts you test over and over will definitely be rock solid, but that means there will be parts of your ap that won’t be tested nearly as much.
KIF can’t see off the screen
Since KIF relies on <UIAccessibility> and that relies on view drawing, KIF can’t see things that are off the screen. You’ll end up frequently doing something like this:
[scenario addStep:[KIFTestStep stepToScrollToItemWithAccessibilityLabel:@"Settings"]];
Fork it
KIF is a great framework, but Square has an “Individual Contributor License Agreement (CLA)” that you must sign before any of your code can go into core. This means that there are a number of great things that developers have made, but never got merged in (like the JUnit code).
In Summary
Does writing tests help you write better code? Yes.
Will testing add to development time? Yes.
Is it worth the extra time? Yes.
If you’re interested in seeing some sample tests, head over to my Github repo.