Test Logs in Xcode
Apple spent the week of WWDC touting the new testing features in Xcode 7. One thing they failed to mention both this year and last was the persistence of test results to disk. These files and folders are not officially documented, so if you decide to rely on them make sure to check a new version of Xcode doesn’t break anything. This post will be updated to reflect changes as needed.
If you run a set of tests in Xcode 6/7 or with xcodebuild test via the command line, a new folder inside of the app’s folder in DerivedData is created, called Test. It contains similar files to the other folders, but Xcode 7 adds some new things as well.
.xcactivitylog
An .xcactivitylog is a gzipped copy of the build log that’s generated any time a project is built. Each file is uniquely named with a UUID. Unfortunately for us, the UUID generated is unique per folder, so correlating a file in Build and Test cannot be done by file name alone. These appear to be segments of the output, not complete logs. At least one of these is generated per build.
TestSummaries.plist
This is the exciting part. This file contains the results of the tests in a machine readable format. At the root of this plist is 3 things: FormatVersion, RunDestination, and TestableSummaries.
FormatVersion
This is a string that is used to version the TestSummaries file format. Between Xcode 6 and 7, the version number did not change, though additional information is added to the files generated in 7. With this, we can assume that the version is not incremented when additions are made to the objects contained.
RunDestination
This contains information about what computer, device, and sdk were used to run the tests. All the items are fairly self-explanatory.
TestableSummaries
Each test target is an item in this array. As of Xcode 7, running test will automatically run all test targets associated with a project.
Within the Tests array there are records of the test, but they are quite buried. The top-level item distinguishes between All tests or Selected tests. The only difference between these two is how many levels of Subtests exist between the top level Test object and the individual test results.
In All tests the identifier hierarchy looks like so
All tests
└── AppTests.xctest
└── AppTests
└── Individual tests
For Selected tests the .xctest level is skipped.
Selected tests
└── AppTests
└── Individual tests
Once we get down into individual tests, the Identifier is automatically generated based on the method name. so -(void)testExample becomes AppTests/testExample. When using Specta, the names are mangled slightly more.
Take this test for example:
describe(@"successful request", ^{
it(@"calls success block when request succeeds", ^{
...
});
});
The TestIdentifier generated will look something like this: AppTests/test_successful_request__calls_success_block_when_request_succeeds.
Tests written in swift will always end with ().
Xcode 7 adds ActivitySummaries and FailureSummaries
For Unit test targets, the useful piece of information here is the reason the test fails. For UI test targets, this is where the steps to perform the UI tests are recorded, along with any extra information about them. When running a UI test, each action taken on a UI object generates one of these sub activities.
Take this test:
func testOne() {
let app = XCUIApplication()
app.buttons["One"].tap()
let aTextFieldTextField = app.textFields["A Text Field"]
aTextFieldTextField.tap()
aTextFieldTextField.typeText("asd")
}
Sidenote: the test must tap a textField before it can enter text into it.
This test generates 4 activities, with sub activities auto-generated:
t = 1.77s Wait for app to idle
t = 2.42s Tap the "One" Button
t = 2.43s Wait for app to idle
t = 2.68s Find the "One" Button
t = 2.72s Dispatch the event
t = 2.97s Wait for app to idle
t = 3.62s Tap the "A Text Field" TextField
t = 3.62s Wait for app to idle
t = 3.62s Find the "A Text Field" TextField
t = 3.64s Dispatch the event
t = 3.87s Wait for app to idle
t = 4.49s Type 'asd' into the "A Text Field" TextField
t = 4.49s Wait for app to idle
t = 4.49s Find the "A Text Field" TextField
t = 4.55s Dispatch the event
t = 4.66s Wait for app to idle
The step we care about most is “Dispatch the event” because it contains the attachments generated from the tests.
Attachments
Xcode 7 introduced UI testing, a big part of which is screenshots of the tests as they happen. The Attachments object contains information about where those screenshots live. But there’s more! In addition to a .png of the screen during that particular test step, there’s also a snapshot. What is a snapshot? It’s a archive containing the entire view hierarchy on screen when the screenshot was taken as a binary plist.
Failures
When a test fails, a summary is added to the FailureSummaries key. This is just a shortcut to the item in ActivitySummaries which has almost the same information.
Xcode vs xcodebuild
xcodebuild
Thankfully for those of us who automate builds, xcodebuild test does the same thing that Xcode does, though it generates fewer files. Here’s what you’ll find in Logs after running on the command line:
├── Build
│ ├── Cache.db
│ └── FCDDA053-2DA1-4C1C-8716-B6C339CB5815.xcactivitylog
└── Test
├── 21ED62CB-4F14-4A87-805E-C594936F24E6_TestSummaries.plist
└── Attachments
├── Screenshot_6CD674C6-4E0D-4DAC-BF66-016198C32780.png
├── Screenshot_8126D5ED-88B9-4F6D-8F38-A339F3A14AE5.png
├── Screenshot_94513284-AA2F-48DC-A9D7-C56A0EF9BC91.png
├── Screenshot_E6EBC6AC-7311-49C2-829D-AF9880141C4B.png
├── Snapshot_5638F5F5-C04C-4F5E-AE42-C517B6B1FCBC
├── Snapshot_57F67566-9E22-4005-B7D6-B822EF08CD2E
├── Snapshot_7AA349A8-CF54-4AC4-8139-A676167C8801
└── Snapshot_A065B515-A4DE-431C-B72F-87952AA133A6
One caveat: The .xcactivitylog in the Build folder does not contain the full build log. It is missing the logs generated when tests are run.
Now this seems great, but remember this is inside of DervivedData so there will be many other files, so finding it programmatically is not easy. One solution is to grep the console output for the sessionIdentifier which is included in the XCTestConfiguration:
Found configuration <XCTestConfiguration: 0x7ff38a5894e0>
testBundleURL:file:///Users/michele/Library/Developer/Xcode/DerivedData/Buttons-amswwgmmhfjuzcegszalugciwdsj/Build/Products/Debug-iphonesimulator/ButtonsUITests-Runner.app/PlugIns/ButtonsUITests.xctest
productModuleName:ButtonsUITests
testsToSkip:(null)
testsToRun:(null)
reportResultsToIDE:YES
sessionIdentifier:<__NSConcreteUUID 0x7ff38a58bf70> CC966A22-0E28-4880-B445-9E401BB24CDB
pathToXcodeReportingSocket:(null)
disablePerformanceMetrics:no
treatMissingBaselinesAsFailures:no
baselineFileURL:(null)
targetApplicationPath:/Users/michele/Library/Developer/Xcode/DerivedData/Buttons-amswwgmmhfjuzcegszalugciwdsj/Build/Products/Debug-iphonesimulator/Buttons.app
targetApplicationBundleID:com.mrt.Buttons
Manually sorting through Xcode’s logs is The Hard Way™ and finally with Xcode 7 there is an easier way to get that information.
There is a new xcodebuild argument resultBundlePath which will save the .xcactivitylog, TestSummary.plist, and related files to a directory of your choosing. As of Xcode 7b1, this must be an absolute path.
$ xcodebuild test -scheme 'Buttons' -destination 'platform=iOS Simulator,name=iPhone 6,OS=9.0' -resultBundlePath '~/Desktop/build'
Outputs:
├── 1_Test
│ ├── Attachments
│ │ ├── Screenshot_1236AC16-AF8C-4E87-A748-4867EAB120FC.png
│ │ ├── Screenshot_52E63346-D71F-4A6B-ACA9-736AD54854F3.png
│ │ ├── Screenshot_7A8F0CD6-3981-47FD-BC2D-DC6B35863656.png
│ │ ├── Screenshot_E4792BBC-9717-4BD1-8CD0-D560E19A491D.png
│ │ ├── Snapshot_273B4980-BE03-4EA8-81B7-03F237772558
│ │ ├── Snapshot_86786328-081A-4314-ADC9-213648471F5D
│ │ ├── Snapshot_AA3A1ADC-84C8-4FCC-91A6-590431B89210
│ │ └── Snapshot_CDCA9CA4-847D-46DF-852F-8BCEF0C468AE
│ ├── action.xcactivitylog
│ ├── action_TestSummaries.plist
│ └── build.xcactivitylog
├── Attachments
│ ├── Screenshot_2750BC84-575E-4849-9A39-820FB8D8E998.png
│ ├── Screenshot_59E390A6-9B39-4FD5-AA13-6915C211F6F8.png
│ ├── Screenshot_8F1BB0EE-1146-4CE0-BB5B-1ED473A12173.png
│ ├── Screenshot_DF5341F9-2763-47C6-821B-2B157E88AEC7.png
│ ├── Snapshot_21D3F473-0F28-4F4A-97FD-F30263461422
│ ├── Snapshot_5AF107B1-7CA0-4B14-9EEB-3918E4C4B5E9
│ ├── Snapshot_5F74DD59-6225-40D7-955A-21B366A563BD
│ └── Snapshot_FD2ED335-1729-4489-8B7F-0777E73D4EE8
├── Info.plist
└── TestSummaries.plist
There is duplicate data here. The TestSummaries.plist at the root of the directory is version 1.1, but inside 1_Test is the correct version. This also adds both build and test .xcactivitylog files conveniently in one place. Using CI? This argument will make your life a whole lot easier.
Xcode
It is common knowledge that when Xcode builds a project, it does not do the same thing that xcodebuild does. Just like xcodebuild, building in Xcode creates a Test folder inside Logs, and saves summary plists and attachments. However, Xcode itself creates a lot more logs.
├── Build
│ ├── Cache.db
│ └── D0C828D2-21CA-4182-9A0E-C80DE1F8FA86.xcactivitylog
├── Debug
│ ├── 130CC81E-789E-4FDD-A156-DB75FBEBC770.xcactivitylog
│ ├── 44845FEA-0D87-4266-AD62-502E0F4C30B2.xcactivitylog
│ └── Cache.db
├── Issues
│ ├── 0F1BB6CD-F28B-48DB-9A09-B3948257D698.xcactivitylog
│ ├── 4AFC3EE7-12EC-4E4C-BDF9-2ED7BCA6887E.xcactivitylog
│ ├── C32CA334-AAF9-43B4-AB00-D10373391A08.xcactivitylog
│ └── Cache.db
└── Test
├── 9309CF06-A5BB-418C-91D5-299788C685A3.xcactivitylog
├── 9309CF06-A5BB-418C-91D5-299788C685A3_TestSummaries.plist
├── Attachments
│ ├── Screenshot_556D54FE-235B-4A3E-A00E-C34ED00AF043.png
│ ├── Screenshot_786546BC-98C4-4788-B7F1-9A66DFA0EF51.png
│ ├── Screenshot_976F8BF7-C2A6-4E8F-8BFC-EE6A4FA6D78E.png
│ ├── Screenshot_BC2D4BF7-DC31-459F-B726-84651CFF52A3.png
│ ├── Snapshot_31A23466-8A83-42B6-8AD7-4CBD2FB34BB7
│ ├── Snapshot_626616A0-436F-4D45-822B-41E21E7E80CF
│ ├── Snapshot_73A33739-49A0-4D8E-9295-338BDAE08C5C
│ └── Snapshot_83BF8F36-C1EF-45E0-B55E-3320F7FE55D7
└── Cache.db
When opening up the multiple .xcactivitylog files, there is some overlap. It appears that Xcode will split the different sections of the build into different activity logs. Unfortunately, because the UUIDs are unique and not related, putting a full log back together from these pieces is fairly difficult.
Conclusion
These new files and flags will make automated builds even better with Xcode 7. Apple has finally rounded out the toolset for automating tests, and CI providers are already starting to offer builds using Xcode 7.
The sample project used in this post is available on Github.
Thanks
- Lawrence Lomax for telling me about the new
resultBundlePathoption. - Brian Nickel for fixing the sample project’s storyboard overriding accessibilityLabel.