Monthly Archives: February 2018

Xcode’s Secret Performance Tests

I was inspired today, by a question from another developer, to dig into Xcode’s performance testing. This developer had observed that XCTestCase exposes a property, defaultPerformanceMetrics, whose documentation strongly suggests can be used to add additional performance metrics:

This method returns XCTPerformanceMetric_WallClockTime by default. Subclasses of XCTestCase can override this method to change the behavior of measureBlock:.

If you’re not already familiar, the basic approach to using Xcode’s performance testing infrastructure is you add unit tests to your project that wrap code with instructions to measure performance. From the default unit test template:

func testPerformanceExample() {
	// This is an example of a performance test case.
	self.measure {
		// Put the code you want to measure the time of here.
	}
}

Depending on the application under test, one can imagine all manner of interesting things that might be useful to tabulate during the course of a critical length of code. As mentioned in the documentation, “Wall Clock Time” is the default performance metric. But what else can be measured?

Nothing.

At least, according to any header files, documentation, WWDC presentations, or blunt Googling that I have encountered. There is exactly one publicly documented Xcode performance testing metric, and it’s XCTPerformanceMetric_WallClockTime.

I was curious whether supporting additional, custom performance metrics might be possible but under-documented. To test this theory, I added “beansCounted” to the list of performance metrics returned from my XCTestCase subclass. For some reason I couldn’t get Swift to accept the XCTPerformanceMetric pseudo-type, but it allowed me to override as returning an array of String:

override static func defaultPerformanceMetrics() -> [String] {
	return ["beansCounted"]
}

When I build and test, this fails with a runtime exception “Unknown metric: beansCounted”. The location of an exception like this is a great clue about where to go hunting for information about whether an uknown metric can be made into a known one! If there’s a trick to implementing support for my custom “beansCounted” metric, the answer lies in the method XCTestCase’s “measureMetrics(_: automaticallyStartMeasuring: forBlock:)”, which is where the exception was thrown.

By setting a breakpoint on this method and stepping through the assembly in Xcode, I can watch as the logic unfolds. To simplify what happens: first, a list of allowable metrics is computed, and then the list of desired metrics is iterated. If any metric is not in the list? Bzzt! Throw an exception.

I determined that things are relatively hardcoded such that it’s not trivial to add support for a new metric. I was hoping I could implement some magic methods in my test case, like “startMeasuring_beansCounted” and “stopMeasuring_beansCounted”

but that doesn’t appear to be the case. The performance metrics are supported internally by a private Apple class called XCTPerformanceMetric, and the list of allowable metrics is derived from a few metrics hardcoded in the “measureMetrics…” method:

  • “com.apple.XCTPerformanceMetric_WallClockTime”
  • “com.apple.XCTPerformanceMetric_UserTime”
  • “com.apple.XCTPerformanceMetric_RunTime”
  • “com.apple.XCTPerformanceMetric_SystemTime”

As well as a bunch of others exposed by a private “knownMemoryMetrics” method:

  • “com.apple.XCTPerformanceMetric_TransientVMAllocationsKilobytes”
  • “com.apple.XCTPerformanceMetric_TemporaryHeapAllocationsKilobytes”
  • “com.apple.XCTPerformanceMetric_HighWaterMarkForVMAllocations”
  • “com.apple.XCTPerformanceMetric_TotalHeapAllocationsKilobytes”
  • “com.apple.XCTPerformanceMetric_PersistentVMAllocations”
  • “com.apple.XCTPerformanceMetric_PersistentHeapAllocations”
  • “com.apple.XCTPerformanceMetric_TransientHeapAllocationsKilobytes”
  • “com.apple.XCTPerformanceMetric_PersistentHeapAllocationsNodes”
  • “com.apple.XCTPerformanceMetric_HighWaterMarkForHeapAllocations”
  • “com.apple.XCTPerformanceMetric_TransientHeapAllocationsNodes”

How interesting! There are a lot more metrics defined than the single “wall clock time” exposed by Apple. So, should we use them? Official answer: no way! This is private, unsupported stuff, and can’t be relied upon. Punkass Daniel Jalkut answer? Why not? They’re your tests, and your the only one who will get hurt if they suddenly stop working. In my opinion taking advantage of private, undocumented system behavior for private, internal gain is much different than shipping public software that relies upon such undocumented behaviors.

I modified my unit test subclass to return a custom array of tests based on the discoveries above, just to test a few:

override static func defaultPerformanceMetrics() -> [String] {
	return [XCTPerformanceMetric_WallClockTime, "com.apple.XCTPerformanceMetric_TransientHeapAllocationsKilobytes", "com.apple.XCTPerformanceMetric_PersistentVMAllocations", "com.apple.XCTPerformanceMetric_UserTime"]
}

The tests build and run with no exception. That’s a good sign! But these “secret peformance tests” are only useful if they can be observed and tracked the way the wall clock time can be. How does Xcode hold up? I made my demonstration test purposefully impactful on some metrics:

func testPerformanceExample() {
	self.measure {
		for _ in 1..<100 {
			print("wasting time")
		}
		let _ = malloc(3000)
	}
}

Now when I build and test, look what shows up in the Test navigator’s editor pane:

Screenshot of performance metrics after reducing the size of allocations and length of run.

Look at all those extra columns! And if I click the “Set Baselines…” button, then tweak my function to make it substantially less performant:

func testPerformanceExample() {
	self.measure {
		for _ in 1..<10000 {
			print("wasting time")
		}
		let _ = malloc(300000)
	}
}

Now the columns have noticably larger numbers:

Screenshot of Xcode's test results after running tests with

But more importantly, the test fails:

Screenshot of test errors generated by failing to meet performance baselines.

I already mentioned that by any official standard, you should not take advantage of these secret metrics. They are clearly not supported by Apple, may be inaccurate or have bugs, and could outright stop working at any time. I also said that, in my humble opinion, you should feel free to use them if you can take advantage of them. The fact that they are supported so well in Xcode probably implies that groups internal to Apple are using them and benefiting from them. Your mileage may vary.

The only rule is this: if Apple does do anything to change their behavior, or you otherwise ruin your day by deciding to play with them, you shouldn’t blame Apple, and you can’t blame me!

Enjoy.

Accessible Frames

I love the macOS system-wide dictionary lookup feature. If you’re not familiar with this, just hold down the Control, Command, and D keys while hovering with the mouse cursor over a word, and the definition appears. What’s amazing about this feature is it works virtually everwyhere on the system, even in apps where the developers made no special effort to support it.

Occasionally, while solving a crossword puzzle in my own Black Ink app, I use this functionality to shed some light on one of the words in a clue. As alluded to above, it “just works” without any special work on my part:

Screen shot of Black Ink's interface, with a highlighted word being defined by macOS system-wide dictionary lookup.

Look carefully at the screenshot above, and you can see that although the dictionary definition for “Smidgen” appears, it seems to have highlighted the word in the wrong location. What’s going on here?

The view in Black Ink that displays the word is a custom NSTextField subclass called RSAutosizingTextField. It employs a custom NSTextFieldCell subclass in order to automatically adjust the font size and display position to best “fill up” the available space. In short: it behaves a lot like UITextField on iOS.

To achieve this behavior, one of the things my custom cell does is override NSTextFieldCell’s drawingRect(forBounds:). This allows me to take advantage of most of NSTextField’s default drawing behavior, but to nudge things a bit as I see fit. It’s this mix of customized and default behaviors that leads to the drawing bug seen above. I’ve overridden the drawing location, but haven’t done anything to override the content hierarchy as it’s reflected by the accessibility frameworks.

What do the accessibility frameworks have to do with macOS system-wide dictionary lookup? A lot. Apparently it’s the “accessibilityFrame” property that dictates not only where the dictionary lookup’s highlighting will be drawn, but also whether the mouse will even be considered “on top of” a visible word or not. So in the screenshot above, if the mouse is hovering over the lower half of the word “Smidgen”, then the dictionary lookup doesn’t even work.

The fix is to add an override to NSTextField’s accessibilityFrame method:

public override func accessibilityFrame() -> NSRect {
	let parentFrame = super.accessibilityFrame()
	guard let autosizingCell = self.cell as? RSAutosizingTextFieldCell else { return parentFrame }

	let horizontalOffset = autosizingCell.horizontalAdjustment(for: parentFrame)

	// If we're flipped (likely for NSTextField, then the adjustments will be inverted from what we
	// want them to be for the screen coordinates this method returns.
	let flippedMultiplier = self.isFlipped ? -1.0 as CGFloat : 1.0 as CGFloat
	let verticalOffset = flippedMultiplier * autosizingCell.verticalAdjustment(for: parentFrame)

	return NSMakeRect(parentFrame.origin.x + horizontalOffset, parentFrame.origin.y + verticalOffset, parentFrame.width, parentFrame.height)
}

Effectively I take the default accessibility frame, and nudge it by the same amount that my custom autosizing text cell is nudging the content during drawing. The result is only subtly difference, but makes a nice visual refinement, and a big improvement to usability:

Screen shot of dictionary lookup UI with properly aligned word focus.

I thought this was an interesting example of the accessibility frameworks being leveraged to provide a service that benefits a very wide spectrum of Mac users. There’s a conventional wisdom about accessibility that emphasizing the accessibility of apps will make the app more usable specifically for users who take advantage of screen readers and other accommodations, but more generally for everybody who uses the app. This is a pretty powerful example of that being the case!

Playground Graphs

I was playing around with the Swift standard library’s “map” function, when I noticed a cool feature of Xcode Playgrounds. Suppose you are working with an array of numbers. In the Xcode Playgrounds “results” section, you can either click the Quick Look “eye” icon, or click the little results rectangle to get an inline results view of the expression you’re viewing:

Screenshot of Xcode Playgrounds's inline results view, revealing the values of an array of numb ers.

The linear list of values is revelatory and easy to read, but wouldn’t it be easier to understand as a graph? It turns out simply passing these values through the map function does just that:

Screenshot of the Xcode Playgrounds's result of the map function when returning numeric values.

I thought I had stumbled on some magical secret of Xcode, but it turns out the behavior is well documented, and applies to more than just the “map” function. You can even grab the edges of the result view and resize it to better suit your data. In fact, any looping numeric value seems to trigger the availability of this handy graphing functionality:

Screenshot of Xcode Playgrounds showing a graph of the results of Fibonacci sequence.

I am still frustrated by a lot of behaviors of Xcode Playgrounds, but little gems like these are nice to stumble upon.