Category Archives: Hacking

Xcode’s Secret Performance Tests

I was inspired today, by a question from another developer, to dig into Xcode’s performance testing. This developer had observed that XCTestCase exposes a property, defaultPerformanceMetrics, whose documentation strongly suggests can be used to add additional performance metrics:

This method returns XCTPerformanceMetric_WallClockTime by default. Subclasses of XCTestCase can override this method to change the behavior of measureBlock:.

If you’re not already familiar, the basic approach to using Xcode’s performance testing infrastructure is you add unit tests to your project that wrap code with instructions to measure performance. From the default unit test template:

func testPerformanceExample() {
	// This is an example of a performance test case.
	self.measure {
		// Put the code you want to measure the time of here.

Depending on the application under test, one can imagine all manner of interesting things that might be useful to tabulate during the course of a critical length of code. As mentioned in the documentation, “Wall Clock Time” is the default performance metric. But what else can be measured?


At least, according to any header files, documentation, WWDC presentations, or blunt Googling that I have encountered. There is exactly one publicly documented Xcode performance testing metric, and it’s XCTPerformanceMetric_WallClockTime.

I was curious whether supporting additional, custom performance metrics might be possible but under-documented. To test this theory, I added “beansCounted” to the list of performance metrics returned from my XCTestCase subclass. For some reason I couldn’t get Swift to accept the XCTPerformanceMetric pseudo-type, but it allowed me to override as returning an array of String:

override static func defaultPerformanceMetrics() -> [String] {
	return ["beansCounted"]

When I build and test, this fails with a runtime exception “Unknown metric: beansCounted”. The location of an exception like this is a great clue about where to go hunting for information about whether an uknown metric can be made into a known one! If there’s a trick to implementing support for my custom “beansCounted” metric, the answer lies in the method XCTestCase’s “measureMetrics(_: automaticallyStartMeasuring: forBlock:)”, which is where the exception was thrown.

By setting a breakpoint on this method and stepping through the assembly in Xcode, I can watch as the logic unfolds. To simplify what happens: first, a list of allowable metrics is computed, and then the list of desired metrics is iterated. If any metric is not in the list? Bzzt! Throw an exception.

I determined that things are relatively hardcoded such that it’s not trivial to add support for a new metric. I was hoping I could implement some magic methods in my test case, like “startMeasuring_beansCounted” and “stopMeasuring_beansCounted”

but that doesn’t appear to be the case. The performance metrics are supported internally by a private Apple class called XCTPerformanceMetric, and the list of allowable metrics is derived from a few metrics hardcoded in the “measureMetrics…” method:

  • “”
  • “”
  • “”
  • “”

As well as a bunch of others exposed by a private “knownMemoryMetrics” method:

  • “”
  • “”
  • “”
  • “”
  • “”
  • “”
  • “”
  • “”
  • “”
  • “”

How interesting! There are a lot more metrics defined than the single “wall clock time” exposed by Apple. So, should we use them? Official answer: no way! This is private, unsupported stuff, and can’t be relied upon. Punkass Daniel Jalkut answer? Why not? They’re your tests, and your the only one who will get hurt if they suddenly stop working. In my opinion taking advantage of private, undocumented system behavior for private, internal gain is much different than shipping public software that relies upon such undocumented behaviors.

I modified my unit test subclass to return a custom array of tests based on the discoveries above, just to test a few:

override static func defaultPerformanceMetrics() -> [String] {
	return [XCTPerformanceMetric_WallClockTime, "", "", ""]

The tests build and run with no exception. That’s a good sign! But these “secret peformance tests” are only useful if they can be observed and tracked the way the wall clock time can be. How does Xcode hold up? I made my demonstration test purposefully impactful on some metrics:

func testPerformanceExample() {
	self.measure {
		for _ in 1..<100 {
			print("wasting time")
		let _ = malloc(3000)

Now when I build and test, look what shows up in the Test navigator’s editor pane:

Screenshot of performance metrics after reducing the size of allocations and length of run.

Look at all those extra columns! And if I click the “Set Baselines…” button, then tweak my function to make it substantially less performant:

func testPerformanceExample() {
	self.measure {
		for _ in 1..<10000 {
			print("wasting time")
		let _ = malloc(300000)

Now the columns have noticably larger numbers:

Screenshot of Xcode's test results after running tests with

But more importantly, the test fails:

Screenshot of test errors generated by failing to meet performance baselines.

I already mentioned that by any official standard, you should not take advantage of these secret metrics. They are clearly not supported by Apple, may be inaccurate or have bugs, and could outright stop working at any time. I also said that, in my humble opinion, you should feel free to use them if you can take advantage of them. The fact that they are supported so well in Xcode probably implies that groups internal to Apple are using them and benefiting from them. Your mileage may vary.

The only rule is this: if Apple does do anything to change their behavior, or you otherwise ruin your day by deciding to play with them, you shouldn’t blame Apple, and you can’t blame me!


Touch Bar Crash Protection

I wrote previously about crashes related to Apple’s Touch Bar. These crashes seem to affect all apps that were built with a certain toolchain. Most likely it affects all apps that were built against an SDK of a certain vintage. For example, some of my apps that are still built against a 10.6 SDK crash on Touch Bar Macs, either frequently or infrequently, depending upon the user.

I had hoped that we might see a fix from Apple in macOS 10.12.2, but alas the issue is still there. This left me feeling obligated to my customers to find a solution that I can deploy soon. I don’t know if Apple considers the crashes a problem worth pursuing, and if so, how soon they plan to deliver a fix.

Poking around the AppKit infrastructure supporting the Touch Bar, I discovered a secret NSUserDefaults setting, NSFunctionBarAPIEnabled, which seems to determine whether the system exposes an app to the Touch Bar at all. It defaults to YES, but if it’s set to NO for an app, I think the app remains more or less invisible to the Touch Bar.

I have very reproducible test cases for many apps, including Apple’s own SystemUIServer process, so I decided to play around with the NSFunctionBarAPIEnabled user default and see how things go. To my satisfaction, setting the value explicitly to NO for any of the affected apps completely eliminates the crashes:

defaults write NSFunctionBarAPIEnabled -bool NO

SystemUIServer is an interesting example, because I can’t honestly imagine what I’m giving up by disabling Touch Bar support in the app. It’s probably a case where having the default on by default is exposing it to bugs in the Touch Bar infrastructure, even though it will never benefit by having Touch Bar support enabled.

Other apps are not so clear cut: you might have an affected app on your Mac that “works with the Touch Bar,” even though it doesn’t do anything special to support it yet. My own app, MarsEdit, is one such app. The Touch Bar works when you’re focused in on some system-standard UI element such as a text view, but it doesn’t do anything special throughout most of the app. In a situation like this, if you are suffering many crashes as a user, you might decide to do something like the above, writing a custom NO setting to the NSFunctionBarAPIEnabled value. Be aware, however, if you do this that you’ll lose Touch Bar functionality for that app forever, or at least until you remember you set this funny default value.

Getting back to my motivation to eliminate these crashes as soon as possible for my customers, I think that I will ship an update to MarsEdit that disables the Touch Bar, but does so in a transient manner. By registering a default value in the app itself I will not force users to save any permanent value in preferences, and will also give them the ability to override my judgement as they see fit. If you wanted to do something like this in an app, you could add a few lines like this to main.m:

NSDictionary* myDict = [NSDictionary dictionaryWithObject:[NSNumber numberWithBool:NO] forKey:@"NSFunctionBarAPIEnabled"];
[[NSUserDefaults standardUserDefaults] registerDefaults:myDict];

You want to put this early in your app’s launch, so that it’s registered before AppKit’s Touch Bar infrastructure loads up. When it sees that NSFunctionBarAPIEnabled is set to NO, it will kindly avoid initializing the classes which are evidently making many apps prone to crashes on Touch Bar Macs.

I haven’t decided for sure yet whether to ship with this in place, but unless I find a more suitable workaround, I think I will. Disabling Touch Bar support entirely in the short term will be preferable to subjecting my customers to unpredictable crashes that are out of my control.

Xcode 6 On Sierra

Xcode 6 and Xcode 7 are not supported by Apple on macOS Sierra, and should not be used for production work.

But what if you have a good reason for running one or the other? Maybe you want to compare a behavior in the latest Xcode 8 with an earlier version of the app. Instead of keeping a virtual machine around, or a second partition with an older OS release, it is liberating to be able to run and test against older versions of Xcode.

So far, it appears that Xcode 7 “mostly works” in spite of being unsupported by Apple. I’ve run into some launch-time crashes, but reopening the app tends to succeed.

Xcode 6 will flat out fail to launch, because one of its internal plugins depends on a private framework (Ubiquity.framework) that is no longer present on macOS Sierra. If you were willing to hack a copy of Xcode 6, however, you could get it running. You definitely shouldn’t do this, but if you’re curious how it could be done, here’s how:

  1. Always have a backup copy of any data that is important to you.
  2. Locate a copy of /System/Library/PrivateFrameworks/Ubiquity.framework from the previous OS X release.
  3. Copy the framework to within Xcode 6’s own Contents/Frameworks bundle subfolder:
    ditto /my/old/System/Library/PrivateFrameworks/Ubiquity.framework ./
  4. Navigate to the problematic Xcode plugin and modify its library lookup table so that it points to the app-bundled copy of Ubiquity.framework, instead of the non-existent system-installed copy.
    install_name_tool -change /System/Library/PrivateFrameworks/Ubiquity.framework/Versions/A/Ubiquity @rpath/Ubiquity.framework/Versions/A/Ubiquity ./iCloudSupport
  5. Now that you've modified Xcode, its code signature is invalid. You can repair it by signing it with your own credentials or with an ad hoc credential:
    codesign --deep -f -s - ./
  6. Did I mention you really shouldn't do this?

Apple has good reason to warn people off running older versions of Xcode, but if you absolutely need to get something running again, it's often possible.


I’m working on some heavy NSTextView, NSScrollView, NSClipView type stuff in MarsEdit. This stuff is fraught with peril because of the intricate contract between the three classes to get everything in a text view, including its margins, scrolling offset, scroll bars, etc., all working and looking just right.

When faced with a problem I can’t solve by reading the documentation or Googling, I often find myself digging in at times, scratching my head, to Apple’s internal AppKit methods, to try to determine what I’m doing wrong. Or, just to learn with some certainty whether a specific method really does what I think the documentation says it does. Yeah, I’m weird like this.

I was cruising through -[NSClipView scrollToPoint:] today and I came across an enticing little test (actually in the internal _immediateScrollToPoint: support method):

0x7fff82d1e246 <+246>:  callq  0x7fff82d20562            ; _NSDebugScrolling

0x7fff82d1e24b <+251>:  testb  %al, %al

0x7fff82d1e24d <+253>:  je     0x7fff82d20130            ; <+8160>

0x7fff82d1e253 <+259>:  movq   -0x468(%rbp), %rdi

0x7fff82d1e25a <+266>:  callq  0x7fff8361635e            ; symbol stub for: NSStringFromSelector

0x7fff82d1e25f <+271>:  movq   %rax, %rcx

0x7fff82d1e262 <+274>:  xorl   %ebx, %ebx

0x7fff82d1e264 <+276>:  leaq   -0x118d54fb(%rip), %rdi   ; @“Exiting %@ scrollHoriz == scrollVert == 0”

0x7fff82d1e26b <+283>:  xorl   %eax, %eax

0x7fff82d1e26d <+285>:  movq   %rcx, %rsi

0x7fff82d1e270 <+288>:  callq  0x7fff83616274            ; symbol stub for: NSLog


Hey, _NSDebugScrolling? That sounds like something I could use right about now. It looks like AppKit is prepared to spit out some number of logging messages to benefit debugging this stuff, under some circumstances. So how do I get in on the party? Let’s step into _NSDebugScrolling:


0x7fff82d20562 <+0>:   pushq  %rbp

0x7fff82d20563 <+1>:   movq   %rsp, %rbp

0x7fff82d20566 <+4>:   pushq  %r14

0x7fff82d20568 <+6>:   pushq  %rbx

0x7fff82d20569 <+7>:   movq   -0x11677e80(%rip), %rax   ; _NSDebugScrolling.cachedValue

0x7fff82d20570 <+14>:  cmpq   $-0x2, %rax

0x7fff82d20574 <+18>:  jne    0x7fff82d20615            ; <+179>

0x7fff82d2057a <+24>:  movq   -0x116a7ad9(%rip), %rdi   ; (void *)0x00007fff751a9b78: NSUserDefaults

0x7fff82d20581 <+31>:  movq   -0x116d5df8(%rip), %rsi   ; “standardUserDefaults”

0x7fff82d20588 <+38>:  movq   -0x1192263f(%rip), %rbx   ; (void *)0x00007fff882ed4c0: objc_msgSend

0x7fff82d2058f <+45>:  callq  *%rbx

0x7fff82d20591 <+47>:  movq   -0x116d5fa0(%rip), %rsi   ; “objectForKey:”

0x7fff82d20598 <+54>:  leaq   -0x118ab0cf(%rip), %rdx   ; @“NSDebugScrolling”

0x7fff82d2059f <+61>:  movq   %rax, %rdi

0x7fff82d205a2 <+64>:  callq  *%rbx


Aha! So all i have to do is set NSDebugScrolling to YES in my app’s preferences, and re-launch to get the benefit of this surely amazing mechanism. Open the Scheme Editor for the active scheme, and add the user defaults key to the arguments passed on launch:

Screenshot 3 29 16 3 50 PM

You can see a few other options in there that I sometimes run with. But unlike those, NSDebugScrolling appears to be undocumented. Googling for it yields only one result, where it’s mentioned offhand in a Macworld user forum as something “you could try.”

I re-launched my app, excited to see the plethora of debugging information that would stream across my console, undoubtedly providing the clues to solve whatever vexing little problem led me to stepping through AppKit assembly code in the first place. The results after running and scrolling the content in my app?

Exiting _immediateScrollToPoint: without attempting scroll copy ([self _isPixelAlignedInWindow]=1)

I was a little underwhelmed. To be fair, that might be interesting, if I had any idea what it meant. Given that I’m on a Retina-based Mac, it might indicate that a scrollToPoint: was attempted that would have amounted to a no-op because it was only scrolling, say, one pixel, on a display where scrolling must move by two pixels or more in order to be visible. I’m hoping it’s nothing to worry about.

But what else can I epect to be notified about by this flag? Judging from the assembly language at the top of this post, the way Apple imposes these messages in their code seems to be based on a compile-time macro that expands to always call that internal _NSDebugScrolling method, and then NSLog if it returns true. Based on the assumption that they use the same or similar macro everywhere these debugging logs are injected, I can resort to binary analysis from the Terminal:

cd /System/Library/Frameworks/AppKit.framework
otool -tvV AppKit | grep -C 20 _NSDebugScrolling

This dumps the disssembly of the AppKit framework binary, greps for _NSDebugScrolling, and asks that 20 lines of context before and after every match be provided. This gives me a pretty concise little summary of all the calls to _NSDebugScrolling in AppKit. It’s pretty darned concise. In all there are only 7 calls to _NSDebugScrolling, and given the context, you can see the types of NSLog strings would be printed in each case. None of it seems particularly suitable to the type of debugging I’m doing at the moment. It’s more like plumbing feedback from within the framework that would probably mainly be interesting from an internal implementor’s point of view. Which probably explain why this debugging key is not publicized, and is only available to folks who go sticking their nose in assembly code where it doesn’t belong.