Monthly Archives: December 2016

Touch Bar Crash Protection

Hacking, Mac

I wrote previously about crashes related to Apple’s Touch Bar. These crashes seem to affect all apps that were built with a certain toolchain. Most likely it affects all apps that were built against an SDK of a certain vintage. For example, some of my apps that are still built against a 10.6 SDK crash on Touch Bar Macs, either frequently or infrequently, depending upon the user.

I had hoped that we might see a fix from Apple in macOS 10.12.2, but alas the issue is still there. This left me feeling obligated to my customers to find a solution that I can deploy soon. I don’t know if Apple considers the crashes a problem worth pursuing, and if so, how soon they plan to deliver a fix.

Poking around the AppKit infrastructure supporting the Touch Bar, I discovered a secret NSUserDefaults setting, NSFunctionBarAPIEnabled, which seems to determine whether the system exposes an app to the Touch Bar at all. It defaults to YES, but if it’s set to NO for an app, I think the app remains more or less invisible to the Touch Bar.

I have very reproducible test cases for many apps, including Apple’s own SystemUIServer process, so I decided to play around with the NSFunctionBarAPIEnabled user default and see how things go. To my satisfaction, setting the value explicitly to NO for any of the affected apps completely eliminates the crashes:

defaults write com.apple.systemuiserver NSFunctionBarAPIEnabled -bool NO

SystemUIServer is an interesting example, because I can’t honestly imagine what I’m giving up by disabling Touch Bar support in the app. It’s probably a case where having the default on by default is exposing it to bugs in the Touch Bar infrastructure, even though it will never benefit by having Touch Bar support enabled.

Other apps are not so clear cut: you might have an affected app on your Mac that “works with the Touch Bar,” even though it doesn’t do anything special to support it yet. My own app, MarsEdit, is one such app. The Touch Bar works when you’re focused in on some system-standard UI element such as a text view, but it doesn’t do anything special throughout most of the app. In a situation like this, if you are suffering many crashes as a user, you might decide to do something like the above, writing a custom NO setting to the NSFunctionBarAPIEnabled value. Be aware, however, if you do this that you’ll lose Touch Bar functionality for that app forever, or at least until you remember you set this funny default value.

Getting back to my motivation to eliminate these crashes as soon as possible for my customers, I think that I will ship an update to MarsEdit that disables the Touch Bar, but does so in a transient manner. By registering a default value in the app itself I will not force users to save any permanent value in preferences, and will also give them the ability to override my judgement as they see fit. If you wanted to do something like this in an app, you could add a few lines like this to main.m:

NSDictionary* myDict = [NSDictionary dictionaryWithObject:[NSNumber numberWithBool:NO] forKey:@"NSFunctionBarAPIEnabled"];
[[NSUserDefaults standardUserDefaults] registerDefaults:myDict];

You want to put this early in your app’s launch, so that it’s registered before AppKit’s Touch Bar infrastructure loads up. When it sees that NSFunctionBarAPIEnabled is set to NO, it will kindly avoid initializing the classes which are evidently making many apps prone to crashes on Touch Bar Macs.

I haven’t decided for sure yet whether to ship with this in place, but unless I find a more suitable workaround, I think I will. Disabling Touch Bar support entirely in the short term will be preferable to subjecting my customers to unpredictable crashes that are out of my control.

Installing Symbols For WatchOS

Bug Reports, watchOS, Xcode

For months I have been plagued by Xcode’s persistent failure to successfully install WatchOS device support on my Mac. If I open Xcode, and my iPhone (paired with an Apple Watch) is attached to the Mac, I am greeted by a progress indicator such as this:

InterfaceController swift

Because I don’t actually do any watchOS development, this has mostly been a mild annoyance. The bug doesn’t seem to affect the performance of my Mac, and other actions I perform in Xcode continue without delay. However, for those of you who are actually doing Watch development, if you run into this problem, you’ll be perpetually greeted by this error when you try to debug on a real device:

Xcode panel refusing to install on a device because of missing Apple Watch symbols.

The worst side-effect of this bug for me, and the problem that finally encouraged me to hunker down and solve the issue, is the fact that each attempt to “Install Symbols” is associated with a fresh download of a larger-than-200MB file from Apple to my Mac. This is no big deal on my home network, but when I’m roaming and tethered to my bandwidth-metered iPhone, it can lead to costly overage charges.

I’ve been to hell and back tracking down exactly how Xcode installs these symbols, where it fails, and how to work around the problem. I’ll give an overview of how Xcode’s symbol installation process works, and walk you through one concrete method for working around the problem, getting those precious symbols installed, and breaking the cycle of perpetual failed installations.

Symbol Installation Overview

First I want to give you a high level idea of how Xcode downloads and installs symbols. Here are the steps that take place:

  1. A device suitable for development is connected.
  2. Xcode looks for existing symbols in the suitable home directory location
  3. If symbols are not already installed, the symbol-installation process begins.
  4. A disk image file containing the required materials is downloaded to a temporary folder.
  5. The disk image file is copied to a Caches subfolder in your home directory.
  6. The disk image is mounted at a temporary location on your Mac
  7. A macOS Installer package on the disk image is used to install symbol files directly into your home folder.
  8. The temporary disk image is unmounted.
  9. The cached disk image file is deleted.

In my tests, the failure to successfully install watchOS symbols broke down in step 7, where the package is supposed to successfully copy symbol files into my home folder. I was able to determine that the Apple infrastructure responsible for the installation is failing on my Mac with a cryptic authentication error:

Could not download and install Symbols for watchOS 3.1 (14S471). Authorization is required to install the packages

The failure is not happening in Xcode per se, but in a helper tool that is part of the private PackageKit.framework. I filed Radar #29568241, in the hopes that somebody at Apple will have insight as the root cause of the problem, and how it can be fixed for good in either Xcode, or in the configurations of affected users’ Macs.

Fix It Yourself

To manually accomplish what Xcode is failing to do, we need to:

  1. Get a copy of the downloaded disk image.
  2. Copy the pertinent files out of the disk image’s installer package.
  3. Copy the pertinent files into the correct installation path in ~/Library/Developer/Xcode/

Because simply creating the directory at the installation path is enough to stop Xcode attempting to install symbols, I’m going to cover that first. Folks who don’t need or want the symbols for Watch development will solve the problem sufficiently after this step.

Determine the Installation Path

Unfortunately, the path is quite specific to the Watch device at hand, and may be difficult to guess. Here’s one way of figuring it out. Making sure you have an iPhone connected that triggers the symbol installation in Xcode, do the following:

  1. Quit Xcode.
  2. From the Terminal, navigate to Xcode’s binary executable folder: 
    cd /Applications/Xcode.app/Contents/MacOS/
  3. Run Xcode with a special flag to cause additional logging to display: 
    ./Xcode -DVTDownloadableLogLevel 3
  4. Wait for Xcode to launch and start installing symbols.
  5. Scan the output in Terminal for a line like this: 
    Starting a download for Watch1,2 3.1 (14S471), to file path /Volumes/Data/daniel/Library/Developer/Xcode/watchOS DeviceSupport/Watch1,2 3.1 (14S471)/Symbols

    Of course, your output will be slightly different, but depending on the version of Watch and watchOS you have, the important part is relative to your home folder. In my case, I need to make sure this folder exists:

    ~/Library/Developer/Xcode/watchOS DeviceSupport/Watch1,2 3.1 (14S471)/Symbols

If you don’t care about Watch development, this is your big chance: just create that folder now. Quit and relaunch Xcode again, and you’ll see that your problems are over. No more symbol installations required. If you skim the logging output from above you’ll see that the “InstalledIfAllPathsArePresent” key in one of the dictionaries indicates that the path merely being present is enough to convince Xcode it’s “installed.”

If you do care about Watch development, you’re going to want those symbols. Let’s tackle that problem next.

Getting the Disk Image

Remember back in “Symbol Installation Overview,” I described the installation failing in step 7: where a package on a mounted volume is attempted to be installed. Unfortunately, this failure does not prevent steps 8 and 9 from proceeding, so when the whole procedure is over, there are no obvious artifacts left around to work with. The disk image, that was copied to ~/Library/Caches/com.apple.dt.Xcode/Downloads in step 5, is available for such a short period of time, you’re unlikely to nab it before Xcode deletes it upon failure.

You may have noticed in the Xcode logging output that many URLs to web resources are passed around. It’s likely that one of these contains the required disk image content, but I thought it would be safer to catch Xcode in the act and use precisely the disk image file that it itself was intending to use.

Yet another “secret” Xcode user default key can help us here. Once again, navigate to the Terminal to execute Xcode manually. This time, we’ll pass a flag that instructs it explicitly not to delete the coveted disk image file after it’s done:

./Xcode -DVTDownloadableLeaveTemporaryFiles 1

This time you’ll have to wait again until the process tries, and fails, but when you’re done you should find a suitable disk image file waiting for you at:

~/Library/Caches/com.apple.dt.Xcode/Downloads/Xcode.Symbols.Watch1,2.14S471-3.1.dmg

(Or with whatever name is suitable for the version of Watch and OS you are accommodating.)

Copying the Symbol Files Out

Double-click the disk image to mount it on your Mac. You’ll find a volume appears with a suitable device-oriented name, and a matching file within called e.g. “Watch1,1_Watch1,2.pkg”.

Warning: Do not open and install this package willy-nilly without reading the following carefully. This is a standard Installer package, and double-clicking it on your Mac will launch the system-standard Installer application. However, agreeing to let it “just install” on your Mac could have unintended consequences. The files in this package are named very similarly to many system files on your Mac, but of course these files are binaries for a ARM based Watch device, and not an x86_64 personal computer.

There are many techniques for getting the files out of an Installer Package without actually, you know, installing. You could use an app like Pacifist to extract them directly to a folder on your disk. There may also be a way to invoke the command-line “installer” tool in such a way that it simply spits the files out where you want them. For that matter, it might be safe to simply “Choose Folder…” from the Installer app, and trust that it will install everything in that folder instead of on the root of the volume.

Me? I decided to strike a semi-paranoid compromise. I used the Installer app, but instead of installing to my Mac’s main volume, I created a writable disk image in Disk Utility, and targeted that volume explicitly:

WatchHack

When the installation was complete, I had a mounted volume, filled with the contents that Xcode has presumably been intending to copy into my home folder.

WatchHack

Finish the Job

At this point you know correct install location in your home folder, and you have a copy of all the files that should be there. It should be as simple as selecting the files from “WatchHack” and dragging them to the “Symbols” folder in the appropriate ~/Library/Developer subfolder.

When the symbol files have been copied in, quit and relaunch Xcode. Not only should it avoid that nasty “Installing Symbols” phase, but building and running on your physical watchOS device should once again work as expected.

Touch Bar Crashers

Bug Reports

Since the MacBook Pro with Touch Bar was released, I’ve noticed a small but steady influx of new crash reports against my apps. Typically, the crashes in something CALayer related, and the correlated crash log often descends from “-[DFRElement dealloc]”:

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libobjc.A.dylib               	0x00007fffea5211cf objc_class::demangledName(bool) + 33
1   com.apple.CoreFoundation      	0x00007fffd58b6c20 ___forwarding___ + 144
2   com.apple.CoreFoundation      	0x00007fffd58b6b08 _CF_forwarding_prep_0 + 120
3   com.apple.QuartzCore          	0x00007fffdb7012a8 -[CALayer actionForKey:] + 95
4   com.apple.QuartzCore          	0x00007fffdb6fbccf CA::Layer::mark_visible(CA::Transaction*, bool) + 345
5   com.apple.QuartzCore          	0x00007fffdb6fbe8c CA::Layer::set_visible(unsigned int) + 306
6   com.apple.QuartzCore          	0x00007fffdb6eef21 CA::Context::invalidate() + 75
7   com.apple.framework.DFRFoundation	0x00007fffe180c7bd -[DFRElement dealloc] + 113
[...]

That DFR prefix rang a bell for me, because it’s the same prefix as some of Apple’s private frameworks that support the Touch Bar simulator in Xcode, which I also use in my free, standalone Touch Bar simulator, Touché.

User comments associated with these crashes often mentioned connecting or disconnecting an external monitor, or with waking their MacBook from sleep.

I speculated that the crashes might represent some subtle bug that is only likely to happen when the Touch Bar infrastructure on the Mac is forced to stop and start again. I would be surprised if, when the computer goes to sleep, the physical Touch Bar on the MacBook didn’t also stop streaming content from the computer. When the computer wakes, it would make sense for it to trigger a resumption of that streaming.

Hmm, if only I had a Touch Bar Mac, so I could try to reproduce these crashes. I would open my apps, then I would close the laptop lid, and open it again, say 100,000 times or so, until I reproduce the problem.

Since I had already written a Touch Bar simulator app, which itself has the ability to stop and start streaming of the Touch Bar content, I speculated that I might be able to provoke the same crashes from within my app. Indeed, some of my users who reported these crashes were not using Touch Bar Macs, but were running Touché.

I rigged up a special version of my app that would simply show and hide the Touch Bar simulator window repeatedly, perhaps thousands of times per second. Maybe that would trigger the crashing bug that only rarely affects my customers.

It crashes my apps all right, usually in less than a second. The funny thing is it seemed to only crash my apps. None of Apple’s apps seemed to be dying. Other 3rd party apps on my Mac were also holding up fine.

Finally I came across another app that also crashed, and came up with a theory: only apps that are built with a suitably older SDK are crashing. For reasons I won’t go into, most of my apps are still built on a Mac OS X 10.6 Mac running Xcode 6. Sure enough, as I went out scouting for apps that I suspected were of a similar vintage, I was able to crash them in “less than a second” as well.

If you’re on a Touch Bar Mac, or running Touché, and seeing crashes in apps that culminate in CALayer, I think there’s a very good chance it represents a bug in Apple’s Touch Bar support on the Mac. I’ve filed a bug with a reprodeucable test case: Radar #29537507.