All posts by Daniel Jalkut

Swatch Your Step

Shortly after macOS 10.13 was released, I received an oddly specific bug report from a customer, who observed that the little square “swatches” in the standard Mac color panel no longer had any effect on MarsEdit’s rich text editor.

Screenshot of the macOS standard color panel.

I was able to reproduce the problem in the shipping 3.7.11 version of MarsEdit, which for various reasons is still built using an older version of Xcode, against the 10.6 SDK. The MarsEdit 4 Beta, which is built against the 10.12 SDK, does not exhibit the problem.

It’s not unusual for the behavior of Apple’s frameworks to vary based on the version of SDK an application was built against. The idea is usually to preserve the old behaviors of frameworks, so that any changes do not defy the expectations of a developer who has not been able to build and test their app against a later SDK. Sometimes, the variations in behavior lead to bugs like this one.

Using a totally straightforward demo app, consisting only of an NSTextView and a button to bring up the color panel, I was able to confirm that the bug affects an app that links against the macOS 10.9 SDK, but does not affect an app that links against the 10.10 SDK.

I filed Radar #34757710: “NSColorPanel swatches don’t work on apps linked against 10.9 or earlier.” I don’t know of a workaround yet, other than compiling against a later SDK.

Xcode 9 Signing Workarounds

I wrote on Monday about issues with Xcode 9 relating to code signing. Although the gist of that post involved sandboxed Mac applications that launch sandboxed child processes, the fundamental issue is a bit broader: Xcode 9 adds a “com.apple.security.get-task-allow” entitlement to any binary it signs. For the majority of developers, this is probably not an issue, because the entitlement is removed when an Xcode archive is exported for distribution. Most developers, and particularly iOS developers, use Xcode archives.

For folks who don’t, side effects of this additional entitlement include, but may not be limited to:

  1. Inability to launch sandboxed child processes.
  2. Rejection from the Mac App Store.
  3. Unknown consequences of shipping with an unintended entitlement.

So, if you’re a developer who doesn’t use archives, what are your options? I’ve come up with four workarounds, and I present them here, roughly sorted by advisability and level of tedium:

  1. Use Xcode 8. The simplest solution is to not upgrade to Xcode 9 unless and until you need to. Xcode 8’s signing process does not impose the unintended entitlement, so there is no risk of shipping a product that has it, unless you add it yourself. The downside to sticking with Xcode 8 is you won’t enjoy any of the new features of Xcode 9, you’ll have to work to support either Swift 4, macOS 10.13, or iOS 11 SDK features in your app.

  2. Manually re-sign the built-product. Code signing is code signing, and you’re free to sign anything you like to suit your needs, using the “codesign” command line tool. It frankly sounds like a pain in the neck to recursively re-sign every binary in the app bundle, ensuring that the suitable entitlements (minus the unwanted one) are preserved, but I’m sure it can be done.

  3. Use Xcode archives. It strikes me as a little obnoxious to have to use Xcode archives when they don’t offer any added benefits for my dibstrution workflow. But as a long term solution, this is probably the safest bet. The new behavior in Xcode 9 strongly suggests that Apple expects most developers to use archives, and joining the crowd is usually a good idea when it comes to avoiding trouble with Apple’s developer tools.

    If you are using Xcode archives for the first time, particularly with a complex project, you might discover that the resulting archives are not suitable for exporting a signed application. If you get a “Generic Xcode Archive” after running Build -> Archive, you know you’ve got a problem. By default the archive process builds all targets with an “install” option, rendering their built products into a file hierarchy that will be used to build the archive. If your project includes helper apps, for example, they will be “installed” alongside your main app, resulting in a generic archive of two apps, instead of the expected archive of a single app.

    The solution for this problem is to ensure that the “SKIP_INSTALL” build setting is set to YES for any such helper app. Just archive your main app, export the “Built Products” from the resulting archive, and look at the file hierarchy to determine whether you have subtargets that need to have installation disabled.

  4. Hack Xcode 9. In a hurry to ship an update to your app, and you’ve only got Xcode 9 handy? It turns out the imposition of this “com.apple.security.get-task-allow” entitlement is controlled by a single property list file inside Xcode’s application bundle. As a test, I edited the file:

    Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/Library/Xcode/PrivatePlugIns/IDEOSXSupportCore.ideplugin/Contents/Resources/BaseEntitlements.plist
    

    It contains a single entitlement, the one that’s causing our grief. I deleted the entitlement from the list, saved the file, and relaunched Xcode. After doing so, everything is “back to normal.”

    I can’t strongly encourage you to hack your copy of Xcode because I don’t know what the consequences might be. “It seems fine,” but you’re on your own if you decide to do this.

This small change in Xcode 9 causes a lot of unexpected grief for folks who don’t use Xcode archives. I am curious to know how widespread the problem is, and enthusiastic to get the word out about it so that affected folks can work around the problem, or at least be aware of it. Myself, I’ll probably end up adopting the workaround of using Xcode archives, but I’m hopeful that Apple will see the merit of providing an option in an update to Xcode 9 that supports disabling the addition of this entitlement without archiving and exporting a built product.

Unordered Directory Contents

Since I updated to macOS 10.13 High Sierra, some of my unit tests broke. Examining the failures more carefully, I discovered that they were making assumptions about the order that Foundation’s FileManager.contentsOfDirectory(atPath:) would return items.

I wrote a quick playground to test the behavior on a 10.12 machine:

import Foundation

let array = try! FileManager.default.contentsOfDirectory(atPath: "/Applications/Utilities")
print("\(array.debugDescription)")

The results come back alphabetically ordered by file name:

[".DS_Store", ".localized", "Activity Monitor.app", "Adobe Flash Player Install Manager.app", "AirPort Utility.app", "Audio MIDI Setup.app", "Bluetooth File Exchange.app", "Boot Camp Assistant.app", "ColorSync Utility.app", "Console.app", "Digital Color Meter.app", "Disk Utility.app", "Grab.app", "Grapher.app", "Keychain Access.app", "Migration Assistant.app", "Script Editor.app", "System Information.app", "Terminal.app", "VoiceOver Utility.app"]

The same playground on 10.13 tells a different story:

["AirPort Utility.app", "VoiceOver Utility.app", "Terminal.app", "Activity Monitor.app", ".DS_Store", "Grapher.app", "Audio MIDI Setup.app", ".localized", "System Information.app", "Keychain Access.app", "Grab.app", "Migration Assistant.app", "Script Editor.app", "ColorSync Utility.app", "Console.app", "Disk Utility.app", "Bluetooth File Exchange.app", "Boot Camp Assistant.app", "Digital Color Meter.app"]

I thought at first this might have been related to the APFS conversion that 10.13 applied to my boot volume, but the same ordering discrepancy occurs for items on my HFS+ volumes as well.

After checking the 10.13 release notes for clues, and finding none, I consulted the documentation. Well, what do you know?

The order of the files in the returned array is undefined.

So, mea culpa. The test code in question probably shouldn’t have ever made assumptions about the ordering of items returned from this method. While it has evidently always been undefined, it appears they are only making good on that promise in 10.13. You have been warned!

Update: It turns out I have some real bugs in my apps, not just in my tests, because of assuming the results of this call will be reasonably sorted. Luckily I use a bottleneck method for obtaining the list of files, and I can impose my own sorting right at the source. If you’re looking to make the same kinds of changes to your app, be sure to heed Peter Maurer’s advice and use “localizedStandardCompare” (available since macOS10.6/iOS4) to obtain Finder-like ordering of the results.

Sandbox Inheritance Tax

I ran into a subtle bug with Xcode 9 that I think is worth sharing. Specifically, this bug affects Mac applications that:

  1. Are sandboxed.
  2. Launch a sandboxed subprocess with NSTask (or posix_spawn).
  3. Configure the subprocess to inherit the parent’s sandbox.

When such an app is compiled with Xcode 9, the subprocess will crash whenever the parent process launches it. A canonical example of something that might suffer from this problem is a bundled crash-monitor. I embed one with my apps to keep an eye on the running status of the parent process, and to present a crash-reporting interface to users if the host app terminates prematurely. When I build and run my app with Xcode 9, the bundled crash monitor dies instantly upon being launched.

It took me a while to realize that the subprocess is dying because it fails to satisfy the contract for inheriting a sandbox. From Apple’s “Enabling App Sandbox Inheritance“:

To enable sandbox inheritance, a child target must use exactly two App Sandbox entitlement keys: com.apple.security.app-sandbox and com.apple.security.inherit. If you specify any other App Sandbox entitlement, the system aborts the child process.

Well, that’s funny because my child process does specify only those two keys, but the system is aborting it anyway. It turns out that Xcode 9 is inserting a third entitlement without my permission. Clicking on the detail of the “Process Product Packaging” build phase in Xcode’s log navigator, I can see that there are three entitlements for my target:

Xcode build log detail showing the wrong entitlements.

When my subprocess is launched, the system sees that extra “com.apple.security.get-task-allow” entitlement in the context of “com.apple.security.inherit”, and unceremoniously crashes my the child process.

I’m not sure what Apple’s reasoning is for imposing this entitlement on sandboxed targets, but it appears to be doing so across the board, for literally every sandboxed target in my app. I confirmed that all of my apps, XPC processes, helper tools, etc., are all getting this bonus entitlement.

I searched Xcode’s files, and discovered the entitlement listed in this file inside the Xcode app bundle:

Contents/Developer/Platforms/MacOSX.platform/Developer/Library/Xcode/PrivatePlugIns/IDEOSXSupportCore.ideplugin/Contents/Resources/BaseEntitlements.plist

Putting aside the question of whether it’s appropriate for Xcode to surreptitiously add entitlements that are not specified by the developer’s own list of permissions, the addition of the entitlement for these particular targets, ones that inherit their parent’s sandbox, turns out to be a fatal move.

Ideally I would be able to work around this by adding a custom build phase to manually tweak the generated entitlements file, removing the unwanted key. But the “Process Product Packaging” build phase happens so late in the build process that it’s after the last user-specified custom build phase. There’s no room in Xcode’s current design for fixing up the problematic entitlements before they are incorporated into the signed product. As far as I can tell the only clean workaround would be to redundantly re-sign the child app with a custom script, and corrected entitlements, after Xcode’s build process is completed.

I filed Radar #34628449, “Sandboxed project build with Xcode 9 cannot launch child process.”

Update: Colin Barrett pointed out on Twitter that the entitlement in question here, “com.apple.security.get-task-allow”, may be required in order to attach to and debug a process. If true, then I think this is something that was handled in a different way in Xcode 8. I can confirm that my apps do not have the entitlement imposed on them by Xcode 8, yet I am able to attach to and debug them.

If Apple changed the debugger infrastructure in Xcode 9 so that the relationship between the debugger and target processes is more locked down, requiring a specific entitlement, then that’s probably a good thing. But if this change was made without thinking about the implications for the above-cited “strict two entitlement” rule for sandbox inheritance, then probably some flexibility needs to be applied to that rule.

Finally, as I noted above the entitlement is being applied to all my targets. What I didn’t clarify is that the entitlement is added even when Building and Archiving. A release build’s binaries are endowed with this additional entitlement, which may also bring additional security vulnerabilities to the app.

I would not ship a sandboxed Mac app that is built with Xcode 9, until we understand more about when Xcode applies this entitlement, and whether it can be prevented for Release builds at the very least.

Update 2: I’ve learned that Xcode’s “Export Archive” functionality causes the unwanted entitlement to be removed. Apparently the assumption is that everybody creates Xcode archives as part of their build and release process. I am sure this is true for most (all?) iOS deployments, but for Developer-ID signed apps on the Mac, there has traditionally been less of an incentive to do this. Got a properly signed Mac application? Zip it up, put it on a web server, and you’re done.

I’m not sure yet whether I’ll switch my build process to use archiving, or whether I’ll pull some other stunt to redo the code signing with corrected entitlements. In any case this has been quite an adventure today getting to the bottom of this. I updated my bug report with Apple to request that they provide some standard build flag that would prevent the problematic entitlement from being added from the start. In the mean time, I’ll explore one of the workarounds and get my builds back to fully functional!

Better Swift Completion

Apple released Xcode 9 earlier this week, and in spite of a few glitches here and there, I have found the update to be an overall improvement over Xcode 8. It’s nice that Apple continues to invest in the core tools for Mac and iOS developers.

I’ve been dabbling in more and more Swift development lately, and it’s brought to light a shortcoming in Xcode’s code completion which has unfortunately not improved in Xcode 9: completion of Swift function calls when there is a large quantity of candidates.

Take for example NSAttributedString. If I want to initialize a new instance in Swift, I type “NSAttributedString(” to bring up the list of compatible init methods I can choose from:

SwiftCompletion

The problem at this point is that I have to navigate the menu by hand. I can’t narrow down the list of completions any further by typing, because the very next character I type will be interpreted as the manual filling out of parameters of the NSAttributedString initializer.

BadCompletion

This is a situation where Objective-C gets much nicer treatment in the editor. Because completion in Objective-C begins when I start typing “init”, and because the named first parameter is part of the init message name, I can winnow down the results quite a bit:

Pasted Image 9 22 17 11 24 AM

Better still, because Xcode performs a fuzzy match on the typing, I can proceed to type the names of additional parameters to zero in completely on the variation I want:

MEAppController AppDelegate m Edited

When I accept the completion, all of my typing is replaced with the expected, templated parameter placeholders for the chose initializer.

I filed Radar #34594940 requesting better completion for Swift.

JavaScript OSA Handler Invocation

When Apple added support to macOS to support JavaScript for Automation, they did so in a way that more or less allows folks who invoke AppleScripts to invoke JavaScript for Automation scripts as if they were exactly the same. An abstraction in Apple’s Open Script Architecture (OSA) makes it easy for script-running tools to theoretically handle any number of scripting languages without concern for the implementation details of those languages.

This mostly works, but I recently received a bug report that shed light on a problem with Apple’s implementation of JavaScript with respect to invoking a specific named handler. The OSA provides a mechanism for loading and running a specific handler, or function, within a script. My app FastScripts takes advantage of this to query a script about whether it would prefer to be invoked in another process or not. Unfortunately, when it comes to JavaScript, Apple’s implementation runs the whole script in addition to running just the specific, named handler.

If you’ve got Xcode handy, you can use this simple playground content to observe the problem:

import OSAKit

if let javaScriptLanguage = OSALanguage(forName: "JavaScript") {
   let scriptSource = "Application('Safari').activate();" +
         "function boo() { ObjC.import('Cocoa'); $.NSBeep(); }"
   let myScript = OSAScript(source: scriptSource, language: javaScriptLanguage)

  // Only the behavior of boo should be observed
  myScript.executeHandler(withName: "boo", arguments: [], error: nil)
}

// Give time for the beep to sound
RunLoop.current.run(until: Date(timeIntervalSinceNow:5))

The named function “boo()” only invokes NSBeep, so when this playground is run, all that should happen is a beep should be emitted from the Mac. Instead, when it runs Safari becomes the active application. This is because in addition to running the “boo()” handler, it also runs the whole script at the top level.

A workaround to the bug is to wrap the top level functionality of a script in a “run()” handler, so where the scriptSource is declared above, instead use:

   let scriptSource = "function run() { Application('Safari').activate(); }" +
         "function boo() { ObjC.import('Cocoa'); $.NSBeep(); }"

I hope this helps the one other person on earth who cares about invoking JavaScript for Automation methods indvidually! (Radar #33962901, though I’m not holding my breath on this one!)

Xcode GitHub Integration

Apple’s beta release of Xcode 9 features impressive improvements to its source control features, including streamlined integration with GitHub. There’s even a fancy “Open in Xcode” button when you go to clone a project:

Screen capture of the GitHub interface for cloning a project

This integration is amazing. You just click the button, specify a save folder in Xcode, and boom! You’re off and …

Screen capture of build failure indicating a missing signing certificate

Oh, right. Code signing. The otherwise stellar GitHub integration in Xcode underscores a longstanding deficiency in how it manages code signing identities for multi-team, collaborative projects. Precisely the kinds of projects you’re liable to find on GitHub.

The problem could be solved, or at least diminished greatly, by providing some mechanism for declaring that a project should be code signed “with the user’s own default developer team.” The default branch of any open source project targeting Apple platforms, would specify the DEVELOPMENT_TEAM as something like:

DEVELOPMENT_TEAM = Automatic

Xcode would provide a user-level setting for “Default Development Team”, and in the absence of any overriding setting, that team would be used whenever a project was configured as above.

I wrote about this problem once before, but with all the work being put into streamlining the experience of cloning from and pushing to GitHub, now is an ideal time for Apple to embrace a fix. Radar #32614751.

Another issue that stops short the cloning, and immediate building and running, of open source projects, is the need to fulfill external dependencies. In some cases this might require manually downloading and installing libraries, or cloning projects, but in the vast majority of cases the dependencies will be specified using built-in Git submodule support, or a popular package manager. In each of these cases, it should be trivial for Xcode to detect that the project it has just cloned also has dependencies:

  • Git submodules: there is a .gitmodules directory.
  • Carthage: there is a Cartfile file.
  • CocoaPods: there is a Podfile file.
  • Swift Package Manager: there is a Swift.package file.

If Xcode sees evidence of any of these techniques at play, it could do the favor of checking them out immediately after cloning the project. Radar #32615265.

The GitHub integration coming in Xcode 9 provides a nearly effortless capability for cloning, building, and running open source projects that target Apple platforms. Ideally it would also go the extra mile and provide for variable, dynamic development teams, as well as conduct a rudimentary check for dependencies that must be checked out before commencing work on the project.

Evergreen Images

Brent Simmons, the original developer of MarsEdit and NetNewsWire, is building a new feed reader app called Evergreen:

Evergreen is an open source, productivity-style feed reader for Macs.

It’s at a very early stage – we use it, but we don’t expect other people to use it yet.

I’ve never been one to shy away from early-stage software, so of course I ran to the GitHub project page, cloned the repository, and built it immediately on my own Mac.

Screenshot of Evergreen about box without a custom icon.

Ahh, the tell-tale sign of a young app: the generic about box. Personally, I like to give apps-in-progress an icon, even if only a placeholder image, as soon as possible. It occurred to me that Apple has done the favor of providing a pretty-darned-suitable image for “Evergreen” in the form of its Emoji glyph of the same name:

🌲

Since I have the source code right here, why don’t I render that tree at a large size in a graphics app, resize it to a million different resolutions, bundle it up and check it in to the Evergreen source base?

Because that’s not nearly as fun as doing it in code. I dove into the Evergreen application delegate class, adding the following function:

private func evergreenImage() -> NSImage? {
	var image: NSImage? = nil
	let imageWidth = 1024
	let imageHeight = 1024
	let imageSize = NSMakeSize(CGFloat(imageWidth), CGFloat(imageHeight))

	if let drawingContext = CGContext(data: nil, width: imageWidth, height: imageHeight, bitsPerComponent: 8, bytesPerRow: 0, space: CGColorSpaceCreateDeviceRGB(), bitmapInfo: CGImageAlphaInfo.premultipliedFirst.rawValue) {

		let graphicsContext = NSGraphicsContext(cgContext: drawingContext, flipped: false)
		NSGraphicsContext.saveGraphicsState()
		NSGraphicsContext.setCurrent(graphicsContext)

		let targetRect = NSRect(origin: NSZeroPoint, size: imageSize)
		NSString(string: "🌲").draw(in: targetRect, withAttributes: [NSFontAttributeName: NSFont.systemFont(ofSize: 1000)])

		NSGraphicsContext.restoreGraphicsState()

		if let coreImage = drawingContext.makeImage() {
			image = NSImage(cgImage: coreImage, size: imageSize)
		}
	}

	return image
}

In summary this code: creates a CoreGraphics drawing context, renders a huge evergreen Emoji glyph into it, and creates an NSImage out of it.

Then from the “applicationDidFinishLaunching()” function:

if let appIconImage = evergreenImage() {
	appIconImage.setName("NSApplicationIcon")
	NSApplication.shared().applicationIconImage = appIconImage
}

Give the newly created image the canonical name, used by AppKit, for looking up the application icon, and immediately change the application’s icon image to reflect the new value. It worked a treat:

EvergreenEmoji

In programming there is usually a hard way, an easy way, and a fun way. Be sure to take the third option as often as possible.

Resolving Modern Mac Alias Files

When a user selects a file in the Mac Finder and chooses File -> Make Alias, the resulting “copy” is a kind of smart reference to the original. It is similar to a POSIX symbolic link, but whereas a symbolic link references the original by full path, an alias has historically stored additional information about the original so that it stands a chance of being resolved even if the original full path no longer exists.

An alias, for example, can usually withstand being copied from one volume to another. To test this: create a Folder in your home folder called “Test”, and a file within it called “File”. Now, make an alias to “File”. You should have a folder hierarchy that looks like this:

Test
	File
	File alias

Click on the “File Alias” item, then choose “File” -> “Show Original” from the Mac menu bar to see how it resolves to the original.

Now, copy the whole folder to another volume, for example to a thumb drive or other external drive on your Mac. Click on the alias file and “Show Original” again. Instead of resolving to the file on your home volume, it resolves relative to the location on the new volume.

That’s pretty neat.

In the old days, if a developer needed to resolve an alias file on disk, they would use a Carbon function such as FSResolveAliasFile. In recent years, particularly as Application Sandboxing was introduced on the Mac, Apple has shifted away from aliases as a developer-facing concept, towards the use of “Bookmark Data”. As a result, an “alias file” created in the Finder is no longer a traditional alias, but a blob of bookmark data. You can confirm this by examining the hex content of the “File Alias” you created above. From the Terminal:

% xxd File\ alias
00000000: 626f 6f6b 0000 0000 6d61 726b 0000 0000  book....mark....
00000010: 3800 0000 3800 0000 f003 0000 0000 0410  8...8...........
00000020: 0000 0000 0061 0000 768c 979b 7ed8 be41  .....a..v...~..A
00000030: 0000 0000 ff7f 0000 0403 0000 0400 0000  ................
00000040: 0303 0000 0004 0000 0700 0000 0101 0000  ................

It was nice of them to design the format with the tell-tale “book….mark” data right there in the header!

Although you can still use FSResolveAliasFile on these beasts, the function is deprecated and Xcode will warn you about such behavior. The way forward is to use the newer

-[NSURL URLByResolvingBookmarkData:...]

method in Objective-C, or

URL.init(resolvingBookmarkData: ...)

in Swift. Just load the NSData from the alias file’s URL, and pass it to NSURL/URL as appropriate.

There’s a big catch, however, which is that you must take care to pass the alias file’s URL as the “relativeTo:” parameter when resolving the bookmark. Otherwise the bookmark will resolve as expected in typical scenarios, but will fail to resolve in all the scenarios where bookmarks really shine, as for example in the case of moving a bookmark and its target to another volume. So, for example, to resolve an alias file you know exists at “/private/tmp/Test/SomeAlias”:

var targetURL: URL? = nil
do {
	let aliasFileURL = URL(fileURLWithPath: "/private/tmp/Test/SomeAlias")
	let thisBookmarkData = try URL.bookmarkData(withContentsOf: aliasFileURL)
	var ignoredStaleness: Bool = false
	targetURL = try URL(resolvingBookmarkData: thisBookmarkData, options: .withoutUI, relativeTo: aliasFileURL, bookmarkDataIsStale: &ignoredStaleness)
}
catch {
	print("Got error: \(error)")
}

Notice how I ignore the staleness? It’s because I think this notion of staleness is tied more to resolving data with security scope, or in any case a bookmark that you are holding as pure Data, and not one that persists as a file on disk. The safety of ignoring staleness is supported by the fact that, starting in macOS 10.10, there is a new convenience method on NSURL specifically for resolving “alias files”:

var targetURL2: URL? = nil
do {
	let aliasFileURL2 = URL(fileURLWithPath: "/private/tmp/Test/SomeAlias")
	targetURL2 = try URL(resolvingAliasFileAt: aliasFileURL2)
}
catch {
	print("Got error: \(error)")
}

Which takes care of ensuring the “relativeTo:” information is considered, on your behalf. If you don’t need to support Mac OS X 10.9 or earlier, you should use the newest method! Otherwise, I hope the information preceding is of some use.

Better GitHub Searching

Sometimes when I’m searching a GitHub repository, I end up with a ton of uninteresting results because there are, for example, tests or documentation in the repository that are not pertinent to what I’m searching for.

For example, in the Apple Swift repository, searching for “struct String” currently yields 22 results, many of which are in the “test/” subdirectory. I’m not interested in these at the moment.

To search any subpath, just modify the search with the “path:” flag: “struct String” path:/stdlib. Six results, all pertinent to the actual implementation of “struct String”. Just what I was looking for.

There are lots of fancy constraints you can apply to GitHub searches, I simply hadn’t thought to look them up until now. Maybe some of them will make your exploration easier, too.