Category Archives: Mac

What To Do About Code Signing

I wrote about the confusion arising from Apple’s poor communication about new code signature requirements, and then just earlier today I wrote about the revelation that many apps have been reprieved by an Apple whitelist.

I tend to write long, so let’s cut to the chase: what do you need to do if you are a Mac developer who ships software directly to customers or via the Mac App Store?

  1. You must ensure your next software release is signed with a Version 2 code signature. The easiest way to do this is to build and sign your product on Mac OS X 10.9 or later.
  2. You must test your existing, published apps to see that they can be downloaded and opened without issue on 10.9.5 and 10.10. To be safe, try to test once from a machine/install that has never seen the app before. If you’re concerned, read the details in my previous posts to assure yourself that your existing app was whitelisted.
  3. If your existing app does not open or in particular if it receives an “obsolete resource envelope” assessment from the “spctl” tool, you must either release a new version of your app signed with a Version 2 code signature, or re-sign your existing app. Otherwise, people who download the app will be greeted by a warning that the software is “not trusted.”

That’s it, folks. Everybody has to start signing with the modern code-signing infrastructure. In the interim, there’s a good chance your app has been whitelisted to operate as usual during the transition, but that courtesy will probably not extend to your next release.

Gatekeeper’s Opaque Whitelist

I wrote previously about the confusion that arose when many developers, trying to comply with Apple’s new code signing rules, ran across strange system behavior in which version 1 signatures seemed to work, yielding the curious system policy message “accepted CDHash.”

I can’t believe I didn’t think until now to check Apple’s open source Security framework for clues about this. Poking around today I found something very curious. In the policyengine.cpp source file, search for “consult the whitelist” and you’ll find the clump of code that very deliberately avoids rejecting an app for its “obsolete resource envelope” if it passes some whitelist check:

if (mOpaqueWhitelist.contains(code, rc, trace))
	allow = true;

Well I’ll be darned. Could this explain the fact that many, many people observed that their apps with old, V1 signatures continue to pass Gatekeeper’s scrutiny e.g. on 10.9.5, even though Apple stated that V2 code signatures would be required?

The whitelist database is stored on your Mac in the following location:

/var/db/gkopaque.bundle/Contents/Resources/gkopaque.db

Go ahead, poke around at it. It’s big! It will be a bit easier to play with if you have any experience at all using the sqlite3 command line tool. With it, for example I was able to discover that there’s a table called “whitelist” within it, and that it contains two columns: “current” and “opaque”. And there are a LOT of rows in the table:

sqlite> select COUNT(*) from whitelist
68101

So what does a typical row from the table look like?

sqlite> select quote(current), quote(opaque) \
           from whitelist limit 1;
X'000327ECE1FB5A27B5F5C51A009900B1E4854BB7'|
X'CBE56B9784974E0A1C0159C41F392B77421B4D23'

By scrutinizing the code and poking around, I’ve determined that the “current” column is not unique, and corresponds to the “CDHash” of a given code object being analyzed. For example, a version of MarsEdit that shipped with a V1 code signature and which does not seem affected by the changes in Apple’s Gatekeeper policy has a CDHash of “D1FBA2AB9A4814877BE8C1D2A8615FB48D8D4026”, and on my system anyway there are two rows corresponding to that CDHash.

I don’t really get what the “opaque” column is about, and my ability to scrutinize Security source code isn’t great enough to easily be able to tell by reading the source, either. But it seems to me that it must somehow be a way of informing the security system that certain specific (possibly modified?) instances of an app are still essentially the same as the “current” CDHash being tested.

Using some more sqlite3 magic, we can determine the number of unique values in the “current” column:

sqlite> select count(distinct current) from whitelist;
36215

OK, I run a lot of software, but I’m quite positive I have not run 36K unique parcels of code in recent memory. My suspicion is that in the run-up to the major changes Apple has made to Gatekeeper, they painstakingly accumulated a list of 36215 “trusted” hashes and deposited them on everybody’s Mac so that the effect of 10.9.5’s stricter code signing checks would be mitigated.

One test I did to confirm that the database is not just a personalized list of the apps I have used, was to download an app that I don’t use regularly, Panic’s Unison, but that I thought would be popular and reputable enough to appear on Apple’s whitelist. I downloaded it, and before running it even once, I checked its CDHash (“35d9c847ebb7461aee5f08bb8e017b5a3891bc0f”) with the database. Sure enough, 7 rows match this CDHash. Perhaps accommodating 7 distinct releases of Unison? Again, I’m not sure. I then took the extra step of logging into another machine entirely, one which has never seen Unison even as a downloaded file. The database on that machine also contains the hash.

Want to see if your apps are “whitelisted” or not? It’s pretty easy to do from the Terminal:

  1. Find your CDHash by running “codesign -dvvv Your.app” and searching for the CDHash value in the output.
  2. Grep the database for your value:
    sqlite3 /var/db/gkopaque.bundle/Contents/Resources/gkopaque.db "select quote(current), quote(opaque) from whitelist" | grep -i [yourCDHashHere]
    

This whitelist offers a significant amount of explanation as to why some apps are allowed to launch without issue on 10.9.5 and 10.10. I don’t understand the database completely, particularly the meaning of those “opaque” CDHash values in the second column, but I feel as though a lot of mysterious behavior on all of our Macs is suddenly a lot more understandable.

Update later on Oct 6: After publishing this entry, Ed Marczak chimed in on Twitter with some information about the database:

In other words, according to Ed, Apple gathered the list of “whitelisted CDHashes” by surveying the software that people actually run, and reporting that data to Apple in the form the unique code signature hashes. I’m not sure what criteria they applied in the end to decide which of those apps are included in the whitelist, but it sounds reasonable to assume that if nobody ran your app on 10.9.4, Apple did not have the opportunity to consider including you in the whitelist.

(Thanks to Jeff Johnson for talking through some of the discoveries that led to this post).

JavaScript Bug Traps

I am the kind of developer who leans heavily on automated systems for the improvement of my code. For this reason I tend to enable the maximum number of warnings, rely upon static analysis to discover nuanced bugs, develop and run unit tests to automatically confirm expected behaviors, and I encourage my apps to crash hard if they are going to crash at all.

I would characterize all of these behaviors as part of an overall attitude of vigilance against software defects. Learning about problems in code quickly and clearly is a key step maintaining an overall high level of code quality.

Most of my software is developed in Objective-C, which enables me to leverage compile-time and run-time tools from Apple that catch my errors quickly. Between compile-time warnings and errors, and runtime exceptions and crashes, the vast majority of my programming errors in Objective-C are caught within minutes of writing them.

On the other hand, MarsEdit features a rich HTML text editor largely implemented in JavaScript, a language which tends not to afford any of these convenient alerts to my mistakes. I have often stared in bewilderment at the editor window’s perplexing behavior, only to discover after many lost minutes or hours, that some subtle JavaScript error has been preventing the expected behavior of my code. I’ll switch to the WebKit inspector for the affected WebView, and discover the console is littered with red: errors being dutifully logged by the JavaScript runtime, but ignored by everybody.

Wouldn’t it be great if the JavaScript code crashed as hard as the Objective-C code does?

In fact, these days mine does. By a combination of JavaScript hooks that are alerted to the errors, Objective-C bridges that propagate them to the host app, and the judicious use of undocumented WebKit Inspector methods, development versions of my WebView-based apps now automatically present an inspector window the moment any error occurs in the view’s underlying JavaScript code.

For those of you who also develop JavaScript-heavy WebKit apps, I’ll share the steps for enabling this same type of behavior in your app. These steps all apply to Mac-based WebViews, but most of the same techniques could be used with iOS if you are willing to use some private methods to establish a bridge between the WebView and host app (for debug builds only, of course).

Step 1: Catch the error in JavaScript.

In whatever the main source content is for your WebView-based user interface, add some script code to register a callback for JavaScript errors:

<script type="text/javascript">
   function gotJavaScriptError(errorEvent)
   {
      console.log("Error: " + errorEvent.message);
   }

   window.addEventListener("error", gotJavaScriptError,  false);
</script>

Now if you encounter some error and happen to be attached with the web inspector, you’ll see additional confirmation that the error has in fact been noticed. if you’re following along in your own code while you read, I strongly encourage you to add a testing mechanism for generating errors. A simple “click” event callback (or “touchstart” on iOS) does the trick well:

<script type="text/javascript">
   function gotClick(theEvent)
   {
      blargh
   }

   window.addEventListener("click", gotClick, false);
</script>

Now when you click, you blargh. And when you blargh, the error is logged. Of course, just logging the error isn’t very useful because it’s quiet and goes ignored. Just like JavaScript errors usually do…

Step 2: Propagate the error to the host app.

You could do some interesting stuff in the WebView itself. For example you might convert the error into a string and display a prominent alert message. But since there’s so much more you can do once the host app is in charge, it’s worth going the extra mile for that functionality.

To tie the host app to the WebView we need to wait for the WebView’s frame to finish preparing its “window script object,” and then set a named attribute on that object that refers to an Objective-C instance. See that your WebView has a frameLoadDelegate configured, and within that delegate object implement this method:

- (void)webView:(WebView *)webView 
        didClearWindowObject:(WebScriptObject *)windowObject
        forFrame:(WebFrame *)frame
{
   // Expose ourselves as a JavaScript window attribute
   [[webView windowScriptObject] setValue:self forKey:@"appDelegate"];
}

At this point any JavaScript code, including our error-handling callback, can reference the Objective-C delegate directly. But in order to make good use of the delegate we’ll have to add a couple other methods. The first lets the WebKit runtime know that we are not fussy about which of our methods are accessible to the WebView:

+ (BOOL) isSelectorExcludedFromWebScript:(SEL)aSelector
{
   return NO;
}

Now JavaScript code within the WebView can call any method it chooses on the delegate object. Implement something suitable for catching the news of the JavaScript error:

- (void) javaScriptErrorOccurred:(WebScriptObject*)errorEvent
{
   NSLog(@"WARNING JavaScript error: %@", errorEvent);
}

And back in your WebView HTML source file, amend the JavaScript error handler to call through to Objective-C instead of pointlessly logging to the JavaScript console:

<script type="text/javascript">
   function gotJavaScriptError(errorEvent)
   {
      console.log("Error: " + errorEvent.message);
      window.appDelegate.javaScriptErrorOccurred_(errorEvent);
   }

   ...
</script>

Notice that the colons in Objective-C method names are simply replaced with underscores to make them compatible with the JavaScript function naming rules. The errorEvent argument in this case is translated to a DOMEvent object instance in Objective-C, where it can be further interrogated. Alternatively you could pull the parts of the event you want out in JavaScript, and pass along only those bits.

Now we’ve done a considerable amount of work, only to upgrade our easy to ignore JavaScript errors from littering the web console to littering the system console. At least we’d notice these errors in the Xcode debugger console, and might eventually get around to taking a closer look. But we can still do much better.

Step 3: Get face to face with the error.

Of course from Objective-C there are any manner of ways you could handle the error. In a shipping app it might make sense to prompt the user with news of the issue and offer, much like a crash reporter would, to send information about the error to you. Or if the contents and behavior of your WebView are critical enough, maybe it’s worth forcing the app to quit just as it would for a native code crash. But for debugging builds I’ve find it very helpful simply to have the error force the Web Inspector open so I am no longer able to quietly ignore all the red console logging. In the delegate class, go back to the javaScriptErrorOccurred: method and replace the NSLog call with this string of fancy mumbo-jumbo

- (void) javaScriptErrorOccurred:(id)errorEvent
{
#if DEBUG
   id inspector = [[self webView] performSelector:@selector(inspector)];
   [inspector performSelector:@selector(showConsole:) withObject:nil];
#endif
}

That’s it. Now when you run into a WebView-based JavaScript error, the web inspector appears and the list of pertinent errors is front-and-center.

I encourage you to leave the DEBUG barrier in place, because as the prolific use of “performSelector” may suggest, these are private WebKit methods that would probably not be viewed as acceptable by e.g. App Store reviewers. Anyway, you probably don’t want customers being pushed into the Web Inspector.

I hope this technique proves useful for those of you with extensive JavaScript source bases of your own. For everybody else, perhaps I’ve helped to drive home the idea that we should be as vigilant as possible against software defects. All developers write bugs, all the time. It’s only fair that we try to balance the scales and give ourselves a fighting chance by also being alerted to the bugs we write, all the time.

Brent’s Coding Nits

I agree with every one of Brent Simmons’s coding nits, even if I don’t practice what he preaches completely. Everybody has to make some mistakes to keep life interesting, right?

There’s also one point he makes:

All of your class names should have a prefix. All of them.

Which I think is particularly applicable to shared code such as the kind he’s reviewing on GitHub when compiling this list of nits. It’s not nearly so important, and in fact not really that important at all, for a project’s private classes to have prefixes. For example if you make a new app called “Motorcycle Derby,” you should be forgiven for calling your app’s delegate class “AppDelegate.m” if that is what suits you. By this point in history, Apple takes virtual namespacing seriously enough that you’re unlikely to clash with an unprefixed name from them, and so long as everybody distributing shared library code follow’s Brent’s rule above, you won’t clash with them either.

Still, you can’t go wrong by adding prefixes to be extra sure that you won’t conflict with anybody.

AppleScript XML-RPC

My mind was fairly well blown this morning to learn that for more than ten years, AppleScript on Mac OS X has included a built-in command for communicating with XML-RPC and SOAP endpoints on the web.

XML-RPC is a well-known semi-standard method for communicating between processes over the internet. As it happens, a large number of blogging APIs including the WordPress API and many others, were also designed around XML-RPC. For this reason, XML-RPC is a major component of MarsEdit, my Mac-based blogging application.

The funny thing is, I knew that Apple had developed built-in support for SOAP: one of my teammates at Apple was responsible for it! But I either blocked out the XML-RPC support, or disregarded it as uninteresting at the time. And I don’t think I ever knew that the support had been extended to AppleScript in such a native fashion.

XML-RPC isn’t, as they say, rocket science. However, it’s pretty cool that with an off-the-shelf Mac one can throw together a simple script to, for example, grab the latest post off your blog, build a link to it by its title, and copy the HTML to the pasteboard:

set myBlogUsername to "sweatertest"
set myBlogPass to "xxx"

tell application "http://sweatertest.wordpress.com/xmlrpc.php"
	set myPosts to call xmlrpc {method name:"wp.getPosts", parameters:{"1", myBlogUsername, myBlogPass, "1"}}

	set myPost to item 1 of myPosts

	set theLink to link of myPost
	set theTitle to post_title of myPost
	set myPostLink to "<a href='" & theLink & "'>" & theTitle & "</a>"
end tell

set the clipboard to myPostLink

Wire it up with a FastScripts keyboard shortcut and you’re really cooking! This example may be a bit contrived, but one can imagine writing similar scripts to query a blog for information, or even to fire off short blog posts after prompting for content. (Note that if you do automate something like this you will probably want to store the password securely in the keychain).

People love to hate AppleScript, but this is one example of how many nifty little treats lurk within it. The fact that it’s omnipresent on Mac OS X makes it an excellent resource for providing simple solutions, when a simple solution will do.