I wrote previously about the confusion that arose when many developers, trying to comply with Apple’s new code signing rules, ran across strange system behavior in which version 1 signatures seemed to work, yielding the curious system policy message “accepted CDHash.”
I can’t believe I didn’t think until now to check Apple’s open source Security framework for clues about this. Poking around today I found something very curious. In the policyengine.cpp source file, search for “consult the whitelist” and you’ll find the clump of code that very deliberately avoids rejecting an app for its “obsolete resource envelope” if it passes some whitelist check:
if (mOpaqueWhitelist.contains(code, rc, trace)) allow = true;
Well I’ll be darned. Could this explain the fact that many, many people observed that their apps with old, V1 signatures continue to pass Gatekeeper’s scrutiny e.g. on 10.9.5, even though Apple stated that V2 code signatures would be required?
The whitelist database is stored on your Mac in the following location:
/var/db/gkopaque.bundle/Contents/Resources/gkopaque.db
Go ahead, poke around at it. It’s big! It will be a bit easier to play with if you have any experience at all using the sqlite3 command line tool. With it, for example I was able to discover that there’s a table called “whitelist” within it, and that it contains two columns: “current” and “opaque”. And there are a LOT of rows in the table:
sqlite> select COUNT(*) from whitelist 68101
So what does a typical row from the table look like?
sqlite> select quote(current), quote(opaque) \
from whitelist limit 1;
X'000327ECE1FB5A27B5F5C51A009900B1E4854BB7'|
X'CBE56B9784974E0A1C0159C41F392B77421B4D23'
By scrutinizing the code and poking around, I’ve determined that the “current” column is not unique, and corresponds to the “CDHash” of a given code object being analyzed. For example, a version of MarsEdit that shipped with a V1 code signature and which does not seem affected by the changes in Apple’s Gatekeeper policy has a CDHash of “D1FBA2AB9A4814877BE8C1D2A8615FB48D8D4026”, and on my system anyway there are two rows corresponding to that CDHash.
I don’t really get what the “opaque” column is about, and my ability to scrutinize Security source code isn’t great enough to easily be able to tell by reading the source, either. But it seems to me that it must somehow be a way of informing the security system that certain specific (possibly modified?) instances of an app are still essentially the same as the “current” CDHash being tested.
Using some more sqlite3 magic, we can determine the number of unique values in the “current” column:
sqlite> select count(distinct current) from whitelist; 36215
OK, I run a lot of software, but I’m quite positive I have not run 36K unique parcels of code in recent memory. My suspicion is that in the run-up to the major changes Apple has made to Gatekeeper, they painstakingly accumulated a list of 36215 “trusted” hashes and deposited them on everybody’s Mac so that the effect of 10.9.5’s stricter code signing checks would be mitigated.
One test I did to confirm that the database is not just a personalized list of the apps I have used, was to download an app that I don’t use regularly, Panic’s Unison, but that I thought would be popular and reputable enough to appear on Apple’s whitelist. I downloaded it, and before running it even once, I checked its CDHash (“35d9c847ebb7461aee5f08bb8e017b5a3891bc0f”) with the database. Sure enough, 7 rows match this CDHash. Perhaps accommodating 7 distinct releases of Unison? Again, I’m not sure. I then took the extra step of logging into another machine entirely, one which has never seen Unison even as a downloaded file. The database on that machine also contains the hash.
Want to see if your apps are “whitelisted” or not? It’s pretty easy to do from the Terminal:
- Find your CDHash by running “codesign -dvvv Your.app” and searching for the CDHash value in the output.
- Grep the database for your value:
sqlite3 /var/db/gkopaque.bundle/Contents/Resources/gkopaque.db "select quote(current), quote(opaque) from whitelist" | grep -i [yourCDHashHere]
This whitelist offers a significant amount of explanation as to why some apps are allowed to launch without issue on 10.9.5 and 10.10. I don’t understand the database completely, particularly the meaning of those “opaque” CDHash values in the second column, but I feel as though a lot of mysterious behavior on all of our Macs is suddenly a lot more understandable.
Update later on Oct 6: After publishing this entry, Ed Marczak chimed in on Twitter with some information about the database:
@danielpunkass Meant to clue you into this. 10.9.4 ran an agent that uploaded these to Apple. That's where they get the mass hash list from.
— Edward Marczak (@marczak) October 6, 2014
In other words, according to Ed, Apple gathered the list of “whitelisted CDHashes” by surveying the software that people actually run, and reporting that data to Apple in the form the unique code signature hashes. I’m not sure what criteria they applied in the end to decide which of those apps are included in the whitelist, but it sounds reasonable to assume that if nobody ran your app on 10.9.4, Apple did not have the opportunity to consider including you in the whitelist.
(Thanks to Jeff Johnson for talking through some of the discoveries that led to this post).