Category Archives: Apple

Open URL From Today Extension

A friend of mine mentioned in passing that he was having trouble getting an obvious, well-documented behavior of his Today extension to work … as documented. According to Apple, a Today extension should use NSExtensionContext when it wants to open its host app, e.g. to reveal a related data item from the Today widget, in the context of the host application.

A widget doesn’t directly tell its containing app to open; instead, it uses the openURL:completionHandler: method of NSExtensionContext to tell the system to open its containing app.

They even cite one of their own apps as an example:

In some cases, it can make sense for a Today widget to request its containing app to open. For example, the Calendar widget in OS X opens Calendar when users click an event.

The idea is you should be able to use a simple line of code like this from within your Today extension. E.g., when a user clicks on a button or other element in the widget, you call:

[[self extensionContext] openURL:[NSURL URLWithString:@"marsedit://"] completionHandler:nil];

Unfortunately this doesn’t work for me, for my friend, or I’m guessing, most, if not all people who try to use it. Maybe it’s only broken on Mac OS X? I was curious, so I stepped into the NSExtensionContext openURL:completionHandler: method, and observed that it essentially tries to pass the request directly to the “extension host proxy”:

0x7fff8bd83b03:  movq   -0x15df0b7a(%rip), %rax   ; NSExtensionContext.__extensionHostProxy

But in my tests, this is always nil. At least, it’s always nil for a Today widget configured out of the box the way Xcode says it should be configured by default. So, when you call -[NSExtensionContext openURL:completionHandler:] from a Today widget on OS X, chances are it will pass the message along to … nil. The symptom here is the URL doesn’t open, your completion handler doesn’t get called. You simply get nothing.

Getting back to the fact that Apple used Calendar as their example in the documentation, I thought I’d use my debugging skills to poke around at whether they are in fact calling the same method they recommend 3rd party developers use. If you caught my Xcode Consolation post a while back, it will come as no surprise that an lldb regex breakpoint works wonders here to how the heck Apple’s extension is actually opening URLs. First, you have to catch the app extension while it’s running. It turns out Today widgets are killed pretty aggressively, so if you haven’t used it very recently, it’s liable to be gone. Click on the Today widget to see e.g. Apple’s Calendar widget, then quickly from the Terminal:

ps -ax | grep .appex

To see all running app extensions. Look for the one of interest, ah there it is:

56052 ??         0:00.29 /Applications/Calendar.app/Contents/PlugIns/com.apple.iCal.CalendarNC.appex/Contents/MacOS/com.apple.iCal.CalendarNC

That’s the process ID, 56052 in this case, at the beginning of the line. Quickly click the Notification Center again to keep the process alive, and then from the Terminal:

lldb -p 56052

If all goes well you’ll attach to Apple’s Calendar app extension, where you can set a regex breakpoint on openURL calls, then resume:

(lldb) break set -r openURL
(lldb) c

Now quickly go back to the Notification Center, and click a calendar item for today. If you don’t have a calendar item for today, whoops, go to Calendar, add one, and start this whole dance over ;) Once you’ve clicked on a calendar item in Notification Center and are attached with lldb, you’ll see the tell-tale evidence:

(lldb) bt
* thread #1: tid = 0xd1a4d, 0x00007fff87f22f1f AppKit`-[NSWorkspace openURL:], queue = 'com.apple.main-thread', stop reason = breakpoint 1.8
  * frame #0: 0x00007fff87f22f1f AppKit`-[NSWorkspace openURL:]
    frame #1: 0x00007fff9213f763 CalendarUI`-[CalUIDayViewGadgetOccurrence mouseDown:] + 221
...

So Apple’s Calendar widget, at least, is not using -[NSExtension openURL:completionHandler:]. It’s using plain-old, dumb -[NSWorkspace openURL:]. And when I change my sample Today extension to use NSWorkspace instead of NSExtensionContext, everything “just works.” I suspect it will for my friend, too.

I’m guessing this is a situation where the functionality of Today extensions might be more fleshed out on iOS than on Mac, and the documentation just hasn’t caught up with reality yet. There are a lot of platform-specific caveats in the documentation, and perhaps one of them should be that, for the time being anyway, you should use NSWorkspace to open URLs from Mac-based Today extensions.

Yosemite’s Dark Mode

If you’re a Mac developer you’re no doubt feeling that restlessness that comes when a major new OS release is nigh. Mac OS X 10.10 Yosemite has been cycling through “GM Candidate” releases for the past couple weeks, and many people seem to think it’s likely to be released to the public later this month.

One of the big user-facing visual changes is an optional “dark mode,” accessible from the General tab of System Preferences. When a user ticks this on, high level UI elements such as the menu bar and dock adopt a different style of drawing which you can roughly characterize as being white-on-black instead of the default black-on-white Mac appearance.

One hiccup for developers of third party apps is if you have an NSStatusItem or other custom menu bar item with a custom icon, you probably designed it so that it looks good on the white or light default menu bar.

Unfortunately, Apple has not provided a sanctioned method for detecting whether the user has switched to “dark mode,” so it’s non-trivial to tune your art to suit each of the modes at runtime. Instead, Apple recommends that all such icons should be designed such that Apple can effectively color them as appropriate for the current mode.

Luckily it’s pretty easy to get this behavior for custom status bar icons. You probably already call -[NSStatusItem setIcon:yourImage] with your custom image. Now, just make sure to call -[yourImage setTemplate:YES], and you’re done. The only caveat I would add here is that Mac OS X 10.9 and earlier do not seem to respect the “template” attribute when drawing status items, so I think the safest bet is to check for 10.10 at runtime, and omit the call to setTemplate if you’re running any earlier system:

BOOL oldBusted = (floor(NSAppKitVersionNumber) <= NSAppKitVersionNumber10_9)
if (!oldBusted)
{
	// 10.10 or higher, so setTemplate: is safe
	[myIcon setTemplate:YES]
}

Now your status item icon looks great on 10.10, where in fact its coloring may even be slightly altered in light mode, and in 10.9, where it looks, well, like it always did. Update: thanks to Rick Fillion and Will Cosgrove for explaining that my problems on 10.9 are probably because my image is not designed right for use as a template. I’ll leave the solution to that as an exercise for the writer.

Unfortunately, this promise does not hold true for regular NSMenuItem items that are installed at the highest level in the menu bar. These are relatively uncommon, but for example you might find some apps with a gear icon, a script icon, or other similar graphic in lieu of a text-based menu title. When an icon is supplied to such a menu item, the template nature is not meaningfully respected in dark mode, so the icon draws more or less as-is, and is likely to almost disappear because of the lack of contrast. If you happen to be running 10.10 you can witness this problem by running iTunes and seeing that at least one script is in ~/Library/iTunes/Scripts. Its script menu icon appears right next to the “Help” menu.

I reported the issue to Apple and it came back as a duplicate. As of the latest 10.10 GM candidate, it has not been fixed, but I got a reassuring tweet from Apple’s own Eric Schlegel today:

So in summary, if you use icons as the “titles” of menus either inside or outside of the status bar area of the menu bar, be sure to call setTemplate:YES on it when running on 10.10. If you happen to have a menu bar icon affected by the bug I reported, I recommend creating a test project that installs an NSStatusItem with the same icon, so you can get a sense for how it’s going to look in dark mode when its template nature is suitably handled.

What To Do About Code Signing

I wrote about the confusion arising from Apple’s poor communication about new code signature requirements, and then just earlier today I wrote about the revelation that many apps have been reprieved by an Apple whitelist.

I tend to write long, so let’s cut to the chase: what do you need to do if you are a Mac developer who ships software directly to customers or via the Mac App Store?

  1. You must ensure your next software release is signed with a Version 2 code signature. The easiest way to do this is to build and sign your product on Mac OS X 10.9 or later.
  2. You must test your existing, published apps to see that they can be downloaded and opened without issue on 10.9.5 and 10.10. To be safe, try to test once from a machine/install that has never seen the app before. If you’re concerned, read the details in my previous posts to assure yourself that your existing app was whitelisted.
  3. If your existing app does not open or in particular if it receives an “obsolete resource envelope” assessment from the “spctl” tool, you must either release a new version of your app signed with a Version 2 code signature, or re-sign your existing app. Otherwise, people who download the app will be greeted by a warning that the software is “not trusted.”

That’s it, folks. Everybody has to start signing with the modern code-signing infrastructure. In the interim, there’s a good chance your app has been whitelisted to operate as usual during the transition, but that courtesy will probably not extend to your next release.

Gatekeeper’s Opaque Whitelist

I wrote previously about the confusion that arose when many developers, trying to comply with Apple’s new code signing rules, ran across strange system behavior in which version 1 signatures seemed to work, yielding the curious system policy message “accepted CDHash.”

I can’t believe I didn’t think until now to check Apple’s open source Security framework for clues about this. Poking around today I found something very curious. In the policyengine.cpp source file, search for “consult the whitelist” and you’ll find the clump of code that very deliberately avoids rejecting an app for its “obsolete resource envelope” if it passes some whitelist check:

if (mOpaqueWhitelist.contains(code, rc, trace))
	allow = true;

Well I’ll be darned. Could this explain the fact that many, many people observed that their apps with old, V1 signatures continue to pass Gatekeeper’s scrutiny e.g. on 10.9.5, even though Apple stated that V2 code signatures would be required?

The whitelist database is stored on your Mac in the following location:

/var/db/gkopaque.bundle/Contents/Resources/gkopaque.db

Go ahead, poke around at it. It’s big! It will be a bit easier to play with if you have any experience at all using the sqlite3 command line tool. With it, for example I was able to discover that there’s a table called “whitelist” within it, and that it contains two columns: “current” and “opaque”. And there are a LOT of rows in the table:

sqlite> select COUNT(*) from whitelist
68101

So what does a typical row from the table look like?

sqlite> select quote(current), quote(opaque) \
           from whitelist limit 1;
X'000327ECE1FB5A27B5F5C51A009900B1E4854BB7'|
X'CBE56B9784974E0A1C0159C41F392B77421B4D23'

By scrutinizing the code and poking around, I’ve determined that the “current” column is not unique, and corresponds to the “CDHash” of a given code object being analyzed. For example, a version of MarsEdit that shipped with a V1 code signature and which does not seem affected by the changes in Apple’s Gatekeeper policy has a CDHash of “D1FBA2AB9A4814877BE8C1D2A8615FB48D8D4026”, and on my system anyway there are two rows corresponding to that CDHash.

I don’t really get what the “opaque” column is about, and my ability to scrutinize Security source code isn’t great enough to easily be able to tell by reading the source, either. But it seems to me that it must somehow be a way of informing the security system that certain specific (possibly modified?) instances of an app are still essentially the same as the “current” CDHash being tested.

Using some more sqlite3 magic, we can determine the number of unique values in the “current” column:

sqlite> select count(distinct current) from whitelist;
36215

OK, I run a lot of software, but I’m quite positive I have not run 36K unique parcels of code in recent memory. My suspicion is that in the run-up to the major changes Apple has made to Gatekeeper, they painstakingly accumulated a list of 36215 “trusted” hashes and deposited them on everybody’s Mac so that the effect of 10.9.5’s stricter code signing checks would be mitigated.

One test I did to confirm that the database is not just a personalized list of the apps I have used, was to download an app that I don’t use regularly, Panic’s Unison, but that I thought would be popular and reputable enough to appear on Apple’s whitelist. I downloaded it, and before running it even once, I checked its CDHash (“35d9c847ebb7461aee5f08bb8e017b5a3891bc0f”) with the database. Sure enough, 7 rows match this CDHash. Perhaps accommodating 7 distinct releases of Unison? Again, I’m not sure. I then took the extra step of logging into another machine entirely, one which has never seen Unison even as a downloaded file. The database on that machine also contains the hash.

Want to see if your apps are “whitelisted” or not? It’s pretty easy to do from the Terminal:

  1. Find your CDHash by running “codesign -dvvv Your.app” and searching for the CDHash value in the output.
  2. Grep the database for your value:
    sqlite3 /var/db/gkopaque.bundle/Contents/Resources/gkopaque.db "select quote(current), quote(opaque) from whitelist" | grep -i [yourCDHashHere]
    

This whitelist offers a significant amount of explanation as to why some apps are allowed to launch without issue on 10.9.5 and 10.10. I don’t understand the database completely, particularly the meaning of those “opaque” CDHash values in the second column, but I feel as though a lot of mysterious behavior on all of our Macs is suddenly a lot more understandable.

Update later on Oct 6: After publishing this entry, Ed Marczak chimed in on Twitter with some information about the database:

In other words, according to Ed, Apple gathered the list of “whitelisted CDHashes” by surveying the software that people actually run, and reporting that data to Apple in the form the unique code signature hashes. I’m not sure what criteria they applied in the end to decide which of those apps are included in the whitelist, but it sounds reasonable to assume that if nobody ran your app on 10.9.4, Apple did not have the opportunity to consider including you in the whitelist.

(Thanks to Jeff Johnson for talking through some of the discoveries that led to this post).

Fixing 90-Day Apple ID Expirations

I wrote a week ago that I was cautiously optimistic about a solution to a long-standing ordeal with my Apple ID password expiring every 90 days.

Today I got a very encouraging response from the Apple support representative who was shepherding the problem through to the engineering team that evidently knew how to fix the underlying issue with my account.

An update on the ticket I submitted to the Engineers states that you should now no longer receive those password reset prompts. Please contact us back referencing our case number: 663194617 if for any reason the issue persists after 90 days.

I have raised my cautious optimism to outright good cheer at the sight of evidence from Apple that “Engineering” saw a problem, claims to have fixed it, and both they and the support representative are confident about the solution. Furthermore, they offer a very special kind of 90-day warranty by inviting me to get back in touch for another round of effort should they be mistaken in their assessment.

I’ve been complaining about this issue on Twitter for such a long time and with such repetition that I’ve had the opportunity to hear feedback from tens if not a hundred or more people who have let me know that they too suffer the problem, and are eager to know of a solution. In the wake of last week’s post, at least one person was inspired to get in touch with Apple, only be to be shunted from Apple’s developer relations team to AppleCare, where I had previously failed to obtain any successful result.

As helpful as the representative was who helped me, I don’t feel comfortable sharing her name or direct contact information. But the fact that she seemed genuinely interested in solving this problem made me realize that it would be ideal if others who go down this path could stand at least a better chance of finding somebody as empathetic to the problem as she was.

So I asked her if she had any problem with me sharing my case number with “everybody on Twitter.” I didn’t exactly want to sic a mob of frustrated developers on Apple, but I do genuinely wish that everybody who has been as annoyed by this problem as I have will have a path of hope towards seeing it resolved. To my satisfaction she agreed enthusiastically with my proposition:

I don’t have an issue at all with you mentioning this case. I’d be more than happy for our group to help anyone in this situation if/when we can.

So in summary, this long journey has led me to a position where I now believe I can offer what is the most definitive set of instructions for developers with Apple Developer Connection accounts who would like to eliminate the annoying 90-day password resets:

  1. Go to the Apple Developer Program Support page.
  2. Select “Access Issues” from the Subject popup.
  3. Briefly explain the problem with your Apple ID password requiring a reset every 90 days, and reference my case #663194617 as a likely comparable issue which has been handled by developer support.

The fact that you’ll be in touch with a support staff that is empowered to consult with the pertinent engineering team, combined with the fact that they have persistent access to the documentation for my case will hopefully be a recipe that sets you up for the same success (finally!) that I have had.

Accepted CDHash

A bit over a month ago, Apple announced big changes to the way Mac OS X versions 10.9.5 and 10.10 will recognize the code signatures of 3rd party applications, hinting very strongly that consequences would be dire for any developer neglecting to re-sign their apps:

Important: For your apps to run on updated versions of OS X they must be signed on OS X version 10.9 or later and thus have a version 2 signature.

Few details were given as to why the old signatures would no longer be respected, leaving developers with little to go on except to take Apple’s word that we should drop everything and update our code signing processes, which for some of us was a non-trivial amount of work. Those of us willing to grant Apple the benefit of the doubt assumed that there was some greater purpose to our collective suffering. Surely this nuisance is in the name of preserving or increasing security for all of Apple’s and our mutual customers.

That, and the fact that Apple stated bluntly that if we don’t make these changes, our apps will not pass Gatekeeper’s assessment and thus will not be allowed to launch without considerable work by users. This is an unacceptable user experience to any developer worth her or his salt, so of course the vast majority of us complied.

Some of us complied even when the logic of doing so was comically bent. For example Apple implied that developers of apps for the Mac App Store also needed to update their code signatures, in spite of the fact that all Mac App Store apps are signed by Apple, not by the original developers. The process of submitting software for sale on the App Store does include a code signing phase, but the signature is replaced by Apple before it is distributed to customers. So, if there is a security issue with version 1 code signatures, Apple is in a position to remedy the problem without the involvement of 3rd party developers. Some of us sought clarification from Apple on this point. Questions in the Apple Developer Forums along the lines of “do we really have to re-sign apps and submit a new version just to accommodate this new requirement?” were met by terse restatements from Apple along the lines of “all apps must be signed with version two code signatures.”

Sigh.

To make matters yet more confusing, a developer who has signed off on the chore of complying with Apple’s requests would not necessarily be able to verify the job was done right, because for example on pre-release builds of 10.9.5 and 10.10, many apps with “old and busted” version 1 signatures unexpectedly passed the system’s Gatekeeper check, contrary to the firm indication from Apple that they shouldn’t. Apple’s documentation provided a specific command to run from the Terminal that would verify or reject any specific app’s code signing:

spctl -a -t exec -vv Foo.app

For many of us with very old version 1 code signatures, the command line came back with a cryptic “accepted cdhash”, and the system happily opened and ran the apps without issue.

I reported the problem as a security bug because it seemed to suggest that apps with broken, insecure code signatures would still be allowed to run. I waited to see whether, with each subsequent beta release, Apple would finally ratchet things down to give me a taste of how customers would experience trying to run these apps with outdated signatures. But each beta release continued to allow them to run without incident.

Finally, a few days ago I got my bug sent back to me to be closed. Oh, they finally fixed it? Nope. It was returned with the classification “Behaves as intended.” Then yesterday, 10.9.5 shipped and lo and behold, contrary to every warning and veiled threat from Apple, many of these apps with old, version 1 code signatures, of the variety that yield a cryptic “accepted cdhash” assessment from the spctl tool, well, they just work. They’re fine. This happens to apply to my entire line of apps and this is probably also the case for many other developers. In short? I dropped everything, spent hours revising my code signing process, investigating unexpected results, contacting Apple for clarification, trying to make sense of Apple’s terse replies, and finally doing my best to comply regardless of doubt. And it turns out I didn’t actually have to lift a finger.

Which is not to say that nobody had to lift a finger. Any developer whose code signatures involved the use of “custom resource rules,” and (I think) any developer whose version 1 signature was signed with a new enough version of Xcode to escape the “accepted cdhash” loophole, but not new enough to be a “version 2” signature, did need to re-sign their apps.

This was a classic case of Apple communicating far too poorly about a situation that purported to affect potentially every Mac developer. Many of us spent way too much time trying to decode and make sense of the situation when Apple could have done so for us through careful clarification of the specific code signatures that needed updating, how they could be reliably verified, and what the actual consequences of inaction would be.

Cautious Optimism

For years I have been suffering a relatively benign yet still infuriating problem with my Apple ID: the password expires every 90 days, like clockwork, forcing me to choose a new one.

It sounds like a minor inconvenience, but it’s made somewhat worse by the fact that my Apple ID and its password are tied to countless different Apple services, each of which saves a copy of the credentials separately from the others. Long story short? I have to enter the password umpteen different times, every 90 days. A litany of authorization panels appear to let me know that, when I least expected it, Messages, iCloud, calendar syncing, iTunes connect, iTunes itself, the Apple Store, Xcode’s ADC integration, etc., etc., all need to be reauthorized. And for many of these services I must carry out this dance on my Mac, iPad, and iPhone. Oh, and my Apple TV.

A couple months ago I got it in mind that I would finally take the plunge and see what AppleCare, Apple’s famously courteous and helpful customer support team, could do for me. I was impressed from the outset by the seriousness with which they took my request, and by the assiduousness of the attention they gave to my problem. I was passed up the ranks of the support team until I was on a first name basis with a very helpful agent out of Austin, TX, who, though she seemed unable to solve the problem, also seemed unwilling to give up until she could. She liaised with various groups within and outside of AppleCare, keeping me posted about the status of this, that, or other approach that may or may not get to the bottom of things. Her colleagues who specialized in AppleID problems offered various suggestions and she diligently came back to me with questions about whether I was a member of this or that program. This went on for a week or two, but it felt invigorating. Even though it was tedious, it felt as though we would end up at a solution. Eventually this relentless, passionate agent would figure out the problem and give me the blessed call to let me know that, at last, everything was going to be OK.

And then she never called me again.

I don’t know what happened to my case. Maybe it’s sitting open in the system, in her queue, collecting dust. Maybe she’s moved on to another job and her tickets are collateral damage. Or maybe she just got tired of trying and one day decided to close it without saying another word.

My 90 day anniversary came up again a couple days ago, and I was inspired to, you’ll never believe this, gripe about it on Twitter. Usually my griping is met by a choir of fellow sufferers who also wish they could eliminate this hex on their Apple ID account. This time was no different, although there was one reply that offered an optimistic take. Rosyna Keller assured me that the problem is well known as an ADC-specific issue:

Sure enough, everybody I’ve ever known who suffered the problem has an ADC account, and most if not all of the afflicted are very long-time members. But still I sighed: I anticipated another weeks-long ordeal possibly ending in me being no better off than I was.

All the same, yesterday I decided to take the initiative to send a note to Apple’s Developer Relations support team. After all I’ve done it would be a shame to live with this problem for yet another 90 days if there are simple steps that could prevent it.

I went to the Contact Us page for the Apple Developer Program, selected “Access Issues” as the subject, and entered the following message:

For years I have been forced to reset my Apple ID ([My AppleID]) password every 90 days. I recently went through an unproductive, several-days-long support interaction with AppleCare that ended in no change. I have recently been encouraged to believe that actually my affiliation with ADC may be the source of the security restriction.

Is it possible for you to lift the password reset requirement on my account? It’s frustrating to have to change every 90 days especially because so many different Apple services, many of which do not share a centralized, common keychain entry, require the password to function at all. Every 90 days is marked by a sudden explosion of password dialogs on my Macs, iPhone, iPads, etc.

My team ID is [My Team ID]. Thank you for any help you can provide,

Daniel Jalkut
Red Sweater Software

That was yesterday. Today, less than 24 hours after I submitted the request, I got a call on my phone from an area code (916) that I recognized as coming from far Northern California (Sacramento and above). I struggled to imagine who could be calling me from that part of the world, but was relieved when I picked up and met my senior advisor from Apple’s Developer Relations support team.

She indicated clearly that she was filing a ticket on my behalf with the technical team, asking them to look for any flag on my account that could cause this. She suggested that probably what will happen is they will find something, eradicate it, and then they will ask her to keep the ticket open for 90 days to ensure that the problem has in fact been addressed. In the mean time, she promises to keep my apprised of any news.

The main difference between this interaction and the previous interaction with AppleCare is that my contact in the developer support team seems to actually recognize this is an issue and seems confident that it can be fixed. I asked her a little more about it and she said it did sound familiar but she hadn’t run into the problem for a long time. She tended to think it has something to do with an old option for password expiration that was available in the past but no longer is. Curiously, she said she had run across the issue a few times suddenly just in the past few days. I wonder… maybe I’m not the only one who got inspired by Rosyna’s tweet.

I’m not 100% confident that my problems with the 90-day Apple ID password expiration are over, but I would definitely go so far as to say I’m cautiously optimistic. If you’ve suffered with this problem and felt there was nothing to be done about it, maybe it’s time to get in touch with developer support and see if there is enough optimism to go around.