So You Want to Ship a Command-Line Tool for macOS

A word of advice: don’t.

At work, I’ve written a command-line tool which sets up the developer environment. It installs the Nix package manager, sets up a local Postgres instance, and handles all the complex bits of configuration. It does all of this on Linux and macOS, and it supports bash, zsh, and fish for shell configuration.

We build and publish releases with GitHub actions, so that engineers can download and run the latest version of the tool when they need to set up a new machine or repair the development environment on an existing machine.

On Linux, this is all fine and dandy (the usual headaches with running anything on NixOS aside), but macOS requires that programs be code signed and notarized.

Code signing on macOS is a nightmare. Knowing a little bit about asymmetric encryption, I expected the process to be roughly like this:

Generate a public and private key (this is a blob of binary data, or roughly “a file”).
Get Apple to make a certificate signed with my public key (this certificate is also a file).
Run a program, pointing it to the files containing my private key and certificate, which “signs” an executable for distribution.
When users go to run my program, the operating system can see that it’s signed with a valid Apple certificate and run it without issue.

This is not, in fact, how it works.

The first problem is that Apple provides no less than eight different sorts of certificates, with little documentation on what they’re used for, so expect to generate a half-dozen certificates and revoke several before you have a working one.

The next issue is that the codesign command-line tool has a really opaque and complex interface. There’s several options you need to actually run code that aren’t really indicated as such (-o runtime and -timestamp, for instance, are mandatory for notarization).

There’s few instructions for getting certificates into the keychain without GUI access (like, for instance, to sign code in CI), and Apple doesn’t have much documentation for signing anything that’s not built with Xcode.

I was able to solve most of the issues with actually signing code with the fantastic rcodesign tool written by Gregory Szorc, which has a very reasonable command-line interface that actually takes all the keys and certificates as files. I’ll excerpt a paragraph from the linked blog post that rings true to me (emphasis my own):

I’ve learned way too much around minutia around how Apple code signing actually works. The mechanism is way too complex for something in the security space. There was at least one high profile Gatekeeper bug in the past year allowing improperly signed code to run. I suspect there will be more: the surface area to exploit is just too large.

macOS does not support running command-line tools from Finder

Once I had succeeded in signing an executable and verifying it, I immediately ran into another issue: Gatekeeper blocks command-line tools from running when clicked. Apparently this is a “known bug in macOS”, though because it’s filed in Apple’s proprietary “Radar” bug-tracker, I can’t see any of the details or if there are any plans to fix it.

This is especially unfortunate because when an engineer downloads the binary from GitHub on macOS, the downloads pop up from the bottom of the screen, just begging to be clicked on.

The solution, Quinn from the Apple Developer Forums tells us, is to “embed your tool in an application.” This, as far as I can tell, doesn’t work either, but let’s run through it.

There’s a long guide on Embedding a Command-Line Tool in a Sandboxed App, so I followed that, and then slowly, painfully, factored Xcode out of it, so that I wouldn’t have to figure out how to get a 10GB Xcode install onto the CI machine (remember, you need to be signed in to an Apple ID to download Xcode, and there’s no way to do it from the command-line).

With Cassie’s help, I produced a short Swift script which (I hoped) would do what I want: First, it finds the embedded command-line tool binary with the Bundle.url(forAuxiliaryExecutable:) method. Then, that URL is passed to the NSWorkspace.open(urls:, withApplicationAt:) method to run the embedded command-line tool in a new window. Finally, the completionHandler closes the app once the terminal window is open.

(There’s a bug in Swift that makes the @main attribute, mandatory for SwiftUI apps, not work, so I needed to run swiftc myself, adding a mysterious -parse-as-library option to fix the bug. I also needed to read the Swift source code to determine the possible values for the -target command-line option.)

Now, we have to code sign:

The Swift wrapper script.
The original command-line tool.
The .app containing both of the above.
The .dmg containing the .app, because an .app is just a directory, so you need to zip it up to distribute it.

We also need to notarize the .app and the .dmg. Interestingly, you can only notarize .pkg, .dmg, and .app files (in .zips) — command-line tools can only be notarized if they’re embedded in one of the listed containers.

Most of the notarization docs tell you to use altool, which, only once you actually get it to upload something for notarization, will tell you that it’s deprecated and replaced by notarytool. notarytool will print status: Invalid if anything fails, with no additional details. (There’s a separate notarytool subcommand you can use to fetch logs with more information, but nothing in the output tells you this is an option, including turning on verbose/debug logging.)

Anyways, once the app was assembled and passing all of Apple’s validation tools, it, uh, continued to not work!

You can double click the app to run it, but when it tries to launch the embedded command-line tool, we get an error that “mytool-cli can’t be opened because the identity of the developer cannot be confirmed“, followed by “The application ‘Terminal’ can’t be opened. -128”.

I do get one error with Apple’s tooling on these files; though the .app itself passes all the checks, the embedded tool fails spctl’s validation:

$ spctl -a -v --raw ./mytool
./mytool: rejected (the code is valid but does not seem to be an app)
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>assessment:authority</key>
	<dict>
		<key>assessment:authority:flags</key>
		<integer>0</integer>
		<key>assessment:authority:source</key>
		<string>obsolete resource envelope</string>
		<key>assessment:authority:weak</key>
		<true/>
	</dict>
	<key>assessment:cserror</key>
	<integer>-67002</integer>
	<key>assessment:remote</key>
	<true/>
	<key>assessment:verdict</key>
	<false/>
</dict>
</plist>

Googling for “obsolete resource envelope” gives several different bugs over a period of years, none of which apply, and anyways spctl -a -v --raw shows the same error for the /bin/ls that gets shipped with macOS.

That’s where we’re at today. I’ve reported this issue on the Apple Developer Forums, but I don’t expect to get any actionable advice back, seeing as this distribution pathway is already attempting to work around several “known” macOS bugs.

The tools are bad

All of these tools and all of their error messages are garbage. (rcodesign, being not written by Apple, is a notable exception.) Here are some of the commands you might run to check code signatures:

spctl -a -v --raw mytool.app
codesign -verify -vvvv mytool.app
codesign -d -vvv --entitlements :- mytool.app
codesign -vvvv -R=notarized --check-notarization mytool.app

Just absolute spews of letters with no discernible meaning and certainly no reasonable intuition.

On the Apple Developer Forums, MirrorMan posts about roughly the same issue with frustration:

If the product is signed, notarised, and stapled correctly, everything should work. If not, you’ll need to investigate why Gatekeeper is unhappy (2), fix that, and then retest.

“Fix that” !!!!!! No! Just no! There MUST be a way for Gatekeeper/spctl to render an exact description of whether or not it will run something (apps, command line tools, …) on arbitrary customer machines, immediately, right away, from the development machine, bypassing any caching or anything else. How many thousands of 3rd party developer days need to be lost for one Apple developer to spend a few days to update spctl to produce 100% accurate and useful messaging? For example, the messaging out of notarytool was very good! It told me which of the half-dozen hoops I had to jump through next (keychains! app-specific passwords!). It just takes too too long to do the research to figure out what to do at all, only to be confronted with an inexplicable error. $237billion a year, and developers have to guess? Apple can do better. Sorry for the rant but it is necessary.

I don’t have much more to add, except that I’ve gained a lot of empathy for my friends who have turned their backs on the Apple ecosystem entirely for exactly this sort of developer-hostile behavior.

Moving on

So, what can we do with all of this? We have a few options, but none of them are particularly appealing.

Maintain the status quo; tell users to chmod +x and xattr -d com.apple.quarantine the downloaded executable. This bypasses the code signing mechanism but requires the user copy/paste magic commands.

It’s fine, ultimately — the users are engineers, so they can manage a terminal or we can teach them to — but it’s not particularly “clean”, especially when Apple (theoretically) offers code signing mechanisms for this exact purpose, which my employer already pays for.
Download the file with curl. Unix command-line tools like curl and tar don’t add the quarantine bit to the files they create on macOS, so it’s pretty easy to run software downloaded with them.

Unfortunately, the repository where this tool is developed is private, so unauthenticated curl downloads won’t work. I’m seriously considering asking the CTO if we can make the repository public solely so we can use curl as a distribution mechanism. I could even write a platform-detecting shell script!
Reimplement the entire thing in Swift and distribute it as an App built with Xcode without any embedded tools.

Hahahaha. Just kidding.
Distribute the tool as a custom Homebrew tap or something similar.

This tool is responsible for installing Homebrew, and moreover I work at a Nix startup so plenty of engineers refuse to install Homebrew. I think this is a little bit silly, but it’s my job to support them and deliver the smoothest possible developement experience, so we can’t really distribute this with Homebrew.

I think I’ll probably distribute a code-signed executable, and macOS will say it can’t figure out who made it even though their own tools say the signature is fine, and I’ll cry myself to sleep at night.

Conclusions

Do not attempt to “create software” for macOS. They don’t want you to. If you want to run your own programs, install Linux and suffer like you’re supposed to.
If you need to code sign software for Apple computers, use rcodesign.