Announcing git-prole

TL;DR: I’ve built git-prole, an ergonomic git-worktree manager. Check out the user manual to get started! If you don’t know what a Git worktree is or why you might want one, read on.

If you’re anything like me, you have a couple repositories cloned that are cumbersome enough to clone more than once:

$ ls -l
drwxr-xr-x     - rebeccat  15 Oct 11:13 mwb
drwxr-xr-x     - rebeccat  11 Oct 14:19 mwb2
drwxr-xr-x     - rebeccat  30 Sep 10:06 mwb3
drwxr-xr-x     - rebeccat  19 Oct 13:31 mwb4
drwxr-xr-x     - rebeccat   2 Aug 16:45 mwb5
drwxr-xr-x     - rebeccat   3 Oct 12:10 mwb6

This feels a little bit silly, but there’s some good reasons for it:

If builds are expensive, then switching branches can incur a significant time cost.
You might want to check out a colleague’s branch, but you have uncommitted work you’re hesitant to commit or stash.
It’s often convenient to compare your code against the repository’s main branch. git diff can do this, of course, but chances are it’s not very well integrated into your editor, or you need to be able to run the code in addition to reading it.

At the same time, there’s some costs to this approach:

The repositories have to be cloned separately, which is annoying. You can clone from a local repository, but then you have to be careful to add the correct remote, remove the old remote, and set up a bunch of remote-tracking branches again.
You can’t easily share work between the repositories. I often want to split a couple of small fixes off from a current branch, but there’s not a great way to do this without generating a patch and applying it manually or pushing your work to a remote.
The clones are separate repositories, so it’s not easy to add new checkouts or remove unused ones. You effectively have to decide how many branches you want to be able to check out at the same time in advance.

Git worktrees, then, are a feature that allows you to associate multiple checkouts with a single repository, like this:

my-repo/
  main/        # A checkout for the main branch
    .git/      # The main .git directory.
    README.md
  feature1/    # A checkout for work on a feature
    .git       # A file containing `gitdir: /Path/to/my-repo/main/.git`
    README.md

And, as it turns out, with a little fiddling, you can also use a bare checkout for the .git directory; this means that none of the worktrees are “special” and they can all be deleted (if you try to set up a repository like this yourself, you will have problems; more on this later).

my-repo/
  .git/        # A bare repository
  main/        # A checkout for the main branch
    .git       # A file with the path to the bare repository
    README.md
  feature1/    # A checkout for work on a feature
    .git
    README.md

Features

git-prole includes a number of tools to make using multiple worktrees as ergonomic as possible:

git prole clone URL [DESTINATION] will clone a repository into the multiple-worktrees layout described above.
git prole convert will convert the current repository into a worktree layout. (Use --dry-run to see what will happen.)
git prole add will add a new worktree:
- git prole add feature1 will create a feature1 directory next to the rest of your worktrees; git worktree add feature1, in contrast, will create a feature1 subdirectory nested under the current worktree.
- Branches created with git prole add will start at and track the repository’s main branch by default.
- git prole add will copy untracked files to the new worktree by default, making it easy to start a new worktree with a warm build cache.
- git prole add can run commands when a new worktree is created, so that you can warm up caches by running a command like direnv allow.
- git prole add can perform regex substitutions on branch names to compute a directory name, so that you can run git prole add -b myname/team-1234-my-ticket-with-a-very-long-title and get a directory name like my-ticket.
- git prole add respects the -c/--create option (to match git switch); git worktree add only allows -b (with no long-form option available).

Why hadn’t I heard of worktrees before?

This is a pretty cool feature, and once we have some tooling like git-prole to paper over the ergonomic challenges it makes a great workflow! So why haven’t you heard of worktrees before?

For reasons I can’t fully determine, worktrees have received basically zero fanfare except from your coworker who runs nix-darwin. The git-worktree command was first present in Git 2.5.0, released in 2015. However, the release notes for Git 2.5.0, which include dozens of updates like “Clarify in the Makefile a guideline to decide use of USE_NSEC”, have nothing to say about worktrees. GitHub actually published a blog post titled “Git 2.5, including multiple worktrees and triangular workflows”, which received, as far as I can tell, one Reddit post with 35 upvotes and two top-level comments, neither of which were excited for the new feature.

Recently, worktrees have started to garner a little more attention (see, for example, matklad’s “How I Use Git Worktrees”), but the second search result for “git worktrees” is a Stack Overflow question titled “What would I use git-worktree for?”

Trying and failing with worktrees

A couple weeks ago, I finally decided to take a couple hours and figure out how to use worktrees. Let’s create a worktree for a new branch, which I’ll call feature1:

$ git worktree add feature1
Preparing worktree (new branch 'feature1')
branch 'feature1' set up to track 'main'.
HEAD is now at b1b2888 Fix static binary builds in releases (#86)

Great, that was easy! Now, let’s just type git status, which I enter reflexively after nearly any Git command:

$ git status
On branch main
Your branch is up to date with 'origin/main'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        feature1/

nothing added to commit but untracked files present (use "git add" to track)

…huh. The first argument to git worktree add is used as a branch name, but it’s actually a path, and it will happily create a worktree nested in another worktree, despite (presumably) nobody ever wanting to do this.

The actual invocation you want is something like this, where Git uses the last component of the path as the branch name:

$ git worktree add ../feature1

And then we get to the next ergonomic issue with Git worktrees: what do you name them? If you’re not a worktree user already, you probably have your repository cloned in some “projects” directory, containing a whole bunch of repositories. So when you start working on feature1, you should really disambiguate it:

$ git worktree add ../my-repo-feature1

This is a bad experience! You have to type a path manually, prefix the path with the repository name, and then you have to type the repository name again when you cd to the checkout later. Can we do better? Of course!

A better layout for a worktree repository

If we look at the example workflow from the GitHub blog post above, we can see that they expect you to have the workflow layout I showed you at the start of the post:

$ cd ../main
$ rm -rf ../hotfix

GitHub must expect you to have a containing directory named my-repo and then worktrees directly inside it, named corresponding to the branches they have checked out. git-prole works well with this layout; git prole add hotfix will create a new worktree in the same directory as the main worktree.

Let’s set up one of our repositories like that. First, we’ll move our main working tree from /path/to/my-repo to /path/to/my-repo/main:

$ git worktree move . main
fatal: '.' is a main working tree

Oh. I guess we’ll do it the hard way: renaming my-repo to my-repo.bak, creating a new my-repo directory, and then moving my-repo.bak to my-repo/main. A three-step barrier to entry for worktrees, and that’s if you only have one to start with. If you’ve already created some worktrees in the parent directory, you’ll have to figure out how to reassociate them with the main worktree once you move everything. Theoretically this is possible with git worktree repair, but I found myself having to edit files in the .git directory manually.

git-prole will handle this conversion process for you, even with multiple worktrees, un-committed work, or any number of other complications:

$ git worktree list
/path/to/git-prole  719ed92 [main]
$ nix run github:9999years/git-prole -- convert

• Converting ~/git-prole to a worktree repository at ~/git-prole.
  I'll move the following worktrees to new locations:
  • ~/git-prole -> ~/git-prole/main
  Additionally, I'll convert the repository to a bare repository.

Preparing worktree (checking out 'main')
• ~/git-prole has been converted to a worktree checkout
• You may need to `cd .` to refresh your shell
$ cd .
$ git worktree list
/path/to/git-prole       (bare)
/path/to/git-prole/main  719ed92 [main]

Eventually, I got the repository set up the way I wanted, but there was still one thing that bothered me: the .git directory. If you create a worktree layout by moving your existing worktrees around, you end up with one special “main” worktree, which contains the full .git directory shared by the other worktrees and cannot be renamed or removed.

Having a main worktree is fine, but it’s smelly. I don’t like that all of the worktrees are ephemeral and short-lived except for one. The solution to this is to have the main worktree be a bare repository: a .git directory with no accompanying working tree. A repository is bare if its configuration sets the core.bare option to true:

$ git status
On branch main
nothing to commit, working tree clean
$ git config set --local core.bare true
$ git status
fatal: this operation must be run in a work tree

So we create the my-repo directory and do a bare clone into it:

$ git clone --bare git@github.com:9999years/git-prole.git
Cloning into bare repository 'git-prole.git'...

Great! Now let’s set up a working tree:

$ cd git-prole.git/
$ git worktree add ../main
Preparing worktree (checking out 'main')
HEAD is now at 719ed92 Fix links to `git-worktree(1)` docs (#89)
$ cd ../main/
$ git pull
There is no tracking information for the current branch.
Please specify which branch you want to rebase against.
See git-pull(1) for details.

    git pull <remote> <branch>

If you wish to set tracking information for this branch you can do so with:

    git branch --set-upstream-to=origin/<branch> master

$ git branch --set-upstream-to=origin/master master
fatal: the requested upstream branch 'origin/master' does not exist
hint:
hint: If you are planning on basing your work on an upstream
hint: branch that already exists at the remote, you may need to
hint: run "git fetch" to retrieve it.
hint:
hint: If you are planning to push out a new local branch that
hint: will track its remote counterpart, you may want to use
hint: "git push -u" to set the upstream config as you push.
hint: Disable this message with "git config advice.setUpstreamFailure false"

…huh. Apparently, as someone pointed out on Stack Overflow to a user with the same problem, git clone --bare doesn’t just skip creating a working directory. According to the man page:

Also the branch heads at the remote are copied directly to corresponding local branch heads, without mapping them to refs/remotes/origin/.

This means that instead of (e.g.) creating a remote-tracking branch origin/main, Git creates a local branch main. Then, when you go to push or pull, Git errors out when it can’t find a matching remote branch. You can sort of fix this by setting a refspec for your remote (e.g., git config set --local remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*", good luck remembering that the next time you need it), but then you have to rename all of the refs manually, transforming refs/heads/foo to refs/remotes/origin/foo, which is a pain to say the least!

Converting an existing repository to a bare one is similarly painful, although for different reasons (it requires a song and dance to reassociate the main worktree with the repository). Eventually, I realized that the only way to fix these issues would be with a new tool to make setting up and working with multiple worktrees easy.

Easier worktrees with git-prole

For a breath of fresh air, let’s clone a repository with git-prole, taking advantage of the (configuration-gated) gh support to let us enter a GitHub repository slug instead of a full URL:

$ git prole clone 9999years/git-prole
Cloning into '/Users/wiggles/test-repo/git-prole'...
remote: Enumerating objects: 668, done.
remote: Counting objects: 100% (502/502), done.
remote: Compressing objects: 100% (215/215), done.
remote: Total 668 (delta 331), reused 420 (delta 286), pack-reused 166 (from 1)
Receiving objects: 100% (668/668), 277.44 KiB | 1.66 MiB/s, done.
Resolving deltas: 100% (407/407), done.

• • Move git-prole/.git to $TMPDIR/.tmpwdCaYS/.git
  • In $TMPDIR/.tmpwdCaYS/.git, set core.bare=true
  • Move git-prole to $TMPDIR/.tmpwdCaYS/main
  • Create directory git-prole
  • Move $TMPDIR/.tmpwdCaYS/.git to git-prole/.git
  • In git-prole/.git, create but don't check out a worktree for main at git-prole/main
  • In git-prole/main, reset the index state
  • Move git-prole/main/.git to $TMPDIR/.tmpwdCaYS/main/.git
  • Remove git-prole/main
  • Move $TMPDIR/.tmpwdCaYS/main to git-prole/main

Preparing worktree (checking out 'main')
• git-prole has been converted to a worktree checkout

$ cd git-prole
$ git worktree list
/path/to/git-prole       (bare)
/path/to/git-prole/main  719ed92 [main]

Nice! Let’s add a new worktree:

$ git prole add add feature1
• Creating worktree in ~/git-prole/feature1 for feature1 tracking origin/main
Preparing worktree (new branch 'feature1')
branch 'feature1' set up to track 'origin/main'.
HEAD is now at 719ed92 Fix links to `git-worktree(1)` docs (#89)

New branches automatically start at the repository’s default branch, instead of the current commit.

That’s the core of git-prole’s functionality, along with some goodies like copying untracked files to new worktrees to keep your build caches warm and your local configuration intact.

The workflow enabled by cheap checkouts is super nice and I encourage you to give it a try! To get started with git-prole, check out the user manual for installation instructions.