git bisect run: finding the commit that broke it, with skips, merges and flaky tests

Something worked three weeks ago and does not work now. There are 200 commits in between. You could read them, or you could let Git binary-search them: about eight checks instead of two hundred.

git bisect does the search. git bisect run removes you from the loop, so a script decides good or bad at every step. This article covers the parts that decide whether the answer is right: the exit-code contract, commits that cannot be built, merge commits, and bugs that do not reproduce every time.

Every command and script below was run against throwaway repositories on Git 2.43. The behaviour described is documented in git help bisect.

The basic loop

Give Git a bad commit and a good one. It checks out the midpoint, you test it and say which it is, and the range halves. With a script, the whole thing is two commands:

git bisect start HEAD v1.4      # bad first, then good
git bisect run ../check.sh      # script kept outside the repo, see below
# ...
# <hash> is the first bad commit
git bisect reset                # back to where you started

The script’s exit code is the whole interface:

• 0 means this commit is good.
• 1 to 127, except 125, means this commit is bad.
• 125 means this commit cannot be tested, so skip it.
• 128 and above aborts the bisect.

That last rule matters. A script that dies from a signal, or calls exit(-1) (which becomes 255), stops the run instead of marking a commit bad. We checked: a script exiting 129 produced bisect run failed: exit code 129 ... is < 0 or >= 128 and left the bisect in progress for you to inspect. That is the right behaviour for a tool whose answer you are going to act on.

Run the script by hand on a known-good and a known-bad commit before you start. Exit code 127 (command not found) is an ordinary “bad” to the search. If your script fails that way on every commit, Git catches it at the end with bogus exit code 127 for good revision. If it fails only on some commits, for example a tool missing in old ones, nothing warns you and the search is misdirected.

Keep the script outside the repository. Git rewrites the working tree at each step, so a script inside it is rewound to its old versions, or vanishes, on older commits. Outside the tree, you can write the test today and run it against code from before the test existed.

For short checks, git bisect run sh -c 'make || exit 125; ../check.sh' works inline.

Name the ends sensibly

“Good” and “bad” are confusing when you are hunting for the commit that fixed something, or changed a performance figure. Git lets you rename them:

git bisect start --term-old=working --term-new=broken HEAD v1.4
git bisect broken      # instead of "bad"
git bisect working     # instead of "good"

bisect run is unaffected, because it only reads the exit code: 0 for the old state, non-zero for the new one. When you hunt for a fix, the mapping is counter-intuitive: with --term-old=broken --term-new=fixed, the script exits 0 while the bug is still present.

Commits that cannot be tested: exit 125

A commit that does not compile tells you nothing about your bug. Answer “bad” or “good” for it and you point Git at the wrong place. Answer 125 and Git skips it and picks a neighbour:

#!/bin/sh
make -s || exit 125          # unbuildable: skip, don't judge
./run-test.sh                # exits 0 or 1

Skipping has a limit that is worth knowing about before you rely on it. Git can only name the culprit if it can tell good from bad on both sides of it. If the commit that introduced the bug sits inside a run of unbuildable commits, the answer is not available, and Git says so. In one of our test repositories the build broke at commit 4, was fixed at commit 7, and the real bug arrived at commit 6. The output was:

There are only 'skip'ped commits left to test.
The first broken commit could be any of:
(four commit hashes)

The honest answer is a list of four candidates, and the real culprit was one of them. You now read four commits instead of the whole range. Do not make the script treat a build failure as “bad” to force a single result, because it will happily return the wrong commit.

To skip a known-bad stretch without running anything, hand Git a range:

git bisect skip v1.2..v1.3

Merges: use –first-parent

On a repository with merge commits, bisect may land inside a feature branch. Branches often contain half-finished commits that do not build or that fail for reasons unrelated to your bug, even when the merge itself was fine. That is a recipe for false results and a lot of skipping.

--first-parent follows only the main line, so each merge counts as one step:

git bisect start --first-parent HEAD v1.4
git bisect run ../check.sh

We built a history where a feature branch with five commits was merged, and the fifth set the bad state. With --first-parent, bisect named the merge commit (Merge feat 2) as first bad. Without it, bisect named the exact commit on the branch (feat 5 sets bad).

Use --first-parent first when your team merges branches that were green in CI. It is faster and avoids half-finished branch commits. Then, if you need the exact commit, bisect inside the merge:

M=<sha of the merge found above>
git bisect start $M^2 $(git merge-base $M^1 $M^2)
git bisect run ../check.sh

The bad end is the branch tip ($M^2) and the good end is where the branch left the main line. In our test this returned feat 5 sets bad.

Intermittent bugs: one pass does not prove “good”

One wrong answer from the script makes the whole search wrong, because every later step builds on it. There are two cases.

The test is flaky and the code is fine. A good commit fails by chance and gets labelled bad. We simulated a test that fails about one run in five on good code. On a short history, bisect named an innocent commit.

The bug is intermittent. A bad commit passes by chance and gets labelled good. This is more dangerous, because a pass looks like evidence. We simulated a bug that fails one run in four, in a short history. Over six bisects with one run per commit, all six named the wrong commit. Running the check 20 times per commit and stopping at the first failure fixed it:

#!/bin/bash
# many.sh: bad if ANY of 20 runs fails
for _ in $(seq 20); do
  ../once.sh || exit 1
done

With that script, six out of six bisects named the right commit. A bad commit that fails 25% of the time passes 20 runs with probability 0.75^20, about 0.3%. A bug that fails one run in twenty needs about 115 runs for the same confidence, so estimate the reproduction rate first. That 0.3% is per commit tested, and a bisect tests several, so the risk of a wrong answer overall is a few per cent. Failure on any run means bad, which only works if the test is not flaky on good code. For a flaky test, fix or quarantine it first, or require several consecutive failures before calling a commit bad. If the rate is very low, raise the load that triggers the bug (more threads, a tighter timeout, a seed known to fail) before bisecting.

Faster runs: –no-checkout

Checking out every midpoint touches the working tree. With --no-checkout, Git only updates a ref called BISECT_HEAD, and your script reads the commit directly:

#!/bin/sh
git show BISECT_HEAD:config/limits.yml | grep -q 'max: 100' && exit 1
exit 0

git bisect start --no-checkout HEAD v1.4
git bisect run ../check.sh

This suits checks that only read files, such as a config value or a lockfile. Anything that runs the code needs a checkout.

A checklist before you start

1. Run the script by hand on a known-good and a known-bad commit. The answers must differ.
2. Return 125 for anything that stops you testing (build, install, missing fixture), never 1.
3. Repeat the check if the bug is intermittent, and fail on the first failure.
4. Use --first-parent when merges are the unit of change.
5. Read git bisect log, which records every decision.
6. Check the result: revert the commit on a branch and confirm the bug goes away. Bisect finds where behaviour changed, not why.

Writing the failing check is usually most of the work, so keep it afterwards as a regression test. Tracking down a regression in a production service, such as a slow query or a webhook that stopped firing, is part of how we build and maintain APIs and integrations; see our backend development service. Related: idempotent payment webhooks, for the failures that are hardest to reproduce.