Git Bisect Found Our Regression in Six Steps Instead of Six Hours

A performance regression report came in: a report page that used to load in under a second now took four to five seconds, and nobody could say exactly when it had gotten slow — just that it had, sometime in the last few weeks. Between the last known-good state and now sat 140 commits. Reading through 140 commits' diffs looking for something that "could" cause a slowdown is a bad plan; git bisect turns that into a binary search, and this is the actual six-step session that found it.

Starting the bisect: marking the known boundaries

git bisect start
git bisect bad HEAD              # current commit is slow
git bisect bad HEAD              # confirmed bad
git bisect good v2.4.0            # this tagged release was known-fast

Marking one commit bad (has the problem) and one good (doesn't have it) is all bisect needs to start narrowing the range. Git immediately checks out the commit roughly halfway between the two and waits for a verdict on that midpoint.

Step 1: testing the midpoint manually

Bisecting: 69 revisions left to test after this (roughly 6 steps)
[a3f9c21] Merge pull request #412 from feature/search-filters

Git checked out a commit roughly 70 commits into the range and reported "roughly 6 steps" remaining — that's log2(140), the whole point of a binary search over a linear history instead of reading commits in order. Loading the report page against this checkout: still slow, about 4 seconds.

git bisect bad

Step 2: narrowing further

Bisecting: 34 revisions left to test after this (roughly 5 steps)
[7c1de88] Add caching layer for dashboard widgets

This commit's page load: fast, under a second.

git bisect good

This is the step that narrows the actual suspect range the most, and it's worth pausing on: the regression is now known to exist somewhere between commit 7c1de88 (good) and a3f9c21 (bad) — roughly 35 commits, half of the previous range, found with two manual page loads instead of reading two commits' worth of diffs.

Steps 3 through 5: continuing the same pattern

Bisecting: 17 revisions left to test after this (roughly 4 steps)
[d82a04f] Refactor report query builder    -> tested: SLOW -> bad

Bisecting: 8 revisions left to test after this (roughly 3 steps)
[f0339ab] Update report date-range filter UI -> tested: fast -> good

Bisecting: 3 revisions left to test after this (roughly 2 steps)
[9e77c15] Add report export to CSV feature   -> tested: SLOW -> bad

Each step is the same two actions repeated: load the report page against the checked-out commit, then run either git bisect good or git bisect bad based on what happened. No diff-reading required at any of these steps — the test itself (page load time) is the only signal bisect needs.

Step 6: the culprit

Bisecting: 0 revisions left to test after this (roughly 0 steps)
[c44b291] Add subquery for export row count estimate

git bisect bad

c44b291a9f... is the first bad commit
commit c44b291a9f...
Author: ...
Date:   ...

    Add subquery for export row count estimate

 app/Services/ReportQueryBuilder.php | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

Six manual tests, and git named the exact commit — a change that added a subquery to estimate row counts for the CSV export feature, which turned out to run on every report page load, not just when someone actually clicked "export," because the estimate was computed eagerly rather than lazily on demand. Reading that one commit's twelve-line diff immediately made the bug obvious, but only because bisect had already pointed at precisely the right twelve lines out of 140 commits' worth of changes.

Automating the test instead of doing it by hand

Manually loading a page and typing good or bad six times was fine here, but for a bug with an automatable reproduction, git bisect run does the entire search unattended:

git bisect start
git bisect bad HEAD
git bisect good v2.4.0
git bisect run ./test-report-load-time.sh

Where the script exits 0 for good and non-zero for bad:

#!/bin/bash
# test-report-load-time.sh
START=$(date +%s%N)
curl -s -o /dev/null http://localhost:8000/reports/quarterly
END=$(date +%s%N)
ELAPSED_MS=$(( (END - START) / 1000000 ))

if [ "$ELAPSED_MS" -gt 2000 ]; then
    exit 1   # bad: took longer than 2 seconds
else
    exit 0   # good
fi

With a script like this, the entire six-step search runs in the time it takes six page loads to happen automatically, with zero manual judgment calls — genuinely useful for a regression that's tedious or slow to test by hand, or for re-running the same bisect later if a similar regression shows up again in a different feature.

Ruling out non-code causes before you start bisecting

Before starting the bisect at all, it's worth a few minutes confirming the regression is actually a code change and not an environment or infrastructure difference masquerading as one — bisecting 140 commits to find nothing, because the real cause was a changed database instance size or a different deployment region, wastes far more time than the check that would have ruled it out. In this case, a quick sanity check involved confirming both the "known good" and "known bad" states were being tested against the same database and the same server region — cross-checking the request's actual origin with a proper IP lookup tool confirmed both test runs really were hitting the same regional deployment, not two different ones that happened to have different underlying hardware performance. Only once that was ruled out did the bisect proceed on the assumption that a code change, not an infrastructure change, was the actual cause.

This kind of check is easy to skip when you're eager to start narrowing commits, and it's exactly the kind of assumption worth stating and verifying explicitly rather than trusting silently — the same instinct behind stating assumptions before trusting AI-generated output, covered in five prompt engineering patterns that actually improve output quality, applies just as well to debugging assumptions as to prompting ones. Five minutes ruling out the obvious non-code explanation is cheap insurance against an hour spent bisecting a regression that was never actually in the commit history at all, chasing a phantom through 140 individually innocent commits that were never going to explain a slowdown caused by something outside the repository entirely.

Skipping commits that can't be tested

Real commit histories aren't always cleanly bisectable — occasionally a commit in the range won't build, or represents a genuinely broken intermediate state (a multi-commit feature branch merged without every individual commit being independently runnable). For exactly that case, bisect has a third verdict beyond good and bad:

git bisect skip

Marking a commit skip tells git "I can't test this one, keep narrowing around it" rather than forcing a good/bad answer git would otherwise wrongly treat as data. Bisect adjusts its next guess to avoid landing exactly on unbuildable commits where possible, though a range with many consecutive unbuildable commits can still slow the search down or occasionally leave a slightly wider final range than a perfectly clean history would.

Reviewing the search afterward with bisect log

git bisect log

Running this immediately after finishing the search prints every good/bad/skip verdict made during the session, in order — a genuinely useful artifact to attach to the bug report or commit message fixing the regression, since it documents exactly how the culprit commit was found rather than leaving that context to fade from memory within a week of anyone actually needing it again. It also doubles as a script: piping a saved log back into git bisect replay reruns the exact same sequence of checkouts on demand, useful for walking a teammate through how a particular regression was found without them having to redo the search from scratch.

What made this bisectable in the first place

• A clear, repeatable test. "Page takes noticeably longer to load" is testable at any commit; a vague symptom like "feels less stable lately" generally isn't, and bisect needs a test that gives a clean good/bad answer at every point in the range.
• A known-good reference point. Having the v2.4.0 tag as a confirmed-fast baseline meant the search had a real starting boundary instead of guessing how far back to look.
• Commits small enough to build and run individually. If a commit in the range doesn't build or doesn't run cleanly, bisect lets you mark it skip rather than good or bad, but a history full of broken intermediate states makes the whole process much slower.

Why this beats reading history in order

Reading 140 commits' diffs top to bottom, hoping something jumps out as "probably slow," doesn't scale, and it's also just a worse search strategy even when it eventually works — you're reading every commit whether or not it's relevant, rather than eliminating half the remaining suspects with each single test. The same logarithmic-search instinct that makes a sliding-window rate limiter worth reasoning through carefully rather than guessing at applies just as directly here: a systematic search beats an intuition-driven one whenever the search space is large enough that intuition stops being reliable, and 140 commits is well past that point.

The other habit worth taking from this: bisect only works well when history is reasonably clean, which is itself an argument for smaller, more atomic commits with honest messages describing what actually changed and why — not because atomic commits are a virtue in the abstract, but because a future regression hunt through your own history is a real, concrete beneficiary of the discipline, even if that payoff is invisible on the day you're writing the commit. A history of large, mixed-purpose commits doesn't make bisect impossible, it just means each individual verdict is noisier, since a "bad" commit that bundled five unrelated changes tells you far less about which of the five actually mattered.