Skip to content

Commit e7b474c

Browse files
authored
perf(worktree): stop worktree session creation blocking on large-repo scans and deletes (#3141)
## Problem Starting a new worktree session in a large repo (posthog/posthog) gets slower and slower over a week of use. Three compounding causes, all measured on a real posthog checkout: 1. **~20s of ls-files walks per create.** `processWorktreeLink`/`processWorktreeInclude` each run `git ls-files --ignored --others --directory --exclude-from=<file>`. Because `--exclude-from` replaces the standard excludes, git cannot collapse `node_modules/` and recurses the entire ignored tree looking for pattern matches (9.6s + 9.8s, warm cache, growing with ignored-file count). The walk also matched strays like `node_modules/.pnpm/psl@1.9.0/node_modules/psl/.env`, and copying that pre-created `node_modules/.pnpm/` in the fresh worktree — silently defeating posthog's own `.husky/post-checkout` bootstrap check (`[ ! -d node_modules/.pnpm ]`). 2. **Creation blocks on suspending the least-recent task once at the worktree cap.** `doCreateWorkspace` awaited `suspendLeastRecentIfOverLimit()` before creating: a full checkpoint capture plus deletion of a ~6GB, node_modules-laden worktree, inline, before `git worktree add` even starts. Under the cap (early in the week) this branch is skipped entirely, which is why creation "became" slow after days of use. 3. **Worktree deletes rm multi-gigabyte trees inside the repo write lock.** Every suspend, archive, and hourly inactivity sweep serialized with worktree creation on the per-repo write lock while unlinking ~1M files. ## Changes - `getIgnoredPathsFromExcludeFile` now lists candidates with `--exclude-standard --exclude-from=<file>` (so git collapses standard-ignored trees instead of recursing them) and re-applies the exclude file's own patterns in-process via a new gitignore-subset matcher (`exclude-patterns.ts`: comments, negation, dir-only, anchoring, `*`/`?`/`**`/`[...]`). Matches buried inside standard-ignored trees no longer surface, fixing the node_modules bootstrap bug. Repos without `.worktreelink`/`.worktreeinclude` now skip the git calls entirely. - The worktree cap is enforced *after* workspace creation as a fire-and-forget, instead of blocking creation. `suspendLeastRecentIfOverLimit` now suspends exactly the excess over the cap, oldest first (previously it fired when merely *at* the cap, pre-create). Net behavior change: the active count can briefly reach `maxActiveWorktrees + 1` while the background suspend runs. - `deleteWorktree` renames the worktree into `<worktreeBase>/.trash/` (instant, same volume), runs `git worktree prune` under the write lock, and does the recursive removal in the background. Falls back to the old `git worktree remove --force` path when the rename cannot work (cross-volume, already gone). Boot sweeps `.trash/` for leftovers from interrupted deletes, alongside the existing orphan cleanup. Measured on posthog/posthog: the `.worktreeinclude` scan went from ~10s to ~140ms with identical output minus the node_modules stray; worktree deletion no longer holds the write lock for the duration of a multi-gigabyte rm. ## How did you test this? - 36 new parameterized matcher tests (`exclude-patterns.test.ts`). - New real-git integration test: `.envrc` linked, `.env` copied, and `node_modules/` never created in a fresh worktree (regression for the bootstrap bug). - New suspension tests: at-cap does nothing, multi-excess suspends oldest first. - New `sweepTrash` test; existing worktree lifecycle (add/remove/prune) tests pass unchanged against the trash-based delete. - Full suites green: `@posthog/git` 306 tests, `@posthog/workspace-server` 612 tests. Typecheck and Biome clean. - Verified the new scan end-to-end against the real posthog repo via the built package: `.worktreelink -> []` (comments only), `.worktreeinclude -> [".env"]` in ~140ms. ## Automatic notifications - [ ] Publish to changelog? - [ ] Alert Sales and Marketing teams? 🤖 Generated with [Claude Code](https://claude.com/claude-code)
1 parent fcd52e3 commit e7b474c

9 files changed

Lines changed: 639 additions & 32 deletions

File tree

Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
import { describe, expect, it } from "vitest";
2+
import {
3+
matchesExcludePatterns,
4+
parseExcludePatterns,
5+
} from "./exclude-patterns";
6+
7+
function matches(content: string, entry: string): boolean {
8+
return matchesExcludePatterns(entry, parseExcludePatterns(content));
9+
}
10+
11+
describe("parseExcludePatterns", () => {
12+
it.each([
13+
["empty content", ""],
14+
["only comments", "# a comment\n# another"],
15+
["only blank lines", "\n\n \n"],
16+
["a lone negation marker", "!"],
17+
["a lone slash", "/"],
18+
])("produces no patterns from %s", (_label, content) => {
19+
expect(parseExcludePatterns(content)).toEqual([]);
20+
});
21+
});
22+
23+
describe("matchesExcludePatterns", () => {
24+
it.each([
25+
["basename pattern matches at root", ".env", ".env", true],
26+
["basename pattern matches nested", ".env", "config/sub/.env", true],
27+
[
28+
"basename pattern does not match other names",
29+
".env",
30+
".env.local",
31+
false,
32+
],
33+
["comment lines never match", "# .env", ".env", false],
34+
["star glob within a segment", "*.local", ".env.local", true],
35+
[
36+
"star glob matches a nested file via the non-anchored prefix",
37+
"*.local",
38+
"a/b.local",
39+
true,
40+
],
41+
["star does not span slashes", "a*b", "a/b", false],
42+
["question mark matches one char", ".env?", ".envX", true],
43+
["question mark does not match slash", ".env?", ".env/", false],
44+
["anchored pattern matches from root only", "/build", "build", true],
45+
["anchored pattern rejects nested path", "/build", "sub/build", false],
46+
[
47+
"middle-slash pattern anchors to root",
48+
"config/secrets",
49+
"config/secrets",
50+
true,
51+
],
52+
[
53+
"middle-slash pattern rejects nested",
54+
"config/secrets",
55+
"app/config/secrets",
56+
false,
57+
],
58+
["dir-only pattern matches directory entry", ".flox/", ".flox/", true],
59+
["dir-only pattern rejects plain file", ".flox/", ".flox", false],
60+
["dir pattern matches files beneath it", ".flox", ".flox/cache/data", true],
61+
[
62+
"dir-only pattern matches files beneath it",
63+
".flox/",
64+
".flox/cache/data",
65+
true,
66+
],
67+
["double star prefix matches any depth", "**/logs", "a/b/logs", true],
68+
["double star suffix matches contents", "logs/**", "logs/a/b.txt", true],
69+
[
70+
"double star suffix does not match the dir itself",
71+
"logs/**",
72+
"logs",
73+
false,
74+
],
75+
["middle double star spans directories", "a/**/b", "a/x/y/b", true],
76+
["middle double star matches zero directories", "a/**/b", "a/b", true],
77+
["character class matches", ".env.[ab]", ".env.a", true],
78+
["negated character class rejects", ".env.[!ab]", ".env.a", false],
79+
[
80+
"negation un-matches an earlier pattern",
81+
".env*\n!.env.example",
82+
".env.example",
83+
false,
84+
],
85+
[
86+
"negation only affects matching entries",
87+
".env*\n!.env.example",
88+
".env",
89+
true,
90+
],
91+
["later pattern wins over earlier negation", "!.env\n.env", ".env", true],
92+
["escaped bang matches literal bang", "\\!important", "!important", true],
93+
["escaped hash matches literal hash", "\\#file", "#file", true],
94+
["trailing spaces are trimmed", ".env ", ".env", true],
95+
["CRLF line endings do not defeat matching", ".env\r\n", ".env", true],
96+
[
97+
"consecutive double-star segments collapse",
98+
"**/**/logs",
99+
"a/b/logs",
100+
true,
101+
],
102+
])("%s", (_label, content, entry, expected) => {
103+
expect(matches(content, entry)).toBe(expected);
104+
});
105+
106+
it("skips a malformed pattern line instead of dropping the whole file", () => {
107+
// An unterminated char class on one line must not throw out the valid ones.
108+
const patterns = parseExcludePatterns(".env\n[\n.envrc");
109+
expect(matchesExcludePatterns(".env", patterns)).toBe(true);
110+
expect(matchesExcludePatterns(".envrc", patterns)).toBe(true);
111+
});
112+
113+
it("matches a pathological consecutive-double-star pattern in bounded time", () => {
114+
// Regression for ReDoS: a run of `**/` used to compile to that many
115+
// overlapping backtracking groups, blowing up exponentially with path depth.
116+
const pattern = `${Array(30).fill("**").join("/")}/NOMATCH`;
117+
const patterns = parseExcludePatterns(pattern);
118+
const deepPath = `${Array.from({ length: 24 }, (_, i) => String.fromCharCode(97 + (i % 26))).join("/")}/`;
119+
const start = performance.now();
120+
expect(matchesExcludePatterns(deepPath, patterns)).toBe(false);
121+
expect(performance.now() - start).toBeLessThan(1000);
122+
});
123+
124+
it("never matches entries only reachable through unrelated names", () => {
125+
expect(matches(".env", "node_modules/")).toBe(false);
126+
expect(matches(".env", "dist/")).toBe(false);
127+
});
128+
});
Lines changed: 162 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,162 @@
1+
export interface ExcludePattern {
2+
negated: boolean;
3+
dirOnly: boolean;
4+
regex: RegExp;
5+
}
6+
7+
/**
8+
* Parses gitignore-style pattern lines (comments, negation, dir-only trailing
9+
* slash, root anchoring, `*`/`?`/`**`/`[...]` globs). Used to re-apply an
10+
* exclude file's patterns in-process against paths git has already listed, so
11+
* callers can avoid asking git to walk huge ignored trees.
12+
*/
13+
export function parseExcludePatterns(content: string): ExcludePattern[] {
14+
const patterns: ExcludePattern[] = [];
15+
16+
for (const rawLine of content.split("\n")) {
17+
const line = trimUnescapedTrailingSpaces(rawLine);
18+
if (!line || line.startsWith("#")) continue;
19+
20+
let pattern = line;
21+
let negated = false;
22+
if (pattern.startsWith("!")) {
23+
negated = true;
24+
pattern = pattern.slice(1);
25+
} else if (pattern.startsWith("\\!") || pattern.startsWith("\\#")) {
26+
pattern = pattern.slice(1);
27+
}
28+
29+
let dirOnly = false;
30+
if (pattern.endsWith("/")) {
31+
dirOnly = true;
32+
pattern = pattern.slice(0, -1);
33+
}
34+
if (!pattern) continue;
35+
36+
const anchored = pattern.includes("/");
37+
if (pattern.startsWith("/")) {
38+
pattern = pattern.slice(1);
39+
}
40+
41+
// A single malformed pattern must not drop the whole exclude file: skip the
42+
// offending line rather than letting a RegExp throw propagate out.
43+
let regex: RegExp;
44+
try {
45+
regex = globToRegExp(pattern, anchored);
46+
} catch {
47+
continue;
48+
}
49+
patterns.push({ negated, dirOnly, regex });
50+
}
51+
52+
return patterns;
53+
}
54+
55+
/**
56+
* Whether a path matches the pattern list, last match wins (gitignore
57+
* semantics). `entry` may carry a trailing slash to mark a directory, as in
58+
* `git ls-files --directory` output. A pattern matching a parent directory
59+
* matches everything beneath it.
60+
*/
61+
export function matchesExcludePatterns(
62+
entry: string,
63+
patterns: ExcludePattern[],
64+
): boolean {
65+
const isDir = entry.endsWith("/");
66+
const entryPath = isDir ? entry.slice(0, -1) : entry;
67+
68+
let matched = false;
69+
for (const pattern of patterns) {
70+
if (patternMatches(pattern, entryPath, isDir)) {
71+
matched = !pattern.negated;
72+
}
73+
}
74+
return matched;
75+
}
76+
77+
function patternMatches(
78+
pattern: ExcludePattern,
79+
entryPath: string,
80+
isDir: boolean,
81+
): boolean {
82+
if ((isDir || !pattern.dirOnly) && pattern.regex.test(entryPath)) {
83+
return true;
84+
}
85+
86+
let separatorIndex = entryPath.indexOf("/");
87+
while (separatorIndex !== -1) {
88+
if (pattern.regex.test(entryPath.slice(0, separatorIndex))) {
89+
return true;
90+
}
91+
separatorIndex = entryPath.indexOf("/", separatorIndex + 1);
92+
}
93+
return false;
94+
}
95+
96+
function trimUnescapedTrailingSpaces(line: string): string {
97+
// Drop a trailing CR first so CRLF-terminated exclude files don't bake a \r
98+
// into every pattern (which would make the compiled regex match nothing).
99+
return line.replace(/\r$/, "").replace(/(?<!\\) +$/, "");
100+
}
101+
102+
function globToRegExp(pattern: string, anchored: boolean): RegExp {
103+
let source = anchored ? "^" : "^(?:.*/)?";
104+
let i = 0;
105+
106+
while (i < pattern.length) {
107+
const char = pattern[i];
108+
if (char === "*") {
109+
if (pattern[i + 1] === "*") {
110+
if (pattern[i + 2] === "/") {
111+
// Collapse a run of consecutive `**/` into one `(?:.*/)?`. They are
112+
// semantically equivalent, and emitting one group per segment would
113+
// stack overlapping backtracking `.*` groups — catastrophic on a
114+
// slash-heavy path that fails the final literal (ReDoS).
115+
source += "(?:.*/)?";
116+
i += 3;
117+
while (
118+
pattern[i] === "*" &&
119+
pattern[i + 1] === "*" &&
120+
pattern[i + 2] === "/"
121+
) {
122+
i += 3;
123+
}
124+
} else {
125+
source += ".*";
126+
i += 2;
127+
}
128+
} else {
129+
source += "[^/]*";
130+
i += 1;
131+
}
132+
} else if (char === "?") {
133+
source += "[^/]";
134+
i += 1;
135+
} else if (char === "[") {
136+
const classEnd = pattern.indexOf("]", i + 2);
137+
if (classEnd === -1) {
138+
source += "\\[";
139+
i += 1;
140+
} else {
141+
let charClass = pattern.slice(i + 1, classEnd);
142+
if (charClass.startsWith("!")) {
143+
charClass = `^${charClass.slice(1)}`;
144+
}
145+
source += `[${charClass}]`;
146+
i = classEnd + 1;
147+
}
148+
} else if (char === "\\" && i + 1 < pattern.length) {
149+
source += escapeRegExp(pattern[i + 1]);
150+
i += 2;
151+
} else {
152+
source += escapeRegExp(char);
153+
i += 1;
154+
}
155+
}
156+
157+
return new RegExp(`${source}$`);
158+
}
159+
160+
function escapeRegExp(char: string): string {
161+
return /[.*+?^${}()|[\]\\/]/.test(char) ? `\\${char}` : char;
162+
}

0 commit comments

Comments
 (0)