Token Savings
Token Savings: LSP vs Grep/Read¶
How much context does an AI agent consume when navigating code with grep/read versus structured LSP calls? This experiment measures the bytes flowing through each approach on real codebases across three languages and four projects.
Key findings¶
| Codebase | Language | Lines | Overall | /lsp-rename |
Precision | Tokens saved |
|---|---|---|---|---|---|---|
| agent-lsp | Go | 15K | 5x | 96x | 92% noise | ~1.3M |
| Hono | TypeScript | 24K | 13x | 1,441x | 93% noise | ~1.2M |
| FastAPI | Python | 33K | 2x | 116x | 97% noise | ~693K |
| Next.js | TypeScript | 196K | 5x | 52x | 98% noise | ~1.1M |
| HashiCorp Consul | Go | 319K | 34x | 97x | 99% noise | ~13.1M |
/lsp-renamesaves 92-1,441x consistently. The grep agent must read every file containing the symbol to safely rename it; LSP does it atomically in 3 calls.- 92-99% of grep results are false positives. Grep matches strings; LSP returns
only actual symbol references. On consul (319K lines), grep for
Closereturned 1,156 matches; only 12 are real references. - Savings scale with codebase size. 5x at 15K lines, 21x at 319K lines.
- Speculative execution (
/lsp-safe-edit): 60x. Preview an edit without touching disk. The grep agent must edit, build (1.3s), revert, re-build. LSP answers in 2ms. - Code Map (
/lsp-understand): 6x, 936 grep calls reduced to 162. Full file analysis with hover, references, and call hierarchy for every exported symbol. - Edit safety: 23-656x. Structured diagnostics vs raw compiler output.
- Interface implementations: 1,002-1,813x (Go). Grep cannot determine which types satisfy an interface without reading every file.
- Multi-hop call chains: 9-12x, 87-311x faster (Go).
Methodology:
- Grep/Read = total bytes of grep output + file content read + build/test output.
- LSP = JSON response bytes from each LSP call (normalized to compact relative
paths, matching what agent-lsp returns to clients).
- Token estimate: 1 token per 4 bytes (standard approximation for code).
- LSP startup cost excluded (amortizes over a session).
- Go: 13 tasks covering 7 agent skills. Python/TypeScript: 11 tasks.
- Skill workflows test the full multi-step sequence (e.g. /lsp-refactor = blast-radius +
rename + verify), not individual tool calls. The grep side models the equivalent
multi-step agent workflow for the same task.
- Generated by go run ./experiments/token-savings.
agent-lsp (15,850 lines, 82 files)¶
Simple tasks
| Task | Grep/Read | LSP | Ratio | Round trips | Time |
|---|---|---|---|---|---|
Find callers of Shutdown |
3,115 | 1,133 | 3x | 1 vs 1 | 79ms vs 2ms |
Type signature of Shutdown |
4,872 | 393 | 12x | 1 vs 1 | 54ms vs 0ms |
| Edit safety check (break build, measure output) | 75,740 | 1,171 | 65x | 3 vs 3 | 0ms vs 0ms |
Skill workflows and advanced tasks (10 tasks)
| Task | Grep/Read | LSP | Ratio | Round trips | Time |
|---|---|---|---|---|---|
Skill: /lsp-refactor rename Shutdown |
28,304 | 30,733 | 1x | 17 vs 5 | 0ms vs 0ms |
Skill: /lsp-impact on client.go (52 exports) |
2,082,537 | 419,978 | 5x | 795 vs 109 | 0ms vs 0ms |
Skill: /lsp-rename Shutdown (14 files) |
220,455 | 2,285 | 96x | 16 vs 3 | 0ms vs 0ms |
Skill: /lsp-dead-code on client.go (52 exports, 5 dead) |
679,099 | 239,761 | 3x | 57 vs 57 | 0ms vs 0ms |
Skill: /lsp-understand Code Map of client.go (52 exports) |
3,250,695 | 554,820 | 6x | 936 vs 162 | 924ms vs 30ms |
Skill: /lsp-safe-edit (speculative edit + verify + revert) |
75,740 | 1,269 | 60x | 4 vs 3 | 1241ms vs 2ms |
Skill: /lsp-verify (diagnostics + build + tests) |
75,804 | 27,406 | 3x | 3 vs 3 | 5157ms vs 0ms |
Precision: Close (61 grep vs 5 LSP refs, 56 false+) |
4,958 | 517 | 10x | 1 vs 1 | 59ms vs 22ms |
Multi-hop: callers of callers of Shutdown |
41,680 | 4,637 | 9x | 25 vs 2 | 1097ms vs 2ms |
Interface: implementations of logSender (1 found) |
98,221 | 98 | 1002x | 4 vs 1 | 171ms vs 27ms |
Total: 6,641,220 grep/read vs 1,284,201 LSP = 5x savings (~1,339,254 tokens saved)
Hono (24,178 lines TypeScript, 185 files)¶
Simple tasks
| Task | Grep/Read | LSP | Ratio | Round trips | Time |
|---|---|---|---|---|---|
Find callers of Variables |
7,205 | 84 | 86x | 1 vs 1 | 59ms vs 1ms |
Type signature of Variables |
0 | 168 | 0x | 1 vs 1 | 37ms vs 1ms |
| Edit safety check (break build, measure output) | 79,359 | 121 | 656x | 3 vs 3 | 0ms vs 0ms |
Skill workflows and advanced tasks (8 tasks)
| Task | Grep/Read | LSP | Ratio | Round trips | Time |
|---|---|---|---|---|---|
Skill: /lsp-refactor rename Variables |
49,578 | 447 | 111x | 27 vs 5 | 0ms vs 0ms |
Skill: /lsp-impact on types.ts (41 exports) |
1,625,162 | 129,096 | 13x | 610 vs 86 | 0ms vs 0ms |
Skill: /lsp-rename Variables (24 files) |
492,954 | 342 | 1441x | 26 vs 3 | 0ms vs 0ms |
Skill: /lsp-dead-code on types.ts (41 exports, 8 dead) |
694,757 | 129,014 | 5x | 45 vs 45 | 0ms vs 0ms |
Skill: /lsp-understand Code Map of types.ts (41 exports) |
1,922,818 | 142,249 | 14x | 748 vs 128 | 475ms vs 14ms |
Skill: /lsp-safe-edit (speculative edit + verify + revert) |
79,359 | 218 | 364x | 4 vs 3 | 201ms vs 1ms |
Skill: /lsp-verify (diagnostics + build + tests) |
86,227 | 68 | 1268x | 3 vs 3 | 930ms vs 0ms |
Precision: Close (15 grep vs 1 LSP refs, 14 false+) |
1,051 | 91 | 12x | 1 vs 1 | 33ms vs 6ms |
Total: 5,038,470 grep/read vs 401,898 LSP = 13x savings (~1,159,143 tokens saved)
fastapi-bench (32,564 lines, 621 files)¶
Simple tasks
| Task | Grep/Read | LSP | Ratio | Round trips | Time |
|---|---|---|---|---|---|
Find callers of middleware |
7,928 | 572 | 14x | 1 vs 1 | 91ms vs 53ms |
Type signature of middleware |
0 | 156 | 0x | 1 vs 1 | 91ms vs 0ms |
| Edit safety check (break build, measure output) | 181,470 | 4,200 | 43x | 3 vs 3 | 0ms vs 0ms |
Skill workflows and advanced tasks (8 tasks)
| Task | Grep/Read | LSP | Ratio | Round trips | Time |
|---|---|---|---|---|---|
Skill: /lsp-refactor rename middleware |
24,665 | 4,202 | 6x | 25 vs 5 | 0ms vs 0ms |
Skill: /lsp-impact on routing.py (38 exports) |
1,460,499 | 802,062 | 2x | 418 vs 79 | 0ms vs 0ms |
Skill: /lsp-rename middleware (22 files) |
416,042 | 3,601 | 116x | 24 vs 3 | 0ms vs 0ms |
Skill: /lsp-dead-code on routing.py (38 exports, 22 dead) |
904,821 | 626,555 | 1x | 41 vs 41 | 0ms vs 0ms |
Skill: /lsp-understand Code Map of routing.py (38 exports) |
1,482,337 | 825,217 | 2x | 346 vs 118 | 1454ms vs 146ms |
Skill: /lsp-safe-edit (speculative edit + verify + revert) |
181,470 | 4,297 | 42x | 4 vs 3 | 0ms vs 3ms |
Skill: /lsp-verify (diagnostics + build + tests) |
379,102 | 3,659 | 104x | 4 vs 3 | 0ms vs 0ms |
Precision: validate (64 grep vs 2 LSP refs, 62 false+) |
6,465 | 204 | 32x | 1 vs 1 | 86ms vs 24ms |
Total: 5,044,799 grep/read vs 2,274,725 LSP = 2x savings (~692,518 tokens saved)
consul-bench (319,072 lines, 1468 files)¶
Simple tasks
| Task | Grep/Read | LSP | Ratio | Round trips | Time |
|---|---|---|---|---|---|
Find callers of GetNode |
10,815 | 5,956 | 2x | 1 vs 1 | 315ms vs 27ms |
Type signature of GetNode |
20,817 | 464 | 45x | 1 vs 1 | 381ms vs 0ms |
| Edit safety check (break build, measure output) | 173,498 | 7,477 | 23x | 3 vs 3 | 0ms vs 0ms |
Skill workflows and advanced tasks (10 tasks)
| Task | Grep/Read | LSP | Ratio | Round trips | Time |
|---|---|---|---|---|---|
Skill: /lsp-refactor rename GetNode |
46,874 | 14,055 | 3x | 21 vs 5 | 0ms vs 0ms |
Skill: /lsp-impact on catalog.go (57 exports) |
17,745,627 | 841,371 | 21x | 5534 vs 119 | 0ms vs 0ms |
Skill: /lsp-rename GetNode (18 files) |
802,815 | 8,286 | 97x | 20 vs 3 | 0ms vs 0ms |
Skill: /lsp-dead-code on catalog.go (57 exports, 22 dead) |
7,133,669 | 348,452 | 20x | 62 vs 62 | 0ms vs 0ms |
Skill: /lsp-understand Code Map of catalog.go (57 exports) |
27,288,671 | 315,737 | 86x | 6661 vs 178 | 5424ms vs 473ms |
Skill: /lsp-safe-edit (speculative edit + verify + revert) |
173,498 | 7,575 | 23x | 4 vs 3 | 25829ms vs 28ms |
Skill: /lsp-verify (diagnostics + build + tests) |
173,557 | 1,551 | 112x | 3 vs 3 | 17149ms vs 0ms |
Precision: Close (1156 grep vs 12 LSP refs, 1144 false+) |
83,702 | 1,337 | 63x | 1 vs 1 | 281ms vs 223ms |
Multi-hop: callers of callers of GetNode |
134,480 | 15,901 | 8x | 29 vs 2 | 3687ms vs 33ms |
Interface: implementations of ExportFetcher (1 found) |
177,668 | 98 | 1813x | 4 vs 1 | 312ms vs 134ms |
Total: 53,965,691 grep/read vs 1,568,260 LSP = 34x savings (~13,099,357 tokens saved)
Reproduce¶
# Run on any Go project (13 tasks, 7 skills)
go run ./experiments/token-savings --root /path/to/go/project
# Run on any Python project (11 tasks, 7 skills)
go run ./experiments/token-savings --root /path/to/python/project --language python
# Run on any TypeScript project (11 tasks, 7 skills)
go run ./experiments/token-savings --root /path/to/ts/project/src --language typescript
# Append results to the doc
go run ./experiments/token-savings --root /path/to/project --output docs/token-savings.md
Prerequisites: language server on PATH (gopls, pyright-langserver, or typescript-language-server).
Source: experiments/token-savings/main.go
src (196,523 lines, 1201 files)¶
Simple tasks
| Task | Grep/Read | LSP | Ratio | Round trips | Time |
|---|---|---|---|---|---|
Find callers of RouteCacheEntry |
9,014 | 352 | 26x | 1 vs 1 | 138ms vs 1ms |
Type signature of RouteCacheEntry |
7,896 | 310 | 25x | 1 vs 1 | 114ms vs 1ms |
| Edit safety check (break build, measure output) | 110,830 | 6,386 | 17x | 3 vs 3 | 0ms vs 0ms |
Skill workflows and advanced tasks (8 tasks)
| Task | Grep/Read | LSP | Ratio | Round trips | Time |
|---|---|---|---|---|---|
Skill: /lsp-refactor rename RouteCacheEntry |
22,660 | 7,000 | 3x | 9 vs 5 | 0ms vs 0ms |
Skill: /lsp-impact on cache.ts (43 exports) |
1,493,510 | 649,755 | 2x | 477 vs 90 | 0ms vs 0ms |
Skill: /lsp-rename RouteCacheEntry (6 files) |
345,596 | 6,635 | 52x | 8 vs 3 | 0ms vs 0ms |
Skill: /lsp-dead-code on cache.ts (43 exports, 0 dead) |
698,926 | 641,104 | 1x | 47 vs 47 | 0ms vs 0ms |
Skill: /lsp-understand Code Map of cache.ts (43 exports) |
1,958,789 | 803,213 | 2x | 612 vs 134 | 1871ms vs 55ms |
Skill: /lsp-safe-edit (speculative edit + verify + revert) |
111,343 | 6,483 | 17x | 4 vs 3 | 1778ms vs 1ms |
Skill: /lsp-verify (diagnostics + build + tests) |
111,963 | 6,066 | 18x | 3 vs 3 | 1847ms vs 0ms |
Precision: Close (158 grep vs 3 LSP refs, 155 false+) |
13,279 | 357 | 37x | 1 vs 1 | 103ms vs 93ms |
Total: 4,883,806 grep/read vs 2,127,661 LSP = 2x savings (~689,036 tokens saved)