They’re too wasteful.

If you give a task to your coding agent where the requirements may not be spelled out to a tee, it will spend a lot of time and tokens figuring out what it needs to do. I suppose that’s a good thing, since you need to measure twice, cut once. But for every “cut” the AI makes, it takes all the measurements again.

They don’t hold context of the codebase

You know how when you keep working on a codebase, you start to remember which functionality is implemented in which files? Well, coding agents do not. To figure out where to make the right changes, they search your codebase from scratch in every new conversation. If your respository is not named or structured properly, this leads to a lot more wastage.

I’ll concede that humans also gather information from codebases in a similar way, but we usually remember it for later. Our memory doesn’t reset when we move on to a separate task. Imagine you just explained a function thoroughly to your co-worker, but they forget everything when you look away. When we find out happens inside a file or a function we tuck it into a neat little corner in our head. So that next time we come across this file, we at least have a vague idea of what happens within it.

Have you seen how an agent searches for context in your codebase?

They only have surface-level understanding.

An agent will run tools like ls, cat and grep to structurally navigate and find keywords within your code that could match what you’ve described. But since this search is purely text-based, it is at the mercy of the codebase being structured and named properly. And lets be real, most programmers are not good at naming things:

There are only two hard things in Computer Science:

Cache invalidation, naming things, and off-by-one errors.

Let’s say someone has placed a file named middleware.js in the /src/utils/ directory. From the name alone, it’s possible that this file could be handling authorizationn for the routes. So the agent spends time and tokens reading it, only to find out it implements route-level caching turns raw HTTP requests into objects for your web framework, but has no logic related to auth.

You know how people say “Time is money, don’t waste it.”? Well what if you wasted time AND money! That’s what just happened here. The agent spent a lot of time, used a lot of tokens trying to infer semantic information, but you’re not gonna get back all that time and tokens spent.

They’re susceptible to context rot.

When the agents perform their text-based searching, they dump all the text generated by that process into your context window. You’ll see a lot of:

[tool_use] grep: authorization | authentication | middleware | routes

Great! I got results in middleware.js, let me read it to see if it implements logic to handle authorization.

[tool_use] read_file: '/src/middleware.js'

1
...
2
100s of lines of middleware.js
3
...

Hmm… I didn’t find the relevant functions in the middleware.js file, let me refine my search criteria.

And on and on it goes…

If you don’t have your files and folders named perfectly (as is common), this will happen very frequently and all of it will be dumped into your context window. So any task that you give your agent will come burdened with this excess baggage. But thats not really an issue, is it? LLMs now support context windows as large as a million tokens, so shouldn’t you be able to just keep adding to your context without any issues?

Unfortunately in the real world, LLMs’ performance degrades as the input length increases. The linked paper shows noticeable reduction in accuracy once the context length crosses 32k tokens:

But wait a second! this paper was published in August, 2024. Even if we assume that the models have gotten 2x better at handling long context lengths — which they haven’t — that only brings us to upto a skimpy 64k tokens. You will easily blow past that threshold if you’ve got a codebase that’s more than just a weekend project.

[SIDEBAR]: One decent solution to this context rot issue is sub-agents. They quietly do their work in the background and present you with a result, but I digress.

Summarise and remember

What if the agent had semantic understanding of all functions? Instead of moving through your codebase like a troll under a bridge, it could come up with a rough meaning of what it wants and look for that within the search space. So you generate a summary of the functions using an LLM and store that in a database. Anytime the agent wants to search functions by their semantics, it can search from within the summaries.

Riddle me this — What happens if your agent is searching for words like authentication or authorization but your summary doesn’t contain any of them?

1
cat = [1.5, -0.4, 7.2, 19.6, 3.1, ..., 20.2]
2
kitty = [1.5, -0.4, 7.2, 19.5, 3.2, ..., 20.8]

1
export async function processFunctionParseQueue() {
2
  while (functionParseQueue.length > 0) {
3
    const item = functionParseQueue[0];
4
    if (!item) break;
5

6
    try {
7
      await parseFunctionDeclaration(item.node);
8
      functionParseQueue.shift();
9

10
      if (functionParseQueue.length > 0) {
11
        await new Promise((resolve) =>
12
          setTimeout(resolve, API_RATE_LIMIT_DELAY),
13
        );
14
      }
15
    } catch (error) {
16
      // handle error
17
    }
18
  }
19
}

1
This function processes a queue of function declarations asynchronously with rate limiting. Here's what it does:
2

3
# Main Process:
4

5
* Continuously processes items from `functionParseQueue` until it's empty
6
* For each item, it calls `parseFunctionDeclaration()` on the item's node property
7
* Removes successfully processed items from the front of the queue
8

9
# Rate Limiting:
10

11
* After processing each item, it checks if more items remain in the queue
12
* If so, it waits for `API_RATE_LIMIT_DELAY` milliseconds before processing the next item
13
* This prevents overwhelming an API or service with too many rapid requests
14

15
# Error Handling:
16

17
* Wraps the parsing operation in a try-catch block (though the error handling is not implemented in the shown code)
18

19
The function essentially provides a controlled, sequential way to parse function declarations while respecting rate
20
limits, likely for an API that analyzes or processes code.

1
SELECT
2
  name,
3
  1 - (embedding <=> $1::vector) as similarity_score
4
FROM functions
5
WHERE embedding IS NOT NULL
6
ORDER BY embedding <=> $1::vector

Dependency graphs

Functions rarely stand on their own, you almost always need context about how that function is connected to other parts of your code. Especially when you want to investigate memory leaks, you need to investigate all possible branches within that path that allocate memory.

So if a function named set_cache_key has a memory leak — which could depend on what the input is — I really need to know which upstream functions could be involved. To err on the side of caution, I want all the call stacks that end up calling this function. Good luck doing that manually in a codebase like this (1100+ function calls across 700+ functions):

Figure 2: Call graphs in the official Svelte repo (github.com/sveltejs/svelte)

Similar to how you attempt a semantic search for functions, if you tell an agent to get all call graphs that eventually call push_stack it will spend a lot of tokens but the result will not be exhaustive.

1
function first() {
2
    second();
3
}
4

5
function second() {
6
    third();
7
}
8

9
function third() {
10
    // ...
11
}

1
|----------|      |----------|      |----------|
2
|   First  |----->|  Second  |----->|  Third   |
3
|----------|      |----------|      |----------|

1
match path = (start:Function)-[:CALLS*..]->(end:Function { name: "get_stack", path: "/packages/svelte/src/internal/client/dev/tracing.js" })
2
return path;

How much did I save, really?

Alright then. I’ve been waxing lyrical about being so much more efficient than just raw LLMs, but how much more efficient is it?

A lot of problems have already been solved in computer science. Using LLMs for everything is like replacing a forklift with a swiss army knife — a lot more general purpose and appealing — but a lot less powerful. My aim for this project was to reduce the amount of tokens being used for tasks that can be trivially soved by applying just a little bit of computer science knowhow.

Linked below are a couple of rough benchmarks I performed by comparing the methods I’ve just described using Zed editor’s coding agent. I added these optimisations to an MCP server and exposed that to the agent. All the benchmarks compare the amount of tokens used with and without the MCP.

All of these tests were run on github.com/sveltejs/svelte repository using Claude Sonnet 4 with thinking enabled in Zed editor’s coding agent.

1
# System prompt
2

3
You are a distinguished software engineer analyzing a codebase. Your task is to identify functions that are semantically related to a given description, even if they don't directly perform that action.
4

5
This includes functions that:
6
- Directly perform the described action
7
- Handle, validate, or process the described action
8
- Warn about or prevent improper usage of the described action
9
- Are part of the workflow or lifecycle of the described action
10

11
Be inclusive rather than restrictive in your evaluation.
12

13
# User prompt
14

15
Description: "Functions that manage state, state transitions."
16

17
Functions to evaluate:
18

19
  Function: update_reaction
20
  Path: /packages/svelte/src/internal/client/runtime.js:253-356
21
  Summary: Here's a summary of the update_reaction function:
22

23
  This function manages the update cycle of a reaction in a reactive system. It:
24
    1. Saves the previous state/context variables
25
    2. Updates dependency tracking flags and states
26
    3. Executes the reaction's function
27
    4. Handles dependency management by:
28
      * Removing old reaction references
29
      * Adding new dependencies
30
      * Updating dependency arrays
31
    5. Handles untracked writes and possible effect self-invalidation
32
    6. Manages error handling
33
    7. Restores the previous state/context when complete
34

35
The function appears to be part of a reactive programming system, possibly handling side effects and dependency tracking in a component-based architecture.
36

37
...
38

39
For each function, determine if its summary is related to the description. Return true if related, false otherwise.

References

1. RULER: What’s the Real Context Size of Your Long-Context Language Models?
2. SemTools: Are coding agents all you need?