Tokensparsamkeit for coding assistants

One of the first steps toward Tokensparsamkeit is to compress tokens sent to the underlying LLM while keeping the same data. But what are tokens? It’s a gross oversimplification, but for the sake of explanation, let’s consider a word is a token. Read this deep dive if you want more details.

If we consider tokens to be words, we could remove articles and similar words from the payload to decrease the number of tokens. "Find the distance between the Earth and the moon" becomes "Find distance between Earth and moon". For all intents and purposes, the data received is the same, with fewer words.

The trick is to set a proxy between the client and the LLM backend. I’m using rtk myself:

The tool works across file commands, git, gh, test runners, build/lint commands, aws, docker, kubectl, etc. Note that it’s not a magical recipe, as rtk itself mentions:

The second step toward Tokensparsamkeit is to avoid stuffing the context with irrelevant data.

Most people who start using coding assistants assume the context only consists of the system prompt and user prompts. There actually is a lot more. Anthropic’s Effective context engineering for AI agents article mentions:

Claude Code introduced the option to compact (or clear?) the context before each interaction. It explicitly asked with each interaction whether to do it. I liked it, but they removed it a week or so later. Perhaps too many people didn’t understand what it entailed? In any case, make good use of the /compact command that most assistants provide: it will reduce the conversation history to reduce its token usage, while keeping the relevant bits and pieces.

Also, notice that tools and MCP servers use tokens. The more you configure, the more tokens are used. Some MCP servers are so easy to set up that it’s tempting to stuff your assistant with them. Don’t. Or enable them either on a case-by-case basis or at the project level. Why enable the Vaadin MCP on a Rust project?

The same goes for tools, although I don’t think many use them a lot in comparison to MCP servers.

Token usage is very important for cloud-based models, because usage is billed per million tokens. We care about it less if we use a local model. More tokens means more memory and slower inference, but the cost is now decoupled. There are several ways to use local models, including AI gateways. In the scope of this article, I’ll keep it simple.

I want to keep Claude Code as the client, because it’s really good. At the same time, I want to use my own hardware with a local model. Hardware cost is upfront, but with zero recurring costs, except for power consumption.

If you want to just do it, How to Run Local LLMs with Claude Code is where I found the solution. Continue reading the section if you want to read about the issues I faced.

I tried initially to run Qwen3 32B via Ollama in Docker. Docker containers cannot access Apple’s Metal GPU framework, so the model ran entirely on CPU. It loaded successfully but crashed during inference with a 500 error; CPU-only inference on a 32B model is simply too slow to be usable.

I have been using Ollama as the default, because others did. Then I stumbled upon Friends Don’t Let Friends Use Ollama. I switched from Ollama to llama-cpp, which enabled low-level configuration.

The biggest hurdle was the context window size. Claude Code sends lots of tokens to the backend. On the OpenTelemetry tracing demo, it’s around 35k on each request.

I started with Qwen3 models. The default token size wasn’t big enough. When a model received more tokens than its maximum, llama server immediately rejects the request. I tried to increase the limit with the --ctx-size option, to no avail. Qwen3 models are trained with 32,768 tokens. It’s a hard limit baked into the GGUF file metadata. Llama server abides by it.

Llama server is meant to serve multiple requests simultaneously. It turns out that the count of available tokens is shared equally across all possible requests. If the number of max tokens is T and the server can handle x requests in parallel, each request only has T/x tokens available. For this reason, I set the parallelism with --parallel 1.

Despite all of the above, it still didn’t work.

I was using a dense model, which is what we use regularly. Dense models load all in memory at once. The alternative is to use a Mixture of Experts model.

In layman’s terms, a MoE segments its weights/parameters into separate specialized submodels called experts. A routing layer activates only the necessary experts depending on the request. Compared to regular dense models, instead of computing across the entire model of size T, only a small subset of experts is activated per request. The combined size of activated experts t is much smaller than T, even though the sum of all experts together is larger than T.

The Qwen3.5-35B-A3B model is a MoE that works perfectly on my machine. Unlike the dense Qwen3 models capped at 32k, it supports up to 262k tokens, well above what Claude Code uses.

We still lack a couple of elements to reach the goal of Claude Code using a local backend.

To better interact with Claude Code, the model should return structured content. That’s what the --jinja flag is for. For better performance, you should also use Flash Attention. It’s an optimized algorithm for computing the attention mechanism in Transformer models. It’s faster, more memory-efficient, and more scalable than standard attention. Activate it via --flash-attn on. The last configuration parameter is to offload as many layers as possible to the GPU with --n-gpu-layers 99.

The final server command line is:

On the Claude Code side, we need to set several environment variables:

At this point, you can use Claude Code, which will query your local model. Here’s a sample server output for a query, for information.

While the underlying model is important, most people undervalue the client. I used both Claude Code and Copilot CLI with the same underlying model, Claude Sonnet 4.6. I found Claude Code superior by far across several sessions.

Keeping Claude Code while hosting the model locally is a great cost-savvy alternative. You only need to pay for the hardware once. Granted, it’s slower, but it’s a business model worth considering if you have a limited budget, like at home. If you have well-designed working autonomous agents, you can run them during the night anyway.