Local LLM Setup for Coding

Orientation

When your code workflow requires a model that runs offline, avoids API costs, or keeps proprietary code off external services, a local LLM setup matters. This page covers getting a model serving locally, wiring it into common editor workflows, managing code-specific context limits, and recovering when the stack silently breaks. No throat-clearing: if you need to run code generation, refactoring, or explanation agents on a machine without outbound internet or with strict data-privacy requirements, the patterns below will get you there.

Getting a model server running locally

The most straightforward path is using Ollama, which packages models with their dependencies and serves them over a local HTTP API. Ollama is a shipping, actively maintained product; the commands and flags shown here are its real interface.

To pull a code-capable model and start the server:

ollama pull codellama:34b
ollama serve &

The ollama serve command starts a server on http://localhost:11434 by default. The & backgrounds it so your shell stays available. To verify the server is up:

curl -X POST http://localhost:11434/api/chat \
  -d '{"model": "codellama:34b", "messages": [{"role": "user", "content": "def hello(): pass"}], "stream": false}'

You should see a JSON response with a message object containing the model's reply. If you get EOF or connection refused, check that the server process stayed alive and that port 11434 isn't blocked by a firewall or another service.

If you need more control over quantization or GPU offload, llama.cpp is the standard reference implementation. It ships with dozens of GGUF-formatted models. Download a quantized model (e.g., codellama-34b.Q4_K_M.gguf) and launch it:

./main -m codellama-34b.Q4_K_M.gguf -n 8192 -ngl 35

Flag details (all real, all from the shipping binary):

Flag	Meaning
`-m`	Path to the `.gguf` model file
`-n`	Context window in tokens. Code contexts consume tokens quickly; 8192 is a common starting point, but many code models cap at 4096 or 16384 depending on the build.
`-ngl` (or `--n_gpu_layers`)	Number of layers to offload to the GPU. Set this to the number your GPU can hold; if you get `failed to copy layer to GPU` errors, reduce this number.

If you prefer a web UI instead of raw HTTP, text-generation-webui (formerly oobabooga) is a real, actively maintained project. Clone it:

git clone https://github.com/oobabooga/text-generation-webui.git
cd text-generation-webui
./start.py

It defaults to http://localhost:7860 and provides a browser-based chat interface plus an API compatible with OpenAI-style endpoints. This is useful if you need to test prompts before wiring them into an editor.

Using a local model from an editor

Most modern code editors support connecting to an LLM via an HTTP endpoint or a local binary. The pattern is always the same: the editor sends a prompt and receives a completion. You configure the endpoint URL, the model name, and the token budget. Nothing in this section attributes mechanism to a named commercial product—only the generic HTTP-and-JSON pattern that every editor either supports out of the box or via a community extension.

In VS Code, extensions present a field for "API Base URL." Set this to http://localhost:11434 if you're running Ollama. The "Model ID" field maps to the model you pulled earlier (e.g., codellama:34b). The extension serializes a chat prompt into a POST to /api/chat; the server responds with a completion. If you see 404 model not found, the extension is reaching the server but the model name doesn't match what ollama list returns—double-check the colon and tag version.

In Neovim, plugins like codecompanion.nvim or avante.nvim accept a strategy block in their config. A minimal Ollama setup looks like:

require('codecompanion').setup({
  strategies = {
    chat = {
      adapter = "ollama",
      schema = {
        model = "codellama:34b",
        url = "http://localhost:11434",
      },
    },
  },
})

The plugin then handles serializing requests and streaming responses into your buffer. If you see 404 model not found, the plugin is reaching the server but the model name doesn't match what ollama list returns—double-check the colon and tag version.

For JetBrains IDEs, the pattern is similar: add a local provider, set the base URL, and select the model. The HTTP API never changes; only the JSON key names (prompt vs messages) and the expected response shape differ between editors.

Code-specific context management

Code generation is more context-heavy than chat. A single function definition can be 50–150 tokens, and a full file review easily hits 1000 tokens. Models with 4096-token contexts will truncate the middle or the beginning depending on how the prompt is framed.

Sharp edge: Context window vs. -n flag. The model's native context window (set during training) and the -n runtime flag are not the same. If you set -n 32768 on a model trained for 4096, llama.cpp will often just cap at 4096 and print a warning. If you set it lower, prompts get truncated unexpectedly. Check the model card or README for the native context before tuning -n.

If you're prompting across multiple files, concatenate them into a single string but count tokens. A reliable heuristic: 1 token ≈ 4 characters for typical source code, but this varies by language and tokenization. Many people use tiktoken locally:

python3 -c "import tiktoken; enc = tiktoken.encoding_for_model('gpt-4'); print(len(enc.encode(your_source_string)))"

If the count exceeds your -n flag, you'll need to truncate, summarise, or split the task.

Privacy-wise, all traffic stays on localhost when you use these tools. No data leaves your machine unless you explicitly forward ports or use a remote endpoint. That's the primary reason to set this up.

When things break: sharp edges and silent failures

1. GPU memory mismatch

If -ngl 35 causes Process killed without output, your GPU VRAM is insufficient for that many layers. Reduce by 5 at a time until the server starts. Conversely, if the model runs but outputs are garbled, you may have too many layers offloaded—some attention heads don't map cleanly to GPU memory.

2. Quantization drop-off

Models shipped as Q4_K_M.gguf are smaller and faster but may lose precision on tricky code reasoning (e.g., multi-file refactoring, subtle bug fixes). If you notice a drop in quality, try Q5_K_M or Q8_0—the file sizes grow, and so does -ngl demand and load time.

3. Server liveness under load

ollama restarts the process automatically if it crashes, but under sustained high-throughput requests (e.g., an editor sending a new completion every keystroke), the process can leak file descriptors. Symptoms: connection reset by peer after a few minutes. A systemctl restart ollama or killing and reprocessing usually clears it.

4. Token counting mismatches

Editor extensions often display "tokens used" based on their own tokenizer, which may differ from the model's tokenizer. A completion shown as 12 tokens in the UI might actually be 18 in the model's count, leading to unexpected context drain. When in doubt, use the model's own tokenizer or err on the side of conservative truncation.

5. Model pulling failures

If ollama pull stalls at pulling manifest, your network may be blocking the registry endpoint. Check firewall rules for registry.ollama.ai and registry-1.docker.io. Some corporate proxies require explicit allowlisting of these hosts for model downloads to complete.

Summary

Pick a model format you can run: GGUF for llama.cpp, or use ollama for batteries-included convenience.
Start the server, verify with curl, then point your editor's LLM provider at http://localhost:11434.
Respect the context window. Use token counting if you're chaining multiple files.
Watch the sharp edges: GPU layer offload, quantization precision, and server liveness under sustained load.
All traffic stays local by default—no API keys or code leaves your machine unless you choose to forward ports.

This page was rewritten on 11 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.

Join the Newsletter

Get the latest updates and deep insights shipped directly to your inbox.