Local LLMs for coding enable developers to run large language models directly on their machines, eliminating latency, enabling offline work, and reducing reliance on cloud APIs. This guide walks through setting up, tuning, and maintaining local LLMs specifically for code generation, refactoring, documentation, and interactive debugging—essential tasks where low latency and context awareness matter.

Choosing and Preparing the Model

Select a model that balances size, speed, and code understanding. Models like codellama-7b, phi3:mini, and deepseek-coder:33b are commonly available and optimized for code. Use ollama to manage model downloads and inference, a tool with a consistent CLI and well-documented API.

ollama pull codellama:7b
ollama list

# NAME             SIZE     MODIFIED
# codellama:7b     4.2 GB   2024-03-15 12:03

To verify the model is ready, run a minimal prompt:

ollama run codellama:7b
> Write a Python function to reverse a linked list.
def reverse_linked_list(head):
    prev = None
    curr = head
    while curr:
        next_temp = curr.next
        curr.next = prev
        prev = curr
        curr = next_temp
    return prev

The model should return contextually accurate code in under 1.5 seconds on a modern laptop.

Running the LLM as a Service

Start the model as a long-running service using ollama serve, which exposes a REST API on localhost:11434.

ollama serve &
# Background the process; verify it's running
ps aux | grep ollama
# Output:
# user    1234  0.3  0.9  234567  89012  ??  Ss   10:00AM   0:03.20 ollama serve

Ensure the service is accessible:

curl -X GET http://localhost:11434/api/tags
# Output:
# {
#   "models": [
#     {
#       "name": "codellama:7b",
#       "size": 4414382592,
#       "modified": "2024-03-15T12:03:24Z"
#     }
#   ]
# }

Failures here often stem from missing dependencies: libcuda.so not found (common on Linux), or cudnn mismatch. Use ldd to inspect:

ldd /path/to/ollama | grep cuda
# Expected output includes:
# libcuda.so.1 => /usr/lib/x86_64-linux-gnu/libcuda.so.1
# libcudnn.so.8 => /usr/lib/x86_64-linux-gnu/libcudnn.so.8

If libcuda.so.1 is missing, install NVIDIA drivers or ensure CUDA_HOME is set.

Configuring the Model for Coding Workflows

The default model behavior is often too general. Tune it for code-specific tasks by creating a config.json file in the model’s directory:

{
  "temperature": 0.3,
  "top_p": 0.8,
  "max_tokens": 1024,
  "stop": [
    "###",
    "```",
    "```python",
    "```javascript"
  ],
  "seed": 42,
  "num_ctx": 4096,
  "num_gpu_layers": 32,
  "num_threads": 8
}

Place this file at ~/.ollama/models/codellama:7b/config.json. This config ensures:

  • Low temperature (0.3) for deterministic, focused code generation.
  • top_p of 0.8 to balance diversity and coherence.
  • max_tokens of 1024 to allow lengthy code blocks and explanations.
  • stop tokens that end responses at code blocks or section markers.
  • A fixed seed for reproducibility across runs.
  • Increased context (num_ctx) to preserve file-wide context.
  • GPU acceleration via num_gpu_layers (32 layers on a 24GB GPU).
  • Multi-threading for CPU-heavy tasks.

Apply the config with:

ollama create codellama-coder -f config.json

The create command generates a new model variant named codellama-coder, which can be run independently.

Integrating with Editor and Build Systems

Use ollama as a backend in editor plugins or build scripts. In Neovim with nvim-lspconfig, configure the LSP client to use the local Ollama server:

require('lspconfig').ollama.setup({
  cmd = { 'ollama', 'serve' },
  settings = {
    ollama = {
      model = 'codellama-coder',
      temperature = 0.3,
      max_tokens = 1024
    }
  }
})

In a Makefile, integrate ollama calls for automated code reviews:

.PHONY: review
review:
	@echo "Running code review with local LLM..."
	@cat << EOF | ollama run codellama-coder --prompt
Review this code for performance, readability, and correctness:

```python
def process_data(data):
    result = []
    for i, item in enumerate(data):
        if item['value'] > 10:
            processed = item['value'] * 2
            if i % 2 == 0:
                result.append(processed)
    return result
EOF

When make review runs, it pipes the code into the model and returns a review, often catching subtle issues like missing edge cases or inefficient loops.

Handling Model Failures and Performance Degradation

Local LLMs fail silently. Monitor for common issues:

  • Model not loaded: The ollama serve process is running, but curl http://localhost:11434/api/generate returns {"error": "model not found"}. Verify the model name and case:
ollama list | grep codellama
# Expected:
# codellama:7b
# codellama-coder

Ensure the model name in ollama run codellama-coder matches exactly. A typo like codellama:7b instead of codellama-coder fails silently.

  • Memory overflow: The model runs slowly or crashes. Check with htop or nvidia-smi:
nvidia-smi -l 1
# Output:
# +-----------------------------------------------------------------------------+
# | GPU  Name        Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
# | Fan  Temp  Perf  Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
# |===============================+======================+======================|
# |   0  NVIDIA A5000        On  | 00000000:01:00.0 Off |                  N/A |
# |  30%   45C    P0    50W / 200W |   18120MiB / 24576MiB |     15%      Default |
# +-----------------------------------------------------------------------------+

If GPU memory exceeds 20GB, reduce num_gpu_layers or switch to codellama:7b instead of codellama:34b.

  • Incomplete or truncated responses: The model stops generating early, often cutting off after max_tokens. Add a stream flag to curl calls:
curl -X POST http://localhost:11434/api/generate \
  -d '{
    "model": "codellama-coder",
    "prompt": "Write a test suite for the reverse_linked_list function.",
    "stream": true
  }' | jq -r '.response'

Streaming ensures the client receives the full output as it’s generated, avoiding partial results.

Updating and Managing Model Versions

Model updates should be automated. Use cron to periodically pull the latest model version:

# Add to crontab: `crontab -e`
# Run every Monday at 2 AM
0 2 * * 1 ollama pull codellama:7b

To track changes, maintain a models/ directory with versioned configs:

mkdir -p models/codellama-7b-2024-03
cp config.json models/codellama-7b-2024-03/
cp models/codellama-7b-2024-03/config.json ~/.ollama/models/codellama:7b/config.json

Recreate the model with:

ollama create codellama:7b-2024-03 -f models/codellama-7b-2024-03/config.json

This enables rolling back to a known-good model configuration when a new update breaks a workflow.

Using Models with Git and Code Reviews

Integrate the LLM into pull request workflows. Create a script pr-review.sh:

#!/bin/bash
# pr-review.sh
# Usage: ./pr-review.sh <PR_TITLE> <CHANGED_FILES>

PR_TITLE="$1"
CHANGED_FILES="$2"

echo "Generating PR review for: $PR_TITLE"
cat << EOF | ollama run codellama-coder --prompt
Review this pull request:

Title: $PR_TITLE

Changes:
$(cat $CHANGED_FILES)

Provide feedback on:
- Code quality
- Performance implications
- Readability and maintainability
- Edge cases missed
- Suggested improvements
EOF

Run this on CI with:

# .github/workflows/pr-review.yml
name: LLM Code Review
on: pull_request

jobs:
  review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: |
          sudo apt-get update
          sudo apt-get install -y curl jq
          curl -L https://ollama.com/download/ollama-linux-amd64 -o ollama
          sudo install ollama /usr/local/bin
          sudo systemctl start ollama
          ./pr-review.sh "$GITHUB_HEAD_REF" <(git diff --name-only $GITHUB_BASE_REF)

This pipeline ensures every PR gets a detailed, context-aware review from the local LLM.

Troubleshooting Common Pitfalls

  • Model not found: The ollama serve process runs, but curl http://localhost:11434/api/generate returns {"error": "model not found"}. Check the model name exactly: codellama:7b vs codellama:7b-2024, and ensure the model directory exists in ~/.ollama/models/.
  • Slow startup: The LLM takes 3–5 seconds to respond on first call. This is often due to the model being loaded from disk. Use ollama run codellama-coder to warm the model:
ollama run codellama-coder --prompt "Warmup"
  • Context mismatch: The model forgets earlier parts of a file. Use num_ctx and temperature to control context retention. Lower temperature (0.2) for strict, context-aware responses.
  • Streaming cuts off: When streaming responses, the client receives incomplete JSON. Ensure the Content-Type header is application/json and the stream is properly formatted:
curl -X POST http://localhost:11434/api/generate \
  -H "Content-Type: application/json" \
  -d '{
    "model": "codellama-coder",
    "prompt": "Explain this code.",
    "stream": true
  }' | while IFS= read -r line; do
    echo "$line" | jq -r '.response // ""'
done

This loop ensures the client processes each chunk correctly.

Summary

Setting up local LLMs for coding involves selecting a model, running it as a service, tuning it with precise configuration, and integrating it into workflows. The key is consistency: the same model, same config, same prompt format, used across editors, CI, and interactive sessions. Failures are often subtle—missing GPU layers, incorrect model names, or unstreamed responses—but diagnosable with standard tools and clear error messages. With this setup, developers gain a powerful, always-available coding partner that works offline, at speed, and with deep context.

This page was rewritten on 11 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.