Performance golf is the disciplined practice of iteratively refining a system’s performance through micro-optimizations, where every byte, millisecond, and round-trip counts. It matters when a system is already functional but under pressure: response times degrade under load, latency spikes appear unpredictably, and resource usage exceeds expectations. This is the phase after “it works,” where the goal is “it works *well*.”

Identify the Bottleneck

The first step is to isolate the slowest components. Use real measurements from production or a close-to-production environment.

Start with a system-wide latency profile using perf and pidstat:

perf record -F 99 -a -g -o perf.data -- sleep 60
perf script -i perf.data | stackcollapse-perf.pl | flamegraph.pl > perf-flame.svg

The resulting flame graph reveals the call stack at the time of sampling. Look for wide, deep stacks. A slow database query might show a deep stack with pg_sleep, pg_fetch, and memcpy dominating.

Use systemd-logs to correlate timestamps:

journalctl --since "2024-04-15 12:00:00" --until "2024-04-15 12:05:00" --unit=app.service | grep -E "(slow|latency|timeout)"

Look for patterns: a spike in app.service logs coinciding with a spike in nginx access logs. Then, extract a single high-latency request trace using zipkin or opentelemetry via HTTP:

GET /api/v1/orders/12345?include=items,shipments HTTP/1.1
Host: api.example.com
X-Trace-ID: 9a3b2c1d-4e5f-6a7b-8c9d-0e1f2a3b4c5d

The response includes a X-Trace-ID header and a Trace-Context in the Link header:

Link: <https://trace.example.com/v1/trace/9a3b2c1d-4e5f-6a7b-8c9d-0e1f2a3b4c5d>; rel=trace

Query the trace store to inspect the timeline:

{
  "trace_id": "9a3b2c1d-4e5f-6a7b-8c9d-0e1f2a3b4c5d",
  "spans": [
    {
      "name": "HTTP Request",
      "start": 1713240000000000,
      "duration": 124000000,
      "span_id": "1a2b3c4d",
      "parent_id": null,
      "attributes": {
        "http.method": "GET",
        "http.url": "/api/v1/orders/12345",
        "http.status_code": 200
      }
    },
    {
      "name": "Database Query",
      "start": 1713240000250000,
      "duration": 115000000,
      "span_id": "2a3b4c5d",
      "parent_id": "1a2b3c4d",
      "attributes": {
        "db.statement": "SELECT * FROM orders WHERE id = $1",
        "db.type": "PostgreSQL"
      }
    }
  ]
}

A span with duration > 100ms and start offset from the parent’s start indicates a bottleneck. The slowest span is the target.

Pinpoint the Slowest Operation

Once the bottleneck is identified, drill down into the slowest operation—typically a database query, a cache miss, or a remote service call.

For a slow database query, run EXPLAIN ANALYZE on the actual query:

EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON)
SELECT o.id, o.created_at, i.product_name, s.status
FROM orders o
JOIN order_items i ON o.id = i.order_id
JOIN shipments s ON o.id = s.order_id
WHERE o.created_at > '2024-04-15'
  AND o.status = 'shipped'
ORDER BY o.created_at DESC
LIMIT 100;

The output shows:

[
  {
    "Plan": {
      "Node Type": "Sort",
      "Startup Cost": 4567.89,
      "Total Cost": 4568.00,
      "Actual Time": 123.456,
      "Rows": 100,
      "Width": 120,
      "Plans": [
        {
          "Node Type": "Index Scan",
          "Parent Relationship": "InitPlan",
          "Startup Cost": 0.00,
          "Total Cost": 321.00,
          "Actual Time": 12.345,
          "Rows": 150,
          "Width": 120,
          "Index Name": "orders_status_created_at_idx",
          "Filter": "status = 'shipped' AND created_at > '2024-04-15'",
          "Buffers": {
            "Shared Hit": 23,
            "Shared Read": 12,
            "Local Hit": 1,
            "Local Read": 0
          }
        }
      ],
      "Buffers": {
        "Shared Hit": 112,
        "Shared Read": 23,
        "Local Hit": 5,
        "Local Read": 1
      }
    }
  }
]

The Actual Time of the sort step is 123.456ms. The Total Cost is much higher than Startup Cost, indicating that the sort is expensive. The Buffers section shows 112 shared buffer hits—so the query is mostly in memory—but 23 shared reads, meaning the data is not in memory and must be fetched from disk.

Look for Bitmap Heap Scan and Bitmap Index Scan patterns: they indicate that the planner is combining multiple index scans. If Bitmap Index Scan shows high Buffers and Actual Time, the index is large and not fully cached.

The Filter step in the Index Scan shows that the filter condition is applied after the index lookup. If Rows is much larger than Filtered Rows, the filter is expensive. Use EXPLAIN (VERBOSE) to see the actual filter expressions:

EXPLAIN (VERBOSE, COSTS, BUFFERS, FORMAT JSON)
SELECT o.id, o.created_at, i.product_name, s.status
FROM orders o
JOIN order_items i ON o.id = i.order_id
JOIN shipments s ON o.id = s.order_id
WHERE o.created_at > '2024-04-15'
  AND o.status = 'shipped'
  AND i.quantity > 1
ORDER BY o.created_at DESC
LIMIT 100;

Now, the Filter step includes i.quantity > 1 and o.status = 'shipped'. If Filter time is high and Rows is large, consider moving the i.quantity > 1 condition into the index.

Optimize the Query

Rewrite the query to reduce I/O and computation. First, ensure the correct index exists:

CREATE INDEX orders_shipped_recent_idx ON orders (status, created_at)
WHERE status = 'shipped';

This index is a *partial index* optimized for the common filter. It reduces the number of rows scanned and improves cache locality.

Next, add a composite index on order_items to support the join:

CREATE INDEX order_items_order_id_quantity_idx ON order_items (order_id, quantity)
INCLUDE (product_name);

The INCLUDE clause stores product_name in the index without needing a secondary lookup. This reduces the number of Heap Fetch operations.

Now, rewrite the query using LATERAL JOIN to push down filtering:

SELECT o.id, o.created_at, i.product_name, s.status
FROM orders o
LATERAL (
  SELECT i.product_name, i.quantity
  FROM order_items i
  WHERE i.order_id = o.id
    AND i.quantity > 1
  ORDER BY i.created_at
  LIMIT 10
) i
JOIN shipments s ON o.id = s.order_id
WHERE o.created_at > '2024-04-15'
  AND o.status = 'shipped'
ORDER BY o.created_at DESC
LIMIT 100;

This version performs a *materialized subquery* for each order. The LATERAL join ensures that the order_items scan is tailored to each order, reducing the number of rows processed.

Run EXPLAIN ANALYZE again. The Bitmap Index Scan on orders_shipped_recent_idx now shows Filtered Rows close to Rows, indicating that the filter is efficient. The LATERAL join has Actual Time and Rows that scale with the number of orders.

Reduce Round-Trips and Payloads

After optimizing the query, focus on reducing round-trips between services and shrinking payloads.

Use curl to test a service endpoint with a verbose trace:

curl -v -H "Accept: application/json" \
  -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c" \
  http://api.example.com/v1/orders/12345?include=items,shipments

Inspect the Transfer-Encoding: chunked and Content-Length in the response headers. A large JSON payload with redundant data suggests inefficiency.

Use jq to analyze the response:

curl -s http://api.example.com/v1/orders/12345?include=items,shipments | jq '{
  "total_size": (. | length),
  "order_count": (.orders | length),
  "item_count": (.items | length),
  "shipment_count": (.shipments | length),
  "unique_products": (.items | map(.product_name) | unique | length)
}'

This shows that the response includes 100 order objects, each with 5 items, and 50 shipments. But product_name appears in every item, and many items share the same product.

To reduce payload size, change the API to use id references instead of inline data:

{
  "id": 12345,
  "created_at": "2024-04-15T10:00:00Z",
  "status": "shipped",
  "items": [
    { "id": 101, "order_id": 12345, "product_id": 2001, "quantity": 3 },
    { "id": 102, "order_id": 12345, "product_id": 2002, "quantity": 1 }
  ],
  "shipments": [
    { "id": 2001, "order_id": 12345, "status": "delivered", "tracking_number": "TRK123456" }
  ],
  "metadata": {
    "product_ids": [2001, 2002],
    "shipment_ids": [2001]
  }
}

Now, the client can fetch the products and shipments in separate requests:

GET /api/v1/products?ids=2001,2002 HTTP/1.1
Host: api.example.com
Accept: application/json

The response is smaller and more cacheable. Use Vary: Accept, Content-Type in the response headers:

HTTP/1.1 200 OK
Content-Type: application/json
Vary: Accept, Content-Type
ETag: "abc123"
Cache-Control: public, max-age=3600

This allows CDN and client-side caches to serve different representations based on the Accept header.

Monitor and Iterate

Performance golf is not a one-time task. Set up a continuous monitoring loop:

  • Use prometheus to scrape http_request_duration_seconds and pg_stat_statements metrics.
  • Create a dashboard with grafana showing p95, p99, and median latency over time.
  • Set up alerts with alertmanager:
groups:
  - name: performance
    rules:
      - alert: HighLatencyOrderAPI
        expr: rate(http_request_duration_seconds{job="api", endpoint="/orders"}[5m]) > 0.5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Order API p95 latency above 500ms"
          description: "The p95 latency for /orders endpoint has exceeded 500ms for 5 minutes."

When an alert fires, check the trace and query plan. Then, apply the next optimization—perhaps a new index, a caching layer, or a batching strategy.

Each iteration reduces latency, increases throughput, and reduces resource consumption. The process never ends.

Common Pitfalls and Silent Failures

  • Index not used despite EXPLAIN: The planner uses Seq Scan even when an index exists. The cause is often missing or outdated ANALYZE statistics. Run ANALYZE orders; and ANALYZE order_items;.
  • Partial index not selected: The query filter is not WHERE but AND. The index is defined on status = 'shipped', but the query uses status = 'shipped' AND created_at > '...'. The planner fails to match the WHERE condition.
  • Lateral join not materialized: The LATERAL subquery is evaluated for every order, but the planner chooses Nested Loop instead of Hash Join. Use SET enable_hashjoin = on; and SET enable_nestloop = off;.
  • Payload size not optimized: The client fetches order, items, and shipments but does not deduplicate product_name. The JSON response contains product_name repeated 100 times. A Map<String, Product> on the client side reduces memory usage and improves rendering speed.
  • Cache miss due to inconsistent keys: The cache key for orders is "/orders/{id}", but the client sends "/orders/12345" and "/orders/12345/". The cache misses on the trailing slash. Use nginx or traefik to normalize paths:
location /orders/ {
  rewrite ^/orders/(.*)/$ /orders/$1 last;
}

These small but critical details define the difference between "good" and "excellent" performance.

Performance golf is not about big wins. It is about the relentless pursuit of the small ones.

This page was rewritten on 12 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.