Load Testing & Performance Engineering: Find Bottlenecks Before Users Do

Your app works fine with 10 users. Then you launch. 10,000 users hit simultaneously. Everything breaks.

Load testing finds these limits before production. This guide covers practical tools and strategies.

Why Load Testing Matters

Users        | Response Time | Status
-------------|---------------|--------
10           | 50ms          | ✅
100          | 52ms          | ✅
1,000        | 150ms         | ⚠️
5,000        | 500ms         | ❌
10,000       | 5000ms+       | 🔥

Load testing reveals the breaking point. Then you fix it.

Tool 1: k6 (Modern, Easy)

k6 is scripted in JavaScript. Lightweight, fast feedback.

// load-test.js
import http from 'k6/http';
import { check, sleep } from 'k6';

export let options = {
  // Stage 1: Ramp up to 100 users over 30 seconds
  stages: [
    { duration: '30s', target: 100 },
    // Stage 2: Hold at 100 users for 1 minute
    { duration: '1m', target: 100 },
    // Stage 3: Ramp down over 10 seconds
    { duration: '10s', target: 0 }
  ],
  
  // Alert if error rate > 5%
  thresholds: {
    'http_req_duration': ['p(95)<500'], // 95th percentile < 500ms
    'http_req_failed': ['rate<0.05'],   // Error rate < 5%
  }
};

export default function() {
  // Test endpoint
  let response = http.get('http://localhost:3000/api/users');
  
  // Validate response
  check(response, {
    'status is 200': (r) => r.status === 200,
    'response time < 200ms': (r) => r.timings.duration < 200,
    'has data': (r) => r.json('data') !== null
  });
  
  sleep(1); // Wait 1 second between requests
}

Run test:

k6 run load-test.js

Output:

    data_received..............: 512 kB
    data_sent..................: 128 kB
    http_req_blocked...........: avg=1ms
    http_req_connecting........: avg=0ms
    http_req_duration..........: avg=125ms  p(95)=320ms  p(99)=450ms
    http_req_failed............: 0.00%  ✅
    http_req_receiving.........: avg=50ms
    http_req_sending...........: avg=5ms
    http_req_tls_handshaking...: avg=0ms
    http_req_waiting...........: avg=70ms
    http_reqs..................: 3600
    iteration_duration.........: avg=1.12s
    iterations.................: 3600
    vus........................: 0
    vus_max....................: 100

Tool 2: Locust (Python-Based, Flexible)

Locust simulates user behavior with Python classes.

# locustfile.py
from locust import HttpUser, task, between
import random

class WebsiteUser(HttpUser):
    wait_time = between(1, 3)  # Wait 1-3 seconds between requests
    
    @task(3)  # 3x more likely than other tasks
    def get_users(self):
        self.client.get("/api/users")
    
    @task(1)
    def get_user_detail(self):
        user_id = random.randint(1, 1000)
        response = self.client.get(f"/api/users/{user_id}")
    
    @task(1)
    def create_order(self):
        self.client.post(
            "/api/orders",
            json={"user_id": 42, "total": 99.99}
        )
    
    def on_start(self):
        # Login before requests
        response = self.client.post(
            "/api/login",
            json={"email": "test@example.com", "password": "password123"}
        )
        self.token = response.json()["token"]
        self.client.headers = {"Authorization": f"Bearer {self.token}"}

Run test:

locust -f locustfile.py --host=http://localhost:3000 --users=1000 --spawn-rate=50

Web UI shows live metrics:

RPS        | Failures | Avg Response | 95% Response | 99% Response
-----------|----------|--------------|--------------|---------------
150        | 0        | 125ms        | 320ms        | 450ms

Finding Bottlenecks

1. CPU Bottleneck

If CPU maxes out but memory/disk are fine:

# Check CPU usage
top -b -n 1 | grep app

# High CPU = inefficient algorithm or query
# Fix: Optimize hot paths, add caching

2. Memory Bottleneck

# Check memory leaks
free -h

# If memory grows over time:
# Fix: Find leak with profiler
node --inspect app.js
# Open chrome://inspect in Chrome DevTools

3. Database Bottleneck

-- Monitor queries
SELECT pid, usename, state, query, query_start
FROM pg_stat_activity
WHERE state = 'active';

-- Slow queries
SELECT query, calls, mean_exec_time
FROM pg_stat_statements
ORDER BY mean_exec_time DESC LIMIT 10;

-- Fix: Add indexes, optimize queries, read replicas

4. I/O Bottleneck

# Check disk I/O
iostat -x 1

# If await > 20ms, disk is slow
# Fix: Use faster storage (SSD), optimize queries

5. Network Bottleneck

# Check bandwidth
iftop

# Network saturated?
# Fix: Enable compression, CDN, caching

Load Testing Strategy

Phase 1: Baseline (Single User)

export let options = {
  vus: 1,
  duration: '1m'
};

Expected: Sub-100ms response time.

Phase 2: Gradual Ramp-Up

export let options = {
  stages: [
    { duration: '2m', target: 100 },
    { duration: '5m', target: 500 },
    { duration: '10m', target: 1000 },
    { duration: '5m', target: 0 }
  ]
};

Find where performance degrades.

Phase 3: Sustained Load

export let options = {
  stages: [
    { duration: '1m', target: 1000 },  // Ramp up
    { duration: '10m', target: 1000 }, // Hold (watch for memory leaks)
    { duration: '1m', target: 0 }      // Ramp down
  ]
};

Run 10 minutes. If memory/errors grow, you have a leak.

Phase 4: Stress Testing (Break It)

export let options = {
  stages: [
    { duration: '2m', target: 100 },
    { duration: '2m', target: 500 },
    { duration: '2m', target: 1000 },
    { duration: '2m', target: 2000 }, // Beyond expected peak
    { duration: '2m', target: 5000 }, // Stress it
    { duration: '1m', target: 0 }
  ]
};

Find breaking point. Your infrastructure should scale before breaking.

Performance Targets

Metric	Target
P95 latency	< 500ms
P99 latency	< 1000ms
Error rate	< 0.1%
CPU usage	< 80%
Memory usage	< 80%
Disk I/O	< 20ms

Real Example: Bottleneck Found

Load test reveals: Response time jumps at 500 users.

CPU:   30% → 85% (maxed)
Memory: 60% → 65% (stable)
DB:    100 conn → 100 conn (maxed)

Problem: Database connection pool exhausted.

Fix:

# Increase connection pool
resource "aws_db_instance" "postgres" {
  allocated_storage            = 100
  max_allocated_storage        = 200  # Auto-scaling
  multi_az                     = true # High availability
  
  # CloudWatch monitoring
  enable_cloudwatch_logs_exports = ["postgresql"]
}

# Add read replicas for reads
resource "aws_db_instance" "postgres_read" {
  replicate_source_db = aws_db_instance.postgres.id
}

# Connection pooling (PgBouncer)
resource "aws_elasticache_cluster" "pgbouncer" {
  engine           = "elasticache_cluster"
  parameter_group  = "pgbouncer"
  
  # Max 200 connections from app
  # PgBouncer -> 20 connections to database
}

After fix: 5,000 users, response time stays 125ms. ✅

Automation in CI/CD

# .github/workflows/performance.yml
name: Performance Test

on:
  push:
    branches: [main]

jobs:
  load-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Run k6 load test
        run: |
          k6 run load-test.js --vus 100 --duration 5m --out json=results.json
      
      - name: Check performance thresholds
        run: |
          # Fail if P95 > 500ms
          p95=$(jq '.metrics.http_req_duration.values.p95' results.json)
          if (( $(echo "$p95 > 500" | bc -l) )); then
            echo "Performance degradation detected: P95=$p95ms"
            exit 1
          fi
      
      - name: Comment on PR
        if: failure()
        uses: actions/github-script@v6
        with:
          script: |
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: '❌ Performance test failed: P95 latency > 500ms'
            })

Fail builds that degrade performance.

Checklist

Conclusion

Load testing is your safety net. It prevents embarrassing outages and expensive infrastructure over-provisioning.

Start simple: 1 user, 10 users, 100 users. Watch metrics. Fix bottlenecks.

By the time users hit your app, you'll already know it works.

SUBID