Production mistakes every backend developer should understand

Table of Contents

Introduction

When I first started learning full-stack development, my only goal was to make things work.
If the feature worked and the API returned a response, I considered the job done.
But as I built more projects and started looking deeper into how real systems work, my focus slowly shifted from just making things work to making them work well.
That’s when I started noticing something important: Most production issues don’t come from complex bugs.
They come from small decisions we don’t think about early on. Things like missing timeouts, blocking operations, or poor logging don’t usually break an app immediately.
But at scale, they quietly hurt performance, reliability, and debuggability. This post is a collection of such mistakes I learned about while studying backend systems, reading engineering blogs, and experimenting on my own.
They’re simple things, but they matter a lot once traffic grows.

1. Missing Timeouts on API Calls

One of the most common mistakes is not setting timeouts on API Calls. Whenever HTTP request is made we don’t know due to any XYZ reasons request is taking a lot of time to respond due to which it affects server more than client & if request like these keep piling up it can lead to server crash. We should also add retries if:

• The request is idempotent (‘GET’, ‘HEAD’, ‘OPTIONS’, ‘PUT’, ‘DELETE’)
• The error is retryable (network error, timeout, 5xx status codes, 429 status code)
• attempt < retries

But this create another problem called Thundering Herd, where if let’s say a service went down & 1,000 clients all retry at exactly 1.0 seconds, they’ll just crash the server again.
To fix this we can add Jitter to our retry intervals, which adds a bit of randomness to your retry intervals, you spread the load and give the service room to breathe.
Also its good idea to add exponential backoff so that each retry waits longer than the last one, its done to overwhelm the server with same request periodically.

Below is typescript function that is wrapper on axios for timeouts, retries with exponential backoff and jitter.

  import axios from 'axios';

  async function axiosWithRetry({
    timeout = 5000,
    retries = 3,
    retryDelay = 1000,
    idempotent = false,
    onRetry,
    ...axiosConfig
  }) {
    const method = (axiosConfig.method || 'GET').toUpperCase();
    const isSafeMethod = ['GET', 'HEAD', 'OPTIONS', 'PUT', 'DELETE'].includes(method);
    const shouldRetryMethod = idempotent || isSafeMethod;

    for (let attempt = 0; attempt <= retries; attempt++) {
      try {
        return await axios({ ...axiosConfig, timeout });
      } catch (error) {
        const status = error.response?.status;
        const isTimeout = error.code === 'ECONNABORTED' || error.message.includes('timeout');
        const isNetworkError = !error.response && !!error.request; 
        const isRetryableStatus = status >= 500 || status === 429;

        const canRetry = shouldRetryMethod && (isTimeout || isNetworkError || isRetryableStatus);
        
        if (!canRetry || attempt === retries) {
          throw error;
        }

        const backoff = retryDelay * Math.pow(2, attempt);
        const jitter = backoff * 0.2 * Math.random();
        const delay = backoff + jitter;

        onRetry?.(attempt + 1, retries, delay, error);
        await new Promise(res => setTimeout(res, delay));
      }
    }
  }

  // Usage:
  try {
    const response = await axiosWithRetry({
      url: 'https://api.example.com/users',
      method: 'POST',
      data: { name: 'Aayushmaan Soni', email: 'aayushmaansoni.dev@gmail.com' },
      headers: {
        'Authorization': 'Bearer token123',
        'Idempotency-Key': 'unique-request-uuid-12345',
      },
      timeout: 5000,
      retries: 3,
      retryDelay: 1000,
      idempotent: true, 
      onRetry: (attempt, maxRetries, delay, error) => {
        logger.warn(`Attempt ${attempt} failed. Retrying in ${delay}ms...`);
      },
    });

    logger.info('Success:', response.data);
  } catch (error) {
    logger.error('Final failure after all retries:', error.message);
  }

2. Console statements for Logging

Most people use console statements for logging and don’t make effort to remove them in production but small things like can have silent impact on performance as these statements grow as codebase grows and the issue is they are synchronous in nature and block the event loop until they are flushed to the terminal or log file. Now in case of high traffic any synchronous blocking operation can lead to increased latency and reduced throughput.
To avoid this we can use logging libraries like Winston, Pino, Bunyan etc which are optimized for performace & is asynchronous in nature. They allow us to set log levels, output logs to files or external systems, and format logs in structured ways (like JSON) which makes it easier to analyze later.

3. Don’t run everything on the main thread

Node.js is a single threaded language which can lead to very bad performace even in medium traffic if we run everything on main thread. Things like heavy computations like image processing, encryption, data parsing etc should be offloaded to worker threads or separate microservices. This ensures that the main event loop remains unblocked and can continue handling incoming requests efficiently.

4. Database pool size

Most of dev just set max_pool_size to some amount and think thats it and think during high traffic we will just increase the pool size but these are temprory fixes not efficient and these solution don’t scale.
To make it more efficient & scale:

• Set per query timeouts to avoid long running queries hogging connections.
• Monitor available connection not total connections to understand real usage.
• Use connection pooling libraries that support dynamic resizing based on load.
• Implement circuit breakers to stop sending requests to the DB when it’s overloaded to let it recover.
• Watch out for zombie connections and clean them up periodically.

5. ReDoS: Regex Denial of Service

A poorly written Regular Expression can be an accidental Denial of Service (ReDoS) attack. Patterns like (a+)+b can cause catastrophic backtracking when matched against a long string like aaaaa...ac.
A single malicious string can pin your CPU at 100% forever.
To fix it:

• Validate input lengths before applying regex.
• Use timeouts for regex execution but mostly avoid regex patterns known for backtracking issues.
• Avoid nested quantifiers (plus signs inside plus signs).

Conclusion

These are few mistakes that are commonly made by developers i observed through my learning journey, fixing these issues early on can save a lot of headaches as your application scales. Always think about performance, reliability, and debuggability from the start, not as an afterthought.