How We Processed 100 Million Events Per Day in Node.js

Imagine this scenario: the event processor has been stable for months. Traffic climbs after a new integration goes live, but CPU stays below 50%. Then consumer lag starts growing. Memory moves from 600 MB to 3.8 GB. Kubernetes restarts the pod, the queue redelivers everything that was in flight, and the replacement pod dies faster than the first one. By 09:20, twelve million events are waiting and every restart adds more duplicates.

This is a composite scenario, not a single incident I can point to. The pattern is real. Node.js can move a lot of data, but an asynchronous API does not make capacity unlimited. If producers can create work faster than consumers can finish it, the difference ends up somewhere. Usually in memory. Usually at the worst possible time.

This article covers the architecture we used to process roughly 100 million events per day. I'll walk through streams that keep the pipeline bounded, backpressure that makes overload visible, worker threads that isolate CPU-heavy transformations, queues that absorb bursts without hiding failure, and memory controls that keep a slow downstream service from taking the whole process with it.

First, what 100 million events per day actually means

One hundred million sounds like an enormous throughput number. Divided across 24 hours, it is about 1,157 events per second.

That average is almost useless for capacity planning.

Traffic does not arrive evenly. A partner uploads a backlog. A scheduled job wakes up on the hour. A broker reconnects and releases several minutes of buffered messages. Our design target was the peak, not the daily average: tens of thousands of events per second for short periods, with enough headroom to drain the backlog after the burst.

Event size matters just as much. An event with a 2 KB JSON payload and a cheap validation step is a different workload from a 200 KB document that needs decompression, schema migration, hashing, and four database writes. "Events per second" without payload size, processing cost, and downstream latency is a vanity metric.

The capacity model we used was deliberately simple:

required concurrency = peak events/second × processing time in seconds

20,000 events/s × 0.040 s = 800 concurrent operations

That does not mean opening 800 database transactions. It means the complete system needs 800 units of in-flight capacity at that peak. We split that capacity across broker partitions, Node.js processes, bounded stream buffers, worker threads, and database connection pools.

The word that matters is bounded.

The architecture

The pipeline separated transport, I/O-heavy enrichment, CPU-heavy transformation, and persistence. Each stage had its own concurrency limit and its own failure policy.

No stage was allowed to accept unlimited work. When the database slowed down, the batch writer stopped pulling. The worker pool filled, so the transform stopped reading. The stream paused, so the consumer stopped requesting messages. The backlog stayed in the broker, which was built to hold it.

That feedback path was the system.

Streams: move events without collecting them

The first version processed broker messages in pages. It fetched 10,000 events, parsed all of them, enriched all of them, and wrote the results. It was easy to understand and worked in tests.

It also multiplied memory use at every stage. Ten thousand raw buffers became ten thousand strings, then ten thousand JavaScript objects, then ten thousand transformed objects waiting for the database. A 20 MB batch could occupy several times that amount after parsing and copying.

The fix was a stream pipeline. Each event moved to the next stage as soon as that stage had capacity.

import { pipeline } from 'node:stream/promises';
import { Transform, Writable } from 'node:stream';

const parse = new Transform({
  objectMode: true,
  highWaterMark: 64,
  transform(message: BrokerMessage, _encoding, callback) {
    try {
      callback(null, JSON.parse(message.value.toString('utf8')));
    } catch (error) {
      callback(error as Error);
    }
  },
});

const validate = new Transform({
  objectMode: true,
  highWaterMark: 64,
  transform(event: unknown, _encoding, callback) {
    const result = eventSchema.safeParse(event);

    // BAD input belongs in quarantine, not in an infinite retry loop.
    if (!result.success) return callback(new InvalidEventError(result.error));

    callback(null, result.data);
  },
});

await pipeline(
  brokerMessageStream,
  parse,
  validate,
  enrichTransform,
  workerPoolTransform,
  batchWriter,
);

pipeline() matters because it connects failure and cleanup across the entire chain. If the writer fails, upstream stages are destroyed instead of continuing to read messages nobody can commit. Hand-wiring a dozen data, error, end, and drain handlers is possible. I've done that. It's miserable.

Object mode also changes how highWaterMark works. The value is a number of objects, not bytes. Sixty-four 2 KB events are harmless. Sixty-four 5 MB events are not. For payloads with wide size variation, we rejected oversized messages at ingress and moved large documents to object storage. The event carried a reference.

Backpressure: make "slow down" travel upstream

The most common mistake I see in Node.js pipelines is treating write() as a command. It is a negotiation.

When writable.write(chunk) returns false, the writable has reached its buffer threshold. The producer must stop and wait for drain. Ignoring the return value does not make the consumer faster. It turns process memory into an unbounded queue.

import { once } from 'node:events';

for await (const event of source) {
  const canContinue = destination.write(event);

  if (!canContinue) {
    // GOOD: the backlog stays upstream where it can be observed and replayed.
    await once(destination, 'drain');
  }
}

destination.end();

The same rule applied outside Node streams. A broker consumer had a maximum number of uncommitted messages. The worker pool had a bounded task queue. The database layer had a fixed connection pool and a batch-size limit. Every boundary needed an answer to one question: what happens when the next stage is full?

If the answer is "we keep accepting work," there is no backpressure.

For queue consumers, we paused partitions before local capacity ran out and resumed them after the low-water mark:

class CapacityGate {
  constructor(
    private readonly maxInFlight = 2_000,
    private readonly resumeAt = 1_200,
  ) {}

  private inFlight = 0;

  async acquire(consumer: PartitionConsumer): Promise<void> {
    if (this.inFlight >= this.maxInFlight) {
      consumer.pause();
      await this.waitUntil(() => this.inFlight <= this.resumeAt);
      consumer.resume();
    }

    this.inFlight++;
  }

  release(): void {
    this.inFlight--;
  }

  private async waitUntil(predicate: () => boolean): Promise<void> {
    while (!predicate()) await new Promise(resolve => setTimeout(resolve, 10));
  }
}

The polling loop is intentionally simplified for the article; in production, a semaphore that resolves waiters on release() avoids timers. The important part is the pair of thresholds. Pausing at 2,000 and resuming at 1,200 prevents rapid pause-resume oscillation.

Backpressure also needs metrics. We tracked in-flight events, stream buffer occupancy, partition pause duration, consumer lag, worker queue depth, batch flush time, and database pool wait time. CPU alone told us very little. A process at 35% CPU can still be completely stuck behind a saturated connection pool.

Worker threads: isolate CPU work, not every async function

Node.js is excellent at coordinating I/O because the event loop does not wait for network calls to finish. JSON schema migration, compression, cryptographic hashing, and complex mapping are different. They consume CPU on the JavaScript thread. While that work runs, the consumer cannot acknowledge messages, renew leases, or react to backpressure.

The first bad fix is usually Promise.all().

// BAD: 5,000 promises still execute CPU work on the same JavaScript thread.
await Promise.all(events.map(event => expensiveTransform(event)));

Promises provide concurrency for waiting. They do not provide parallel JavaScript execution.

We moved measured CPU hotspots to a fixed worker thread pool. Fixed is important. Spawning one worker per event replaces event-loop blocking with thread creation, memory pressure, and scheduler contention.

// worker.ts
import { parentPort } from 'node:worker_threads';

parentPort!.on('message', ({ id, event }) => {
  try {
    const result = migrateCompressAndHash(event);
    parentPort!.postMessage({ id, result });
  } catch (error) {
    parentPort!.postMessage({ id, error: serializeError(error) });
  }
});

// processor.ts
const pool = new WorkerPool({
  size: Math.max(1, availableParallelism() - 1),
  maxQueue: 1_000,
});

async function transform(event: DomainEvent): Promise<ProcessedEvent> {
  // Keep one core available for the event loop, GC, and broker heartbeats.
  return pool.run(event);
}

We did not send everything to workers. Database calls stayed on the main thread. Small JSON parses stayed on the main thread. Worker threads have a cost: structured cloning, message passing, and another V8 isolate with its own heap. A 300-microsecond transformation can become slower after the round trip.

Profile first. Move work that is both CPU-heavy and frequent. Measure again.

Large binary payloads got special treatment. Copying a 10 MB ArrayBuffer into a worker briefly creates another 10 MB allocation. Transferable buffers move ownership instead:

worker.postMessage(
  { id, payload: buffer.buffer },
  [buffer.buffer], // GOOD: ownership moves; the main-thread buffer is detached.
);

That reduced copying, but it introduced an ownership rule: the sender must never touch the buffer again. This is the kind of optimization that belongs behind a well-tested abstraction, not scattered through business code.

Queues: absorb bursts without hiding broken consumers

The broker gave us durable backpressure across processes. When traffic exceeded current processing capacity, messages waited on disk instead of in a Node.js heap.

But a queue only buys time. If producers publish 20,000 events per second and consumers sustain 15,000, lag grows by 5,000 every second. No retention setting changes that arithmetic.

We partitioned by an ordering key such as account or aggregate ID. Events for one aggregate stayed ordered, while different aggregates ran in parallel. Random partitioning would have increased throughput slightly and broken the domain invariant we cared about. Throughput that corrupts state is not throughput.

Delivery was at least once. A consumer could finish the database transaction and crash before committing the broker offset, so the same event could arrive again. We made processing idempotent with an event ID and a unique constraint:

BEGIN;

INSERT INTO processed_events (consumer_name, event_id, processed_at)
VALUES ('balance-projection', $1, now())
ON CONFLICT DO NOTHING;

-- Apply the projection only when the insert above created a row.
-- Duplicate delivery is normal operation, not an exceptional incident.

COMMIT;

In the real implementation, the deduplication insert and business update ran in the same transaction, and the application checked whether the insert affected a row. A separate "check then write" query has a race condition.

Retries were bounded. Network timeouts and temporary database failures used exponential backoff with jitter. Invalid schemas, missing required fields, and unsupported event versions went to a quarantine queue immediately. Retrying malformed input 47 times only delays the moment someone has to inspect it.

We also separated retry traffic from fresh traffic. Otherwise, a failing downstream dependency could fill every consumer slot with old messages and starve events that did not need that dependency.

Memory optimization: control retention before tuning V8

When a Node.js process uses too much memory, teams often start with --max-old-space-size. That raises the ceiling. It does not fix the leak or the unbounded buffer.

Our largest gains came from retaining fewer objects for less time.

Keep the in-flight set small

We tuned highWaterMark, worker queue length, broker prefetch, batch size, and database concurrency as one memory budget. Each limit looked reasonable in isolation. Multiplied together across eight pipeline stages and twelve pods, they were not.

A rough per-process budget made the trade-off visible:

2,000 in-flight events × 12 KB retained/event = 24 MB
1,000 queued worker tasks × 12 KB             = 12 MB
500 pending writes × 18 KB                    = 9 MB
worker isolates and application baseline      = measured separately

The retained size is not the JSON payload size. It includes object overhead, parsed strings, closures, tracing context, and references that keep larger graphs alive. We measured heap snapshots under representative load rather than guessing.

Stop copying payloads

Repeated object spread was convenient and expensive:

// BAD: every stage creates a new object and may retain the previous version.
const enriched = { ...event, customer, metadata };
const normalized = { ...enriched, currency: currency.toUpperCase() };

// GOOD: build the final shape once when the intermediate forms add no value.
const normalized = normalizeEvent(event, customer, metadata);

We also stopped converting buffers to strings before we knew the message type, avoided keeping raw and parsed payloads together, and logged IDs instead of whole events. Logging a failed 200 KB payload is still a 200 KB allocation, followed by JSON serialization and another queue inside the logger.

Batch writes, but cap both size and time

Larger database batches improved throughput until they created long transactions and large retained arrays. We flushed on either condition: 500 events or 25 milliseconds since the first event entered the batch.

const writer = new TimedBatchWriter({
  maxItems: 500,
  maxWaitMs: 25,
  write: events => repository.insertBatch(events),
});

The item limit bounded memory. The time limit bounded latency during quiet periods. We tuned both from measurements, not from a universal "best" batch size.

Watch garbage collection as a symptom

GC pauses increased when buffers grew because V8 had more live objects to trace. After we bounded the pipeline, GC time fell without special flags. Only then did we tune heap size per container, leaving memory for native buffers, worker isolates, the code cache, and the operating system.

Setting a 4 GB V8 heap inside a 4 GB container is an eviction plan.

Failure handling: acknowledge only what you can prove

An event was acknowledged after its durable effect committed. Not after parsing. Not after entering the worker queue. Not after the database promise was created.

That rule made shutdown behavior important. On SIGTERM, the consumer stopped fetching new messages, let the bounded in-flight set finish, flushed the current batch, committed completed offsets, and exited before the Kubernetes grace period ended.

process.on('SIGTERM', async () => {
  consumer.pause();

  await capacityGate.drain();
  await batchWriter.flush();
  await consumer.commitCompleted();
  await workerPool.close();

  process.exit(0);
});

If the deadline expired, we exited and accepted redelivery. Idempotency made that safe. Waiting forever during shutdown would only move the failure from duplicate work to a stuck deployment.

The production checklist

What I got wrong (and what I'd do differently)

My first instinct was to increase concurrency whenever lag grew. More promises. More prefetched messages. Larger batches. It worked in short benchmarks because the benchmark ended before the buffers filled.

The production system taught a less exciting lesson: throughput comes from keeping every stage busy without letting any stage own unlimited work. The limit is usually a downstream dependency, and adding concurrency beyond that limit creates waiting objects, longer GC cycles, more timeouts, and a nastier recovery after a restart.

I also reached for worker threads too early. Some transformations looked CPU-heavy in code review and disappeared in the profiler. Others looked harmless and blocked the event loop because they ran for every event. Measurement changed where we spent effort.

The biggest mistake, though, was treating consumer lag as the problem. Lag was the symptom. The useful question was always more specific: which boundary stopped draining, why, and where is the excess work waiting now?

I learned each of these the hard way. You don't have to.