How We Processed 100 Million Events Per Day in Node.js

Imagine this scenario: the event processor has been stable for months. Traffic climbs after a new integration goes live, but CPU stays below 50%. Then consumer lag starts growing. Memory moves from 600 MB to 3.8 GB. Kubernetes restarts the pod, the queue redelivers everything that was in flight, and the replacement pod dies faster than the first one. By 09:20, twelve million events are waiting and every restart adds more duplicates.

This is a composite scenario, not a single incident I can point to. The pattern is real. Node.js can move a lot of data, but an asynchronous API does not make capacity unlimited. If producers can create work faster than consumers can finish it, the difference ends up somewhere. Usually in memory. Usually at the worst possible time.

This article covers the architecture we used to process roughly 100 million events per day. I'll walk through streams that keep the pipeline bounded, backpressure that makes overload visible, worker threads that isolate CPU-heavy transformations, queues that absorb bursts without hiding failure, and memory controls that keep a slow downstream service from taking the whole process with it.

Producers create work in bursts. Every stage downstream of them is bounded, so the stream that reaches the database is paced by what the database can absorb.

First, what 100 million events per day actually means

One hundred million sounds like an enormous throughput number. Divided across 24 hours, it is about 1,157 events per second.

That average is almost useless for capacity planning.

Traffic does not arrive evenly. A partner uploads a backlog. A scheduled job wakes up on the hour. A broker reconnects and releases several minutes of buffered messages. Our design target was the peak, not the daily average: tens of thousands of events per second for short periods, with enough headroom to drain the backlog after the burst.

Event size matters just as much. An event with a 2 KB JSON payload and a cheap validation step is a different workload from a 200 KB document that needs decompression, schema migration, hashing, and four database writes. "Events per second" without payload size, processing cost, and downstream latency is a vanity metric.

The capacity model we used was deliberately simple:

required concurrency = peak events/second × processing time in seconds

20,000 events/s × 0.040 s = 800 concurrent operations

That does not mean opening 800 database transactions. It means the complete system needs 800 units of in-flight capacity at that peak. We split that capacity across broker partitions, Node.js processes, bounded stream buffers, worker threads, and database connection pools.

The word that matters is bounded.

The architecture

The pipeline separated transport, I/O-heavy enrichment, CPU-heavy transformation, and persistence. Each stage had its own concurrency limit and its own failure policy.

No stage was allowed to accept unlimited work. When the database slowed down, the batch writer stopped pulling. The worker pool filled, so the transform stopped reading. The stream paused, so the consumer stopped requesting messages. The backlog stayed in the broker, which was built to hold it.

That feedback path was the system.

Streams: move events without collecting them

The first version processed broker messages in pages. It fetched 10,000 events, parsed all of them, enriched all of them, and wrote the results. It was easy to understand and worked in tests.

It also multiplied memory use at every stage. Ten thousand raw buffers became ten thousand strings, then ten thousand JavaScript objects, then ten thousand transformed objects waiting for the database. A 20 MB batch could occupy several times that amount after parsing and copying.

The fix was a stream pipeline. Each event moved to the next stage as soon as that stage had capacity.

import { pipeline } from 'node:stream/promises';
import { Transform, Writable } from 'node:stream';

const parse = new Transform({
  objectMode: true,
  highWaterMark: 64,
  transform(message: BrokerMessage, _encoding, callback) {
    try {
      callback(null, JSON.parse(message.value.toString('utf8')));
    } catch (error) {
      // Bad input leaves the pipeline sideways, not through the error channel.
      quarantine.send(message, error as Error);
      callback();
    }
  },
});

const validate = new Transform({
  objectMode: true,
  highWaterMark: 64,
  transform(event: DecodedEvent, _encoding, callback) {
    const result = eventSchema.safeParse(event);

    if (!result.success) {
      quarantine.send(event, new InvalidEventError(result.error));
      return callback();
    }

    callback(null, result.data);
  },
});

await pipeline(
  brokerMessageStream,
  parse,
  validate,
  enrichTransform,
  workerPoolTransform,
  batchWriter,
);

pipeline() matters because it connects failure and cleanup across the entire chain. If the writer fails, upstream stages are destroyed instead of continuing to read messages nobody can commit. Hand-wiring a dozen data, error, end, and drain handlers is possible. I've done that. It's miserable.

That same property is why a malformed event must never travel through callback(error). An error in any stage destroys the whole pipeline, the pod exits, the broker redelivers the batch, and the same unparseable message kills the replacement pod. One bad payload becomes a crash loop. A per-event failure is data, so it goes to quarantine and the stage calls callback() with nothing: the event leaves the pipeline and the healthy ones keep moving. The error channel is reserved for conditions that really do invalidate the pipeline, like a broken database connection.

Object mode also changes how highWaterMark works. The value is a number of objects, not bytes. Sixty-four 2 KB events are harmless. Sixty-four 5 MB events are not. For payloads with wide size variation, we rejected oversized messages at ingress and moved large documents to object storage. The event carried a reference.

Backpressure: make "slow down" travel upstream

The most common mistake I see in Node.js pipelines is treating write() as a command. It is a negotiation.

When writable.write(chunk) returns false, the writable has reached its buffer threshold. The producer must stop and wait for drain. Ignoring the return value does not make the consumer faster. It turns process memory into an unbounded queue.

import { once } from 'node:events';

const failed = new AbortController();
destination.once('error', error => failed.abort(error));

try {
  for await (const event of source) {
    const canContinue = destination.write(event);

    if (!canContinue) {
      // GOOD: the backlog stays upstream where it can be observed and replayed.
      // The abort signal matters: a destination that has already errored will
      // never emit 'drain', and an unguarded wait here stalls the consumer
      // forever - no progress, no crash, no alert.
      await once(destination, 'drain', { signal: failed.signal });
    }
  }

  destination.end();
} catch (error) {
  throw failed.signal.aborted ? failed.signal.reason : error;
}

The same rule applied outside Node streams. A broker consumer had a maximum number of uncommitted messages. The worker pool had a bounded task queue. The database layer had a fixed connection pool and a batch-size limit. Every boundary needed an answer to one question: what happens when the next stage is full?

If the answer is "we keep accepting work," there is no backpressure.

For queue consumers, we paused partitions before local capacity ran out and resumed them after the low-water mark:

class CapacityGate {
  private inFlight = 0;
  private paused = false;
  private readonly waiting: Array<() => void> = [];

  constructor(
    private readonly consumer: PartitionConsumer,
    private readonly maxInFlight = 250,
    private readonly resumeAt = 150,
  ) {}

  async acquire(): Promise<void> {
    if (this.inFlight >= this.maxInFlight) {
      // Pause once, not once per blocked caller.
      if (!this.paused) {
        this.paused = true;
        this.consumer.pause();
      }

      // release() hands over the slot, so no increment happens here.
      return new Promise<void>(resolve => void this.waiting.push(resolve));
    }

    this.inFlight++;
  }

  release(): void {
    this.inFlight--;

    if (this.paused && this.inFlight <= this.resumeAt) {
      this.paused = false;
      this.consumer.resume();
    }

    // One freed slot wakes exactly one waiter, so the cap cannot be overshot.
    const next = this.waiting.shift();

    if (next) {
      this.inFlight++;
      next();
    }
  }
}

Two details are easy to get wrong here. The pause and resume calls are guarded by a flag, because a per-message gate has many callers blocked at once and each of them calling resume() independently would reopen the partition while the others are still waiting. And the freed slot is handed directly to one waiter inside release() rather than letting every waiter re-check a counter, which is what keeps the limit exact instead of approximate.

The pair of thresholds is the other half. Pausing at 250 and resuming at 150 prevents rapid pause-resume oscillation.

Those numbers come from the capacity model, not from a round-number instinct. Twelve pods sharing a 20,000 events/second peak give roughly 1,700 events/second each, which is about 67 concurrent operations per pod at 40 milliseconds of processing time. A cap of 250 is roughly 3.5× that, enough headroom to ride out a short burst and keep every stage fed, and small enough that the heap never becomes a second queue. A cap of 2,000 per pod would have meant 24,000 events held in twelve Node.js heaps to serve 800 concurrent operations, which is exactly the backlog the broker is supposed to be holding.

Backpressure also needs metrics. We tracked in-flight events, stream buffer occupancy, partition pause duration, consumer lag, worker queue depth, batch flush time, and database pool wait time. CPU alone told us very little. A process at 35% CPU can still be completely stuck behind a saturated connection pool.

Worker threads: isolate CPU work, not every async function

Node.js is excellent at coordinating I/O because the event loop does not wait for network calls to finish. JSON schema migration, compression, cryptographic hashing, and complex mapping are different. They consume CPU on the JavaScript thread. While that work runs, the consumer cannot acknowledge messages, renew leases, or react to backpressure.

The first bad fix is usually Promise.all().

// BAD: 5,000 promises still execute CPU work on the same JavaScript thread.
await Promise.all(events.map(event => expensiveTransform(event)));

Promises provide concurrency for waiting. They do not provide parallel JavaScript execution.

We moved measured CPU hotspots to a fixed worker thread pool. Fixed is important. Spawning one worker per event replaces event-loop blocking with thread creation, memory pressure, and scheduler contention.

// worker.ts
import { parentPort } from 'node:worker_threads';

parentPort!.on('message', ({ id, event }) => {
  try {
    const result = migrateCompressAndHash(event);
    parentPort!.postMessage({ id, result });
  } catch (error) {
    parentPort!.postMessage({ id, error: serializeError(error) });
  }
});

// processor.ts
import { availableParallelism } from 'node:os';

const pool = new WorkerPool({
  size: Math.max(1, availableParallelism() - 1),
  // Bounded, and sized against the same in-flight budget as the capacity gate.
  maxQueue: 200,
});

async function transform(event: DomainEvent): Promise<ProcessedEvent> {
  // Keep one core available for the event loop, GC, and broker heartbeats.
  return pool.run(event);
}

We did not send everything to workers. Database calls stayed on the main thread. Small JSON parses stayed on the main thread. Worker threads have a cost: structured cloning, message passing, and another V8 isolate with its own heap. A 300-microsecond transformation can become slower after the round trip.

Profile first. Move work that is both CPU-heavy and frequent. Measure again.

Large binary payloads got special treatment. Copying a 10 MB ArrayBuffer into a worker briefly creates another 10 MB allocation. Transferable buffers move ownership instead:

function toOwnedArrayBuffer(chunk: Buffer): ArrayBuffer {
  // A Node Buffer is a view. Transferring chunk.buffer transfers whatever that
  // view sits in - which for anything under Buffer.poolSize >>> 1 is an 8 KB
  // slab shared with unrelated Buffers. Detaching it corrupts all of them.
  const ownsWholeBuffer =
    chunk.byteOffset === 0 && chunk.byteLength === chunk.buffer.byteLength;

  return ownsWholeBuffer
    ? chunk.buffer
    : chunk.buffer.slice(chunk.byteOffset, chunk.byteOffset + chunk.byteLength);
}

const payload = toOwnedArrayBuffer(chunk);

worker.postMessage(
  { id, payload },
  [payload], // GOOD: ownership moves; the main-thread buffer is detached.
);

The slice() branch is a copy, so it defeats the point. It exists as a correctness guard, not as the intended path. To stay on the zero-copy path, allocate the payload so it owns its own ArrayBuffer in the first place. Buffer.allocUnsafeSlow(size) skips the shared pool, and large reads above the pool threshold already get a dedicated allocation.

That reduced copying, but it introduced an ownership rule: the sender must never touch the buffer again. Between that rule and the pooling trap, this is the kind of optimization that belongs behind a well-tested abstraction, not scattered through business code.

Queues: absorb bursts without hiding broken consumers

The broker gave us durable backpressure across processes. When traffic exceeded current processing capacity, messages waited on disk instead of in a Node.js heap.

But a queue only buys time. If producers publish 20,000 events per second and consumers sustain 15,000, lag grows by 5,000 every second. No retention setting changes that arithmetic.

We partitioned by an ordering key such as account or aggregate ID. Events for one aggregate stayed ordered, while different aggregates ran in parallel. Random partitioning would have increased throughput slightly and broken the domain invariant we cared about. Throughput that corrupts state is not throughput.

Delivery was at least once. A consumer could finish the database transaction and crash before committing the broker offset, so the same event could arrive again. We made processing idempotent with an event ID and a unique constraint:

BEGIN;

INSERT INTO processed_events (consumer_name, event_id, processed_at)
VALUES ('balance-projection', $1, now())
ON CONFLICT DO NOTHING;

-- Apply the projection only when the insert above created a row.
-- Duplicate delivery is normal operation, not an exceptional incident.

COMMIT;

In the real implementation, the deduplication insert and business update ran in the same transaction, and the application checked whether the insert affected a row. A separate "check then write" query has a race condition.

Retries were bounded. Network timeouts and temporary database failures used exponential backoff with jitter. Invalid schemas, missing required fields, and unsupported event versions went to a quarantine queue immediately. Retrying malformed input 47 times only delays the moment someone has to inspect it.

We also separated retry traffic from fresh traffic. Otherwise, a failing downstream dependency could fill every consumer slot with old messages and starve events that did not need that dependency.

Memory optimization: control retention before tuning V8

When a Node.js process uses too much memory, teams often start with --max-old-space-size. That raises the ceiling. It does not fix the leak or the unbounded buffer.

Our largest gains came from retaining fewer objects for less time.

Keep the in-flight set small

We tuned highWaterMark, worker queue length, broker prefetch, batch size, and database concurrency as one memory budget. Our first set of limits looked reasonable in isolation. Multiplied together across eight pipeline stages and twelve pods, they were not.

A rough per-process budget made the trade-off visible:

250 in-flight events × 12 KB retained/event = 3.0 MB
200 queued worker tasks × 12 KB             = 2.4 MB
500 pending writes × 18 KB                  = 9.0 MB
worker isolates and application baseline    = measured separately

The retained size is not the JSON payload size. It includes object overhead, parsed strings, closures, tracing context, and references that keep larger graphs alive. We measured heap snapshots under representative load rather than guessing.

Stop copying payloads

Repeated object spread was convenient and expensive:

// BAD: every stage creates a new object and may retain the previous version.
const enriched = { ...event, customer, metadata };
const normalized = { ...enriched, currency: currency.toUpperCase() };

// GOOD: build the final shape once when the intermediate forms add no value.
const normalized = normalizeEvent(event, customer, metadata);

We also stopped converting buffers to strings before we knew the message type, avoided keeping raw and parsed payloads together, and logged IDs instead of whole events. Logging a failed 200 KB payload is still a 200 KB allocation, followed by JSON serialization and another queue inside the logger.

Batch writes, but cap both size and time

Larger database batches improved throughput until they created long transactions and large retained arrays. We flushed on either condition: 500 events or 25 milliseconds since the first event entered the batch.

const writer = new TimedBatchWriter({
  maxItems: 500,
  maxWaitMs: 25,
  write: events => repository.insertBatch(events),
});

The item limit bounded memory. The time limit bounded latency during quiet periods. We tuned both from measurements, not from a universal "best" batch size.

Watch garbage collection as a symptom

GC pauses increased when buffers grew because V8 had more live objects to trace. After we bounded the pipeline, GC time fell without special flags. Only then did we tune heap size per container, leaving memory for native buffers, worker isolates, the code cache, and the operating system.

Setting a 4 GB V8 heap inside a 4 GB container is an eviction plan.

Failure handling: acknowledge only what you can prove

An event was acknowledged after its durable effect committed. Not after parsing. Not after entering the worker queue. Not after the database promise was created.

That rule made shutdown behavior important. On SIGTERM, the consumer stopped fetching new messages, let the bounded in-flight set finish, flushed the current batch, committed completed offsets, and exited before the Kubernetes grace period ended.

// Below terminationGracePeriodSeconds, so we exit before SIGKILL arrives.
const SHUTDOWN_DEADLINE_MS = 25_000;

process.on('SIGTERM', async () => {
  consumer.pause();

  const drained = (async () => {
    await capacityGate.drain();
    await batchWriter.flush();
    await consumer.commitCompleted();
    await workerPool.close();
  })();

  const deadline = new Promise<'timeout'>(resolve => {
    setTimeout(() => resolve('timeout'), SHUTDOWN_DEADLINE_MS).unref();
  });

  const outcome = await Promise.race([
    drained.then(() => 'drained' as const).catch(() => 'failed' as const),
    deadline,
  ]);

  // Exit either way. Unfinished work is redelivered, and that is the safe case.
  process.exit(outcome === 'drained' ? 0 : 1);
});

The deadline is the point of the whole handler. capacityGate.drain() waits on downstream work, and downstream is exactly what tends to be broken during an incident. A hung database call would otherwise hold the process until Kubernetes sends SIGKILL in the middle of a flush. Exiting on the timer and accepting redelivery is safe because processing is idempotent. Waiting forever during shutdown would only move the failure from duplicate work to a stuck deployment.

The production checklist

The order I'd build it in

If I started this again on a Monday, the order would matter more than any individual technique, because each step makes the next one measurable.

Metrics on every boundary come first: in-flight count, stream buffer occupancy, worker queue depth, database pool wait time, consumer lag. Before any optimization, including the ones in this article. A pipeline you cannot see is one you tune by superstition, and CPU utilization tells you almost nothing about a process blocked on a saturated connection pool.

Then one bound, the consumer's in-flight cap, and nothing else yet. Most overload problems disappear at this step. The ones that survive it now point somewhere specific instead of showing up as "memory grows."

Idempotency comes next, before scaling out rather than after the first duplicate incident. Adding it later means reconciling data written by a system that assumed it already had it. That is a migration, not a code change.

Batching waits until write latency is measurably the dominant cost, because below that threshold it adds a flush timer to reason about and nothing else. Worker threads wait until the profiler names the function. Not code review: code review is confidently wrong in both directions about which transformations are expensive. Some hotspots vanish under a profiler, and some harmless-looking mappers block the event loop because they run for every single event.

The step I skipped every time was the first one. Whenever lag grew, my instinct was to raise a number. More promises, more prefetch, bigger batches. It always worked in a short benchmark, because the benchmark ended before the buffers filled. In production it moved the backlog out of a broker built to hold it and into a heap that was not.

That instinct has a tell worth naming: treating lag as the problem. Lag is a symptom, and a slow one. The question that actually resolves an incident is narrower. Which boundary stopped draining, why, and where is the excess work sitting right now? Everything above exists to make that question answerable from the metrics you already have.