Building Idempotent Message Handlers in Symfony Messenger

Imagine this scenario: a Symfony Messenger worker receives PaymentCaptured, creates a merchant payout, and commits the PostgreSQL transaction. The pod disappears before Messenger deletes the message from Amazon SQS. Thirty seconds later, another pod receives the same event. The first payout is correct. The second one is also accepted. Reconciliation catches the duplicates the following morning.

This is a composite scenario, but the failure mode is normal. SQS Standard queues provide at-least-once delivery. Kubernetes replaces pods. Networks lose responses. A worker can finish the business operation and still fail to acknowledge the message. The handler has to make that sequence safe.

This article covers how to make that handler safe. I'll walk through what exactly-once processing actually means, where Redis locks help and where they do not, how a PostgreSQL unique constraint closes the concurrency race, how to build a deduplication table, and how the Outbox Pattern handles the producer side of the same delivery gap.

Duplicate messages across Kubernetes workers, with Redis coordination and PostgreSQL uniqueness enforcing one financial effect.

Exactly once is a system property

A message handler does not control the complete delivery lifecycle.

With SQS, receiving a message makes it invisible for a period. It does not remove it. Messenger deletes it only after the handler returns successfully. That leaves a failure window:

There is no atomic transaction that includes both PostgreSQL COMMIT and SQS DeleteMessage. If the database commits first, a crash can cause duplicate processing. If the message is deleted first, a crash can lose the business operation completely.

This is why "configure retries" is not a solution. Retries are one of the reasons handlers must be idempotent.

To be precise, this article does not promise universal exactly-once delivery. It builds effectively-once business effects from three properties:

1. Every logical event has a stable identity.
2. The business effect and its deduplication record commit in one database transaction.
3. External side effects either accept the same idempotency key or run through a recoverable workflow.

The queue may deliver a message five times. The business result still appears once.

That statement has a scope. It applies to the database-owned effect covered by the transaction and its unique constraints. It does not make the queue, an email provider, an HTTP endpoint, or an external payment rail exactly-once. Those systems need their own idempotency key, reconciliation process, or durable workflow.

Start with a stable message identity

An idempotency key identifies the logical operation, not one attempt to deliver it.

<?php

namespace App\Message;

final readonly class PaymentCaptured
{
    public function __construct(
        public string $eventId,
        public string $paymentId,
        public string $merchantId,
        public int $amountInMinorUnits,
        public string $currency,
    ) {
    }
}

eventId must remain unchanged when the producer retries, when the outbox relay republishes, and when SQS redelivers. Generating it inside the handler defeats the entire design.

// BAD: every attempt becomes a new logical event.
$eventId = Uuid::v7()->toRfc4122();

// GOOD: the producer assigned this ID when the event was created.
$eventId = $message->eventId;

Sometimes the domain already has the right key. paymentId may be enough if a payment can be captured only once. If the same payment can produce several capture events, use the event ID or a composite key such as paymentId + captureSequence.

The choice is a business rule. Infrastructure cannot infer it from a PHP class name.

Prefer naturally idempotent state changes

The cheapest deduplication table is the one you do not need.

Setting an order status to an absolute value is often naturally idempotent:

UPDATE payment
SET status = 'captured', captured_at = :captured_at
WHERE id = :payment_id
  AND status = 'authorised';

Running this statement twice does not apply the transition twice. The second update affects no rows.

An increment is different:

-- BAD: redelivery credits the merchant again.
UPDATE merchant_balance
SET available_amount = available_amount + :amount
WHERE merchant_id = :merchant_id;

Financial systems contain operations that cannot be reduced to a harmless assignment. A payout, ledger posting, inventory decrement, or notification may need an explicit idempotency boundary.

That boundary should be as close to the invariant as possible.

Redis locks reduce overlap, not duplicates

When twelve Messenger pods consume the same queue, a shared Redis lock can stop them from doing the same expensive work concurrently. A local semaphore cannot. Each pod has its own process and memory.

Symfony 8.1 provides message deduplication through DeduplicateStamp and the Lock component:

composer require symfony/lock

# config/packages/lock.yaml
framework:
    lock: '%env(REDIS_LOCK_DSN)%'

REDIS_LOCK_DSN=redis://redis.internal:6379/2

On EKS, that endpoint would normally point to a shared ElastiCache deployment, not a Redis container running beside each worker.

The producer attaches a business-level key:

<?php

use Symfony\Component\Messenger\MessageBusInterface;
use Symfony\Component\Messenger\Stamp\DeduplicateStamp;

$bus->dispatch($message, [
    new DeduplicateStamp(
        key: 'payment-captured.'.$message->eventId,
        ttl: 900,
    ),
]);

The middleware suppresses another dispatch with the same key while the lock exists. The explicit ttl: 900 above gives this example a 15-minute lease. Without that argument, DeduplicateStamp uses a 300-second default. Choose the lease from measured queue delay and handler duration; it is not a retention policy.

By default, the middleware keeps the lock while the message is queued and handled, then releases it after successful handling. When handling fails and Messenger will retry, Symfony's failure listener keeps the lock for the remaining retries. It releases the lock when the message is finally abandoned or moved to a failure transport.

Useful? Yes. A correctness guarantee? No.

DeduplicateStamp does not reacquire the lock when SQS redelivers an already queued envelope. The lock is also released after the handler succeeds, before SQS acknowledgement is a permanent fact. If the handler commits and DeleteMessage fails, a later delivery can execute the handler again.

There are more failure modes:

• The default is a lease, not permanent history. If queue delay plus processing exceeds its TTL, another dispatch can pass.
• A Redis failover can lose a recently acquired lock before asynchronous replication catches up.
• A producer that writes directly to SQS bypasses Messenger's dispatch middleware.
• A long handler needs a lock refresh strategy, but DeduplicateStamp does not refresh its TTL.
• Once a message is moved to the failure transport, its deduplication lock is released. A later manual replay is therefore not deduplicated against a new dispatch with the same key. The durable database invariant must still handle that replay.
• A producer allowed to submit crafted Messenger envelopes can interfere with deduplication keys, so queue publishing permissions remain part of the design.

Use Redis to reduce duplicate load and serialize expensive work. Do not use it to prove that money moved once.

PostgreSQL is a better place for that proof because it already owns the business transaction.

Let PostgreSQL arbitrate the race

Suppose a payment can create one merchant payout. Put that rule in the database:

CREATE TABLE merchant_payout (
    id                    UUID PRIMARY KEY,
    source_payment_id     UUID NOT NULL,
    merchant_id           UUID NOT NULL,
    amount_minor          BIGINT NOT NULL CHECK (amount_minor > 0),
    currency              CHAR(3) NOT NULL,
    created_at            TIMESTAMPTZ NOT NULL DEFAULT now(),
    CONSTRAINT uq_payout_source_payment UNIQUE (source_payment_id)
);

Now two pods can race and PostgreSQL still permits one payout row.

INSERT INTO merchant_payout (
    id,
    source_payment_id,
    merchant_id,
    amount_minor,
    currency
)
VALUES (
    gen_random_uuid(),
    :payment_id,
    :merchant_id,
    :amount,
    :currency
)
ON CONFLICT (source_payment_id) DO NOTHING
RETURNING id;

RETURNING tells the handler whether it created the payout. Under concurrency, the unique index arbitrates the conflict. One transaction wins. The other waits if necessary, then returns no row after the winner commits.

Do not replace this with a check followed by an insert:

// BAD: both pods can observe "not found" before either inserts.
if (!$payoutRepository->existsForPayment($message->paymentId)) {
    $payoutRepository->create($message);
}

The preliminary check may avoid work in a non-concurrent case, but it cannot enforce the invariant. The unique constraint does that.

Domain-specific constraints are my first choice. They document the actual rule and protect every write path, not only Messenger. A generic deduplication table becomes useful when the business table has no natural unique key or one event updates several tables.

Build a transactional deduplication table

The consumer-side table records which logical events a specific consumer has committed:

CREATE TABLE processed_message (
    consumer_name  VARCHAR(150) NOT NULL,
    message_id     UUID NOT NULL,
    processed_at   TIMESTAMPTZ NOT NULL DEFAULT now(),
    PRIMARY KEY (consumer_name, message_id)
);

consumer_name belongs in the key because independent consumers must each process the same event. The settlement projection and compliance projection do not deduplicate each other.

The handler claims the event, applies the business change, and writes any outgoing event inside one PostgreSQL transaction:

<?php

namespace App\MessageHandler;

use App\Message\PaymentCaptured;
use Doctrine\DBAL\Connection;
use Doctrine\DBAL\ParameterType;
use Symfony\Component\Messenger\Attribute\AsMessageHandler;
use Symfony\Component\Messenger\Exception\UnrecoverableMessageHandlingException;

#[AsMessageHandler]
final readonly class CreateMerchantPayoutHandler
{
    private const CONSUMER = 'create-merchant-payout-v1';

    public function __construct(private Connection $connection)
    {
    }

    public function __invoke(PaymentCaptured $message): void
    {
        $this->connection->transactional(function (Connection $connection) use ($message): void {
            $claimedMessageId = $connection->fetchOne(
                <<<'SQL'
                    INSERT INTO processed_message (consumer_name, message_id)
                    VALUES (:consumer, :message_id)
                    ON CONFLICT (consumer_name, message_id) DO NOTHING
                    RETURNING message_id
                SQL,
                [
                    'consumer' => self::CONSUMER,
                    'message_id' => $message->eventId,
                ],
            );

            if ($claimedMessageId === false) {
                // GOOD: this logical event already committed successfully.
                return;
            }

            $payoutId = $connection->fetchOne(
                <<<'SQL'
                    INSERT INTO merchant_payout (
                        id,
                        source_payment_id,
                        merchant_id,
                        amount_minor,
                        currency
                    )
                    VALUES (
                        gen_random_uuid(),
                        :payment_id,
                        :merchant_id,
                        :amount,
                        :currency
                    )
                    ON CONFLICT (source_payment_id) DO NOTHING
                    RETURNING id
                SQL,
                [
                    'payment_id' => $message->paymentId,
                    'merchant_id' => $message->merchantId,
                    'amount' => $message->amountInMinorUnits,
                    'currency' => $message->currency,
                ],
                [
                    'amount' => ParameterType::INTEGER,
                ],
            );

            if ($payoutId === false) {
                // A different event already paid this payment. Retrying cannot fix it.
                throw new UnrecoverableMessageHandlingException(sprintf(
                    'Payment %s already has a merchant payout',
                    $message->paymentId,
                ));
            }

            $connection->executeStatement(
                <<<'SQL'
                    INSERT INTO outbox_event (
                        id,
                        causation_id,
                        event_type,
                        aggregate_id,
                        payload
                    )
                    VALUES (
                        gen_random_uuid(),
                        :causation_id,
                        'MerchantPayoutCreated',
                        :aggregate_id,
                        CAST(:payload AS JSONB)
                    )
                SQL,
                [
                    'causation_id' => $message->eventId,
                    'aggregate_id' => $message->paymentId,
                    'payload' => json_encode([
                        'payoutId' => $payoutId,
                        'paymentId' => $message->paymentId,
                        'merchantId' => $message->merchantId,
                        'amount' => $message->amountInMinorUnits,
                        'currency' => $message->currency,
                    ], JSON_THROW_ON_ERROR),
                ],
            );
        });
    }
}

The ordering inside the transaction matters.

If the payout insert fails, PostgreSQL rolls back the processed_message row. Messenger can retry safely. If the transaction commits but the pod dies before SQS acknowledgement, the next attempt finds the deduplication row and returns without creating another payout.

The second unique boundary catches a different problem: a new event ID trying to pay a payment that already has a payout. The handler marks that case unrecoverable instead of retrying a permanent conflict. It belongs in the failure transport and should trigger an upstream identity investigation.

Do not commit the deduplication row in one transaction and process the message in another. A crash between them turns an unprocessed event into one that every retry skips. That is silent data loss wearing an idempotency badge.

Symfony's doctrine_transaction middleware can wrap handler execution in a transaction. I still prefer an explicit transaction in examples like this because the atomic boundary is visible beside the deduplication rule. Whichever approach you choose, do not nest unrelated transaction policies and assume they compose automatically.

Retention defines the deduplication window

processed_message grows forever unless you define retention. Deleting rows also deletes the proof that an old event was handled.

Keep records longer than every path by which an event can return:

• SQS message retention;
• Messenger failure transport or replay archive retention;
• delayed retries;
• manual replay procedures;
• the period in which an operational restore can reintroduce old messages;
• any audit requirement attached to the business event.

A 14-day cleanup job is unsafe if the team can replay a 30-day-old archive.

At high volume, delete in bounded batches and tune autovacuum for the table. Time partitioning looks attractive, but PostgreSQL requires a partitioned unique constraint to include the partition key. Adding processed_at to the primary key would allow the same (consumer_name, message_id) in two time partitions. Hash partitioning by the deduplication key can preserve uniqueness semantics, but it does not make retention a cheap partition drop.

If retention pressure justifies a more elaborate design, keep an active globally unique deduplication set and archive metadata separately. Do not trade away uniqueness to make cleanup convenient.

External APIs break the database transaction

The previous handler works because the deduplication record, payout, and outbox row live in the same PostgreSQL database.

An HTTP call does not.

// BAD: rollback cannot undo a successful remote payout.
$paymentProvider->createPayout($request);
$connection->insert('processed_message', $deduplicationRecord);

If the provider accepts the payout and the pod dies before the insert commits, the retry calls the provider again. Reversing the order is worse: a crash after recording the event but before the API call loses the payout.

The cleanest solution is a stable idempotency key supported by the provider:

$paymentProvider->createPayout(
    request: $request,
    idempotencyKey: 'merchant-payout.'.$message->eventId,
);

The provider must persist that key and return the original result for subsequent attempts. A request header that the provider ignores is not idempotency.

If the remote API has no idempotency support, model the ambiguity. Persist an attempt with states such as pending, confirmed, and unknown. Store the remote request ID. On timeout, query or reconcile with the provider before deciding to retry. There is no general technique that makes an irreversible, non-idempotent HTTP endpoint exactly once.

This is the part teams tend to hide behind a database transaction. The transaction can remain open for ten seconds while the HTTP call runs. It still cannot roll the call back.

The Outbox Pattern closes the producer gap

Consumer idempotency protects incoming messages. The producer has the opposite problem: it must commit business data and publish an event without losing either one.

Writing to PostgreSQL and SQS directly creates another dual write:

// BAD: the database may commit and SQS publishing may fail.
$entityManager->flush();
$bus->dispatch(new MerchantPayoutCreated(...));

The Outbox Pattern writes the event beside the business state in the same transaction. The previous handler already inserted outbox_event. A separate relay publishes committed rows.

CREATE TABLE outbox_event (
    id              UUID PRIMARY KEY,
    causation_id    UUID NOT NULL,
    event_type      VARCHAR(150) NOT NULL,
    aggregate_id    UUID NOT NULL,
    payload         JSONB NOT NULL,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
    published_at    TIMESTAMPTZ,
    attempt_count   INTEGER NOT NULL DEFAULT 0,
    next_attempt_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    last_error      TEXT
);

CREATE INDEX idx_outbox_unpublished
    ON outbox_event (created_at, id)
    WHERE published_at IS NULL;

Several relay pods can claim different records with SKIP LOCKED:

SELECT id, event_type, aggregate_id, payload
FROM outbox_event
WHERE published_at IS NULL
  AND next_attempt_at <= now()
ORDER BY created_at, id
FOR UPDATE SKIP LOCKED
LIMIT 100;

The SELECT, SQS send, published_at update, and commit must belong to one transaction. The row lock then prevents another relay from selecting the same row concurrently. This simple version holds a database transaction open during the SQS call, so keep batches small. A higher-throughput relay can use an explicit claim token and lease, but it still needs recovery when a claimed relay dies.

For each claimed event, the relay sends the stable outbox ID to SQS and then sets published_at.

It does not make publication exactly once.

SQS can accept the event and the relay can die before PostgreSQL records published_at. The next relay sends it again. Marking the row first creates the opposite failure: a crash can leave an event marked as published even though SQS never received it.

At-least-once publishing is the honest choice. Stable event IDs plus consumer-side deduplication make it safe.

SKIP LOCKED also relaxes ordering. A relay can publish a later row while an earlier row is locked or backing off, and SQS Standard does not preserve order. If aggregate ordering is a domain invariant, add an aggregate sequence and enforce it in the consumer, or use SQS FIFO message groups with their throughput trade-offs.

Failed rows need attempt_count, next_attempt_at, exponential backoff, and an alert or quarantine policy. Published rows also need retention. Monitor both unpublished count and oldest unpublished age; one permanently invalid payload should not consume every relay cycle.

This is also why SQS FIFO deduplication is not the final answer. Its deduplication ID suppresses duplicate sends for a five-minute interval. An outbox row can be retried after that window, and a consumer can still see redelivery after a visibility timeout. FIFO can reduce duplicates and preserve ordering per message group. It cannot join an SQS receipt to your PostgreSQL transaction.

Scaling workers on EKS

The architecture assumes that any pod can receive any message and any pod can disappear after any instruction:

The pods share no correctness state locally. Redis coordinates best-effort locks across replicas. PostgreSQL owns durable uniqueness. SQS owns the backlog.

Visibility timeout and keepalive

SQS hides a received message for its visibility timeout. If processing runs longer, another worker can receive it while the first worker is still active.

Configure a timeout above normal handler duration and let Messenger extend visibility for long work:

# config/packages/messenger.yaml
framework:
    messenger:
        failure_transport: failed

        transports:
            async:
                dsn: '%env(MESSENGER_TRANSPORT_DSN)%'
                options:
                    auto_setup: false
                    # Long sequential handlers should not hide nine prefetched messages.
                    buffer_size: 1
                    visibility_timeout: 120
                    wait_time: 20
                retry_strategy:
                    max_retries: 5
                    delay: 1000
                    multiplier: 2
                    max_delay: 30000
                    jitter: 0.2

            failed: 'doctrine://default?queue_name=failed'

        routing:
            App\Message\PaymentCaptured: async

php bin/console messenger:consume async \
    --keepalive=30 \
    --time-limit=3600 \
    --memory-limit=256M

Symfony implements SQS keepalive with ChangeMessageVisibility. The configured visibility timeout must not be shorter than the keepalive interval. Keepalive reduces premature redelivery. It does not eliminate the duplicate copies that SQS Standard queues are allowed to deliver.

The Symfony SQS transport normally buffers up to 9 messages. For long sequential handlers, those prefetched messages can become visible again before the worker starts them, and keepalive applies to the message currently being handled. buffer_size: 1 favors predictable visibility over prefetch throughput. Measure before increasing it.

SQS visibility cannot be extended beyond 12 hours from the initial receive. Work that can run longer belongs in smaller restartable steps or a workflow service such as AWS Step Functions.

Use an IAM role for the EKS service account. Do not embed AWS access keys in the Messenger DSN.

Graceful shutdown is a performance feature

During a rolling deployment or HPA scale-down, Kubernetes sends SIGTERM, waits for terminationGracePeriodSeconds, then sends SIGKILL.

Symfony Messenger can finish the current message after SIGTERM when the PHP CLI has PCNTL support. The worker must receive the signal, which means PHP should run as the container's direct process or behind an entrypoint that uses exec.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payout-consumer
spec:
  replicas: 3
  template:
    spec:
      serviceAccountName: payout-consumer
      terminationGracePeriodSeconds: 180
      containers:
        - name: worker
          image: example.dkr.ecr.eu-central-1.amazonaws.com/payments:2026-07-14
          command:
            - php
            - bin/console
            - messenger:consume
            - async
            - --keepalive=30
            - --time-limit=3600
            - --memory-limit=256M
          resources:
            requests:
              cpu: 250m
              memory: 192Mi
            limits:
              memory: 384Mi

Set the grace period above the longest credible handler duration, including network timeouts and shutdown overhead. A preStop hook consumes the same grace period. It does not add extra time.

Graceful shutdown reduces redelivery during deployments. Idempotency handles the cases where the node dies, the process is OOM-killed, or the grace period expires.

Use rolling-update limits and, when voluntary disruption must preserve capacity, a PodDisruptionBudget. A PDB does not prevent HPA scale-down or protect against an involuntary node failure, so it complements rather than replaces idempotency.

Scale on backlog, not CPU alone

An I/O-bound worker can sit at 20 percent CPU while the queue grows for an hour. CPU does not describe queue pressure.

For SQS consumers, track:

• ApproximateNumberOfMessagesVisible for queued work;
• ApproximateNumberOfMessagesNotVisible for in-flight work;
• ApproximateAgeOfOldestMessage for user-visible delay;
• measured processing rate per pod.

Kubernetes HPA does not read SQS directly. Use an external metrics adapter or KEDA. Set target backlog per pod from measured throughput and latency, then add scale-down stabilization so a short queue dip does not terminate half the consumers.

The AWS counters are approximate. They are scaling and alerting signals, not reconciliation data.

How the layers fit together

Each mechanism solves a different problem:

The most common mistake I see is asking one layer to do all six jobs. Redis becomes a permanent ledger. SQS FIFO becomes a database transaction. The outbox is described as exactly once. The design looks clean until a pod dies at the wrong instruction.

Use each mechanism for the failure boundary it can actually control.

Test the failure windows

A unit test that calls the handler twice is necessary. It is nowhere near sufficient.

The production test suite should include concurrency and interruption:

public function testRedeliveryDoesNotCreateAnotherPayout(): void
{
    $message = new PaymentCaptured(
        eventId: '0190b6d8-3ae7-7ca2-a653-6ff2d5a6f230',
        paymentId: '5c4080b1-b5d2-4e91-8de8-ec0e119d7647',
        merchantId: '3727331a-b533-4c6e-a0f8-48532237307b',
        amountInMinorUnits: 125_00,
        currency: 'EUR',
    );

    ($this->handler)($message);
    ($this->handler)($message);

    self::assertSame(1, $this->payoutRepository->countForPayment($message->paymentId));
    self::assertSame(1, $this->processedMessageRepository->countFor($message->eventId));
    self::assertSame(1, $this->outboxRepository->countForCausation($message->eventId));
}

Then test what the unit test cannot represent:

1. Start two real PostgreSQL transactions and submit the same event concurrently. Assert one business effect.
2. Kill a worker pod after database commit but before SQS deletion. Assert that redelivery is a no-op.
3. Throw after claiming processed_message but before the business insert. Assert that rollback allows retry.
4. Let a Redis lock expire while the first worker is still running. Assert that PostgreSQL still protects the invariant.
5. Kill the outbox relay after SQS accepts a message but before published_at commits. Assert that duplicate publication remains harmless.
6. Replay a failed message at the edge of the deduplication retention period.

This is where idempotency stops being a code style and becomes an operational property.

The production checklist

What I got wrong (and what I'd do differently)

My first versions of idempotent handlers used a cache check at the top: if the event ID exists in Redis, return; otherwise process it and set the key. It worked in sequential tests. Two workers could still pass the check together, and a Redis restart erased the history I was treating as permanent.

Then I moved the check to PostgreSQL but kept it separate from the business transaction. That closed the concurrency race and opened a crash window. The event could be marked as processed before its effect existed.

The more subtle mistake was calling the result exactly once. The database effect was effectively once, but an email provider and SQS still had their own failure boundaries. Naming the guarantee too broadly made the unresolved parts easy to ignore.

These days I start by drawing every commit, acknowledgement, and remote call. Wherever two durable systems cannot commit together, I assume the operation can repeat. Then I choose the stable identity and put the invariant in the system that owns the data.

I learned each of these the hard way. You don't have to.

References

• Symfony Messenger: Writing Idempotent Handlers
• Symfony Messenger: Message Deduplication
• Symfony Lock: Expiring Locks
• Symfony Amazon SQS Messenger
• Amazon SQS At-Least-Once Delivery
• Amazon SQS Visibility Timeout
• PostgreSQL INSERT and ON CONFLICT
• AWS Transactional Outbox Pattern
• Kubernetes Pod Termination
• KEDA AWS SQS Queue Scaler