Transactional Outbox Pattern with PostgreSQL, Kafka and Debezium

A microservice often needs to update its database and publish an event about the same business action. An order service might insert an order and then send OrderCreated to Kafka. These are two writes to two independent systems. If one succeeds and the other fails, the database and event stream disagree.

The transactional outbox pattern solves this dual-write problem without requiring a distributed transaction between PostgreSQL and Kafka. The service writes both the business change and an outbox record in one local database transaction. A separate relay later publishes the committed outbox record to Kafka. This guide explains the design, schema, polling and change-data-capture options, delivery guarantees, ordering, cleanup, monitoring, and failure testing.

The dual-write failure hidden in simple code

A common implementation saves an order, commits the transaction, and then calls the message broker. If the process crashes after the commit but before publishing, the order exists but no event reaches inventory, billing, or analytics. Retrying the HTTP request may create a duplicate order unless the endpoint is idempotent.

Publishing first is not safer. If the event reaches Kafka and the database transaction then rolls back, downstream services act on a change that never became true in the source system. Wrapping both calls in application code does not make them atomic because PostgreSQL and Kafka have independent commit boundaries.

AWS Prescriptive Guidance describes this as the dual-write problem: a database write and event notification must remain consistent even when either operation or the application fails. The transactional outbox moves both authoritative writes into one database transaction, which is a boundary the database can enforce.

How the transactional outbox works

Inside one PostgreSQL transaction, the application updates its domain tables and inserts a corresponding row into an outbox table. When the transaction commits, both records become visible. If it rolls back, neither record remains. A relay reads committed outbox rows and publishes them to Kafka.

The relay may poll the table or consume database changes through change data capture. After successful publication, a polling relay marks the row as published or removes it according to the retention policy. A CDC pipeline normally advances its own source offset and treats the outbox table as an append-oriented event source.

The pattern guarantees that a committed business change has a durable event record available for publication. It does not automatically guarantee that Kafka receives exactly one copy. A relay can publish successfully and crash before recording completion, so duplicates remain possible. Consumers must be idempotent.

Design a useful outbox table

A practical relational outbox row usually contains event_id, aggregate_type, aggregate_id, event_type, payload, occurred_at, and an ordering value. A polling implementation may also include published_at, attempt_count, next_attempt_at, and last_error. Add trace identifiers when end-to-end observability is required.

Use a globally unique event_id and treat it as immutable. aggregate_id identifies the entity whose state changed, such as an order ID. event_type should describe a business fact in past tense, such as OrderCreated or PaymentAuthorized. A schema_version field helps consumers evolve independently.

Store the payload as JSONB when producers need flexible JSON and PostgreSQL-side inspection. Avoid making the relay reconstruct the event from current domain tables; the state may have changed again before publication. The outbox row should contain the event data that was true at transaction time.

Keep the transaction small. The business update and outbox insert should be fast, with no network call to Kafka inside the transaction. Long transactions hold locks, consume connections, and delay change capture.

PostgreSQL example

An illustrative outbox table can use event_id UUID as the primary key, aggregate_type text, aggregate_id text, event_type text, payload jsonb, occurred_at timestamptz default now(), sequence_no bigint, published_at timestamptz, attempt_count integer default zero, and next_attempt_at timestamptz. Index unpublished rows by next_attempt_at and ordering fields for a polling relay.

When creating an order, begin a transaction, insert the order, insert the outbox event using the same database connection, and commit. Any exception should roll back both statements. Framework annotations such as a transactional service method help only if both writes truly participate in the same local transaction.

Do not let a generic repository silently open a second connection outside the transaction. Integration tests should stop execution after each statement and before commit to prove that rollback leaves neither a partial business row nor a stray outbox event.

Polling publisher implementation

A polling relay queries a limited batch of due, unpublished rows, claims them, publishes each event, and records completion. Multiple relay instances need a safe claiming mechanism. PostgreSQL SELECT FOR UPDATE SKIP LOCKED is commonly used so workers claim different rows without waiting on one another.

One approach starts a short transaction, selects a batch with row locks and SKIP LOCKED, records a lease or processing state, and commits. Publishing then occurs outside the database transaction. After Kafka acknowledges the record, the relay marks the event published. If the relay crashes, an expired lease makes the row eligible again.

Another approach holds locks while publishing, but broker latency then lengthens the database transaction and can reduce throughput. Leasing separates slow network work from row claiming at the cost of more state and duplicate-delivery scenarios. Choose deliberately and test worker crashes.

Use bounded batches, exponential backoff with jitter, maximum retry delay, and a dead-letter or operator-review state for permanent failures. Do not allow one malformed payload to block every later event.

Change data capture with Debezium

Debezium can capture inserts in the outbox table from PostgreSQL’s transaction log and route them to Kafka. Its Outbox Event Router single message transformation is designed for this pattern. The connector should capture the outbox table and transform each inserted row into the intended event record.

The Debezium documentation’s default model uses fields such as id, aggregatetype, aggregateid, type, and payload. aggregateid can become the Kafka record key, which helps route events for one aggregate to the same partition and preserve per-key order.

CDC avoids continuous application-level table polling and uses committed database changes as its source. It also introduces connector operations: replication slots, schema permissions, Kafka Connect availability, source offsets, lag monitoring, and safe handling of database failover.

CDC is not zero-operations magic. A stopped connector can retain PostgreSQL WAL through its replication slot, consuming disk until it resumes or the slot is managed. Monitor connector lag, slot health, Kafka delivery, and database storage.

Polling versus CDC

Polling is easy to understand and can be implemented with the application’s existing database library. It works well at modest volume and gives teams direct control over retries and cleanup. Poor polling queries, however, can add database load and create contention as the table grows.

CDC provides low-latency streaming and separates event publication from application code. It fits organizations already operating Kafka Connect and Debezium. Its complexity is operational rather than conceptual, and the blast radius includes connector configuration and replication infrastructure.

Choose based on event volume, latency requirements, database capability, platform maturity, and ownership. The core pattern remains the same: business state and the durable event record commit together.

Delivery semantics and idempotent consumers

The outbox pattern normally provides at-least-once publication. A relay can publish an event and fail before saving its published status or source offset. On recovery, it sends the event again. This is expected behavior, not necessarily a defect.

Consumers should store processed event IDs or make the business operation naturally idempotent. The duplicate check and consumer-side state change must share a transaction when possible. NeoAI Tech’s idempotency guide explains stable keys, stored outcomes, and safe retries: https://www.neoaitech.com/post/api-idempotency-keys-build-safe-retries-without-duplicate-operations

Be cautious with claims of end-to-end exactly-once processing. Kafka transactions can provide strong guarantees inside supported Kafka workflows, but an external database, email provider, or payment API creates another transaction boundary. Design observable, repeatable side effects instead of relying on a slogan.

Preserve ordering where it matters

Global ordering is expensive and usually unnecessary. Most domains require order only for changes to the same aggregate. Use aggregate_id as the Kafka key so related events reach the same partition. Include a per-aggregate sequence number or version so consumers can detect gaps, duplicates, and stale events.

Concurrent transactions for the same aggregate need a defined versioning rule. Optimistic locking can prevent two updates from producing conflicting versions. Consumers should decide whether to reject an out-of-order event, buffer it briefly, fetch current state, or process events commutatively.

AWS guidance highlights notification order as an important consideration. Timestamps alone may be insufficient because clocks and concurrent commits do not always express business order. Prefer an explicit sequence derived within the source transaction.

Backpressure and retention

If Kafka or the relay slows down, the outbox table grows. This is a durable backlog and needs capacity limits, alerts, and a recovery plan. Monitor the oldest unpublished event, unpublished row count, bytes consumed, publication rate, retry rate, and projected time to drain.

Do not delete rows immediately without an audit and recovery decision. A polling relay can move published rows to a history table or delete them in small batches after a retention period. Large deletes can create table bloat and replication load, so use controlled cleanup and monitor vacuum.

The principles match general event backpressure: incoming work must not grow without operational bounds. NeoAI Tech’s guide covers backlog age, recovery capacity, load shedding, and consumer lag: https://www.neoaitech.com/post/backpressure-in-event-driven-systems-a-practical-guide

Schema evolution and event contracts

Events are long-lived integration contracts. Add fields compatibly, avoid changing the meaning of existing fields, and identify schema_version in the envelope. Consumers should tolerate unknown optional fields and define how long old versions remain supported.

Publishing entire database rows leaks internal structure and couples consumers to the producer’s schema. Prefer a deliberate event contract containing business identifiers and facts. Sensitive fields should be excluded or minimized because events may be retained and copied across systems.

If using Avro or another schema registry format, define compatibility rules and deployment order. The producer’s database migration, application code, connector transformation, registry schema, and consumers may update at different times.

Security controls

Grant the application permission to insert outbox rows but not arbitrarily modify published history. Grant a polling relay only the read and update permissions it needs. A Debezium connector requires replication and table access appropriate to PostgreSQL logical decoding; keep those credentials separate from the application.

Encrypt database and Kafka connections, rotate secrets, restrict network access, and avoid storing unnecessary personal or secret data in payloads. Apply topic authorization so only intended producers, connectors, and consumers can access each event stream.

Validate payload size and content before inserting. A very large or malformed event can increase transaction time, overload Kafka records, and repeatedly fail in the relay.

Observability and operational signals

Every event should carry event_id, aggregate identifiers, event type, occurred time, and trace context. Instrument the business transaction, relay, broker publication, and consumer handling. Distinguish time spent waiting in the outbox from broker delay and consumer processing time.

Important alerts include oldest unpublished event age, publication failures, retry growth, relay throughput below creation rate, Debezium connector failure, source-offset stagnation, replication-slot WAL retention, invalid payloads, and consumer duplicate rate.

Correlate events across logs, metrics, and traces instead of searching by timestamps alone. NeoAI Tech’s OpenTelemetry guide provides a practical starting point: https://www.neoaitech.com/post/opentelemetry-python-flask-observability-guide

Failure tests that prove the design

Kill the application after the business insert but before the outbox insert and confirm the transaction rolls back. Kill it after the outbox insert but before commit and confirm both rows roll back. Kill it immediately after commit and confirm the relay later publishes the durable event.

Kill the relay after Kafka acknowledges but before it records success and confirm the duplicate is harmless. Stop Kafka and verify backlog alerts, bounded retries, and recovery. Feed a malformed event and confirm it does not block unrelated records.

For Debezium, stop the connector, generate transactions, restart it, and verify ordered recovery without loss. Test database failover, replication-slot behavior, schema changes, Kafka partition expansion, and consumer replay from an earlier offset.

Implementation checklist

• Business data and the outbox row commit in one local transaction.

• Every event has a unique immutable ID and explicit schema version.

• Per-aggregate ordering uses a key and sequence or version.

• Relay publication is retryable, bounded, observable, and safe after crashes.

• Consumers are idempotent and record processed event IDs where needed.

• Backlog age, throughput, failures, storage, and connector lag are monitored.

• Cleanup retains enough history for audit and recovery without uncontrolled growth.

• Security permissions and payload data follow least privilege and minimization.

• Crash points, broker outages, duplicates, poison events, and replay are tested.

Frequently asked questions

Does the outbox pattern guarantee exactly-once delivery?

It reliably records events with the business transaction, but relay crashes can produce duplicate publication. Plan for at-least-once delivery and idempotent consumers.

Should we delete an outbox row after publishing?

Use a retention policy based on audit, replay, storage, and compliance needs. Marking then deleting in controlled batches is common for polling implementations.

Is Debezium required?

No. A polling publisher is valid and often simpler at modest scale. Debezium is useful when CDC, low latency, and Kafka Connect operations already fit the platform.

Can the payload be rebuilt from domain tables?

It is safer to store the intended event payload or immutable facts at transaction time. Current domain state may differ when the relay eventually publishes.

How is the outbox pattern different from a saga?

The outbox reliably publishes a local service event. A saga coordinates a business process that spans multiple services and compensating actions. They often work together.

Conclusion

The transactional outbox pattern replaces an unsafe dual write with one atomic database transaction and a recoverable publication pipeline. Its reliability depends on the details: immutable event IDs, ordering rules, idempotent consumers, bounded retries, backlog monitoring, cleanup, and deliberate failure tests.

NeoAI Tech helps teams design production-ready cloud, data, DevOps, and event-driven systems. If your services occasionally lose events or create inconsistent downstream state, contact NeoAI Tech for an architecture review and practical migration plan.

Authoritative sources

AWS Prescriptive Guidance, Transactional Outbox Pattern: https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html

Debezium Documentation, Outbox Event Router: https://debezium.io/documentation/reference/stable/transformations/outbox-event-router.html

AWS sample implementation repository: https://github.com/aws-samples/transactional-outbox-pattern

PostgreSQL transaction documentation: https://www.postgresql.org/docs/current/tutorial-transactions.html

Apache Kafka documentation: https://kafka.apache.org/documentation/