Skip to content
All articles

Message-Delivery Guarantees & ACID in Distributed Transactions

Dean Jain

Dean Jain

Senior Staff Software Engineer · Enterprise AI, Data & Cloud Architect

· 5 min read

Distributed SystemsMessagingSagasArchitecture
---
config:
  theme: dark
  fontSize: 17
  themeVariables:
    fontFamily: "Comic Sans MS, Comic Neue, Chalkboard SE, cursive"
---
flowchart LR
    AMO["⚡ At-most-once<br/>fast, may LOSE messages"]:::warn --- ALO["🔁 At-least-once<br/>safe, may DUPLICATE"]:::obs --- EO["🎯 Exactly-once<br/>ideal, expensive/rare"]:::good
    classDef warn fill:#FFE6A8,stroke:#E0A106,stroke-width:2px,color:#0F172A
    classDef obs fill:#AED6F1,stroke:#2E86C1,stroke-width:2px,color:#0F172A
    classDef good fill:#BFEFC8,stroke:#3FA34D,stroke-width:2px,color:#0F172A

Figure 1: The delivery-guarantee spectrum. Lose messages, duplicate messages, or pay dearly for exactly-once.

The moment your system spans a network, a guarantee you took for granted locally evaporates. You can no longer assume a message is delivered exactly once. Networks drop packets, consumers crash mid-process, acknowledgments get lost. So distributed messaging makes you choose a delivery guarantee, with real trade-offs. The honest answer most production systems land on isn’t “exactly-once” at all. The same network reality breaks ACID transactions across services, which is why sagas exist. Here’s how to reason about both.

TL;DR

  • Exactly-once is the hard one. You realistically pick at-most-once (fast, may lose) or at-least-once (safe, may duplicate).
  • The practical answer: at-least-once + idempotency ≈ exactly-once. Make consumers safe to re-run, and duplicates stop mattering.
  • Ordering is a separate guarantee and it costs throughput, so only demand it where you truly need it.
  • ACID doesn’t cross service boundaries for free. A transaction spanning services can’t just BEGIN…COMMIT.
  • Use sagas, not 2PC. A saga is a sequence of local transactions with compensating actions; choreographed (decentralized) or orchestrated (central coordinator).

1. The five guarantees

When a producer sends a message and a consumer processes it, several things can go wrong between them. The guarantees describe what you’ve decided to tolerate:

  • At-most-once deliver and don’t retry. If it’s lost, it’s lost. Lowest latency and overhead; acceptable for disposable data (a metric tick, a best-effort notification).
  • At-least-once keep retrying until acknowledged. Nothing is lost, but a lost ack means the message gets delivered again so consumers see duplicates. The most common production choice.
  • Exactly-once delivered and processed once, no loss, no duplicates. The ideal and the expensive, often-impractical one, because guaranteeing it end-to-end across a network requires heavy coordination.
  • Idempotent delivery not a level so much as a property: processing the same message twice has the same effect as processing it once.
  • Ordered delivery messages arrive in the order they were sent. Sometimes essential (a sequence of state changes), often not.

The key realization: at-most-once risks correctness (lost data), exactly-once risks performance and complexity. At-least-once sits in the sweet spot if you can handle the duplicates which is exactly what the next section is about.

2. The practical answer: at-least-once + idempotency

Chasing true exactly-once delivery usually isn’t worth the coordination cost. The pattern almost every robust system uses instead:

---
config:
  theme: dark
  fontSize: 17
  themeVariables:
    fontFamily: "Comic Sans MS, Comic Neue, Chalkboard SE, cursive"
---
flowchart LR
    ALO["🔁 At-least-once<br/>(never lose a message)"]:::obs --> ID["🛡️ Idempotent consumer<br/>(re-running is a no-op)"]:::gate
    ID --> EFF["🎯 Effective exactly-once"]:::good
    classDef obs fill:#AED6F1,stroke:#2E86C1,stroke-width:2px,color:#0F172A
    classDef gate fill:#D7C3F2,stroke:#8E5BD0,stroke-width:2px,color:#0F172A
    classDef good fill:#BFEFC8,stroke:#3FA34D,stroke-width:2px,color:#0F172A

Figure 2: The workhorse combination. At-least-once delivery, so nothing is lost, plus idempotent consumers, so duplicates don't matter, gives you exactly-once *effects*.

Choose at-least-once so you never lose a message, then make your consumer idempotent so processing a duplicate has no extra effect. Now the duplicates that at-least-once produces are harmless, and you’ve achieved the effect of exactly-once without the cost of guaranteeing it on the wire. In practice idempotency means one of three things. Dedupe on a message ID you’ve already processed. Use upserts instead of blind inserts. Or design operations that are naturally repeatable, like “set balance to X” rather than “add X”. Exactly-once delivery is the thing you cannot buy. Exactly-once processing is the thing you can build.

Ordering is a separate decision. Guaranteeing messages arrive in send-order constrains how you partition and parallelize, because you often have to pin a key to a single partition and consumer. That caps throughput. So treat ordering as a cost. Demand it only where sequence is semantically required, like a per-account ledger, and relax it everywhere else to scale. Pairing ordered + idempotent handles most stateful event streams correctly.

3. ACID across services: 2PC vs sagas

Delivery guarantees handle individual messages. But what about a business transaction spanning multiple services? Place an order, charge a card, reserve inventory, where you need all-or-nothing across systems that each own their own database. You can’t wrap them in one BEGIN…COMMIT. Two options:

---
config:
  theme: dark
  fontSize: 17
  themeVariables:
    fontFamily: "Comic Sans MS, Comic Neue, Chalkboard SE, cursive"
---
flowchart TB
    subgraph TPC["🔒 Two-Phase Commit"]
        direction LR
        CO["Coordinator"]:::gate -->|"prepare → commit"| N["All nodes lock &amp; wait"]:::warn
    end
    subgraph SAGA["🔄 Saga (local txns + compensation)"]
        direction LR
        T1["Txn A"]:::server --> T2["Txn B"]:::server --> T3["Txn C ❌"]:::danger
        T3 -.->|"compensate"| T2 -.->|"compensate"| T1
    end
    classDef gate fill:#D7C3F2,stroke:#8E5BD0,stroke-width:2px,color:#0F172A
    classDef warn fill:#FFE6A8,stroke:#E0A106,stroke-width:2px,color:#0F172A
    classDef server fill:#A8E6D0,stroke:#2FA37C,stroke-width:2px,color:#0F172A
    classDef danger fill:#FFB3B3,stroke:#D14545,stroke-width:2px,color:#0F172A

Figure 3: 2PC locks all participants until everyone commits; a saga runs local transactions and *compensates* (undoes) earlier ones if a later step fails.

  • Two-Phase Commit (2PC) a coordinator asks every participant to prepare, then commits only if all agree. It gives true atomicity. But it locks resources across services and blocks if the coordinator fails, which is why it doesn’t scale and is avoided in modern microservices.
  • Saga model the business transaction as a sequence of local transactions, each with a compensating action that semantically undoes it. If step C fails, you run the compensations for B and A (refund the charge, release the inventory). You give up isolation (intermediate states are visible) in exchange for no distributed locks. Two styles:
    • Choreographed saga each service listens for events and triggers the next step itself. Decentralized, no single point of control but the overall flow is implicit and harder to follow.
    • Orchestrated saga a central orchestrator directs each step and handles failures. Easier to reason about and monitor; the orchestrator is a dependency.

The mental shift is this. You trade ACID’s isolation for availability and scale, and replace “roll back” with “compensate.”

It’s the same bargain EDA makes everywhere. Accept eventual consistency, and design the undo path deliberately.

One caveat for anyone applying this to AI agents. Everything above assumes a retry is a resend: the same bytes, the same ID, arriving twice. An agent’s retry is a fresh decision, and that breaks the first technique here. I work through what survives in An Agent Doesn’t Retry. It Re-Decides.

Distributed systems quietly revoke two things you relied on locally: exactly-once delivery, and cross-entity transactions. Pretending otherwise is how you ship data loss and stuck orders. Design for it: pick at-least-once + idempotency for messaging, demand ordering only where it’s essential, and use sagas with compensations instead of distributed locks. The network will fail; engineering for that failure up front is the difference between a resilient system and a mysterious one.

Further reading