CAP is often misread as "pick two of three forever and put it in the architecture doc." The consistency-availability trade-off fires during partitions. The trade-off you live with every day, on every query path in a healthy cluster, is latency versus consistency. That everyday axis is what PACELC makes explicit.
If you already know ACID vs BASE and sharding vs replication, this post is the decision frame those pieces sit inside: what C/A/P actually mean, why partitions force CP vs AP, how the consistency spectrum maps to real operations, and how to choose per query instead of per system.
CAP, defined without the folklore
Brewer's theorem says a distributed system cannot simultaneously guarantee all three:
- Consistency (C): here, linearizability. Every read sees the latest successful write; the system behaves as if there is a single up-to-date copy.
- Availability (A): every request to a non-failed node gets a non-error response (it may be stale).
- Partition tolerance (P): the system continues to operate despite network failures that split nodes into isolated groups.
Network partitions are not a rare academic event. Links flap, AZs degrade, GC pauses look like death, DNS lies. In practice you assume P is required. The real choice under a partition is CP or AP.
Important: CAP "consistency" is not ACID's C (constraints). Mixing them up produces meetings that go nowhere. CAP C is about replica agreement under concurrent operations; ACID C is about invariants inside a transaction.
CP vs AP in production language
CP (consistency + partition tolerance). During a split, some clients get errors or waits rather than a wrong answer. Bank ledger debit, inventory reservation that must not oversell, auth token revocation: better down than wrong. After recovery, you do not reconcile conflicting balances; you never created them.
AP (availability + partition tolerance). During a split, reads and writes keep succeeding. Reads may be stale; writes may not be visible everywhere yet. Shopping cart contents, activity feeds, product recommendations: better available with a brief lie than a 503 on Black Friday.
Neither is "more distributed." They encode different business failure modes.
The consistency spectrum (pick per operation)
Linearizability is the top of a ladder. Production systems climb down deliberately:
Most mature SaaS codebases do not pick one rung for the whole system. Payments and auth use strong guarantees. Search indexing, analytics rollups, and "users also bought" tolerate eventual. The architecture doc should list operations, not a single CAP letter.
PACELC: the rest of the time
CAP is silent about healthy clusters. PACELC fills that gap:
If Partition → Availability vs Consistency. Else → Latency vs Consistency.
Why latency vs consistency even when the network is fine? Resilience requires replicas, often across AZs or regions. Strong consistency waits for synchronous replication acknowledgements. Asynchronous replication returns fast and converges later.
Three replicas, synchronous credit of $100:
Latency grows with replica count and RTT. Skip the wait and you have eventual consistency: PACELC's EL (prefer latency) rather than EC (prefer consistency).
How common systems land on the map
The asterisk on MongoDB matters: defaults are not destiny. Quorum and read/write concern knobs move you along the spectrum per call.
Navigation patterns that ship
Tunable consistency. Cassandra/Dynamo-style ONE / QUORUM / ALL (and the dual on writes). Use quorum when the answer must be recent; use ONE when a slightly stale read is fine and latency budgets are tight. With replication factor N, choosing R and W such that R+W > N gives overlap that detects the latest write, without needing every replica on every call.
Async writes with sticky read-your-writes. Acknowledge the write quickly; for a few seconds (or until replication catches up), steer that user's reads to the leader or a session-sticky replica. Most users never notice the underlying eventual model.
Strong only when needed. A useful planning heuristic: roughly 20% of operations genuinely need CP (payments, auth, ledger, entitlement checks); roughly 80% are happier on AP/EL. Check which bucket an endpoint is in before you pay Spanner prices for a feed query.
Multi-tenant SaaS lens
Within a tenant, the domain is often small enough that strong consistency is cheap, especially with DB-per-tenant. Cross-tenant aggregates (billing rollups, platform analytics) almost always accept eventual. Shared-schema multi-tenancy makes CP harder: one hot tenant's quorum traffic shares fate with everyone else. Isolation model and consistency model are coupled decisions.
What belongs in the architecture doc
Not "we are a CP system." Write the operation:
POST /charges: linearizable; fail closed on partition.GET /feed: eventual; prefer availability; read-your-writes for the authoring user.GET /search: eventual; index lag of a few seconds is an SLO, not an incident.
That list drives database choice, quorum settings, retry policy, and what you tell support when an AZ dies.
A worked partition scenario
Two AZs, a ledger service, replication factor three. AZ-B loses connectivity to AZ-A for ninety seconds. Under CP, the minority side refuses writes to the account balance: clients get timeouts or 503s, support sees errors, balances stay correct. Under AP, both sides accept transfers; when the link heals you reconcile conflicting histories (last-write-wins, merge functions, or human ops). Banks choose the first pain. Collaborative carts often choose the second.
Same cluster, healthy network, cross-region replica: every strongly consistent read now waits on the far RTT. That is PACELC's Else branch. Product managers feel it as "the page got slow after we added DR," not as a theorem.
Anti-patterns
- Declaring the company "AP" so every team stops thinking about which endpoints lie.
- Using eventual consistency for entitlement checks ("user still has admin for 30s after revoke") without an explicit risk acceptance.
- Copying Spanner into the diagram when your write rate and team cannot operate it, then running Mongo on defaults anyway.
- Measuring only availability SLOs and ignoring "stale read" as a user-visible failure mode.
Key takeaways
- CAP binds under partition; partitions are inevitable, so the live choice is CP vs AP.
- CAP consistency means linearizability, which is stronger than most product language of "consistent."
- PACELC adds the everyday axis: latency vs consistency when the network is fine.
- Real systems pick per operation along a spectrum, not one letter for the fleet.
- Tunable quorums, sticky read-your-writes, and "strong only for money/auth" are how you navigate without dogma.
- For SaaS: strong inside the tenant where it matters; eventual for feeds, search, and cross-tenant analytics.
Stop arguing about whether the company is CP or AP. Argue about the query.