Module 5.1: What Makes Systems Distributed

Complexity: [MEDIUM]

Time to Complete: 45-60 minutes

Prerequisites: Systems Thinking Track (recommended)

Track: Foundations

What You’ll Be Able to Do

After completing this module, you will be able to:

Diagnose the eight fallacies of distributed computing and identify where each manifests in real cloud-native architectures.
Evaluate a system’s architecture to determine which components introduce distributed-system challenges such as partial failure, network partitions, and clock skew.
Design service boundaries that account for the inherent unreliability of network communication between components.
Implement idempotent operations and retry mechanisms to safely handle transient network failures in microservice environments.
Compare CP and AP architectural patterns to select the appropriate data consistency model for specific business requirements.

Why This Module Matters

In February 2017, an authorized operator working from an established AWS S3 playbook entered an incorrect input while removing a small number of servers from a billing subsystem in US-East-1. The over-broad removal did not directly “take the index offline” in one step; it removed more servers than intended, and the removed servers also supported the index and placement subsystems. Those two large subsystems then required full restarts, and because they held very large amounts of metadata and had not been fully restarted in years, recovery took several hours. The way one subsystem’s failure rippled into hundreds of dependent services is exactly the failure-mode taxonomy and blast-radius analysis taught in Failure Modes and Effects; this module uses the outage only as the doorway into distributed-systems thinking.

The important lesson is not that humans make mistakes, because every operating model must assume that they do. The lesson is that a distributed system turns a local action into a system-wide experiment. A billing maintenance command affected metadata lookup, storage placement, dashboards, build systems, websites, and devices that were not part of the original operator’s mental model. External analysts at Cyence, widely reported at the time, estimated that S&P 500 companies lost roughly $150 million during the interruption, but that number was not an AWS-reported figure and should be read as an outside economic estimate rather than a precise service metric.

That is what makes distributed systems different from ordinary programming. A single process on one machine can still be complicated, but its failure is usually local and its state is usually inspectable from one place. A cloud service is a conversation among machines, networks, clocks, queues, caches, databases, load balancers, controllers, and people. The application presents one coherent experience to the user, while underneath that experience there are independent components that can be slow, unreachable, stale, overloaded, partitioned, or correct in isolation but wrong in combination.

The central analogy for this module is a stage performance with the curtain down. The audience sees one play, but the production depends on actors, lighting, sound, props, ticketing, stage management, and building power all coordinating without the audience seeing the machinery. A distributed system is similar: the user sees “save file” or “place order,” while the system is quietly routing messages across many independently failing parts. Good architecture does not pretend the backstage is simple; it designs the performance so one missed cue does not collapse the whole show.

We accept this complexity because the alternatives are worse for modern workloads. One large machine can be easier to reason about, but it has a finite ceiling for CPU, memory, storage, blast-radius isolation, and geographic reach. Distributed systems let us scale beyond one box, survive some component failures, place work near users, separate teams by service boundary, and evolve parts of a system without replacing the whole. The price is that we must design for uncertainty instead of treating uncertainty as an exception.

Did You Know?

**The external $150 million estimate:** Analysts at Cyence estimated that S&P 500 companies lost roughly$ 150 million during the 2017 AWS S3 disruption. Treat that as a third-party economic estimate, not as an AWS-reported loss number or an exact causal accounting.
Physical speed limits: Light travels about 299,792 km/s in vacuum, but signals in fiber are commonly approximated around 200,000 km/s. New York and London are roughly 5,600 km apart in a straight line, so a perfect fiber path would still have a round-trip floor near 56 ms before routers, queues, encryption, or real cable routes add delay.
The CAP theorem origins: Eric Brewer presented CAP as a conjecture at PODC 2000, and Seth Gilbert with Nancy Lynch later formalized the result in 2002. The practical reading is narrower than the slogan: during a partition, a shared-data system must decide how it trades consistency against availability.
Consensus in Kubernetes: The Kubernetes control plane stores cluster state in etcd, and etcd uses Raft to replicate state among members. Raft was published by Diego Ongaro and John Ousterhout in 2014 with understandability as an explicit design goal compared with Paxos.

Part 1: Defining Distributed Systems

Before we can solve distributed-systems problems, we need a precise definition of what kind of system we are talking about. A distributed system is a collection of independent computational components that coordinate by sending messages across a network and present themselves as one coherent system to users or clients. The components may be physical servers, virtual machines, containers, processes, controllers, databases, message brokers, or managed cloud services. What matters is not the packaging; what matters is that no single process owns all state, all decisions, and all timing.

1.1 What is a Distributed System?

Leslie Lamport’s famous joke-definition captures the emotional truth: “A distributed system is one in which the failure of a computer you didn’t even know existed can render your own computer unusable.” The quote is funny because it is unfair in exactly the way production feels unfair. Your application can be correct, your database schema can be valid, and your deployment can be green, yet a remote dependency, DNS resolver, certificate authority, control-plane quorum, routing table, or object-store subsystem can still decide whether your user gets a working product.

A node is one participant in the system. In a Kubernetes cluster, a worker node is literally called a node, but the concept is broader than Kubernetes. A node can be an API server process, a database replica, a queue consumer, an object-storage metadata service, a sidecar proxy, or a browser tab participating in a collaborative editor. A node has local memory, local disk or durable storage access, local clocks, local CPU scheduling, and local observations. It does not automatically know what every other node knows.

Message passing is the act of coordinating by sending information between nodes. A message might be an HTTP request, a gRPC call, a database replication record, a Raft append entry, a DNS query, a queue event, or a Kubernetes watch notification. The sender can put bytes on the wire, but it cannot force the receiver to process them on time, process them once, process them in order, or process them at all. Once communication crosses a process and network boundary, the sender has only evidence, not certainty.

The single coherent system is the illusion we build for users. A shopper clicks “buy” once and expects one order. A developer runs kubectl apply and expects the cluster to converge toward the declared state. A reader opens a document and expects one current version, even if replicas, caches, and search indexes are involved. The art of distributed systems is maintaining that illusion without lying to yourself about the independent parts underneath it.

A single mainframe or one large database server is easier to reason about because most important state changes happen inside one failure domain. The process can lock memory, write to local disk, and decide an order of events without asking a remote peer. If the machine fails, the failure is often obvious: the whole service stops. This simplicity is valuable, and many systems should stay single-process or single-database as long as their requirements allow it.

Distributed architecture becomes attractive when a single failure domain is no longer acceptable or sufficient. Scale pushes data and work across multiple machines. Fault tolerance demands that one failed component not erase the service. Geography pushes replicas closer to users and into different regions. Organizational growth pushes teams to own separate deployable units. Those are legitimate reasons to distribute a system, but each one buys capability by spending cognitive budget.

The first discipline, then, is to recognize when distribution has entered the room. A web server and a database are already a distributed system because the application can succeed while the database is slow, or the database can commit while the web server times out. A service with a cache is distributed because cache state and database state can diverge. A Kubernetes controller and the API server are distributed because the controller observes desired state, acts, and later learns whether the API server and cluster accepted the change.

1.2 The Boundary Test

Use a boundary test when you are unsure whether a design is distributed: can one part keep running while another part is unavailable, slow, stale, or partitioned? If yes, distributed-systems behavior is present. The boundary may be a network hop, a process boundary, a data-replication boundary, a queue boundary, a cache boundary, or a human-approval boundary. The key question is not whether there are multiple computers; the key question is whether the system can have multiple partial views of reality at the same time.

This matters because distributed failures are rarely polite. They do not arrive as one clean exception with a perfect root cause attached. They arrive as increased latency, missing acknowledgments, duplicate events, old cache entries, leader elections, half-open TCP connections, delayed DNS changes, clock skew, overloaded retry queues, or a metrics dashboard that is still green because the broken path is not the one the dashboard checks. You diagnose such systems by asking what each participant could observe locally, not by assuming the system has one global truth that everyone sees instantly.

1.3 Distribution Should Be Intentional

The strongest distributed-systems design decision is sometimes to avoid distributing a concern at all. A modular monolith with one database may be the right architecture while a product is still discovering its invariants, while one team owns most changes, or while traffic fits comfortably inside one failure domain. Keeping code in one deployable unit does not mean ignoring architecture. It means preserving local reasoning until the organization, scale, or reliability requirement can pay for the added coordination cost.

Accidental distribution is common because cloud platforms make the mechanical act of creating services easy. A team can add a queue, cache, function, database replica, service mesh, and regional failover path with a few configuration changes, but every addition creates a new place where reality can diverge. The queue can accept messages while consumers are broken. The cache can serve old data while the database is correct. The failover region can be reachable while its credentials, schema, or feature flags are not ready for real traffic.

Intentional distribution starts with a reason that can survive an incident review. “We split this service because this team owns a separate invariant and needs independent release control” is a stronger reason than “the class was getting large.” “We replicate this data because users in another region need a bounded recovery objective” is stronger than “multi-region sounds safer.” When the reason is explicit, the design can name the trade-off it is accepting instead of discovering that trade-off during an outage.

Part 2: The Eight Fallacies of Distributed Computing

The eight fallacies of distributed computing are false assumptions often associated with L. Peter Deutsch, James Gosling, and the Sun Microsystems engineering culture that shaped early networked software. They are not a formal theorem, and they are not a checklist to memorize for trivia. They are a compact way to diagnose the assumptions hidden inside architecture diagrams. When an incident surprises a team, one of these fallacies is often quietly present.

Use the fallacies as design-review prompts rather than slogans. For each remote dependency, ask what happens if the network is slow, unavailable, expensive, hostile, changing, owned by another team, or different from the path you tested. The answer does not always need a complex mechanism. Sometimes it is a timeout, sometimes a cached fallback, sometimes a queue, sometimes an explicit error, and sometimes a decision not to make the call in the user path at all. The value is that the assumption becomes visible before production traffic tests it for you.

2.1 The Network Is Reliable

The first fallacy says the network is reliable. In a local program, a function call either returns or throws inside one process. In a distributed system, a remote call can be lost before it reaches the server, processed by the server while the response is lost, delayed behind other traffic, rejected by a load balancer, reset by a proxy, or accepted by a server that crashes before writing durable state. “Request failed” is not a single fact; it is a family of possibilities.

Cloud-native systems often violate this fallacy through optimistic service-to-service calls. Service A calls Service B and assumes that an HTTP 200 means success while any timeout means no work happened. That assumption is dangerous because the timeout belongs to the caller’s local clock, not to the remote side’s state. The safe design treats the network as an unreliable transport and gives every side effect a way to be retried, reconciled, or observed later.

2.2 Latency Is Zero

The second fallacy says latency is zero. Developers fall into this trap when they replace a local method call with a remote service call and keep the same mental model. A local call may take nanoseconds or microseconds; a cross-zone or cross-region call pays serialization, kernel scheduling, queueing, routing, TLS, proxy, and server time. Even when the median is excellent, the tail can dominate the user’s experience because one slow dependency can hold the whole request open.

The cloud-native manifestation is the chatty microservice chain. A page request calls a profile service, which calls an entitlement service, which calls a billing service, which calls a feature-flag service, which calls a database several times. Each hop looks reasonable in isolation, but the end-to-end path becomes an N-plus-one network pattern. Good designs batch, cache carefully, move computation near data, collapse unnecessary hops, and set latency budgets before service boundaries become permanent.

2.3 Bandwidth Is Infinite

The third fallacy says bandwidth is infinite. Network capacity is large enough that small tests hide the problem, then production traffic reveals it through replication lag, slow image pulls, overloaded service meshes, expensive cross-zone transfer, or backups that saturate links during recovery. Bandwidth is also shared: your service is not the only system using the same node, network interface, zone fabric, NAT gateway, or observability pipeline.

A common cloud-native example is treating logs, metrics, traces, and replication streams as free. Teams add high-cardinality labels, full request bodies, debug logs, and cross-region replication without budgeting for volume. During an incident, the observability pipeline can become part of the load spike, and the replication stream can compete with user traffic. Designing for finite bandwidth means reducing unnecessary payloads, applying backpressure, sampling thoughtfully, and knowing which traffic can be shed first.

2.4 The Network Is Secure

The fourth fallacy says the network is secure. Private address space, cluster networking, and internal DNS can create a false sense of safety. A request that came from “inside” may still come from a compromised workload, an over-permissioned service account, a stale credential, a misconfigured proxy, or a tenant path that should never have reached the target. Distributed systems multiply trust decisions because every boundary is a potential policy boundary.

In Kubernetes, this fallacy shows up when teams rely only on namespace separation or on the idea that “only our services can reach this endpoint.” Secure distributed design authenticates callers, authorizes actions, encrypts sensitive paths, rotates credentials, records audit trails, and assumes that topology will change. The network can help enforce policy, but it should not be the only reason a dangerous operation is accepted.

2.5 Topology Does Not Change

The fifth fallacy says topology does not change. Real systems reschedule pods, rotate nodes, replace instances, rebalance shards, move leaders, update DNS, add replicas, remove replicas, and shift traffic during deploys. Even if the application code stays still, the runtime graph around it moves. A design that depends on fixed addresses, fixed replica counts, or fixed leader placement is brittle before it reaches production scale.

Kubernetes makes topology change routine. A Deployment rollout replaces pods gradually. A Service endpoint list changes as readiness probes pass or fail. The API server offers watches so clients can observe changes, but a watch is not magic global memory; clients still need reconnection logic and resource-version handling. Treat topology as data that changes over time, not as a diagram frozen during the design review.

2.6 There Is One Administrator

The sixth fallacy says there is one administrator. In a small system, one team may control code, infrastructure, network policy, credentials, dashboards, and deployment cadence. In a real platform, different teams own the application, cluster, cloud account, identity provider, CI system, database, security policy, and incident process. A distributed system is also a distributed organization.

This fallacy appears when a service assumes another team will coordinate every breaking change manually. It also appears when a platform team rolls out a policy change without considering application-owned retries, clients, or data migrations. Good distributed designs use explicit contracts, versioned APIs, deprecation windows, ownership metadata, runbooks, and observable compatibility signals. The architecture should not require perfect cross-team timing to remain correct.

2.7 Transport Cost Is Zero

The seventh fallacy says transport cost is zero. Transport cost includes money, CPU, memory, sockets, file descriptors, TLS handshakes, serialization, connection pools, load-balancer capacity, and operational attention. A single remote call can be cheap; millions of remote calls per minute across zones or regions can become a material cost center and a reliability risk.

Cloud-native teams often discover this fallacy through synchronous fan-out. An API endpoint fans out to twenty services, each service emits traces and logs, and every hop crosses a mesh proxy that performs encryption and policy checks. The transport layer is doing useful work, but it is not free. Design reviews should ask how many network calls a user action creates, how those calls scale with data size, and whether a lower-cost asynchronous path would meet the product need.

2.8 The Network Is Homogeneous

The eighth fallacy says the network is homogeneous. Real networks differ by region, zone, cloud provider, device, proxy path, MTU, DNS behavior, packet-loss profile, rate limit, and policy. Even inside one cluster, traffic between two pods on the same node behaves differently from traffic that crosses nodes, zones, gateways, or egress controls. Homogeneity is a comforting fiction that disappears under load or during migration.

Hybrid and multi-cloud systems make this fallacy visible quickly. One path may have low latency but strict egress controls. Another may have higher latency but more capacity. A managed database endpoint may behave differently from a self-hosted service behind a Kubernetes Service. Robust systems measure path behavior, expose dependency latency by route, test failover paths before they are needed, and avoid assuming that every network hop has the same semantics.

The eight fallacies are a diagnostic lens. When you diagnose a distributed architecture, ask which fallacy would make the design look simpler than it is. When you evaluate a production incident, ask which fallacy made the failure surprising. When you design a new service boundary, ask what the system must still do when that fallacy is false, because in production it eventually will be.

Part 3: Partial Failure

Partial failure is the defining property of distributed systems. In a single process, many failures are total from the user’s point of view: the process crashes, the request stops, or the machine is unavailable. In a distributed system, some parts keep working while others fail. The database primary may be healthy while a replica lags. The API may accept writes while the search index is stale. A queue may keep receiving messages while consumers are stuck. The load balancer may reach one zone while another zone is isolated.

Partial failure is hard because it creates ambiguity. If you call another service and do not receive a response, you do not know whether the request never arrived, arrived and failed validation, arrived and committed, committed but lost its response, or is still waiting behind other work. The caller’s timeout is a local decision to stop waiting. It is not proof that the remote side is dead, and it is not proof that the remote side did nothing.

This uncertainty is why perfect failure detection is impossible in ordinary asynchronous networks. A detector can suspect a node after a timeout, but slow and dead can look identical from a distance. If messages can be delayed arbitrarily, no observer can always distinguish a crashed node from a node whose messages are merely late. Practical systems use timeouts, heartbeats, leases, and quorum protocols anyway, but they treat those mechanisms as engineering approximations with trade-offs.

Gray failures make the situation worse. A gray failure is not a clean crash; it is a component that is alive enough to pass some checks while broken for meaningful work. A pod might answer TCP connections but hang on database calls. A disk might respond slowly only under write pressure. A service might return success while silently dropping background events. A dependency might be reachable from one zone and unreachable from another. Health checks reduce blind spots, but they cannot prove that every user path is healthy.

Hypothetical scenario: imagine a checkout system with three services: cart, payment, and email. The payment service commits a charge, but the response to checkout times out after two seconds. Checkout retries immediately without an idempotency key, so the payment service sees a second valid request and creates a second charge. Nobody intended a double charge, and no single component “malfunctioned” according to its local logic. The distributed failure was the gap between a committed side effect and a caller that could not observe it.

Partial failure also changes how you model user-visible state. A single-process program can often return either success or failure, but a distributed workflow may need states such as accepted, pending, retrying, committed-but-not-notified, needs-reconciliation, or degraded. Those states can feel awkward in a product conversation, yet they are more honest than pretending every workflow is instantaneous and binary. If the system cannot represent uncertainty, operators and users will discover uncertainty anyway through duplicate tickets, manual database fixes, or inconsistent dashboards.

State machines are useful because they make uncertainty explicit. Instead of writing a checkout flow as a long chain of synchronous calls, you can record an order as accepted, attach an idempotency key, enqueue payment, mark payment as authorized when confirmed, and then enqueue email as a separate effect. That design still has failures, but each failure leaves a durable state that can be retried or inspected. The architecture becomes easier to operate because the system remembers where it was when communication became ambiguous.

The response to partial failure is not despair; it is design. Use idempotent operations when a caller may retry. Store request identifiers so duplicate attempts map to one logical effect. Make state transitions observable so operators can reconcile uncertain outcomes. Prefer asynchronous workflows when the user does not need every downstream effect before receiving a response. Place bulkheads between dependencies so one slow subsystem does not consume every thread, connection, or worker in the caller.

Module 5.4, Partial Failure and Timeouts, goes deeper into timeout selection, retry storms, jitter, and circuit breakers. For now, the important mental model is that partial failure is not an edge case. It is the normal failure mode of systems whose parts can disagree about what just happened.

Part 4: The Network Is Not Reliable

A network partition occurs when some nodes cannot communicate with other nodes, even though at least one side may still be running. Partitions can be clean, where two groups cannot reach each other in either direction, but they can also be asymmetric. Node A may reach Node B while Node B cannot reach Node A through the expected path. One availability zone may reach the database while another zone cannot. A service mesh policy may block traffic in one direction while health checks use a different path and stay green.

Packet loss and reordering add more uncertainty. TCP hides some loss and ordering problems by retransmitting and presenting an ordered byte stream, but that does not make distributed operations reliable. Retransmission takes time, connection pools can fill, proxies can reset connections, deadlines can expire, and application-level retries can duplicate work. UDP-based protocols, DNS behavior, load balancers, queues, and streaming systems each have their own delivery semantics that must be understood rather than assumed.

Timeouts are guesses because they encode a trade-off between waiting too long and giving up too early. A short timeout improves caller responsiveness and frees resources quickly, but it can mark healthy work as failed during a latency spike. A long timeout reduces false suspicion, but it can trap threads, sockets, and users behind a dependency that is not recovering. The right timeout depends on downstream latency distribution, caller budget, retry policy, resource limits, and the business value of waiting.

Retries are useful only when the operation is safe to repeat or when the caller can prove that a previous attempt did not commit. The usual production answer is idempotency: give each logical operation a stable key, store the result under that key, and return the same result for duplicate attempts. Idempotency does not mean “the function has no side effects.” It means repeated delivery of the same logical request produces one logical outcome.

from dataclasses import dataclass, field


@dataclass
class PaymentGateway:
    receipts_by_key: dict[str, str] = field(default_factory=dict)
    first_call_timed_out: bool = False

    def charge(self, idempotency_key: str, amount_cents: int) -> str:
        if idempotency_key in self.receipts_by_key:
            return self.receipts_by_key[idempotency_key]

        receipt = f"receipt-{len(self.receipts_by_key) + 1}"
        self.receipts_by_key[idempotency_key] = receipt

        if not self.first_call_timed_out:
            self.first_call_timed_out = True
            raise TimeoutError("client timed out after the remote side committed")

        return receipt


def charge_with_retry(gateway: PaymentGateway, key: str, amount_cents: int) -> str:
    for _attempt in range(3):
        try:
            return gateway.charge(key, amount_cents)
        except TimeoutError:
            continue
    raise RuntimeError("payment status is unknown and must be reconciled")


gateway = PaymentGateway()
request_key = "checkout-1000"

print(charge_with_retry(gateway, request_key, 5000))
print(charge_with_retry(gateway, request_key, 5000))

receipt-1
receipt-1

The example is intentionally small, but the production principle is large. The first call creates the receipt and then simulates a lost response. The retry uses the same idempotency key, so the gateway returns the existing receipt instead of creating a second charge. A real implementation would persist the key durably, scope it to the actor and operation, expire it carefully, and expose reconciliation paths for cases where the caller exhausts retries without learning the outcome.

Backoff and jitter matter because retries can become a second outage. If every caller retries immediately after a shared dependency becomes slow, the dependency receives more traffic precisely when it has less capacity. Exponential backoff spreads attempts over time, and jitter prevents a synchronized wave of clients from retrying together. Module 5.4 will focus on those mechanics, but you should already connect them to the bigger theme: the network is not reliable, so the caller must be polite, bounded, and repeatable.

Part 5: CAP and the Consistency-Availability Trade-off

CAP is one of the most quoted and most misquoted ideas in distributed systems. In its practical form, CAP says that when a network partition prevents some nodes from communicating, a shared-data system cannot simultaneously provide a fully consistent answer and remain available to every request on every side of the partition. Partition tolerance is not a feature you “pick” in a real distributed system; partitions are a condition the environment can impose. The live design choice during a partition is how you trade consistency against availability.

Availability in CAP also has a stricter meaning than many architecture conversations use. It is not “the status page is green” or “most users can still do something.” In the formal model, availability means every request to a non-failing node eventually receives a response. That is why a CP system can look unavailable during a partition even though the service is making a deliberate safety choice: a node that cannot coordinate may refuse a write rather than accept a value that could violate consistency. In product language, this is often a controlled refusal rather than a crash.

The sloppy slogan “pick two of three” hides the time dimension. When there is no partition, many systems can provide useful consistency and useful availability together. During a partition, however, a system that preserves strong consistency may reject or delay some operations until it can coordinate with the required nodes. That is the CP posture: preserve consistency under partition by sacrificing some availability. A system that preserves availability may accept operations on both sides and reconcile later. That is the AP posture: preserve availability under partition by accepting weaker consistency.

Neither posture is morally superior. CP is appropriate when accepting conflicting writes would violate an invariant that the business cannot repair. Examples include granting the same scarce resource twice, accepting a withdrawal that breaks an account invariant, or changing security policy without a single authoritative order. AP is appropriate when temporary divergence is acceptable and availability matters more than immediate agreement. Examples include shopping carts, like counters, telemetry ingestion, local-first collaboration, recommendations, and workloads where conflicts can be merged or compensated.

Consensus protocols such as Raft and Paxos are common tools for CP decisions because they use quorum agreement to keep replicas in one ordered history. Module 5.2, Consensus and Coordination, explains how those protocols coordinate leaders, logs, and quorums. Eventually consistent replication and conflict-resolution strategies are common AP tools because they let writes continue in more places and converge later. Module 5.3, Eventual Consistency, explains read-your-writes, causal consistency, version vectors, last-writer-wins hazards, and CRDT-style approaches.

PACELC extends the conversation by pointing out that even when there is no partition, replicated systems often trade latency against consistency. A system can wait for remote replicas to acknowledge a write before responding, which improves consistency but adds latency. Or it can respond after a local write and replicate asynchronously, which improves latency but allows stale reads and conflict windows. CAP is the emergency-room framing for partitions; PACELC reminds you that normal operation also contains trade-offs.

The most useful CAP question is not “is this database CP or AP?” The better question is “which operations require which guarantees under which failure modes?” One product may use CP coordination for account ownership, AP replication for activity feeds, asynchronous events for email, and cached reads for public catalog pages. Distributed-system design is often a per-invariant decision, not a single label stamped on the whole architecture.

Read paths deserve the same care as write paths. A system may accept writes through a CP leader but serve reads from followers, caches, or search indexes that lag behind the leader. That can be a good design if the product labels freshness honestly or if stale reads do not break user expectations. It is a bad design if a user changes a security setting, receives a success message, and another page immediately shows the old permission as if the write did not happen. Consistency is not one switch; it is a promise attached to a particular operation and user expectation.

Decision Framework: CP vs AP

Use this framework when you compare CP and AP architectural patterns for a specific business requirement. Start with the invariant, not with the technology. Ask what must never be temporarily false, what can be corrected later, who is harmed by rejection versus stale acceptance, and whether the system has a reliable merge or compensation path after a partition heals.

Requirement	Prefer CP when…	Prefer AP when…	Design consequence
Scarce resource ownership	Two actors must never own the same resource at the same time.	Temporary duplicate claims can be detected and resolved without serious harm.	CP usually needs consensus, leases, or a single authoritative writer.
Money or entitlement change	A wrong acceptance creates legal, financial, or security exposure.	Temporary local acceptance can be reversed or limited safely.	CP favors rejecting some requests during partition instead of accepting unsafe state.
User-facing read latency	Users can tolerate waiting for an authoritative answer.	Users value fast local reads more than perfectly fresh data.	AP often pairs asynchronous replication with visible freshness or conflict handling.
Telemetry and events	Losing or duplicating data would corrupt an invariant.	Approximate, delayed, or duplicate-tolerant data is acceptable.	AP ingestion can buffer locally and reconcile downstream with deduplication.
Collaboration and offline work	Conflicts cannot be merged meaningfully.	Users can work locally and conflicts have domain-specific merge rules.	AP designs need versioning, conflict display, or CRDT-like structures.

The framework deliberately separates business truth from implementation taste. A team may prefer a fast AP database, but if the invariant is “only one active cluster leader can mutate this state,” the design probably needs CP coordination. A team may prefer strong consistency everywhere, but if the workload is high-volume metrics ingestion, forcing every write through a quorum may spend latency and availability on precision the product does not need.

Patterns & Anti-Patterns

Patterns are recurring design moves that make distributed systems survivable. Anti-patterns are recurring shortcuts that make a system appear simpler by hiding uncertainty until production reveals it. The goal is not to apply every pattern everywhere. The goal is to match the pattern to the failure mode and to know which anti-pattern a design review should challenge immediately.

Pattern	Why it helps	Example application
Design for partial failure	Keeps one unhealthy dependency from turning into a total outage.	A checkout path accepts the order, queues email, and lets email recover independently.
Make operations idempotent	Allows safe retry after ambiguous timeout or lost response.	Payment, order creation, and provisioning APIs store a client request key.
Use bulkheads	Limits blast radius by isolating resources for different dependencies or tenants.	Separate worker pools prevent slow search indexing from consuming checkout workers.
Embrace asynchrony	Removes unnecessary synchronous coupling from user-facing paths.	A user upload returns after durable storage, while thumbnails and search indexing run later.
Apply timeouts, backoff, and jitter	Bounds waiting and prevents synchronized retry storms.	Clients retry transient failures with capped exponential backoff and randomized delay.

Anti-pattern	Why it is dangerous	Better approach
Assuming the network is reliable	Converts lost, delayed, or duplicated messages into corrupt user-visible state.	Treat every remote side effect as ambiguous until confirmed or reconciled.
Building synchronous chains across many services	Makes end-to-end availability the product of every hop in the chain.	Collapse unnecessary calls, cache carefully, batch, or move work to asynchronous flows.
Creating a distributed monolith	Keeps tight coupling while adding deployment, network, and ownership failure modes.	Draw service boundaries around stable ownership and data invariants, not around nouns alone.
Ignoring clock skew	Makes event ordering, lease expiry, and last-writer-wins behavior unsafe.	Use monotonic clocks for durations and logical or version-based ordering for causality.
Retrying without idempotency	Turns transient failures into duplicate charges, duplicate jobs, or duplicate state changes.	Store operation keys and make duplicate requests return the original logical result.

One practical review habit is to ask for the failure story of each service boundary. If the boundary exists only because it looks tidy on a diagram, it may be a distributed monolith boundary. If the boundary exists because a team owns a durable invariant, the contract is versioned, and failure behavior is explicit, the distribution is more likely to pay for itself.

Common Mistakes

Mistake	Problem	Solution
Treating an HTTP timeout as proof that no work happened	The remote service may have committed the side effect before the response was lost.	Use idempotency keys, reconciliation reads, and clear unknown-outcome handling.
Splitting services before defining data ownership	The system gains network failure modes while teams still fight over the same invariants.	Define the authoritative owner for each state transition before creating a service boundary.
Calling many services synchronously in one user request	One slow or unavailable dependency can stall the entire request path.	Budget latency per hop, batch calls, cache carefully, or move noncritical work behind a queue.
Assuming health checks prove user-path health	A component can pass a shallow probe while failing a real dependency or data path.	Use readiness checks, synthetic transactions, and dependency-specific observability.
Using retries without backoff or jitter	Clients can amplify load and prevent a recovering dependency from stabilizing.	Cap retries, add exponential backoff, add jitter, and stop retrying when the caller deadline is spent.
Treating CAP as “pick two forever”	The real trade-off happens during partitions, while normal operation also has latency-consistency choices.	Decide per invariant how the system behaves during partition and how it trades latency against freshness otherwise.
Relying on wall clocks for distributed ordering	Clock skew can make a later write appear earlier or expire a lease incorrectly.	Use monotonic clocks for durations and logical versions, consensus, or causality tracking for ordering.
Hiding partial failure from users and operators	Ambiguous state becomes harder to reconcile when the system pretends everything is fine.	Expose pending, retrying, degraded, and unknown states with clear operational paths.

Quiz

Scenario: A checkout API times out while calling the payment service. The payment service may have charged the customer, but the API did not receive the response. What should the checkout API do before retrying?

Answer

The checkout API should implement idempotent operations and retry mechanisms by sending the same stable request key with each attempt. That design handles transient network failures because the payment service can return the original receipt instead of creating a second charge. This answer probes the outcome to implement idempotent operations and also reinforces that a timeout is not proof that the remote side did nothing.
Scenario: A team proposes a user request path that calls twelve services in sequence before returning a page. Each service has good median latency in isolation. What distributed-systems fallacy should you diagnose first?

Answer

You should diagnose the eight fallacies of distributed computing, especially “latency is zero” and “the network is reliable.” The architecture hides a chatty synchronous chain where tail latency and partial failure compound across every hop. A better design would set an end-to-end latency budget, collapse unnecessary calls, batch data access, or move noncritical work to asynchronous paths.
Scenario: During a zone-level network issue, half of a replicated inventory system cannot reach the other half. The business rule says one serialized item must never be sold to two customers. Should this operation prefer CP or AP behavior?

Answer

This operation should prefer CP behavior because the business requirement cannot safely tolerate conflicting ownership of the same scarce item. During a partition, a CP design may reject or delay writes that cannot reach quorum, sacrificing some availability to preserve consistency. This answer probes the outcome to compare CP and AP architectural patterns by starting from the invariant rather than from database branding.
Scenario: A metrics pipeline accepts duplicate events and deduplicates them later by event ID. During a partition, local collectors can keep buffering and accepting events. Why might AP behavior be acceptable here?

Answer

AP behavior can be acceptable because the business requirement values continued ingestion and can tolerate temporary divergence. The system has a merge path: duplicate events are identified later by event ID, and delayed events can still contribute to aggregate views. This does not mean consistency is irrelevant; it means the chosen consistency model matches a workload where availability and low-latency local acceptance matter more than immediate global agreement.
Scenario: A service’s readiness probe returns success because the process can answer HTTP, but real requests hang whenever they need the database. What kind of failure is this, and how should you evaluate the architecture?

Answer

This is a gray failure, a form of partial failure where the component is alive enough to pass some checks but broken for meaningful work. You should evaluate the architecture by identifying which components introduce distributed-system challenges such as partial failure, dependency-specific health, and ambiguous user-path state. Better readiness checks and synthetic transactions can reduce the blind spot, but they still cannot create perfect failure detection.
Scenario: A platform team wants to split a large service into five microservices because smaller repositories feel cleaner. The same team will still own all data, releases, and invariants. What risk should you call out?

Answer

The risk is a distributed monolith: the design adds network communication, partial failure, deployment ordering, and operational overhead without gaining independent ownership or clearer invariants. You should design service boundaries that account for the unreliability of network communication between components and only distribute when the boundary pays for itself. A better first step may be modularizing the codebase while keeping one deployable service until ownership and data boundaries are real.
Scenario: A product manager asks why a cross-region write cannot be both instantly visible everywhere and always accepted during a wide-area partition. How would you explain the trade-off without using the sloppy “pick two” phrase?

Answer

Explain that partition tolerance is not a menu option once the system spans unreliable networks; a partition can be imposed by the environment. During that partition, the system must choose whether to preserve consistency by rejecting or delaying some operations, or preserve availability by accepting operations that may need later reconciliation. PACELC adds that even without a partition, the system often trades lower latency against stronger consistency when deciding how many replicas must acknowledge a write before the user sees success.

Hands-On Exercise

In this exercise, you will observe partial failure in a local Kubernetes cluster. The point is not that Kubernetes makes the problem disappear; the point is that Kubernetes gives you controllers and Services that can mask one pod failure while the distributed system converges. You will create a three-replica Deployment, expose it through a Service, delete one pod, and watch the Service remain available while the ReplicaSet creates a replacement.

Prerequisites: A working local Kubernetes cluster such as kind or another disposable test cluster, plus kubectl configured for that cluster. Use a namespace dedicated to the lab so cleanup is safe.

kubectl create namespace ds-lab
kubectl -n ds-lab create deployment catalog --image=nginx:1.27-alpine --replicas=3
kubectl -n ds-lab expose deployment catalog --port=80 --target-port=80
kubectl -n ds-lab rollout status deployment/catalog
kubectl -n ds-lab get pods -l app=catalog -o wide

Now remove one running pod and immediately observe both the pod list and the Service path. A Service gives clients one stable name, while EndpointSlices and readiness determine which pod IPs receive traffic.

POD_NAME="$(kubectl -n ds-lab get pod -l app=catalog -o jsonpath='{.items[0].metadata.name}')"
kubectl -n ds-lab delete pod "$POD_NAME" --wait=false
kubectl -n ds-lab get pods -l app=catalog -w

In a second terminal, probe the Service while the replacement pod is being created. If your cluster is slow to pull images, wait for at least two pods to be Ready before judging the Service behavior.

kubectl -n ds-lab run catalog-probe \
  --image=curlimages/curl \
  --rm -it --restart=Never \
  -- http://catalog.ds-lab.svc.cluster.local

kubectl -n ds-lab get endpointslice \
  -l kubernetes.io/service-name=catalog \
  -o wide

Clean up the namespace after you finish. The cleanup command removes the Deployment, Service, pods, and temporary probe pod created for this exercise.

kubectl delete namespace ds-lab

Success Criteria:

The initial Deployment reaches three Ready replicas behind one Service name.
Deleting one pod creates a visible partial failure while at least part of the system keeps serving.
The ReplicaSet creates a replacement pod and the Service returns to three healthy endpoints.
You can explain why the client saw one Service even though the backing pod set changed.

Sources

Next Module

Next, continue to Module 5.2: Consensus and Coordination, where we study how distributed systems use quorums, leaders, and replicated logs to make a group of unreliable nodes agree on one ordered history.