An Introduction to Distributed Systems Concepts

At some point, a single server stops being enough. You need more throughput, or you need to survive failures, or you need to serve users across the globe. The moment you have more than one computer cooperating to do a job, you are building a distributed system — and you inherit all its rules. This article is a foundation: the vocabulary, the hard problems, and the mental models that make the field navigable.

Why distributed systems are hard

The core difficulty is not scale. It’s that the components fail independently, and there is no global clock. Two machines that think they hold the same state can disagree, and you can’t just “check the time” to sort it out, because their clocks aren’t synchronized.

This is the source of nearly every distributed systems bug: you design for a world where things are consistent, and the world silently stops being consistent.

The fallacies of distributed computing

Coined by Peter Deutsch at Sun Microsystems, the “Eight Fallacies” are the assumptions that break beginners:

1. The network is reliable
2. Latency is zero
3. Bandwidth is infinite
4. The network is secure
5. Topology doesn’t change
6. There is one administrator
7. Transport cost is zero
8. The network is homogeneous

Almost every distributed systems incident traces back to one of these being treated as true.

Replication: the basic building block

Replication means keeping copies of your data on multiple machines. It buys you availability (a copy survives a machine) and read throughput (reads can hit any copy). The question replication raises is: what happens when copies diverge?

Two models:

• Leader-based replication (primary/secondary): all writes go to one leader; followers copy the log. Simple and predictable.
• Leaderless replication (Dynamo-style): writes go to several nodes; quorum protocols decide consistency.

Write → leader ──→ follower A
              ──→ follower B
              ──→ follower C

Read  → any replica (asynchronous: may be slightly stale)

Consistency: synchronous vs asynchronous

With synchronous replication, a write is acknowledged only after followers confirm — durable, but slower, and one slow follower slows every write.

With asynchronous replication, the leader acknowledges immediately and followers catch up in the background — fast, but a leader crash right after acknowledging can lose data.

Real systems usually use a hybrid. PostgreSQL’s synchronous_commit = remote_write, Kafka’s min.insync.replicas, and Redis replication modes are all dials on this same axis: how much latency you’ll trade for how much durability.

The CAP theorem: a lens, not a law

CAP says a distributed system under network partition can choose only two of three: Consistency, Availability, and Partition tolerance. Because partitions will happen, the real choice is between C and A during a partition.

• CP (e.g., ZooKeeper, etcd, most databases under strong settings): when the network splits, some nodes refuse to serve rather than risk serving stale data.
• AP (e.g., Cassandra, DynamoDB): when the network splits, nodes keep serving, and consistency is eventually restored.

        Network partition happens
        ├── CP: "I can't guarantee freshness, so I won't serve."  → some requests fail
        └── AP: "I'll serve what I have, and reconcile later."    → some reads are stale

Treat CAP as a question to ask about your specific operation, not a label for your whole system. An ordering system is CP-critical for balances; its analytics dashboard is happily AP.

Consensus: getting agreement

Once you replicate, you eventually need nodes to agree — on who’s the leader, on the order of operations, on the value of a counter. That’s consensus, and it’s the deepest rabbit hole in the field.

Protocols like Paxos and Raft solve this: they let a set of machines agree on a value even when some fail or the network partitions, provided a majority is reachable. Raft is the one most engineers actually read, because it’s designed for understandability:

• Leader election: nodes vote; the winner leads
• Log replication: the leader appends entries and replicates to a majority
• Safety: once an entry is committed, it’s never lost or reordered

raft cluster (5 nodes)
  leader  ───→ 3 followers
  commit rule: entry committed after a MAJORITY of nodes store it
  majority = floor(n/2) + 1 = 3  for a 5-node cluster

A 3-node cluster tolerates one failure; a 5-node cluster tolerates two. Odd numbers are deliberate: 3 nodes get you majority with only 2 confirmed writes.

Exactly-once semantics and its cousins

Delivery guarantees are another fundamental vocabulary:

• At-most-once: the message may be lost, but never duplicated
• At-least-once: the message is never lost, but may be delivered twice
• Exactly-once: neither lost nor duplicated — the most expensive to guarantee

Most systems you build should aim for at-least-once plus idempotency: design your consumers so processing the same event twice is harmless. Idempotency keys on API requests are the simplest practical example:

POST /api/charges
Idempotency-Key: 4a7c-11ec-... 

# Retry the same request with the same key → same result, no double charge

Time, ordering, and the sad truth about clocks

You cannot build correct systems on wall-clock time. Clocks drift, leap seconds happen, and NTP corrections jump time backward. Distributed systems use:

• Logical clocks (Lamport timestamps): a counter that orders events causally
• Vector clocks: capture which events causally precede others, enabling merge

The practical lesson: never use timestamps as the source of truth for ordering, and treat “time” as a monotonic clock when you need ordering, and a wall clock only when you need a human-readable label.

When you actually need a distributed system

A healthy warning: most services do not need to be distributed. A single Postgres instance with good indexes, connection pooling, and read replicas serves astonishing traffic. Distributed systems add operational complexity (and debugging pain) that is only worth it when:

• You need to survive a full datacenter/region failure
• Read or write throughput genuinely exceeds one machine
• You need geographic latency you can’t get with caching alone

Conclusion

Distributed systems are not a technology choice; they are a set of constraints that arrive the moment you leave a single machine. Learn to speak the vocabulary — replication, consensus, quorum, idempotency, logical clocks — and you’ll stop being surprised by the failures, which is more than half the battle. When in doubt, start single-machine, measure, and distribute only what the numbers demand.