What distributed systems are and why they are hard

What a distributed system is

A distributed system is a set of separate computers, called nodes, that cooperate over a network and appear to users as a single service. When you search the web, stream a film or pay by card, dozens or thousands of machines handle your request. You never see them individually; you just see an answer.

Organisations build distributed systems for three main reasons:

• Scale: one server can only hold so much data and handle so many requests. Adding more machines (scaling out) raises the ceiling.
• Availability: if one machine fails, others keep the service running.
• Latency: placing nodes near users in different regions cuts response time.

Why they are hard

On one computer, a function call either works or throws an error, and every part of the program shares the same memory and clock. Across a network, none of that holds.

• The network is unreliable. Messages can be delayed, duplicated, reordered or lost. If you send a request and hear nothing, you cannot tell whether the request was lost, the server crashed, or the reply is still on its way.
• Partial failure. Some nodes can be down while others are up. A disk can fill on one machine while the rest run normally. The system must keep working in a half-broken state.
• No global clock. Each machine has its own clock, and they drift apart. Even with NTP (Network Time Protocol) you cannot fully trust timestamps to order events across machines.
• Concurrency. Many clients update shared data at the same time, so conflicting writes must be detected and resolved.

The fallacies of distributed computing

A well-known list from engineers at Sun Microsystems describes false assumptions that newcomers make. In plain terms, do not assume that:

Designing for failure

Good distributed designs accept that failure is normal and plan for it. Common techniques include:

1. Use timeouts on every remote call so a slow node cannot hang the caller.
2. Retry with exponential backoff and jitter, and make operations idempotent so a repeated request does no harm.
3. Replicate data so losing one node does not lose data (see replication and sharding).
4. Use health checks and load balancers to route around failed nodes.
5. Choose the right consistency level for each piece of data (see CAP and PACELC).

Tip: in exams and interviews, the phrase "partial failure" is the single best summary of why distributed systems are harder than single-machine programs.

At work, you meet these ideas when a microservice times out, a database replica lags behind, or two data centres disagree about the latest value. Recognising the underlying cause makes troubleshooting much faster.