Unconventional Introduction
It is midnight, I am sitting in my room in an apartment in Lagos, Nigeria. No coffee, caffiene or any stimulant. Just me, grit, my laptop, and the task of understanding distributed systems and consensus to take a step futher into my journey to becoming a protocol engineer.
I have studied these phase before, I have a good understanding of the concepts, I have also worked on personal projects as relating to these topics. However, learning is not a one-time event, it is a continuous process. I want to take this opportunity to understand these concepts in a more in-depth manner, and to also share my understanding with others.
Core Concepts
1. Distributed System
Distributed system is a collection of independent computers called nodes that work together to perform a task or set of tasks. These nodes communicate with each other and acts as a single cohorent system to the user.
2. Consensus
Consensus is the process of reaching agreement on a single value, decision, or state in a distributed system.
3. Crash Fault
Crash fault is when a node in a distributed system crashes or fails to respond. It is usually as a result of hardware failure, software failure, network failure, or power failure. It is a fail-stop fault.
4. Byzantine Fault
Byzantine fault is a when one or more nodes in a distributed system is rouge or corrupt exhibiting arbitirary or malicous behaviour such as sending incorrect information or intentionally disrupting communication. It is a fail-arbitrary fault.
5. Fault Tolerance
Fault tolerance is a system property that enables a distributed system to function correctly even when components fail. In distributed systems, this is achieved by using consensus mechanisms to agree on a consistent state despite failures, network partitioning, or data corruption.
Types of Fault Tolerance
- Crash Fault Tolerance (CFT)
- Byzantine Fault Tolerance (BFT)
6. Crash Fault Tolerance (CFT)
Crash Fault Tolerance is a property of systems that can handle nodes crashing gracefully, ensuring the system remains reliable and data stays consistent across all healthy nodes.
7. Byzantine Fault Tolerance (BFT)
Byzantine Fault Tolerance is a property of systems that can handle nodes exhibiting Byzantine (arbitrary) faults, ensuring the system remains reliable and consistent even if some nodes act maliciously or fail unpredictably.
8. FLP Impossibility Theorem
The FLP Impossibility Theorem states that in a purely asynchronous network, it is impossible for a deterministic consensus algorithm to achieve both safety and liveness if there is even one node crash failure.
Safety (Agreement)
All non-faulty nodes agree on the same valid value. "Nothing bad happens."
Liveness (Termination)
Nodes eventually reach a decision; the system continues to make progress. "Something good eventually happens."
Fault Tolerance
The algorithm must remain functional despite node failures.
9. CAP Theorem
This theorem states that in a distributed data store, it is impossible to simultaneously provide more than two out of the following three guarantees:
Consistency
Every read receives the most recent write or an error.
Availability
Every request receives a (non-error) response, without the guarantee that it contains the most recent write.
Partition Tolerance
The system continues to operate despite an arbitrary number of messages being dropped or delayed by the network between nodes.
10. Throughput vs Latency vs Bandwidth
Throughput is the rate at which successful operations are completed over a specific duration (e.g., 10,000 transactions per second).
Latency is the time it takes for a single request to be processed and for the response to return (e.g., 50ms).
Bandwidth is the maximum rate of data transfer across a network path (e.g., 1 Gbps). It is about the "width" of the pipe, while throughput is about how much "water" is actually flowing.
11. Asynchronous vs Synchronous Model
Synchronous Model assumes that message transmission and processing delays are bounded by a known constant. This allows for timeouts to reliably detect failures.
Asynchronous Model assumes that delays are unbounded (finite but unknown). In this model, you cannot distinguish between a crashed node and a very slow network link.
Partial Synchrony (The Middle Ground): The system behaves asynchronously for periods but eventually hits a "Global Stabilization Time" (GST) after which it behaves synchronously. Most real-world BFT systems (like Tendermint) rely on this.
12. Network Partition
A network partition is a communication failure between nodes, splitting the system into isolated sub-groups. It is a type of fault that tests the system's "Partition Tolerance."
Consensus Algorithm
1. Paxos
A foundational (and notoriously complex) consensus algorithm for CFT. It ensures agreement on a single value through a multi-phase process (Prepare, Promise, Accept, Accepted).
2. Raft
A CFT consensus algorithm designed to be more understandable than Paxos. It achieves consensus by electing a strong leader who manages a replicated log. If the leader fails, a new election is held.
3. Practical Byzantine Fault Tolerance (PBFT)
One of the first efficient solutions for BFT in asynchronous systems. It uses a three-phase protocol (Pre-prepare, Prepare, Commit) and can tolerate $m$ malicious nodes if the total nodes $n \geq 3m + 1$.
4. Tendermint
A modern BFT consensus engine that uses a simplified version of PBFT and is widely used in the Cosmos ecosystem. It operates under a partial synchrony model.
Consensus Problem And Probable Solutions
Fault
- Crash Fault
- Byzantine Fault
- Network Partition
Probable Solutions
Consensus with at most $m$ crash faults
At least $2m + 1$ nodes are required to reach consensus. This ensures that even if $m$ nodes fail, a majority of $m + 1$ nodes is still available to make progress (Quorum).
Consensus with at most $m$ byzantine faults
At least $3m + 1$ nodes are required. In a BFT environment, you need a $2/3$ majority $(2m + 1)$ to overcome the possibility of $m$ malicious nodes colluding with or confusing $m$ honest but slow nodes.
Consensus with at most $m$ network partitions
This generally follows the CFT requirement of $2m + 1$ total nodes to ensure that only one partition can ever hold a majority (Quorum) at a time, preventing "Split-Brain" scenarios.
Misconceptions
1. Distributed systems And Microservices
Distributed systems are multiple independent nodes working together and appear as a single system.
Microservices is an architectural patten where an application is divided into smaller, independent services that communicate with each other over a network.
All microservices are distributed systems, but not all distributed systems are microservices.