MODULE 07 · SYSTEM ARCHITECTURE

Fault Tolerance

Fault tolerance lets a system continue operating or degrade acceptably when components fail.

01 · What is it?

Fault tolerance is the ability to keep providing useful service when a component, dependency, or operating path fails.

The system may use replicas, checksums, retries, quorum rules, or spare components so the failure is absorbed within the operation. If the normal path cannot continue, the product may safely reduce optional capability, queue the work, or show a clear pending state. Fault tolerance is always bounded by the failures the design was built to handle.

The goal is not to hide every failure. The goal is to protect the most important user and business outcome while the system continues or degrades acceptably.

02 · Think of it like...

Think of a plane with duplicated controls and instruments: one instrument can fail while the pilot still has enough information to fly.

Normal Operation

Component Fails

Fallback Path

Operation Continues

03 · What does it do?

Absorb a failure

Use a spare or alternate path without stopping the whole operation.

Protect correctness

Use validation, checksums, or quorum rules to avoid silently accepting bad results.

Degrade acceptably

Keep the core journey working while an optional feature becomes slower, simpler, or temporarily unavailable.

Make limits explicit

State which failures are covered and what happens when more failures occur.

These protections have trade-offs: replicas and spare capacity cost more, retries can increase load, and a fallback may provide less functionality. A PM should choose the user promise before choosing the mechanism.

04 · Where is it used?

Storage systems

Keep multiple copies or encoded fragments so one disk or node can fail without losing data.

Distributed services

Use retries, timeouts, and fallback behavior for a defined dependency failure.

Payments and orders

Prevent duplicate or lost operations through idempotency, durable records, and safe replay.

Infrastructure

Use redundant power, networks, or hosts when an uninterrupted operation is required.

05 · Example

A payment service receives a timeout after it has sent a charge request. A fault-tolerant design does not blindly charge again.

The payment request has an idempotency key.
The service checks the provider or its durable record before retrying.
If the provider is temporarily unavailable, the order is held safely instead of being marked paid without proof.
The customer sees a clear pending state while recovery continues.

PM Perspective

Ask what failure is covered, what data must remain correct, and whether the user sees a retry, a delay, or an error. Fault tolerance is not permission to hide every failure.

06 · How it connects

Request

Dependency

Failure Signal

Retry / Fallback

Safe Result

Related concepts include redundancy, failover, graceful degradation, timeouts, retries, idempotency, queues, backups, and disaster recovery. Each handles a different part of the failure story; none is a complete substitute for the others.

PM Perspective: Ask which component can fail, what state must remain correct, how long the user can wait, and what the product should say when the normal path is unavailable.

07 · Do Not Confuse

Fault Tolerance

Continue a defined operation through a defined failure, often with little visible interruption.

High Availability

Reduce downtime by detecting failures and moving service to a healthy path. Failover may be visible briefly.

Reliability

The broader probability that a system performs correctly over time.

Disaster Recovery

Restore service after a larger incident that exceeds the normal fault model.

Retry

One recovery technique. Repeating a request is safe only when the operation and its side effects are understood.

Backup

A recoverable copy of data, not a live replacement for the failed service.

08 · Topic Glossary

ENGLISH TERM	中文	SIMPLE ENGLISH	中文小白理解
Fault Tolerance	容错	Continuing a defined operation when a known component fails.	已知某个部件会坏时，系统仍能继续完成规定工作。
Failure Model	故障模型	The failures a design is expected to handle.	系统设计时预先考虑要应对哪些故障。
Fallback	备用路径	An alternate way to complete or safely pause work.	完成工作或安全暂停工作的另一条路径。
Idempotency	幂等性	Repeating an operation does not create an unintended extra effect.	同一个操作重复执行，不会意外多做一次。
Quorum	法定数量	The minimum number of agreeing members needed for a decision.	做决定时必须同意的最少成员数量。
Checksum	校验和	A value used to detect changed or corrupted data.	用来发现数据是否被改动或损坏的值。

09 · What to Remember

Fault tolerance is about continuing through a defined failure, not pretending every failure can be hidden.

A PM should ask which failure model is covered, how correctness is protected, and what the user sees when the limit is reached.