What is Failure Tolerance?

Failure tolerance, on the other hand, involves designing the system in such a way that it can endure failures without significant loss of functionality or data. While the failures may be detected and sometimes visible to users, the system is capable of continuing its operations, possibly in a degraded mode. The system is built to manage, recover from, and adapt to failures.

• Purpose:
  • Failure tolerance aims to ensure that the system can continue operating even in the presence of failures, albeit possibly at a reduced capacity or performance level.
  • Unlike failure masking, failures may be detected, but the system is resilient enough to handle them without complete shutdown.
• Strategies:
  • Graceful Degradation: The system is designed to degrade gracefully when facing failures. This means that certain non-critical functionalities may be temporarily disabled or scaled back to prioritize core operations.
  • Fault Isolation: Systems are compartmentalized or modularized in such a way that if one component fails, it doesn’t bring down the entire system. Failures are contained within their respective modules, allowing other parts of the system to continue functioning.
  • Automatic Recovery: Automated mechanisms are implemented to detect and recover from failures without manual intervention. This could involve restarting failed processes, restoring data from backups, or rerouting traffic to healthy components.
• Example: In a distributed database system, if one node fails, the system can continue to serve read requests using data from other nodes while the failed node is being repaired or replaced.

In distributed systems, dealing with failures is a critical aspect of design and implementation. Since these systems consist of multiple interconnected components, the likelihood of failures increases. Two primary approaches to handling these failures are masking and tolerating them. This article explores the differences between these approaches, their techniques, and their use cases.