Maximum High Availability with acceptable data loss

Community
,
1

Hi,

Are there settings which would create a CrateDB cluster with 3 nodes, so that even if 2 nodes go down, the 1 remaining node retains the write (and read) capability?

Similarly, for a 5-node cluster, can only 2 nodes provide read and write functionality?

The primary goal is to achieve maximum high availability and downtime resiliency.

The data needs to be replicated to all available nodes.

If the write-to node goes down before its recent received data gets replicated throughout the cluster, it is an acceptable loss.

Is this hyper-HA setup possible?

2

Hi @Egor_Sharin,

Welcome to our community!

The core question is around the voting configuration for the different scenarios you described. The concept behind that is described in our documentation: Clustering - CrateDB: Reference

To elect a master among the eligible nodes, a majority (floor(half)+1), also known as quorum, is required among a subset of all master-eligible nodes, this subset of nodes is known as the voting configuration.

For your concrete examples, this means:

From 3 to 2 to 1

Starting from three nodes, floor(3 / 2) + 1 = 2 which means that the loss of one node can be tolerated. After that, the cluster has become a two-nodes cluster, and floor(2 / 2) + 1 = 2. Therefore, the cluster won’t continue operating below two nodes. This is the classical split-brain scenario that distributed systems need to avoid, as there is no way for the remaining node to tell if the other node truly stopped operating or may think that is is the survivor itself and also would keep accepting new data. Such a conflict would not be resolvable when later trying to restore the cluster.

From 5 to 2

Here, floor(5 / 2) + 1 = 3, so the cluster could loose two nodes at the same time.
See also this remark from the documentation:

For instance, if you have a 5-nodes cluster, with all nodes master-eligible, and node 1 is currently the master, and you shutdown node 5, then node 4, then node 3, the cluster will stay available as the voting configuration will have adapted to only have nodes 1, 2, and 3 on it.

The timing of an incident has significant impact. Loosing first two nodes at the same time, and then a few minutes later another node (5 → 3 → 2), is a different scenario compared to loosing three nodes all at once (5 → 2).