How to distribute the primary shards more evenly across multiple nodes?

CrateDB
1

I have stored Prometheus monitoring data in CrateDB 5.10.1, with the table name being doc.metrics . After rolling the upgrade to version 5.10.2, I noticed that most of the primary shards are concentrated on a single node. I checked the documentation that I can use ALTER TABLE REROUTE for specific partitions, but it seems inefficient. Is there a better way to achieve this?
The upgrade process are as follows.

Step 1. SET GLOBAL TRANSIENT "cluster.routing.allocation.enable" = 'new_primaries';
Step 2. systemctl stop crate
Step 3. apt-get install crate=5.10.2
Step 4. systemctl start crate
Step 5. After upgrading all nodes.
SET GLOBAL TRANSIENT "cluster.routing.allocation.enable" = 'all';

image
image2722×1352 162 KB

2

Hi @Jun_Zhou,

This type of distribution can happen when restarting nodes. CrateDB equally balances the total number of shards across nodes, but doesn’t differentiate between primary and replica shards.

In terms of high availability, this won’t cause any problems. Both primary and replica shards acknowledge writes before those are considered successful. Replica shards will take over the role of a primary shard if needed.

Over time, as relocations happen (e.g. on the deletion of partitions, further node restarts, etc.) the situation will likely change and show a less concentrated picture.

Under certain conditions, these primary shard concentrations can have an impact on write performance, this is tracked as an improvement on GitHub at crate/crate#15919.

Do you see any operational impact on your system since the upgrade?

Best
Niklas

3

Thanks @hammerhead, currently, there are no obvious issues. I’m just curious why the primary shards are so concentrated in their distribution, as I am concerned about the future performance of the cluster.