It doesn't work after increasing total_shards_per_node

CrateDB
1

Hi team, I’m using CrateDB 5.10.3 and created more than 150 tables with default value 8 of number_of_shards, so that I encounter below error. I try to set "cluster.routing.allocation.total_shards_per_node" = '5000'; and decrease shards of table, while the error still exist. Did I make wrong configurations?

cr> SET GLOBAL TRANSIENT "cluster.routing.allocation.total_shards_per_node" = '5000';
SET OK, 1 row affected (0.065 sec)

cr> ALTER TABLE spl.state_machine_def SET (number_of_shards = 4);
SQLParseException[Validation Failed: 1: this action would add [4] total shards, but this cluster currently has [4000]/[4000] maximum shards open;]

io.crate.exceptions.SQLParseException: Validation Failed: 1: this action would add [4] total shards, but this cluster currently has [4000]/[4000] maximum shards open;

Caused by: java.sql.SQLException: Exception chain:
org.postgresql.util.PSQLException: ERROR: Validation Failed: 1: this action would add [8] total shards, but this cluster currently has [4000]/[4000] maximum shards open;
2

You might want to adjust cluster.max_shards_per_node
e.g. SET GLOBAL PERSISTENT "cluster.max_shards_per_node" = '1250';

also see:

However there should be a very good reason, if you want to go that high with shards, and you might want to reconsider your general table setup.

1 Like
3

Thanks @proddata. I successfully adjust the max_shards_per_node as your suggestion, but I couldn’t decrease the shards of table.

cr> SET GLOBAL PERSISTENT "cluster.max_shards_per_node" = '1500';
SET OK, 1 row affected (0.078 sec)
cr> ALTER TABLE spl.state_machine_def SET ("blocks.write"=true);
ALTER OK, -1 rows affected (0.195 sec)
cr> ALTER TABLE spl.state_machine_def SET ("number_of_shards" = 4);
IllegalStateException[index spl.state_machine_def must have all shards allocated on the same node to shrink index]

java.lang.IllegalStateException: index spl.state_machine_def must have all shards allocated on the same node to shrink index
        at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.validateShrinkIndex(MetadataCreateIndexService.java:589)
        at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.prepareResizeIndexSettings(MetadataCreateIndexService.java:638)
        at org.elasticsearch.cluster.metadata.MetadataCreateIndexService$IndexCreationTask.execute(MetadataCreateIndexService.java:436)
        at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45)
        at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:665)
        at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:280)
        at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:175)
        at org.elasticsearch.cluster.service.MasterService.lambda$doStart$0(MasterService.java:121)
        at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:170)
        at org.elasticsearch.cluster.service.BatchedTask.run(BatchedTask.java:70)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.lang.Thread.run(Thread.java:1575)