Crate Admin UI couldn't view cluster status

CrateDB
1

Our crate 5.6.2 cluster has five nodes that three nodes are master and five nodes also as data nodes. Now I couldn’t view cluster status in Admin UI, while nodes status are normal by using systemctl status crate command, and also can insert data into cratedb, the crate-infra.log didn’t show any error logs. Any suggestion would be appreciated. Thanks in advance.

crate.yml

path.data: /data/crate
network.host: 0.0.0.0
discovery.seed_hosts:
  - t-cratedb-s-001
  - t-cratedb-s-002
  - t-cratedb-s-003
  - t-cratedb-s-004
  - t-cratedb-s-005
cluster.initial_master_nodes:
  - t-cratedb-s-001
  - t-cratedb-s-002
  - t-cratedb-s-003
auth.host_based.enabled: true
auth:
  host_based:
    config:
      0:
        user: crate
        address: _local_
        method: trust
      99:
        method: password
cluster.name: crate-infra
node.name: "p-cratedb-s-00x"
gateway.recover_after_data_nodes: 5
gateway.expected_data_nodes: 5

image
image2932×1538 180 KB

2

What do the logs say for each of the nodes after startup? (I suspect no cluster has been formed yet.)
Could you explain what install / deployment method you used?
On which host are the nodes running on?
How do you connect to the Admin UI?

4

The cluster has been formed about one month. We installed crate cluster on ubuntu by apt install crate command. The crate services are running on five nodes now.

image
image1920×596 144 KB

I connect to Admin UI by http://10.10.22.212:4200.

All nodes crate-infra.log are uploaded.
crate-infra-002node.log (3.0 MB)
crate-infra-005node.log (18.7 KB)
crate-infra-004node.log (18.7 KB)
crate-infra-003node.log (37.4 KB)
crate-infra-001node.log (20.1 KB)

5

Thank you for sharing those details.
Did this problem appear immediately after upgrading?
Did you do a rolling upgrade with ongoing data ingestion?
What version was the cluster running before?
Did you change/reset cluster.routing.allocation.enable as part of this process?

6

@proddata @hernanc I have locate the root cause that the login user crateadmin missing admin privileges. After granting all privileges to crateadmin, then Admin UI show cluster status. But I’m curious why the privileges of crateadmin suddenly gone.

image
image3000×1608 266 KB

1 Like
7

I’m curious whether the issue you’re encountering could be related to version 5.6 of CrateDB, which introduced ROLEs, significantly altering privilege management. To delve into your specific situation, we need details on how you managed the upgrade process to identify any potential problems that our test cases might have missed.

8

I also think the issue may be related to the upgrade that’s the only change before encountering the issue. The complete upgrade process are as follows.
Step 1. shutdown all nodes
Step 2. apt upgrade crate
Step 3. start all nodes

9

@Jun_Zhou What was the version you were running before the upgrade?

10

Hi @matriv , The version is 5.5.3.

11

Apologies for this long delay, the issue was finally identified and is fixed with: Fix role granting to not loose existing privileges. by seut · Pull Request #16713 · crate/crate · GitHub It will be available with the upcoming 5.8.4 relase