Crate Admin UI couldn't view cluster status

Our crate 5.6.2 cluster has five nodes that three nodes are master and five nodes also as data nodes. Now I couldn’t view cluster status in Admin UI, while nodes status are normal by using systemctl status crate command, and also can insert data into cratedb, the crate-infra.log didn’t show any error logs. Any suggestion would be appreciated. Thanks in advance.

crate.yml /data/crate
  - t-cratedb-s-001
  - t-cratedb-s-002
  - t-cratedb-s-003
  - t-cratedb-s-004
  - t-cratedb-s-005
  - t-cratedb-s-001
  - t-cratedb-s-002
  - t-cratedb-s-003
auth.host_based.enabled: true
        user: crate
        address: _local_
        method: trust
        method: password crate-infra "p-cratedb-s-00x"
gateway.recover_after_data_nodes: 5
gateway.expected_data_nodes: 5

What do the logs say for each of the nodes after startup? (I suspect no cluster has been formed yet.)
Could you explain what install / deployment method you used?
On which host are the nodes running on?
How do you connect to the Admin UI?

The cluster has been formed about one month. We installed crate cluster on ubuntu by apt install crate command. The crate services are running on five nodes now.

I connect to Admin UI by

All nodes crate-infra.log are uploaded.
crate-infra-002node.log (3.0 MB)
crate-infra-005node.log (18.7 KB)
crate-infra-004node.log (18.7 KB)
crate-infra-003node.log (37.4 KB)
crate-infra-001node.log (20.1 KB)

Thank you for sharing those details.
Did this problem appear immediately after upgrading?
Did you do a rolling upgrade with ongoing data ingestion?
What version was the cluster running before?
Did you change/reset cluster.routing.allocation.enable as part of this process?

@proddata @hernanc I have locate the root cause that the login user crateadmin missing admin privileges. After granting all privileges to crateadmin, then Admin UI show cluster status. But I’m curious why the privileges of crateadmin suddenly gone.

1 Like

I’m curious whether the issue you’re encountering could be related to version 5.6 of CrateDB, which introduced ROLEs, significantly altering privilege management. To delve into your specific situation, we need details on how you managed the upgrade process to identify any potential problems that our test cases might have missed.

I also think the issue may be related to the upgrade that’s the only change before encountering the issue. The complete upgrade process are as follows.
Step 1. shutdown all nodes
Step 2. apt upgrade crate
Step 3. start all nodes

@Jun_Zhou What was the version you were running before the upgrade?

Hi @matriv , The version is 5.5.3.