Using crate-operator the installation fails with "AttachVolume.Attach failed for volume"

CrateDB Installation
1

Hi all,
I’m trying to install CrateDB using crate-operator; but the installation fails giving the same error on every PV created:

32s         Warning   FailedAttachVolume      pod/crate-data-hot-my-cluster-1                           AttachVolume.Attach failed for volume "pvc-5061e8de-1426-4b8d-b9d8-c84f35d37735" : rpc error: code = Aborted desc = volume pvc-5061e8de-1426-4b8d-b9d8-c84f35d37735 is not ready for workloads
32s         Warning   FailedAttachVolume      pod/crate-data-hot-my-cluster-2                           AttachVolume.Attach failed for volume "pvc-399575f5-d78d-48a9-aae5-dd55f814e2f4" : rpc error: code = Aborted desc = volume pvc-399575f5-d78d-48a9-aae5-dd55f814e2f4 is not ready for workloads
32s         Warning   FailedAttachVolume      pod/crate-data-hot-my-cluster-2                           AttachVolume.Attach failed for volume "pvc-b945c926-1ae8-4ec3-86ff-bb2877194a3b" : rpc error: code = Aborted desc = volume pvc-b945c926-1ae8-4ec3-86ff-bb2877194a3b is not ready for workloads
32s         Warning   FailedAttachVolume      pod/crate-data-hot-my-cluster-1                           AttachVolume.Attach failed for volume "pvc-9b2d9870-5724-4d39-9465-29981580c6b3" : rpc error: code = Aborted desc = volume pvc-9b2d9870-5724-4d39-9465-29981580c6b3 is not ready for workloads
32s         Warning   FailedAttachVolume      pod/crate-data-hot-my-cluster-0                           AttachVolume.Attach failed for volume "pvc-6b74d8ff-381d-45a0-8eec-3c2e57c7d2f3" : rpc error: code = Aborted desc = volume pvc-6b74d8ff-381d-45a0-8eec-3c2e57c7d2f3 is not ready for workloads
32s         Warning   FailedAttachVolume      pod/crate-data-hot-my-cluster-0                           AttachVolume.Attach failed for volume "pvc-ce217b0a-b18c-48b7-9484-3e7a7c076c89" : rpc error: code = Aborted desc = volume pvc-ce217b0a-b18c-48b7-9484-3e7a7c076c89 is not ready for workloads

I have a K3S cluster with 1 master node and three worker nodes.
For storage I have Longhorn installed on the three worker nodes.
The commands I used to create CrateDB are:

kubectl create namespace crate-operator
kubectl create namespace for-crate
helm install crate-operator crate-operator/crate-operator --namespace crate-operator --set env.CRATEDB_OPERATOR_DEBUG_VOLUME_STORAGE_CLASS=longhorn --set env.CRATEDB_OPERATOR_DEBUG_VOLUME_SIZE=4GiB
kubectl --namespace for-crate create -f my-crate.yaml

where my-crate.yaml contains:

apiVersion: cloud.crate.io/v1
kind: CrateDB
metadata:
  name: my-cluster
  namespace: for-crate
spec:
  cluster:
    imageRegistry: crate
    name: my-crate
    version: 5.8.1
  nodes:
    data:
    - name: hot
      replicas: 3
      resources:
        limits:
          cpu: 4
          memory: 4Gi
        disk:
          count: 1
          size: 16GiB
          storageClass: longhorn
        heapRatio: 0.25

Looking at the PV I see:

sysop@h5a-dev:~$ kubectl get pv -o wide
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                         STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE    VOLUMEMODE
pvc-b375793c-ed9c-4335-9605-4e490e1e5956   10Gi       RWO            Delete           Bound    kube-system/docker-registry                   longhorn       <unset>                          205d   Filesystem
pvc-0279c77c-73df-4ece-a461-a3d091dd7353   8Gi        RWO            Delete           Bound    apisix/data-apisix-etcd-2                     longhorn       <unset>                          137d   Filesystem
pvc-9433785a-2c42-41fa-8be4-5e3cadb5123d   8Gi        RWO            Delete           Bound    apisix/data-apisix-etcd-1                     longhorn       <unset>                          137d   Filesystem
pvc-c4571652-2a78-4f1c-aa12-c08432a20805   8Gi        RWO            Delete           Bound    apisix/data-apisix-etcd-0                     longhorn       <unset>                          137d   Filesystem
pvc-399575f5-d78d-48a9-aae5-dd55f814e2f4   4Gi        RWO            Delete           Bound    for-crate/debug-crate-data-hot-my-cluster-2   longhorn       <unset>                          38m    Filesystem
pvc-ce217b0a-b18c-48b7-9484-3e7a7c076c89   4Gi        RWO            Delete           Bound    for-crate/debug-crate-data-hot-my-cluster-0   longhorn       <unset>                          38m    Filesystem
pvc-5061e8de-1426-4b8d-b9d8-c84f35d37735   4Gi        RWO            Delete           Bound    for-crate/debug-crate-data-hot-my-cluster-1   longhorn       <unset>                          38m    Filesystem
pvc-b945c926-1ae8-4ec3-86ff-bb2877194a3b   16Gi       RWO            Delete           Bound    for-crate/data0-crate-data-hot-my-cluster-2   longhorn       <unset>                          38m    Filesystem
pvc-6b74d8ff-381d-45a0-8eec-3c2e57c7d2f3   16Gi       RWO            Delete           Bound    for-crate/data0-crate-data-hot-my-cluster-0   longhorn       <unset>                          38m    Filesystem
pvc-9b2d9870-5724-4d39-9465-29981580c6b3   16Gi       RWO            Delete           Bound    for-crate/data0-crate-data-hot-my-cluster-1   longhorn       <unset>                          38m    Filesystem
sysop@h5a-dev:~$

What I’m doing wrong?

The complete events transcript:

sysop@h5a-dev:~/h5a/software/pcams/storehouse/cratedb$ kubectl get events --sort-by='.lastTimestamp' -n for-crate
LAST SEEN   TYPE      REASON                  OBJECT                                                    MESSAGE
8m57s       Warning   FailedScheduling        pod/crate-data-hot-my-cluster-1                           0/4 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling.
9m18s       Warning   FailedScheduling        pod/crate-data-hot-my-cluster-1                           0/4 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling.
9m18s       Warning   FailedScheduling        pod/crate-data-hot-my-cluster-2                           0/4 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling.
8m57s       Warning   FailedScheduling        pod/crate-data-hot-my-cluster-0                           0/4 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling.
8m57s       Warning   FailedScheduling        pod/crate-data-hot-my-cluster-2                           0/4 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling.
8m53s       Normal    Scheduled               pod/crate-data-hot-my-cluster-0                           Successfully assigned for-crate/crate-data-hot-my-cluster-0 to h5a-kw2.int.h5a.net
8m52s       Normal    Scheduled               pod/crate-data-hot-my-cluster-2                           Successfully assigned for-crate/crate-data-hot-my-cluster-2 to h5a-kw3.int.h5a.net
8m53s       Normal    Scheduled               pod/crate-data-hot-my-cluster-1                           Successfully assigned for-crate/crate-data-hot-my-cluster-1 to h5a-kw1.int.h5a.net
9m18s       Warning   FailedScheduling        pod/crate-data-hot-my-cluster-0                           0/4 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling.
9m19s       Normal    SuccessfulCreate        statefulset/crate-data-hot-my-cluster                     create Claim data0-crate-data-hot-my-cluster-2 Pod crate-data-hot-my-cluster-2 in StatefulSet crate-data-hot-my-cluster success
9m19s       Normal    SuccessfulCreate        statefulset/crate-data-hot-my-cluster                     create Pod crate-data-hot-my-cluster-2 in StatefulSet crate-data-hot-my-cluster successful
9m19s       Normal    SuccessfulCreate        statefulset/crate-data-hot-my-cluster                     create Claim data0-crate-data-hot-my-cluster-0 Pod crate-data-hot-my-cluster-0 in StatefulSet crate-data-hot-my-cluster success
9m19s       Normal    SuccessfulCreate        statefulset/crate-data-hot-my-cluster                     create Claim debug-crate-data-hot-my-cluster-0 Pod crate-data-hot-my-cluster-0 in StatefulSet crate-data-hot-my-cluster success
9m19s       Normal    NoPods                  poddisruptionbudget/crate-my-cluster                      No matching pods found
9m19s       Normal    SuccessfulCreate        statefulset/crate-data-hot-my-cluster                     create Pod crate-data-hot-my-cluster-0 in StatefulSet crate-data-hot-my-cluster successful
9m19s       Normal    SuccessfulCreate        statefulset/crate-data-hot-my-cluster                     create Claim data0-crate-data-hot-my-cluster-1 Pod crate-data-hot-my-cluster-1 in StatefulSet crate-data-hot-my-cluster success
9m19s       Normal    AppliedDaemonSet        service/crate-my-cluster                                  Applied LoadBalancer DaemonSet kube-system/svclb-crate-my-cluster-3bf67652
9m19s       Normal    SuccessfulCreate        statefulset/crate-data-hot-my-cluster                     create Claim debug-crate-data-hot-my-cluster-1 Pod crate-data-hot-my-cluster-1 in StatefulSet crate-data-hot-my-cluster success
9m19s       Normal    SuccessfulCreate        statefulset/crate-data-hot-my-cluster                     create Pod crate-data-hot-my-cluster-1 in StatefulSet crate-data-hot-my-cluster successful
9m19s       Normal    SuccessfulCreate        statefulset/crate-data-hot-my-cluster                     create Claim debug-crate-data-hot-my-cluster-2 Pod crate-data-hot-my-cluster-2 in StatefulSet crate-data-hot-my-cluster success
9m19s       Normal    Logging                 cratedb/my-cluster                                        Handler 'cluster_create/statefulset_data_hot' succeeded.
9m19s       Normal    Logging                 cratedb/my-cluster                                        Handler 'cluster_create/sql_exporter_config' succeeded.
9m19s       Normal    Logging                 cratedb/my-cluster                                        Handler 'cluster_create/system_user' succeeded.
9m19s       Normal    EnsuringLoadBalancer    service/crate-my-cluster                                  Ensuring load balancer
9m19s       Normal    Logging                 cratedb/my-cluster                                        Handler 'cluster_create/services' succeeded.
9m18s       Error     Logging                 cratedb/my-cluster                                        Handler 'cluster_create/bootstrap' failed temporarily: None
9m18s       Error     Logging                 cratedb/my-cluster                                        ... failed. Status: 400 Message: Invalid response status
9m18s       Normal    Logging                 cratedb/my-cluster                                        Trying to create system user ...
9m18s       Normal    Logging                 cratedb/my-cluster                                        Webhooks not configured. Not processing event WebhookEvent.FEEDBACK.
9m3s        Warning   Logging                 cratedb/my-cluster                                        Failed to ping cluster....
9m3s        Normal    Logging                 cratedb/my-cluster                                        Webhooks not configured. Not processing event WebhookEvent.HEALTH.
9m3s        Normal    Logging                 cratedb/my-cluster                                        Timer 'ping_cratedb' succeeded.
9m          Normal    ExternalProvisioning    persistentvolumeclaim/debug-crate-data-hot-my-cluster-0   Waiting for a volume to be created either by the external provisioner 'driver.longhorn.io' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
9m          Normal    ExternalProvisioning    persistentvolumeclaim/data0-crate-data-hot-my-cluster-2   Waiting for a volume to be created either by the external provisioner 'driver.longhorn.io' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
9m          Normal    ExternalProvisioning    persistentvolumeclaim/data0-crate-data-hot-my-cluster-1   Waiting for a volume to be created either by the external provisioner 'driver.longhorn.io' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
9m          Normal    ExternalProvisioning    persistentvolumeclaim/debug-crate-data-hot-my-cluster-2   Waiting for a volume to be created either by the external provisioner 'driver.longhorn.io' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
9m          Normal    ExternalProvisioning    persistentvolumeclaim/data0-crate-data-hot-my-cluster-0   Waiting for a volume to be created either by the external provisioner 'driver.longhorn.io' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
9m          Normal    ExternalProvisioning    persistentvolumeclaim/debug-crate-data-hot-my-cluster-1   Waiting for a volume to be created either by the external provisioner 'driver.longhorn.io' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
8m59s       Normal    Provisioning            persistentvolumeclaim/data0-crate-data-hot-my-cluster-1   External provisioner is provisioning volume for claim "for-crate/data0-crate-data-hot-my-cluster-1"
8m59s       Normal    Provisioning            persistentvolumeclaim/data0-crate-data-hot-my-cluster-2   External provisioner is provisioning volume for claim "for-crate/data0-crate-data-hot-my-cluster-2"
8m59s       Normal    Provisioning            persistentvolumeclaim/debug-crate-data-hot-my-cluster-1   External provisioner is provisioning volume for claim "for-crate/debug-crate-data-hot-my-cluster-1"
8m59s       Normal    Provisioning            persistentvolumeclaim/debug-crate-data-hot-my-cluster-0   External provisioner is provisioning volume for claim "for-crate/debug-crate-data-hot-my-cluster-0"
8m59s       Normal    Provisioning            persistentvolumeclaim/debug-crate-data-hot-my-cluster-2   External provisioner is provisioning volume for claim "for-crate/debug-crate-data-hot-my-cluster-2"
8m59s       Normal    Provisioning            persistentvolumeclaim/data0-crate-data-hot-my-cluster-0   External provisioner is provisioning volume for claim "for-crate/data0-crate-data-hot-my-cluster-0"
8m57s       Normal    ProvisioningSucceeded   persistentvolumeclaim/debug-crate-data-hot-my-cluster-1   Successfully provisioned volume pvc-5061e8de-1426-4b8d-b9d8-c84f35d37735
8m57s       Normal    ProvisioningSucceeded   persistentvolumeclaim/data0-crate-data-hot-my-cluster-1   Successfully provisioned volume pvc-9b2d9870-5724-4d39-9465-29981580c6b3
8m57s       Normal    ProvisioningSucceeded   persistentvolumeclaim/debug-crate-data-hot-my-cluster-2   Successfully provisioned volume pvc-399575f5-d78d-48a9-aae5-dd55f814e2f4
8m57s       Normal    ProvisioningSucceeded   persistentvolumeclaim/data0-crate-data-hot-my-cluster-0   Successfully provisioned volume pvc-6b74d8ff-381d-45a0-8eec-3c2e57c7d2f3
8m57s       Normal    ProvisioningSucceeded   persistentvolumeclaim/data0-crate-data-hot-my-cluster-2   Successfully provisioned volume pvc-b945c926-1ae8-4ec3-86ff-bb2877194a3b
8m57s       Normal    ProvisioningSucceeded   persistentvolumeclaim/debug-crate-data-hot-my-cluster-0   Successfully provisioned volume pvc-ce217b0a-b18c-48b7-9484-3e7a7c076c89
8m18s       Normal    Logging                 cratedb/my-cluster                                        Webhooks not configured. Not processing event WebhookEvent.FEEDBACK.
8m18s       Normal    Logging                 cratedb/my-cluster                                        Trying to create system user ...
8m18s       Error     Logging                 cratedb/my-cluster                                        ... failed. Status: 500 Message: Invalid response status
8m18s       Error     Logging                 cratedb/my-cluster                                        Handler 'cluster_create/bootstrap' failed temporarily: None
8m3s        Warning   Logging                 cratedb/my-cluster                                        Failed to ping cluster....
8m3s        Normal    Logging                 cratedb/my-cluster                                        Webhooks not configured. Not processing event WebhookEvent.HEALTH.
8m3s        Normal    Logging                 cratedb/my-cluster                                        Timer 'ping_cratedb' succeeded.
7m18s       Normal    Logging                 cratedb/my-cluster                                        Trying to create system user ...
7m18s       Normal    Logging                 cratedb/my-cluster                                        Webhooks not configured. Not processing event WebhookEvent.FEEDBACK.
7m18s       Error     Logging                 cratedb/my-cluster                                        ... failed. Status: 500 Message: Invalid response status
7m18s       Error     Logging                 cratedb/my-cluster                                        Handler 'cluster_create/bootstrap' failed temporarily: None
7m3s        Warning   Logging                 cratedb/my-cluster                                        Failed to ping cluster....
7m3s        Normal    Logging                 cratedb/my-cluster                                        Webhooks not configured. Not processing event WebhookEvent.HEALTH.
7m3s        Normal    Logging                 cratedb/my-cluster                                        Timer 'ping_cratedb' succeeded.
6m18s       Normal    Logging                 cratedb/my-cluster                                        Webhooks not configured. Not processing event WebhookEvent.FEEDBACK.
6m18s       Normal    Logging                 cratedb/my-cluster                                        Trying to create system user ...
6m18s       Error     Logging                 cratedb/my-cluster                                        ... failed. Status: 500 Message: Invalid response status
6m18s       Error     Logging                 cratedb/my-cluster                                        Handler 'cluster_create/bootstrap' failed temporarily: None
6m3s        Warning   Logging                 cratedb/my-cluster                                        Failed to ping cluster....
6m3s        Normal    Logging                 cratedb/my-cluster                                        Webhooks not configured. Not processing event WebhookEvent.HEALTH.
6m3s        Normal    Logging                 cratedb/my-cluster                                        Timer 'ping_cratedb' succeeded.
5m18s       Normal    Logging                 cratedb/my-cluster                                        Webhooks not configured. Not processing event WebhookEvent.FEEDBACK.
5m18s       Normal    Logging                 cratedb/my-cluster                                        Trying to create system user ...
5m18s       Error     Logging                 cratedb/my-cluster                                        ... failed. Status: 500 Message: Invalid response status
5m18s       Error     Logging                 cratedb/my-cluster                                        Handler 'cluster_create/bootstrap' failed temporarily: None
5m3s        Warning   Logging                 cratedb/my-cluster                                        Failed to ping cluster....
5m3s        Normal    Logging                 cratedb/my-cluster                                        Webhooks not configured. Not processing event WebhookEvent.HEALTH.
5m3s        Normal    Logging                 cratedb/my-cluster                                        Timer 'ping_cratedb' succeeded.
4m18s       Normal    Logging                 cratedb/my-cluster                                        Webhooks not configured. Not processing event WebhookEvent.FEEDBACK.
4m18s       Normal    Logging                 cratedb/my-cluster                                        Trying to create system user ...
4m18s       Error     Logging                 cratedb/my-cluster                                        ... failed. Status: 500 Message: Invalid response status
4m18s       Error     Logging                 cratedb/my-cluster                                        Handler 'cluster_create/bootstrap' failed temporarily: None
4m2s        Warning   Logging                 cratedb/my-cluster                                        Failed to ping cluster....
4m2s        Normal    Logging                 cratedb/my-cluster                                        Webhooks not configured. Not processing event WebhookEvent.HEALTH.
4m2s        Normal    Logging                 cratedb/my-cluster                                        Timer 'ping_cratedb' succeeded.
3m17s       Error     Logging                 cratedb/my-cluster                                        Handler 'cluster_create/bootstrap' failed temporarily: None
3m17s       Normal    Logging                 cratedb/my-cluster                                        Trying to create system user ...
3m17s       Error     Logging                 cratedb/my-cluster                                        ... failed. Status: 500 Message: Invalid response status
3m17s       Normal    Logging                 cratedb/my-cluster                                        Webhooks not configured. Not processing event WebhookEvent.FEEDBACK.
3m2s        Warning   Logging                 cratedb/my-cluster                                        Failed to ping cluster....
3m2s        Normal    Logging                 cratedb/my-cluster                                        Webhooks not configured. Not processing event WebhookEvent.HEALTH.
3m2s        Normal    Logging                 cratedb/my-cluster                                        Timer 'ping_cratedb' succeeded.
2m17s       Normal    Logging                 cratedb/my-cluster                                        Webhooks not configured. Not processing event WebhookEvent.FEEDBACK.
2m17s       Normal    Logging                 cratedb/my-cluster                                        Trying to create system user ...
2m17s       Error     Logging                 cratedb/my-cluster                                        ... failed. Status: 500 Message: Invalid response status
2m17s       Error     Logging                 cratedb/my-cluster                                        Handler 'cluster_create/bootstrap' failed temporarily: None
2m2s        Warning   Logging                 cratedb/my-cluster                                        Failed to ping cluster....
2m2s        Normal    Logging                 cratedb/my-cluster                                        Webhooks not configured. Not processing event WebhookEvent.HEALTH.
2m2s        Normal    Logging                 cratedb/my-cluster                                        Timer 'ping_cratedb' succeeded.
77s         Error     Logging                 cratedb/my-cluster                                        Handler 'cluster_create/bootstrap' failed temporarily: None
77s         Normal    Logging                 cratedb/my-cluster                                        Trying to create system user ...
77s         Error     Logging                 cratedb/my-cluster                                        ... failed. Status: 500 Message: Invalid response status
77s         Normal    Logging                 cratedb/my-cluster                                        Webhooks not configured. Not processing event WebhookEvent.FEEDBACK.
62s         Warning   Logging                 cratedb/my-cluster                                        Failed to ping cluster....
62s         Normal    Logging                 cratedb/my-cluster                                        Webhooks not configured. Not processing event WebhookEvent.HEALTH.
62s         Normal    Logging                 cratedb/my-cluster                                        Timer 'ping_cratedb' succeeded.
32s         Warning   FailedAttachVolume      pod/crate-data-hot-my-cluster-1                           AttachVolume.Attach failed for volume "pvc-5061e8de-1426-4b8d-b9d8-c84f35d37735" : rpc error: code = Aborted desc = volume pvc-5061e8de-1426-4b8d-b9d8-c84f35d37735 is not ready for workloads
32s         Warning   FailedAttachVolume      pod/crate-data-hot-my-cluster-2                           AttachVolume.Attach failed for volume "pvc-399575f5-d78d-48a9-aae5-dd55f814e2f4" : rpc error: code = Aborted desc = volume pvc-399575f5-d78d-48a9-aae5-dd55f814e2f4 is not ready for workloads
32s         Warning   FailedAttachVolume      pod/crate-data-hot-my-cluster-2                           AttachVolume.Attach failed for volume "pvc-b945c926-1ae8-4ec3-86ff-bb2877194a3b" : rpc error: code = Aborted desc = volume pvc-b945c926-1ae8-4ec3-86ff-bb2877194a3b is not ready for workloads
32s         Warning   FailedAttachVolume      pod/crate-data-hot-my-cluster-1                           AttachVolume.Attach failed for volume "pvc-9b2d9870-5724-4d39-9465-29981580c6b3" : rpc error: code = Aborted desc = volume pvc-9b2d9870-5724-4d39-9465-29981580c6b3 is not ready for workloads
32s         Warning   FailedAttachVolume      pod/crate-data-hot-my-cluster-0                           AttachVolume.Attach failed for volume "pvc-6b74d8ff-381d-45a0-8eec-3c2e57c7d2f3" : rpc error: code = Aborted desc = volume pvc-6b74d8ff-381d-45a0-8eec-3c2e57c7d2f3 is not ready for workloads
32s         Warning   FailedAttachVolume      pod/crate-data-hot-my-cluster-0                           AttachVolume.Attach failed for volume "pvc-ce217b0a-b18c-48b7-9484-3e7a7c076c89" : rpc error: code = Aborted desc = volume pvc-ce217b0a-b18c-48b7-9484-3e7a7c076c89 is not ready for workloads
17s         Normal    Logging                 cratedb/my-cluster                                        Webhooks not configured. Not processing event WebhookEvent.FEEDBACK.
17s         Normal    Logging                 cratedb/my-cluster                                        Trying to create system user ...
17s         Error     Logging                 cratedb/my-cluster                                        ... failed. Status: 500 Message: Invalid response status
17s         Error     Logging                 cratedb/my-cluster                                        Handler 'cluster_create/bootstrap' failed temporarily: None
2s          Warning   Logging                 cratedb/my-cluster                                        Failed to ping cluster....
2s          Normal    Logging                 cratedb/my-cluster                                        Webhooks not configured. Not processing event WebhookEvent.HEALTH.
2s          Normal    Logging                 cratedb/my-cluster                                        Timer 'ping_cratedb' succeeded.
sysop@h5a-dev:~/h5a/software/pcams/storehouse/cratedb$
2

Hey Mirto,
thanks for giving crate-operator a try. Sorry to hear you are having troubles to spin up a cratedb cluster!

Looking at the RPC errors - but in no way a Longhorn expert - I would assume that this is rather a Longhorn specific issue at this time of bootstrapping cratedb.

PVC not ready?
volume pvc-5061e8de-1426-4b8d-b9d8-c84f35d37735 is not ready for workloads

Would you please share the used versions (k3s/k8s, longhorn, linux, loadbalancer setup)? Would be great if you could post the YAML of the sts here also.

Thanks in advance,
Walter

3

Well I try to gather the informations.

all the nodes are KVM virtual machines with Debian 12 os and kernel 6.1.0-27-amd64.

K3S is v1.29.4+k3s1 version.

Longhorn is 1.6.1 version.

I’m a little confused about the other info.

I don’t know nothing about the statefulset. I think is is created by the operator using the yaml in my previous post.

About the PVC not ready, if you look at my previous post, all the PVC are in the same state and I’m sure that etcd and registry work correctly.

All that is created by the operator is:

sysop@h5a-dev:~$ kubectl get all -n for-crate
NAME                              READY   STATUS     RESTARTS   AGE
pod/crate-data-hot-my-cluster-1   0/2     Init:0/3   0          25h
pod/crate-data-hot-my-cluster-0   0/2     Init:0/3   0          25h
pod/crate-data-hot-my-cluster-2   0/2     Init:0/3   0          25h

NAME                                 TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
service/crate-my-cluster             LoadBalancer   10.43.145.147   <pending>     4200:31784/TCP,5432:30881/TCP   25h
service/crate-discovery-my-cluster   ClusterIP      None            <none>        4300/TCP,4200/TCP,5432/TCP      25h

NAME                                         READY   AGE
statefulset.apps/crate-data-hot-my-cluster   0/3     25h
sysop@h5a-dev:~$

There are other info to gather?

4

I assume you want to look into this ^^ ! Is the longhorn storage class created by you? Please share -o yaml of that one.

5

Well I created the longhorn storage class. Below the commands I used.

helm repo add longhorn https://charts.longhorn.io
helm repo update
helm install longhorn longhorn/longhorn --namespace longhorn-system --create-namespace --version 1.6.1
kubectl patch storageclass longhorn -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

After this I see with kubectl

sysop@h5a-dev:~$ kubectl get all -n longhorn-system
NAME                                                    READY   STATUS    RESTARTS        AGE
pod/engine-image-ei-5cefaf2b-svcsh                      1/1     Running   46 (11m ago)    211d
pod/longhorn-manager-cldvr                              1/1     Running   47 (11m ago)    211d
pod/longhorn-manager-ndf77                              1/1     Running   46 (11m ago)    211d
pod/engine-image-ei-5cefaf2b-ld2t7                      1/1     Running   46 (11m ago)    211d
pod/longhorn-manager-xpfgq                              1/1     Running   46 (11m ago)    211d
pod/engine-image-ei-5cefaf2b-4djp7                      1/1     Running   46 (11m ago)    211d
pod/instance-manager-ea618c718d78e5f30af0157074b931ed   1/1     Running   0               10m
pod/longhorn-ui-746d45fdfb-zzk4f                        1/1     Running   116 (11m ago)   211d
pod/instance-manager-020021b2e6cad031a753c724f4773579   1/1     Running   0               10m
pod/instance-manager-cabd157e12db676c35ba3e7cc325f646   1/1     Running   0               10m
pod/longhorn-ui-746d45fdfb-87np2                        1/1     Running   107 (11m ago)   211d
pod/longhorn-driver-deployer-9f4fc86-dgzhc              1/1     Running   54 (10m ago)    211d
pod/csi-attacher-57689cc84b-jhvpx                       1/1     Running   82 (10m ago)    211d
pod/csi-resizer-7466f7b45f-9b8dt                        1/1     Running   82 (10m ago)    211d
pod/csi-snapshotter-58bf69fbd5-zwb5x                    1/1     Running   82 (10m ago)    211d
pod/csi-provisioner-6c78dcb664-mwmgm                    1/1     Running   82 (10m ago)    211d
pod/longhorn-csi-plugin-kjtw5                           3/3     Running   232 (10m ago)   211d
pod/csi-provisioner-6c78dcb664-fc879                    1/1     Running   72 (10m ago)    211d
pod/csi-attacher-57689cc84b-v4sl9                       1/1     Running   72 (10m ago)    211d
pod/csi-snapshotter-58bf69fbd5-t9n9l                    1/1     Running   70 (10m ago)    211d
pod/csi-resizer-7466f7b45f-cpqsd                        1/1     Running   70 (10m ago)    211d
pod/csi-resizer-7466f7b45f-d72qh                        1/1     Running   58 (10m ago)    211d
pod/longhorn-csi-plugin-ngm6m                           3/3     Running   256 (10m ago)   211d
pod/csi-snapshotter-58bf69fbd5-kwfjn                    1/1     Running   57 (10m ago)    211d
pod/csi-attacher-57689cc84b-5p9tb                       1/1     Running   60 (10m ago)    211d
pod/longhorn-csi-plugin-vw9kh                           3/3     Running   202 (10m ago)   211d
pod/csi-provisioner-6c78dcb664-lvstn                    1/1     Running   58 (10m ago)    211d

NAME                                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/longhorn-engine-manager       ClusterIP   None            <none>        <none>     211d
service/longhorn-replica-manager      ClusterIP   None            <none>        <none>     211d
service/longhorn-backend              ClusterIP   10.43.208.150   <none>        9500/TCP   211d
service/longhorn-conversion-webhook   ClusterIP   10.43.48.78     <none>        9501/TCP   211d
service/longhorn-admission-webhook    ClusterIP   10.43.230.8     <none>        9502/TCP   211d
service/longhorn-recovery-backend     ClusterIP   10.43.106.66    <none>        9503/TCP   211d
service/longhorn-frontend             ClusterIP   10.43.19.143    <none>        80/TCP     211d

NAME                                      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/engine-image-ei-5cefaf2b   3         3         3       3            3           <none>          211d
daemonset.apps/longhorn-manager           3         3         3       3            3           <none>          211d
daemonset.apps/longhorn-csi-plugin        3         3         3       3            3           <none>          211d

NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/longhorn-ui                2/2     2            2           211d
deployment.apps/longhorn-driver-deployer   1/1     1            1           211d
deployment.apps/csi-resizer                3/3     3            3           211d
deployment.apps/csi-snapshotter            3/3     3            3           211d
deployment.apps/csi-attacher               3/3     3            3           211d
deployment.apps/csi-provisioner            3/3     3            3           211d

NAME                                               DESIRED   CURRENT   READY   AGE
replicaset.apps/longhorn-ui-746d45fdfb             2         2         2       211d
replicaset.apps/longhorn-driver-deployer-9f4fc86   1         1         1       211d
replicaset.apps/csi-resizer-7466f7b45f             3         3         3       211d
replicaset.apps/csi-snapshotter-58bf69fbd5         3         3         3       211d
replicaset.apps/csi-attacher-57689cc84b            3         3         3       211d
replicaset.apps/csi-provisioner-6c78dcb664         3         3         3       211d
sysop@h5a-dev:~$

and the storage class definitions is

sysop@h5a-dev:~$ kubectl describe storageclasses longhorn
Name:            longhorn
IsDefaultClass:  Yes
Annotations:     longhorn.io/last-applied-configmap=kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: longhorn
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: driver.longhorn.io
allowVolumeExpansion: true
reclaimPolicy: "Delete"
volumeBindingMode: Immediate
parameters:
  numberOfReplicas: "3"
  staleReplicaTimeout: "30"
  fromBackup: ""
  fsType: "ext4"
  dataLocality: "disabled"
  unmapMarkSnapChainRemoved: "ignored"
,storageclass.kubernetes.io/is-default-class=true
Provisioner:           driver.longhorn.io
Parameters:            dataLocality=disabled,fromBackup=,fsType=ext4,numberOfReplicas=3,staleReplicaTimeout=30,unmapMarkSnapChainRemoved=ignored
AllowVolumeExpansion:  True
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     Immediate
Events:                <none>
sysop@h5a-dev:~$

Do you see anything wrong?

6

Hi, I have foud a longhorn error message: insufficient storage.

Seems that 35Gb free on the disks are not sufficient.

I’ll enlarge the disks and try again.

longhorn-error
longhorn-error1309×792 90.6 KB

7

After enlarging the disk size the installation worked perfectly.

Thanks a lot.