Kubernetes scale statefulsets error: [node.max_local_storage_nodes] (was [1])?

CrateDB
1

Hello,
I’m trying to run a crateDB, version 4.8.1, on a AKS cluster, in statefulsets, and if I try to scale my service,
following the documentation.

But I’ve got that error on the secondary pods:

Defaulted container "crate" out of: crate, init-sysctl (init)
[2023-04-07T13:26:48,051][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [cratedb-crate-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/data/data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
        at org.elasticsearch.bootstrap.StartupExceptionProxy.<init>(StartupExceptionProxy.java:30) ~[crate-app.jar:4.8.1]
        at io.crate.bootstrap.CrateDB.init(CrateDB.java:162) ~[crate-app.jar:4.8.1]
        at io.crate.bootstrap.CrateDB.execute(CrateDB.java:138) ~[crate-app.jar:4.8.1]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:81) ~[crate-server.jar:?]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[crate-cli.jar:?]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[crate-cli.jar:?]
        at io.crate.bootstrap.CrateDB.main(CrateDB.java:91) ~[crate-app.jar:4.8.1]
        at io.crate.bootstrap.CrateDB.main(CrateDB.java:84) ~[crate-app.jar:4.8.1]
Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/data/data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
        at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:301) ~[crate-server.jar:?]
        at org.elasticsearch.node.Node.<init>(Node.java:306) ~[crate-server.jar:?]
        at io.crate.node.CrateNode.<init>(CrateNode.java:49) ~[crate-app.jar:4.8.1]
        at org.elasticsearch.bootstrap.BootstrapProxy$1.<init>(BootstrapProxy.java:192) ~[crate-app.jar:4.8.1]
        at org.elasticsearch.bootstrap.BootstrapProxy.setup(BootstrapProxy.java:192) ~[crate-app.jar:4.8.1]
        at org.elasticsearch.bootstrap.BootstrapProxy.init(BootstrapProxy.java:266) ~[crate-app.jar:4.8.1]
        at io.crate.bootstrap.CrateDB.init(CrateDB.java:158) ~[crate-app.jar:4.8.1]

PVC description:

Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: file.csi.azure.com
               volume.kubernetes.io/storage-provisioner: file.csi.azure.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      10Gi
Access Modes:  RWX
VolumeMode:    Filesystem

What can I do ?
Best regards,

Brice

2

Your PVC has RWX(aka. ReadWriteMany) as access mode. Your PVC is most likely attached to multiple nodes. While CrateDB nodes could share a volume (though not recommended) they can not share the same data folder (anymore - was possible in older versions).

See our documentation: CrateDB and Kubernetes — CrateDB: Tutorials

 volumeClaimTemplates:
    # Use persistent storage.
    - metadata:
        name: data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi

saying ReadWriteOnce for access mode

3

Thank you for your answer, I’ve try but I’ve got the same error. I presum that the problem is also because I use a fileshare…
best regards :slight_smile: