AccessDeniedException on _state

CrateDB
1

Dear forum,

I have just installed a CrateDB 4.6.4 cluster of 3 nodes and it started successful, after a restart I have this error:

[2021-10-21T09:25:31,173][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [crate01] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: org.elasticsearch.ElasticsearchException: failed to bind service
        at org.elasticsearch.bootstrap.StartupExceptionProxy.<init>(StartupExceptionProxy.java:30) ~[crate-app.jar:4.6.4]
        at io.crate.bootstrap.CrateDB.init(CrateDB.java:162) ~[crate-app.jar:4.6.4]
        at io.crate.bootstrap.CrateDB.execute(CrateDB.java:138) ~[crate-app.jar:4.6.4]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:81) ~[crate-server.jar:?]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[crate-cli.jar:?]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[crate-cli.jar:?]
        at io.crate.bootstrap.CrateDB.main(CrateDB.java:91) ~[crate-app.jar:4.6.4]
        at io.crate.bootstrap.CrateDB.main(CrateDB.java:84) ~[crate-app.jar:4.6.4]
Caused by: org.elasticsearch.ElasticsearchException: failed to bind service
        at org.elasticsearch.node.Node.<init>(Node.java:651) ~[crate-server.jar:?]
        at io.crate.node.CrateNode.<init>(CrateNode.java:53) ~[crate-app.jar:4.6.4]
        at org.elasticsearch.bootstrap.BootstrapProxy$1.<init>(BootstrapProxy.java:192) ~[crate-app.jar:4.6.4]
        at org.elasticsearch.bootstrap.BootstrapProxy.setup(BootstrapProxy.java:192) ~[crate-app.jar:4.6.4]
        at org.elasticsearch.bootstrap.BootstrapProxy.init(BootstrapProxy.java:266) ~[crate-app.jar:4.6.4]
        at io.crate.bootstrap.CrateDB.init(CrateDB.java:158) ~[crate-app.jar:4.6.4]
        ... 6 more
Caused by: java.nio.file.AccessDeniedException: /mnt/disk01/nodes/0/_state
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
        at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:439) ~[?:?]
        at java.nio.file.Files.newDirectoryStream(Files.java:476) ~[?:?]
        at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:214) ~[lucene-core-8.8.2.jar:8.8.2 a92a05e195b775b30ca410bc0a26e8e79e7b3bfb - mdrob - 2021-04-06 16:33:27]
        at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:233) ~[lucene-core-8.8.2.jar:8.8.2 a92a05e195b775b30ca410bc0a26e8e79e7b3bfb - mdrob - 2021-04-06 16:33:27]
        at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:700) ~[lucene-core-8.8.2.jar:8.8.2 a92a05e195b775b30ca410bc0a26e8e79e7b3bfb - mdrob - 2021-04-06 16:33:27]
        at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:81) ~[lucene-core-8.8.2.jar:8.8.2 a92a05e195b775b30ca410bc0a26e8e79e7b3bfb - mdrob - 2021-04-06 16:33:27]
        at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:63) ~[lucene-core-8.8.2.jar:8.8.2 a92a05e195b775b30ca410bc0a26e8e79e7b3bfb - mdrob - 2021-04-06 16:33:27]
        at org.elasticsearch.gateway.PersistedClusterStateService.nodeMetadata(PersistedClusterStateService.java:266) ~[crate-server.jar:?]
        at org.elasticsearch.env.NodeEnvironment.loadNodeMetadata(NodeEnvironment.java:416) ~[crate-server.jar:?]
        at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:329) ~[crate-server.jar:?]
        at org.elasticsearch.node.Node.<init>(Node.java:270) ~[crate-server.jar:?]
        at io.crate.node.CrateNode.<init>(CrateNode.java:53) ~[crate-app.jar:4.6.4]
        at org.elasticsearch.bootstrap.BootstrapProxy$1.<init>(BootstrapProxy.java:192) ~[crate-app.jar:4.6.4]
        at org.elasticsearch.bootstrap.BootstrapProxy.setup(BootstrapProxy.java:192) ~[crate-app.jar:4.6.4]
        at org.elasticsearch.bootstrap.BootstrapProxy.init(BootstrapProxy.java:266) ~[crate-app.jar:4.6.4]
        at io.crate.bootstrap.CrateDB.init(CrateDB.java:158) ~[crate-app.jar:4.6.4]
        ... 6 more

This already happen two times, the first time I thought I did something wrong and I have deleted the whole database disks, but this time I need to understand how to recover the problem without loose the data.

What happen here?

Regards,
Stefano

2

How did you install CrateDB?
Does the user that runs CrateDB have sufficient privileges on the directory?

3

Hi,

I have installed the DB via RPM. I am on a Centos 7.
It seems to me that everything is fine:

[root@crate01 ~]# ls -al  /mnt/disk01/nodes/0/_state
total 24
d-wxrw--wt. 2 crate crate  121 Oct 21 09:11 .
d-wxrw--wt. 3 crate crate   37 Oct 21 09:25 ..
--wxrw--wt. 1 crate crate  278 Oct 21 09:11 _7.cfe
--wxrw--wt. 1 crate crate 1776 Oct 21 09:11 _7.cfs
--wxrw--wt. 1 crate crate  379 Oct 21 09:11 _7.si
--wxrw--wt. 1 crate crate  109 Oct 20 11:11 manifest-0.st
--wxrw--wt. 1 crate crate   89 Oct 20 11:11 node-0.st
--wxrw--wt. 1 crate crate  246 Oct 21 09:11 segments_r
--wxrw--wt. 1 crate crate    0 Oct 20 11:11 write.lock

Or not?

Regards,
S.

4

I got it.

The permissions of the directories are wrong, this is due to the Ansible script I have used to configure the DB.

Thanks,
S.

1 Like