Service not start , Format version is not supported

mahoboy468 · January 9, 2022, 8:21am

Hello,
crate service not start after reboot OS,
and have error
but its worked fine before
can anyone help me?
thanks

2022-01-09T08:06:32,460][INFO ][o.e.e.NodeEnvironment    ] [Jungfrau] using [1] data paths, mounts [[/ (/dev/sda2)]], net usable_space [20.4gb], net total_space [39.1gb], types [ext4]
[2022-01-09T08:06:32,477][INFO ][o.e.e.NodeEnvironment    ] [Jungfrau] heap size [1.9gb], compressed ordinary object pointers [true]
[2022-01-09T08:06:33,348][ERROR][o.e.b.BootstrapProxy     ] [Jungfrau] Exception
org.elasticsearch.ElasticsearchException: failed to bind service
	at org.elasticsearch.node.Node.<init>(Node.java:651) ~[crate-server.jar:?]
	at io.crate.node.CrateNode.<init>(CrateNode.java:53) ~[crate-app.jar:4.6.1]
	at org.elasticsearch.bootstrap.BootstrapProxy$1.<init>(BootstrapProxy.java:192) ~[crate-app.jar:4.6.1]
	at org.elasticsearch.bootstrap.BootstrapProxy.setup(BootstrapProxy.java:192) ~[crate-app.jar:4.6.1]
	at org.elasticsearch.bootstrap.BootstrapProxy.init(BootstrapProxy.java:266) [crate-app.jar:4.6.1]
	at io.crate.bootstrap.CrateDB.init(CrateDB.java:158) [crate-app.jar:4.6.1]
	at io.crate.bootstrap.CrateDB.execute(CrateDB.java:138) [crate-app.jar:4.6.1]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:81) [crate-server.jar:?]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [crate-cli.jar:?]
	at org.elasticsearch.cli.Command.main(Command.java:90) [crate-cli.jar:?]
	at io.crate.bootstrap.CrateDB.main(CrateDB.java:91) [crate-app.jar:4.6.1]
	at io.crate.bootstrap.CrateDB.main(CrateDB.java:84) [crate-app.jar:4.6.1]
Caused by: org.apache.lucene.index.IndexFormatTooOldException: Format version is not supported (resource BufferedChecksumIndexInput(SimpleFSIndexInput(path="/var/lib/crate/nodes/0/_state/segments_zm"))): 0 (needs to be between 1071082519 and 1071082519). This version of Lucene only supports indexes created with release 6.0 and later.
	at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:307) ~[lucene-core-8.8.2.jar:8.8.2 a92a05e195b775b30ca410bc0a26e8e79e7b3bfb - mdrob - 2021-04-06 16:33:27]
	at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:291) ~[lucene-core-8.8.2.jar:8.8.2 a92a05e195b775b30ca410bc0a26e8e79e7b3bfb - mdrob - 2021-04-06 16:33:27]
	at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:61) ~[lucene-core-8.8.2.jar:8.8.2 a92a05e195b775b30ca410bc0a26e8e79e7b3bfb - mdrob - 2021-04-06 16:33:27]
	at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:58) ~[lucene-core-8.8.2.jar:8.8.2 a92a05e195b775b30ca410bc0a26e8e79e7b3bfb - mdrob - 2021-04-06 16:33:27]
	at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:720) ~[lucene-core-8.8.2.jar:8.8.2 a92a05e195b775b30ca410bc0a26e8e79e7b3bfb - mdrob - 2021-04-06 16:33:27]
	at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:81) ~[lucene-core-8.8.2.jar:8.8.2 a92a05e195b775b30ca410bc0a26e8e79e7b3bfb - mdrob - 2021-04-06 16:33:27]
	at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:63) ~[lucene-core-8.8.2.jar:8.8.2 a92a05e195b775b30ca410bc0a26e8e79e7b3bfb - mdrob - 2021-04-06 16:33:27]
	at org.elasticsearch.gateway.PersistedClusterStateService.nodeMetadata(PersistedClusterStateService.java:266) ~[crate-server.jar:?]
	at org.elasticsearch.env.NodeEnvironment.loadNodeMetadata(NodeEnvironment.java:416) ~[crate-server.jar:?]
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:329) ~[crate-server.jar:?]
	at org.elasticsearch.node.Node.<init>(Node.java:270) ~[crate-server.jar:?]
	... 11 more
[2022-01-09T08:06:33,377][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [Jungfrau] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: org.elasticsearch.ElasticsearchException: failed to bind service
	at org.elasticsearch.bootstrap.StartupExceptionProxy.<init>(StartupExceptionProxy.java:30) ~[crate-app.jar:4.6.1]
	at io.crate.bootstrap.CrateDB.init(CrateDB.java:162) ~[crate-app.jar:4.6.1]
	at io.crate.bootstrap.CrateDB.execute(CrateDB.java:138) ~[crate-app.jar:4.6.1]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:81) ~[crate-server.jar:?]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[crate-cli.jar:?]
	at org.elasticsearch.cli.Command.main(Command.java:90) ~[crate-cli.jar:?]
	at io.crate.bootstrap.CrateDB.main(CrateDB.java:91) ~[crate-app.jar:4.6.1]
	at io.crate.bootstrap.CrateDB.main(CrateDB.java:84) ~[crate-app.jar:4.6.1]
Caused by: org.elasticsearch.ElasticsearchException: failed to bind service
	at org.elasticsearch.node.Node.<init>(Node.java:651) ~[crate-server.jar:?]
	at io.crate.node.CrateNode.<init>(CrateNode.java:53) ~[crate-app.jar:4.6.1]
	at org.elasticsearch.bootstrap.BootstrapProxy$1.<init>(BootstrapProxy.java:192) ~[crate-app.jar:4.6.1]
	at org.elasticsearch.bootstrap.BootstrapProxy.setup(BootstrapProxy.java:192) ~[crate-app.jar:4.6.1]
	at org.elasticsearch.bootstrap.BootstrapProxy.init(BootstrapProxy.java:266) ~[crate-app.jar:4.6.1]
	at io.crate.bootstrap.CrateDB.init(CrateDB.java:158) ~[crate-app.jar:4.6.1]
	... 6 more
Caused by: org.apache.lucene.index.IndexFormatTooOldException: Format version is not supported (resource BufferedChecksumIndexInput(SimpleFSIndexInput(path="/var/lib/crate/nodes/0/_state/segments_zm"))): 0 (needs to be between 1071082519 and 1071082519). This version of Lucene only supports indexes created with release 6.0 and later.
	at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:307) ~[lucene-core-8.8.2.jar:8.8.2 a92a05e195b775b30ca410bc0a26e8e79e7b3bfb - mdrob - 2021-04-06 16:33:27]
	at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:291) ~[lucene-core-8.8.2.jar:8.8.2 a92a05e195b775b30ca410bc0a26e8e79e7b3bfb - mdrob - 2021-04-06 16:33:27]
	at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:61) ~[lucene-core-8.8.2.jar:8.8.2 a92a05e195b775b30ca410bc0a26e8e79e7b3bfb - mdrob - 2021-04-06 16:33:27]
	at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:58) ~[lucene-core-8.8.2.jar:8.8.2 a92a05e195b775b30ca410bc0a26e8e79e7b3bfb - mdrob - 2021-04-06 16:33:27]
	at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:720) ~[lucene-core-8.8.2.jar:8.8.2 a92a05e195b775b30ca410bc0a26e8e79e7b3bfb - mdrob - 2021-04-06 16:33:27]
	at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:81) ~[lucene-core-8.8.2.jar:8.8.2 a92a05e195b775b30ca410bc0a26e8e79e7b3bfb - mdrob - 2021-04-06 16:33:27]
	at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:63) ~[lucene-core-8.8.2.jar:8.8.2 a92a05e195b775b30ca410bc0a26e8e79e7b3bfb - mdrob - 2021-04-06 16:33:27]
	at org.elasticsearch.gateway.PersistedClusterStateService.nodeMetadata(PersistedClusterStateService.java:266) ~[crate-server.jar:?]
	at org.elasticsearch.env.NodeEnvironment.loadNodeMetadata(NodeEnvironment.java:416) ~[crate-server.jar:?]
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:329) ~[crate-server.jar:?]
	at org.elasticsearch.node.Node.<init>(Node.java:270) ~[crate-server.jar:?]
	at io.crate.node.CrateNode.<init>(CrateNode.java:53) ~[crate-app.jar:4.6.1]
	at org.elasticsearch.bootstrap.BootstrapProxy$1.<init>(BootstrapProxy.java:192) ~[crate-app.jar:4.6.1]
	at org.elasticsearch.bootstrap.BootstrapProxy.setup(BootstrapProxy.java:192) ~[crate-app.jar:4.6.1]
	at org.elasticsearch.bootstrap.BootstrapProxy.init(BootstrapProxy.java:266) ~[crate-app.jar:4.6.1]
	at io.crate.bootstrap.CrateDB.init(CrateDB.java:158) ~[crate-app.jar:4.6.1]

proddata · January 10, 2022, 7:24am

Did you upgrade your CrateDB installation?

mahoboy468 · January 10, 2022, 8:21am

no, CrateDB version is 4.6.1

proddata · January 10, 2022, 5:00pm

Could you share some info about your setup and what you did?
Docker, linux packages, tar ball?

mahoboy468 · January 11, 2022, 8:20am

installed from Linux Packages
Ubuntu 20.04

jayeff · January 11, 2022, 8:25am

@mahoboy468 Any chance you did upgrade the crate package recently? Maybe happened during the reboot? When was your CrateDB cluster set up originally (rough timeframe)?

mahoboy468 · January 11, 2022, 8:30am

@jayeff No, it has not been updated. system worked fine, i think database files crashed after reboot.
Can it be reindex or create new node folder and move prev node files to new node?

jayeff · January 11, 2022, 8:45am

@mahoboy468 It’s difficult to tell what potentially happened or how to proceed without having much insight into your setup.

You could try to move away the data-folders and start with a fresh, empty data-folder. If you have a backup you could potentially restore the files from the backup. If your tables are replicated CrateDB would restore the data from the replica automatically.

mahoboy468 · January 11, 2022, 9:05am

@jayeff

system have 1 node, and i haven’t backup or snapshot
i try this step and crate start fine without data, and Crate create nodes folder automatically

mv /var/lib/crate/nodes /var/lib/crate/nodes_org
systemctl start crate

so,

systemctl stop crate
mv /var/lib/crate/nodes_org/0/indices /var/lib/crate/nodes/0/
systemctl start crate

crate started, but not show any data
Can i fix this problem only with indices folder and without old _state folder ?

jayeff · January 11, 2022, 10:04am

@mahoboy468 without backup or another node with replicated data I don’t know if you have a chance of restoring your data

mahoboy468 · January 12, 2022, 5:43am

@jayeff it’s very bad! other database engine have solutions for fix same problem!

proddata · January 12, 2022, 1:49pm

I am sorry, that we don’t have better news for you.

It is hard to say what really happened here, without getting the full story. It could easily be a hardware fault or an unlucky case when Crate was turned off in a not graceful way (e.g. power loss). The core data layer of CrateDB provides transactional semantics on file level. I would consider a corruption during runtime very very unlikely. If the node would be part of a cluster with replication enabled, such a corruption would most likely self-heal, without users even noticing it.

amotl · January 12, 2022, 2:45pm

Dear @mahoboy468,

your observations indeed indicate that the Lucene data files got corrupted.

It looks like the main directory file, segments_N is corrupted. This probably means that a power loss happened while a commit was running. ^[1]

Other resources like "Format version is not supported" - Elasticsearch - Discuss the Elastic Stack also indicate that there is no way to ordinarily recover from this situation.

In order to recover your data by carving them out of the raw data files, you might want to look at the Apache Luke program, as roughly outlined at The same full-text search engine for different pro... - Atlassian Community. You also might want to check out the Lucene CheckIndex program, see Lucene and Solr's CheckIndex to the Rescue! - DZone.

Please note that running those programs on your data is completely out of scope for any serious recovery procedure related to CrateDB, I am hereby only sharing some resources with you to indicate how others might have rescued the situation as a last resort. There is no guarantee that this will bring back any of your data. Always take backups!

With kind regards,
Andreas.

How to repair corrupted lucene index? - Stack Overflow ↩︎

Topic		Replies	Views
AccessDeniedException on _state CrateDB	3	758	October 21, 2021
TranslogCorruptedException CrateDB	9	528	September 3, 2024
Community Edition 4.0.4 CrateDB	2	1345	September 23, 2019
Cannot start crateDB service CrateDB	1	975	May 28, 2020
Kubernetes scale statefulsets error: [node.max_local_storage_nodes] (was [1])? CrateDB	2	471	April 7, 2023

Service not start , Format version is not supported

Related topics