# Frozen Cluster with FileSystemException **URL:** \ **Category:** CrateDB\ **Created:** [February 11, 2019, 10:43am UTC](https://community.cratedb.com/t/frozen-cluster-with-filesystemexception/144 "2019-02-11T10:43:18Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![boldsmurf](https://avatars.discourse-cdn.com/v4/letter/b/8dc957/32.png) [@boldsmurf](https://community.cratedb.com/u/boldsmurf)\ **Post date:** [February 11, 2019, 10:43am UTC](https://community.cratedb.com/t/frozen-cluster-with-filesystemexception/144/1 "2019-02-11T10:43:18Z")
I encountered an error on my 4 node Crate cluster with the following exception: java.nio.file.FileSystemException: /mnt/crate/nodes/0/node.lock: Read-only file system Writes were impossible on the whole cluster during the period this error persisted. I finally resolved it by rebooting the machines. What could have caused this? Any idea how to prevent this in the future? Thank you. ---
**Author:** ![mfussenegger](https://sea2.discourse-cdn.com/flex020/user_avatar/community.cratedb.com/mfussenegger/32/79_2.png) [@mfussenegger](https://community.cratedb.com/u/mfussenegger)\ **Post date:** [February 12, 2019, 10:02pm UTC](https://community.cratedb.com/t/frozen-cluster-with-filesystemexception/144/2 "2019-02-12T22:02:32Z")
This must have been caused due some event to the system CrateDB is running on. You’d probably have to check the kernel logs to figure out what happened (via `dmesg` or `journalctl -k`. CrateDB itself doesn’t have the capability to change a filesystem to read-only. ---
**Author:** ![boldsmurf](https://avatars.discourse-cdn.com/v4/letter/b/8dc957/32.png) [@boldsmurf](https://community.cratedb.com/u/boldsmurf)\ **Post date:** [February 18, 2019, 11:02am UTC](https://community.cratedb.com/t/frozen-cluster-with-filesystemexception/144/3 "2019-02-18T11:02:24Z")
Hello. Thanks for the prompt response. I got the same problem again in the last days and I have more information that I would present below: The following are the exact exceptions that I encountered (in order): – Logs begin at Sat 2019-02-09 09:16:44 UTC, end at Sun 2019-02-17 09:38:44 UTC. – Feb 16 18:16:50 crate-4 crate[3905]: [2019-02-16T18:16:50,122][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [li Feb 16 18:16:50 crate-4 crate[3905]: java.lang.InternalError: a fault occurred in a recent unsafe memory access operat Feb 16 18:16:50 crate-4 crate[3905]: at org.apache.lucene.codecs.blocktree.SegmentTermsEnumFrame.loadBlock(SegmentTerm Feb 16 18:16:50 crate-4 crate[3905]: at org.apache.lucene.codecs.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum Feb 16 18:16:50 crate-4 crate[3905]: at org.elasticsearch.common.lucene.uid.PerThreadIDAndVersionLookup.getDocID(PerTh Feb 16 18:16:50 crate-4 crate[3905]: at org.elasticsearch.common.lucene.uid.PerThreadIDAndVersionLookup.lookupVersion( Feb 16 18:16:50 crate-4 crate[3905]: at org.elasticsearch.common.lucene.uid.VersionsResolver.loadDocIdAndVersion(Versi Feb 16 18:16:50 crate-4 crate[3905]: at org.elasticsearch.common.lucene.uid.VersionsResolver.loadVersion(VersionsResol Feb 16 18:16:50 crate-4 crate[3905]: at org.elasticsearch.index.engine.InternalEngine.loadCurrentVersionFromIndex(Inte Feb 16 18:16:50 crate-4 crate[3905]: at org.elasticsearch.index.engine.InternalEngine.resolveDocVersion(InternalEngine Feb 16 18:16:50 crate-4 crate[3905]: at org.elasticsearch.index.engine.InternalEngine.compareOpToLuceneDocBasedOnVersi Feb 16 18:16:50 crate-4 crate[3905]: at org.elasticsearch.index.engine.InternalEngine.planIndexingAsNonPrimary(Interna Feb 16 18:16:50 crate-4 crate[3905]: at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:496) ~ Feb 16 18:16:50 crate-4 crate[3905]: at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:558) ~[crate-ap Feb 16 18:16:50 crate-4 crate[3905]: at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:547) ~[crate-ap Feb 16 18:16:50 crate-4 crate[3905]: at io.crate.execution.dml.upsert.TransportShardUpsertAction.shardIndexOperationOn Feb 16 18:16:50 crate-4 crate[3905]: at io.crate.execution.dml.upsert.TransportShardUpsertAction.processRequestItemsOn Feb 16 18:16:50 crate-4 crate[3905]: at io.crate.execution.dml.upsert.TransportShardUpsertAction.processRequestItemsOn Feb 16 18:16:50 crate-4 crate[3905]: at io.crate.execution.dml.TransportShardAction$2.call(TransportShardAction.java:1 Feb 16 18:16:50 crate-4 crate[3905]: at io.crate.execution.dml.TransportShardAction$2.call(TransportShardAction.java:1 Feb 16 18:16:50 crate-4 crate[3905]: at io.crate.execution.dml.TransportShardAction.wrapOperationInKillable(TransportS Feb 16 18:16:50 crate-4 crate[3905]: at io.crate.execution.dml.TransportShardAction.shardOperationOnReplica(TransportS Feb 16 18:16:50 crate-4 crate[3905]: at io.crate.execution.dml.TransportShardAction.shardOperationOnReplica(TransportS Feb 16 18:16:50 crate-4 crate[3905]: at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncR Feb 16 18:16:50 crate-4 crate[3905]: at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncR Feb 16 18:16:50 crate-4 crate[3905]: at org.elasticsearch.index.shard.IndexShardOperationsLock.acquire(IndexShardOpera Feb 16 18:16:50 crate-4 crate[3905]: at org.elasticsearch.index.shard.IndexShard.acquireReplicaOperationLock(IndexShar Feb 16 18:16:50 crate-4 crate[3905]: at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncR Feb 16 18:16:50 crate-4 crate[3905]: at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable Feb 16 18:16:50 crate-4 crate[3905]: at org.elasticsearch.action.support.replication.TransportReplicationAction$Replic Feb 16 18:16:50 crate-4 crate[3905]: at org.elasticsearch.action.support.replication.TransportReplicationAction$Replic Feb 16 18:16:50 crate-4 crate[3905]: at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(Requ Feb 16 18:16:50 crate-4 crate[3905]: at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.jav Feb 16 18:16:50 crate-4 crate[3905]: at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstra Feb 16 18:16:50 crate-4 crate[3905]: at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable Feb 16 18:16:50 crate-4 crate[3905]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149 Feb 16 18:16:50 crate-4 crate[3905]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624 Feb 16 18:16:50 crate-4 crate[3905]: at java.lang.Thread.run(Thread.java:748) [?:1.8.0\_161] Feb 16 18:16:50 crate-4 systemd[1]: crate.service: main process exited, code=exited, status=128/n/a Feb 16 18:16:50 crate-4 systemd[1]: Unit crate.service entered failed state. Feb 16 18:16:50 crate-4 systemd[1]: crate.service failed. And [2019-02-17T09:40:07,584][ERROR][o.e.b.BootstrapProxy] Exception java.lang.IllegalStateException: failed to obtain node locks, tried [[/var/lib/crate/bold, /mnt/crate/bold]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max\_local\_storage\_nodes] (was [1])? at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:263) ~[crate-app-2.3.4.jar:2.3.4] at org.elasticsearch.node.Node.(Node.java:268) ~[crate-app-2.3.4.jar:2.3.4] at io.crate.node.CrateNode.(CrateNode.java:62) ~[crate-app-2.3.4.jar:2.3.4] at org.elasticsearch.bootstrap.BootstrapProxy$1.(BootstrapProxy.java:199) ~[crate-app-2.3.4.jar:2.3.4] at org.elasticsearch.bootstrap.BootstrapProxy.setup(BootstrapProxy.java:199) ~[crate-app-2.3.4.jar:2.3.4] at org.elasticsearch.bootstrap.BootstrapProxy.init(BootstrapProxy.java:282) [crate-app-2.3.4.jar:2.3.4] at io.crate.bootstrap.CrateDB.init(CrateDB.java:138) [crate-app-2.3.4.jar:2.3.4] at io.crate.bootstrap.CrateDB.execute(CrateDB.java:118) [crate-app-2.3.4.jar:2.3.4] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:76) [crate-app-2.3.4.jar:2.3.4] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) [crate-app-2.3.4.jar:2.3.4] at org.elasticsearch.cli.Command.main(Command.java:90) [crate-app-2.3.4.jar:2.3.4] at io.crate.bootstrap.CrateDB.main(CrateDB.java:87) [crate-app-2.3.4.jar:2.3.4] at io.crate.bootstrap.CrateDB.main(CrateDB.java:80) [crate-app-2.3.4.jar:2.3.4] Caused by: java.io.IOException: failed to obtain lock on /mnt/crate/nodes/0 at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:242) ~[crate-app-2.3.4.jar:2.3.4] … 12 more Caused by: java.nio.file.FileSystemException: /mnt/crate/nodes/0/node.lock: Read-only file system at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91) ~[?:1.8.0\_161] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0\_161] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:1.8.0\_161] at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177) ~[?:1.8.0\_161] at java.nio.channels.FileChannel.open(FileChannel.java:287) ~[?:1.8.0\_161] at java.nio.channels.FileChannel.open(FileChannel.java:335) ~[?:1.8.0\_161] at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:113) ~[lucene-core-6.6.1.jar:6.6.1 9aa465a89b64ff2dabe7b4d50c472de32c298683 - varunthacker - 2017-08-29 21:54:39] at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41) ~[lucene-core-6.6.1.jar:6.6.1 9aa465a89b64ff2dabe7b4d50c472de32c298683 - varunthacker - 2017-08-29 21:54:39] at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45) ~[lucene-core-6.6.1.jar:6.6.1 9aa465a89b64ff2dabe7b4d50c472de32c298683 - varunthacker - 2017-08-29 21:54:39] at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:228) ~[crate-app-2.3.4.jar:2.3.4] … 12 more Using the Crate documentation, I found the ALTER CLUSTER statement and executed as follows: `ALTER CLUSTER REROUTE RETRY FAILED;` That reduced the number of shards that were not started, but I still have a couple with the following explanation: `shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2019-02-18T10:59:33.191Z], failed_attempts[13], delayed=false, details[failed to create shard, failure FileSystemException[/mnt/crate/nodes/0/indices/cGCye6v4RS-UVjhFuxE6eA/7/_state/state-1.st.tmp: Read-only file system]], allocation_status[no_attempt]]]` Does this provide more context for the problem? What should I do in this case. Simply restarting everything no longer works. Thanks ---
**Author:** ![boldsmurf](https://avatars.discourse-cdn.com/v4/letter/b/8dc957/32.png) [@boldsmurf](https://community.cratedb.com/u/boldsmurf)\ **Post date:** [February 18, 2019, 12:53pm UTC](https://community.cratedb.com/t/frozen-cluster-with-filesystemexception/144/4 "2019-02-18T12:53:59Z")
Ok. I figured out what the problem was and how to fix it. You were exactly right before, it was disk related. So what i did was the following: sudo service crate stop #stop crate service cp -r /mnt/crate ~/backup/ #backup all my files, of course df -h # note device path, here /dev/mapper/volume-1e719f29o1 sudo umount /mnt/ # unmount device sudo fsck /dev/mapper/volume-1e719f20p1 # check the disk, everything was “clean” in this case sudo mount /dev/mapper/volume-1e719f20p1 # mounted it again sudo service crate start If you encounter an error like this afterwards: Likely root cause: org.apache.lucene.index.CorruptIndexException: misplaced codec footer (file truncated?): length=0 but footerLength==16 (resource=SimpleFSIndexInput(path="/mnt/crate/nodes/0/indices/8OGRcEOxQhqEN\_\_XG1JtBQ/\_state/state-395.st")) check if it is a zero byte file. If it is, simply rename the file and start your crate node again. Everything worked. Thanks for an awesome product.