I am new to Cratedb and trying to use this in my project. for testing I have imported IMDB database and everything was going smooth. I was really impressed with select query speed. But suddenly i could not see any data or tables in the admin console and also when I try to connect from CLI.
Possible Reason: Server was rebooted while the service still running. but it should not be so sensitive that I lost the entire tables and shards.
Cratedb version: crate-4.5.0
Cluster setup: Single node
I could able see the data files in the folder structure. can you please help me recover the tables.
Could you please elaborate what exactly you did.
- Are you running CrateDB locally (OS, docker, …)?
- which install method did you use?
- what folders are you refering to?
- what does the startup logs look like?
Thank you for your kind reply… Please find the below reply…
-
Are you running CrateDB locally (OS, docker, …)?
----yes running in my windows10 laptop with ubuntu subsystem -
which install method did you use?
---- crate-4.5.0.tar.gz untar the file and started using it… -
what folders are you refering to?
----I am referring to “crate-4.5.0/data/nodes/0/indices” folder
-
what does the startup logs look like?
----no error it just starts like a fresh setup and I could not see the created tables.
Startup log-----
[2021-05-21T00:19:54,391][INFO ][o.e.e.NodeEnvironment ] [Monte Turlon] using [1] data paths, mounts [[/mnt/c (C:\)]], net usable_space [318.4gb], net total_space [475.4gb], types [drvfs]
[2021-05-21T00:19:54,424][INFO ][o.e.e.NodeEnvironment ] [Monte Turlon] heap size [3.9gb], compressed ordinary object pointers [true]
[2021-05-21T00:19:54,448][INFO ][o.e.n.Node ] [Monte Turlon] node name [Monte Turlon], node ID [2BItAFyPQwumwyr6X0jrHg], cluster name [crate]
[2021-05-21T00:19:54,541][INFO ][o.e.n.Node ] [Monte Turlon] version[4.5.0], pid[42], build[fb7a533/2021-03-23T14:12:58Z], OS[Linux/4.4.0-18362-Microsoft/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/16/16+36]
[2021-05-21T00:19:55,790][INFO ][i.c.plugin ] [Monte Turlon] plugins loaded: [enterpriseFunctions, jmx-monitoring, lang-js]
[2021-05-21T00:19:58,570][INFO ][o.e.p.PluginsService ] [Monte Turlon] no modules loaded
[2021-05-21T00:19:58,575][INFO ][o.e.p.PluginsService ] [Monte Turlon] loaded plugin [crate-azure-discovery]
[2021-05-21T00:19:58,576][INFO ][o.e.p.PluginsService ] [Monte Turlon] loaded plugin [es-repository-hdfs]
[2021-05-21T00:19:58,576][INFO ][o.e.p.PluginsService ] [Monte Turlon] loaded plugin [io.crate.plugin.BlobPlugin]
[2021-05-21T00:19:58,577][INFO ][o.e.p.PluginsService ] [Monte Turlon] loaded plugin [io.crate.plugin.PluginLoaderPlugin]
[2021-05-21T00:19:58,578][INFO ][o.e.p.PluginsService ] [Monte Turlon] loaded plugin [io.crate.plugin.SrvPlugin]
[2021-05-21T00:19:58,579][INFO ][o.e.p.PluginsService ] [Monte Turlon] loaded plugin [io.crate.udc.plugin.UDCPlugin]
[2021-05-21T00:19:58,579][INFO ][o.e.p.PluginsService ] [Monte Turlon] loaded plugin [org.elasticsearch.analysis.common.CommonAnalysisPlugin]
[2021-05-21T00:19:58,580][INFO ][o.e.p.PluginsService ] [Monte Turlon] loaded plugin [org.elasticsearch.discovery.ec2.Ec2DiscoveryPlugin]
[2021-05-21T00:19:58,584][INFO ][o.e.p.PluginsService ] [Monte Turlon] loaded plugin [org.elasticsearch.plugin.analysis.AnalysisPhoneticPlugin]
[2021-05-21T00:19:58,585][INFO ][o.e.p.PluginsService ] [Monte Turlon] loaded plugin [org.elasticsearch.plugin.repository.url.URLRepositoryPlugin]
[2021-05-21T00:19:58,587][INFO ][o.e.p.PluginsService ] [Monte Turlon] loaded plugin [org.elasticsearch.repositories.azure.AzureRepositoryPlugin]
[2021-05-21T00:19:58,588][INFO ][o.e.p.PluginsService ] [Monte Turlon] loaded plugin [org.elasticsearch.repositories.s3.S3RepositoryPlugin]
[2021-05-21T00:19:58,589][INFO ][o.e.p.PluginsService ] [Monte Turlon] loaded plugin [org.elasticsearch.transport.Netty4Plugin]
[2021-05-21T00:20:00,163][INFO ][o.e.d.DiscoveryModule ] [Monte Turlon] using discovery type [zen] and seed hosts providers [settings]
[2021-05-21T00:20:01,654][INFO ][psql ] [Monte Turlon] PSQL SSL support is disabled.
[2021-05-21T00:20:02,379][INFO ][i.c.p.PipelineRegistry ] [Monte Turlon] HTTP SSL support is disabled.
[2021-05-21T00:20:02,506][INFO ][o.e.n.Node ] [Monte Turlon] initialized
[2021-05-21T00:20:02,508][INFO ][o.e.n.Node ] [Monte Turlon] starting ...
[2021-05-21T00:20:02,908][INFO ][psql ] [Monte Turlon] publish_address {127.0.0.1:5432}, bound_addresses {[::1]:5432}, {127.0.0.1:5432}
[2021-05-21T00:20:02,952][INFO ][o.e.h.n.Netty4HttpServerTransport] [Monte Turlon] publish_address {127.0.0.1:4200}, bound_addresses {[::1]:4200}, {127.0.0.1:4200}
[2021-05-21T00:20:02,991][INFO ][o.e.t.TransportService ] [Monte Turlon] publish_address {127.0.0.1:4300}, bound_addresses {[::1]:4300}, {127.0.0.1:4300}
[2021-05-21T00:20:03,122][WARN ][o.e.b.BootstrapChecks ] [Monte Turlon] I/O exception while trying to read [/proc/sys/vm/max_map_count]
java.nio.file.NoSuchFileException: /proc/sys/vm/max_map_count at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219) ~[?:?]
at java.nio.file.Files.newByteChannel(Files.java:375) ~[?:?]
at java.nio.file.Files.newByteChannel(Files.java:426) ~[?:?]
at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420) ~[?:?]
at java.nio.file.Files.newInputStream(Files.java:160) ~[?:?] at java.nio.file.Files.newBufferedReader(Files.java:2916) ~[?:?]
at java.nio.file.Files.newBufferedReader(Files.java:2948) ~[?:?]
at org.elasticsearch.bootstrap.BootstrapChecks$MaxMapCountCheck.getBufferedReader(BootstrapChecks.java:448) ~[crate-app.jar:4.5.0]
at org.elasticsearch.bootstrap.BootstrapChecks$MaxMapCountCheck.getMaxMapCount(BootstrapChecks.java:426) [crate-app.jar:4.5.0] at org.elasticsearch.bootstrap.BootstrapChecks$MaxMapCountCheck.getMaxMapCount(BootstrapChecks.java:420) [crate-app.jar:4.5.0]
at org.elasticsearch.bootstrap.BootstrapChecks$MaxMapCountCheck.check(BootstrapChecks.java:400) [crate-app.jar:4.5.0]
at org.elasticsearch.bootstrap.BootstrapChecks.check(BootstrapChecks.java:134) [crate-app.jar:4.5.0]
at org.elasticsearch.bootstrap.BootstrapChecks.check(BootstrapChecks.java:91) [crate-app.jar:4.5.0]
at org.elasticsearch.bootstrap.BootstrapChecks.check(BootstrapChecks.java:74) [crate-app.jar:4.5.0]
at org.elasticsearch.bootstrap.BootstrapProxy$1.validateNodeBeforeAcceptingRequests(BootstrapProxy.java:198) [crate-app.jar:4.5.0]
at org.elasticsearch.node.Node.start(Node.java:716) [crate-app.jar:4.5.0]
at org.elasticsearch.bootstrap.BootstrapProxy.start(BootstrapProxy.java:205) [crate-app.jar:4.5.0]
at org.elasticsearch.bootstrap.BootstrapProxy.init(BootstrapProxy.java:269) [crate-app.jar:4.5.0]
at io.crate.bootstrap.CrateDB.init(CrateDB.java:158) [crate-app.jar:4.5.0]
at io.crate.bootstrap.CrateDB.execute(CrateDB.java:138) [crate-app.jar:4.5.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:81) [crate-app.jar:4.5.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [crate-app.jar:4.5.0]
at org.elasticsearch.cli.Command.main(Command.java:90) [crate-app.jar:4.5.0]
at io.crate.bootstrap.CrateDB.main(CrateDB.java:91) [crate-app.jar:4.5.0]
at io.crate.bootstrap.CrateDB.main(CrateDB.java:84) [crate-app.jar:4.5.0]
[2021-05-21T00:20:03,296][WARN ][o.e.b.BootstrapChecks ] [Monte Turlon] initial heap size [268435456] not equal to maximum heap size [4202692608]; this can cause resize pauses and prevents mlockall from locking the entire heap. Set the heap size via the CRATE_HEAP_SIZE environment variable. See https://crate.io/docs/crate/howtos/en/latest/performance/memory.html
[2021-05-21T00:20:03,310][INFO ][o.e.c.c.Coordinator ] [Monte Turlon] cluster UUID [xqhJMLgsQFyheWZo2pKQlQ]
[2021-05-21T00:20:03,342][INFO ][o.e.c.c.ClusterBootstrapService] [Monte Turlon] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing master is discovered
[2021-05-21T00:20:03,466][INFO ][o.e.c.s.MasterService ] [Monte Turlon] elected-as-master ([1] nodes joined)[{Monte Turlon}{2BItAFyPQwumwyr6X0jrHg}{PgLRdPeURwqiG5BnCL_N5A}{127.0.0.1}{127.0.0.1:4300}{http_address=127.0.0.1:4200} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 4, version: 13, reason: master node changed {previous [], current [{Monte Turlon}{2BItAFyPQwumwyr6X0jrHg}{PgLRdPeURwqiG5BnCL_N5A}{127.0.0.1}{127.0.0.1:4300}{http_address=127.0.0.1:4200}]}
[2021-05-21T00:20:03,591][INFO ][o.e.c.s.ClusterApplierService] [Monte Turlon] master node changed {previous [], current [{Monte Turlon}{2BItAFyPQwumwyr6X0jrHg}{PgLRdPeURwqiG5BnCL_N5A}{127.0.0.1}{127.0.0.1:4300}{http_address=127.0.0.1:4200}]}, term: 4, version: 13, reason: Publication{term=4, version=13}
[2021-05-21T00:20:03,627][INFO ][o.e.n.Node ] [Monte Turlon] started
[2021-05-21T00:20:03,760][INFO ][o.e.g.GatewayService ] [Monte Turlon] recovered [0] indices into cluster_state
Did you change the data path (default: #path.data: data) in the crate.yml?
If there is data/files in that folder and the data would be corrupted, CrateDB normally would not even start.
I am sorry for the delay response… Your amazing… i was almost loosing hope on this DB… i tried to configure backup and enabled path.data for backup. after the reboot it was trying to find the files in backup folder… i have changed the config to default and everything back to normal… i could see the entire IMDB records… Amazing… I can confirm CrateDB is crash proof… 
3 Likes
Hi @Shakeel_Ahmed,
great to hear that you got back your data 
For backups on filesystem you want to set the path.repo setting. Afterwards you can do a CREATE REPOSITORY to allow snapshotting.