Runnig CrateDB from Docker it restarts every few days

CrateDB
1

Hello everyone, I am new to working with this database, I would like you to help me verify what I am doing wrong in my test configuration with this database, since I have noticed in the logs that the container is restarting every few days, and I don’t see a reason for this, I don’t know if I’m configuring something wrong, it’s a test installation in a virtual machine, I only have one instance running at the moment, and this are consulting for only 2 applications (grafana one of them). I share the logs and the docker-compose configuration.

  crate-db:
    labels:
      org.demo: 'demo'
    image: crate:${CRATE_VERSION}
    hostname: crate-db
    container_name: db-crate
    ports:
      - "4200:4200"
      - "4300:4300"
    command: crate -Cdiscovery.type=single-node -Cauth.host_based.enabled=false -Ccluster.name=democluster -Chttp.cors.enabled=true -Chttp.cors.allow-origin="*"
    deploy:
      restart_policy:
        condition: on-failure
    environment:
      - CRATE_HEAP_SIZE=2g
    volumes:
      - crate-db:/data


[2022-04-20T21:44:22,377][INFO ][o.e.e.NodeEnvironment    ] [Sass Maòr] using [1] data paths, mounts [[/data (/dev/vda1)]], net usable_space [64.6gb], net total_space [77.3gb], types [ext4]
[2022-04-20T21:44:22,382][INFO ][o.e.e.NodeEnvironment    ] [Sass Maòr] heap size [2gb], compressed ordinary object pointers [true]
[2022-04-20T21:44:22,626][INFO ][o.e.n.Node               ] [Sass Maòr] node name [Sass Maòr], node ID [-kqBrxp5SyWACFTnXFXhVg], cluster name [democluster]
[2022-04-20T21:44:22,644][INFO ][o.e.n.Node               ] [Sass Maòr] version[4.6.8], pid[1], build[4640961/2022-03-10T09:18:42Z], OS[Linux/5.4.0-97-generic/amd64], JVM[Eclipse Foundation/OpenJDK 64-Bit Server VM/16.0.2/16.0.2+7]
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[2022-04-20T21:44:24,156][INFO ][o.e.p.PluginsService     ] [Sass Maòr] no modules loaded
[2022-04-20T21:44:24,158][INFO ][o.e.p.PluginsService     ] [Sass Maòr] loaded plugin [crate-azure-discovery]
[2022-04-20T21:44:24,159][INFO ][o.e.p.PluginsService     ] [Sass Maòr] loaded plugin [crate-functions]
[2022-04-20T21:44:24,159][INFO ][o.e.p.PluginsService     ] [Sass Maòr] loaded plugin [crate-jmx-monitoring]
[2022-04-20T21:44:24,159][INFO ][o.e.p.PluginsService     ] [Sass Maòr] loaded plugin [crate-lang-js]
[2022-04-20T21:44:24,160][INFO ][o.e.p.PluginsService     ] [Sass Maòr] loaded plugin [es-analysis-common]
[2022-04-20T21:44:24,160][INFO ][o.e.p.PluginsService     ] [Sass Maòr] loaded plugin [es-analysis-phonetic]
[2022-04-20T21:44:24,161][INFO ][o.e.p.PluginsService     ] [Sass Maòr] loaded plugin [es-repository-azure]
[2022-04-20T21:44:24,161][INFO ][o.e.p.PluginsService     ] [Sass Maòr] loaded plugin [es-repository-hdfs]
[2022-04-20T21:44:24,161][INFO ][o.e.p.PluginsService     ] [Sass Maòr] loaded plugin [io.crate.plugin.BlobPlugin]
[2022-04-20T21:44:24,161][INFO ][o.e.p.PluginsService     ] [Sass Maòr] loaded plugin [io.crate.plugin.SQLPlugin]
[2022-04-20T21:44:24,161][INFO ][o.e.p.PluginsService     ] [Sass Maòr] loaded plugin [io.crate.plugin.SrvPlugin]
[2022-04-20T21:44:24,162][INFO ][o.e.p.PluginsService     ] [Sass Maòr] loaded plugin [io.crate.udc.plugin.UDCPlugin]
[2022-04-20T21:44:24,162][INFO ][o.e.p.PluginsService     ] [Sass Maòr] loaded plugin [org.elasticsearch.discovery.ec2.Ec2DiscoveryPlugin]
[2022-04-20T21:44:24,162][INFO ][o.e.p.PluginsService     ] [Sass Maòr] loaded plugin [org.elasticsearch.plugin.repository.url.URLRepositoryPlugin]
[2022-04-20T21:44:24,162][INFO ][o.e.p.PluginsService     ] [Sass Maòr] loaded plugin [org.elasticsearch.repositories.s3.S3RepositoryPlugin]
[2022-04-20T21:44:24,163][INFO ][o.e.p.PluginsService     ] [Sass Maòr] loaded plugin [org.elasticsearch.transport.Netty4Plugin]
[2022-04-20T21:44:25,392][INFO ][o.e.d.DiscoveryModule    ] [Sass Maòr] using discovery type [single-node] and seed hosts providers [settings]
[2022-04-20T21:44:26,635][INFO ][psql                     ] [Sass Maòr] PSQL SSL support is disabled.
[2022-04-20T21:44:26,855][INFO ][i.c.p.PipelineRegistry   ] [Sass Maòr] HTTP SSL support is disabled.
[2022-04-20T21:44:27,263][INFO ][o.e.n.Node               ] [Sass Maòr] initialized
[2022-04-20T21:44:27,263][INFO ][o.e.n.Node               ] [Sass Maòr] starting ...
[2022-04-20T21:44:27,450][INFO ][psql                     ] [Sass Maòr] publish_address {172.18.1.2:5432}, bound_addresses {172.18.1.2:5432}, {127.0.0.1:5432}
[2022-04-20T21:44:27,466][INFO ][o.e.h.n.Netty4HttpServerTransport] [Sass Maòr] publish_address {172.18.1.2:4200}, bound_addresses {172.18.1.2:4200}, {127.0.0.1:4200}
[2022-04-20T21:44:27,479][INFO ][o.e.t.TransportService   ] [Sass Maòr] publish_address {172.18.1.2:4300}, bound_addresses {172.18.1.2:4300}, {127.0.0.1:4300}
[2022-04-20T21:44:27,766][INFO ][o.e.c.c.Coordinator      ] [Sass Maòr] cluster UUID [qNV_DGgeRkm7Nzrf-fJ2Pw]
[2022-04-20T21:44:27,852][INFO ][o.e.c.s.MasterService    ] [Sass Maòr] elected-as-master ([1] nodes joined)[{Sass Maòr}{-kqBrxp5SyWACFTnXFXhVg}{Qvy2I2h_TieqjMI1N0pyHA}{172.18.1.2}{172.18.1.2:4300}{http_address=172.18.1.2:4200} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 10, version: 141, reason: master node changed {previous [], current [{Sass Maòr}{-kqBrxp5SyWACFTnXFXhVg}{Qvy2I2h_TieqjMI1N0pyHA}{172.18.1.2}{172.18.1.2:4300}{http_address=172.18.1.2:4200}]}
[2022-04-20T21:44:27,924][INFO ][o.e.c.s.ClusterApplierService] [Sass Maòr] master node changed {previous [], current [{Sass Maòr}{-kqBrxp5SyWACFTnXFXhVg}{Qvy2I2h_TieqjMI1N0pyHA}{172.18.1.2}{172.18.1.2:4300}{http_address=172.18.1.2:4200}]}, term: 10, version: 141, reason: Publication{term=10, version=141}
[2022-04-20T21:44:27,940][INFO ][o.e.n.Node               ] [Sass Maòr] started
[2022-04-20T21:44:28,052][INFO ][o.e.g.GatewayService     ] [Sass Maòr] recovered [4] indices into cluster_state
[2022-04-20T21:44:29,108][INFO ][o.e.c.r.a.AllocationService] [Sass Maòr] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[md_ets_metadata][2], [md_ets_metadata][1], [md_ets_metadata][0]]]).
[2022-04-25T03:44:30,901][INFO ][o.e.e.NodeEnvironment    ] [Zimba] using [1] data paths, mounts [[/data (/dev/vda1)]], net usable_space [64.7gb], net total_space [77.3gb], types [ext4]
[2022-04-25T03:44:30,904][INFO ][o.e.e.NodeEnvironment    ] [Zimba] heap size [2gb], compressed ordinary object pointers [true]
[2022-04-25T03:44:31,513][INFO ][o.e.n.Node               ] [Zimba] node name [Zimba], node ID [-kqBrxp5SyWACFTnXFXhVg], cluster name [democluster]
[2022-04-25T03:44:31,544][INFO ][o.e.n.Node               ] [Zimba] version[4.6.8], pid[1], build[4640961/2022-03-10T09:18:42Z], OS[Linux/5.4.0-97-generic/amd64], JVM[Eclipse Foundation/OpenJDK 64-Bit Server VM/16.0.2/16.0.2+7]
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[2022-04-25T03:44:33,181][INFO ][o.e.p.PluginsService     ] [Zimba] no modules loaded
[2022-04-25T03:44:33,185][INFO ][o.e.p.PluginsService     ] [Zimba] loaded plugin [crate-azure-discovery]
[2022-04-25T03:44:33,185][INFO ][o.e.p.PluginsService     ] [Zimba] loaded plugin [crate-functions]
[2022-04-25T03:44:33,185][INFO ][o.e.p.PluginsService     ] [Zimba] loaded plugin [crate-jmx-monitoring]
[2022-04-25T03:44:33,186][INFO ][o.e.p.PluginsService     ] [Zimba] loaded plugin [crate-lang-js]
[2022-04-25T03:44:33,186][INFO ][o.e.p.PluginsService     ] [Zimba] loaded plugin [es-analysis-common]
[2022-04-25T03:44:33,186][INFO ][o.e.p.PluginsService     ] [Zimba] loaded plugin [es-analysis-phonetic]
[2022-04-25T03:44:33,186][INFO ][o.e.p.PluginsService     ] [Zimba] loaded plugin [es-repository-azure]
[2022-04-25T03:44:33,186][INFO ][o.e.p.PluginsService     ] [Zimba] loaded plugin [es-repository-hdfs]
[2022-04-25T03:44:33,186][INFO ][o.e.p.PluginsService     ] [Zimba] loaded plugin [io.crate.plugin.BlobPlugin]
[2022-04-25T03:44:33,187][INFO ][o.e.p.PluginsService     ] [Zimba] loaded plugin [io.crate.plugin.SQLPlugin]
[2022-04-25T03:44:33,187][INFO ][o.e.p.PluginsService     ] [Zimba] loaded plugin [io.crate.plugin.SrvPlugin]
[2022-04-25T03:44:33,187][INFO ][o.e.p.PluginsService     ] [Zimba] loaded plugin [io.crate.udc.plugin.UDCPlugin]
[2022-04-25T03:44:33,187][INFO ][o.e.p.PluginsService     ] [Zimba] loaded plugin [org.elasticsearch.discovery.ec2.Ec2DiscoveryPlugin]
[2022-04-25T03:44:33,188][INFO ][o.e.p.PluginsService     ] [Zimba] loaded plugin [org.elasticsearch.plugin.repository.url.URLRepositoryPlugin]
[2022-04-25T03:44:33,188][INFO ][o.e.p.PluginsService     ] [Zimba] loaded plugin [org.elasticsearch.repositories.s3.S3RepositoryPlugin]
[2022-04-25T03:44:33,189][INFO ][o.e.p.PluginsService     ] [Zimba] loaded plugin [org.elasticsearch.transport.Netty4Plugin]
[2022-04-25T03:44:34,447][INFO ][o.e.d.DiscoveryModule    ] [Zimba] using discovery type [single-node] and seed hosts providers [settings]
[2022-04-25T03:44:35,298][INFO ][psql                     ] [Zimba] PSQL SSL support is disabled.
[2022-04-25T03:44:35,438][INFO ][i.c.p.PipelineRegistry   ] [Zimba] HTTP SSL support is disabled.
[2022-04-25T03:44:35,816][INFO ][o.e.n.Node               ] [Zimba] initialized
[2022-04-25T03:44:35,817][INFO ][o.e.n.Node               ] [Zimba] starting ...
[2022-04-25T03:44:35,985][INFO ][psql                     ] [Zimba] publish_address {172.18.1.2:5432}, bound_addresses {172.18.1.2:5432}, {127.0.0.1:5432}
[2022-04-25T03:44:36,003][INFO ][o.e.h.n.Netty4HttpServerTransport] [Zimba] publish_address {172.18.1.2:4200}, bound_addresses {172.18.1.2:4200}, {127.0.0.1:4200}
[2022-04-25T03:44:36,017][INFO ][o.e.t.TransportService   ] [Zimba] publish_address {172.18.1.2:4300}, bound_addresses {172.18.1.2:4300}, {127.0.0.1:4300}
[2022-04-25T03:44:36,341][INFO ][o.e.c.c.Coordinator      ] [Zimba] cluster UUID [qNV_DGgeRkm7Nzrf-fJ2Pw]
[2022-04-25T03:44:36,590][INFO ][o.e.c.s.MasterService    ] [Zimba] elected-as-master ([1] nodes joined)[{Zimba}{-kqBrxp5SyWACFTnXFXhVg}{vosOHCo6QrS_ac2CbHvNvA}{172.18.1.2}{172.18.1.2:4300}{http_address=172.18.1.2:4200} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 11, version: 160, reason: master node changed {previous [], current [{Zimba}{-kqBrxp5SyWACFTnXFXhVg}{vosOHCo6QrS_ac2CbHvNvA}{172.18.1.2}{172.18.1.2:4300}{http_address=172.18.1.2:4200}]}
[2022-04-25T03:44:36,841][INFO ][o.e.c.s.ClusterApplierService] [Zimba] master node changed {previous [], current [{Zimba}{-kqBrxp5SyWACFTnXFXhVg}{vosOHCo6QrS_ac2CbHvNvA}{172.18.1.2}{172.18.1.2:4300}{http_address=172.18.1.2:4200}]}, term: 11, version: 160, reason: Publication{term=11, version=160}
[2022-04-25T03:44:36,861][INFO ][o.e.n.Node               ] [Zimba] started
[2022-04-25T03:44:36,967][INFO ][o.e.g.GatewayService     ] [Zimba] recovered [4] indices into cluster_state
[2022-04-25T03:44:38,091][INFO ][o.e.c.r.a.AllocationService] [Zimba] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[md_ets_metadata][1], [md_ets_metadata][0], [md_ets_metadata][3]]]).
2

Hi @johncaipa,

can you check if you can retrieve logs for a crashed docker container and share them? Maybe also check docker events for potential pointers for your issue. Maybe these can give some indication what’s going on

Johannes

3

Dear @jayeff , thanks for your answer, the only logs I have are the ones I shared previously (the same logs are always repeated every time it is restarted), the container is not recreated, it seems that it only restarts in the event of a failure, I don’t know if I have the configuration wrong in my docker-compose, what could you recommend?

4

As the container is running for a couple days before the restart your docker-compose should be fine.

Maybe the crashed container is still around (docker ps -a) and you can try to access logs for the crashed container (docker logs <crashed-container-id>)?

Otherwise maybe you could add a volume and store the logs on host disk to access them after a crash.