# 3 nodes cluster suddenly failing **URL:** \ **Category:** CrateDB\ **Created:** [May 13, 2021, 6:03pm UTC](https://community.cratedb.com/t/3-nodes-cluster-suddenly-failing/674 "2021-05-13T18:03:43Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![cande1gut](https://avatars.discourse-cdn.com/v4/letter/c/65b543/32.png) [@cande1gut](https://community.cratedb.com/u/cande1gut)\ **Post date:** [May 13, 2021, 6:03pm UTC](https://community.cratedb.com/t/3-nodes-cluster-suddenly-failing/674/1 "2021-05-13T18:03:43Z")
Hi, I was having a 3 nodes cluster made from docker and it suddenly stopped working: ![crate](https://us1.discourse-cdn.com/flex020/uploads/crate/original/1X/0ed1b746df283435bdd38d77e9900d73e956d6b0.png) If I go and check the shards, they appear alongside the 3 nodes but if I go and check the cluster page, they are not there. Any help is appreciated, thanks! ---
**Author:** ![proddata](https://sea2.discourse-cdn.com/flex020/user_avatar/community.cratedb.com/proddata/32/1379_2.png) [@proddata](https://community.cratedb.com/u/proddata)\ **Post date:** [May 13, 2021, 6:29pm UTC](https://community.cratedb.com/t/3-nodes-cluster-suddenly-failing/674/2 "2021-05-13T18:29:01Z")
Are any of the docker containers still running? Do you have any logs? ---
**Author:** ![cande1gut](https://avatars.discourse-cdn.com/v4/letter/c/65b543/32.png) [@cande1gut](https://community.cratedb.com/u/cande1gut)\ **Post date:** [May 13, 2021, 7:15pm UTC](https://community.cratedb.com/t/3-nodes-cluster-suddenly-failing/674/3 "2021-05-13T19:15:48Z")
This is what the log was looking while trying to reinitialize the cluster: ``` cratedb01_1 | [2021-05-13T18:17:33,300][INFO][o.e.e.NodeEnvironment] [cratedb01] using [1] data paths, mounts [[/data (/dev/vdc)]], net usable_space [2.7tb], net total_space [2.8tb], types [ext4] cratedb01_1 | [2021-05-13T18:17:33,304][INFO][o.e.e.NodeEnvironment] [cratedb01] heap size [2gb], compressed ordinary object pointers [true] cratedb01_1 | [2021-05-13T18:17:33,347][INFO][o.e.n.Node] [cratedb01] node name [cratedb01], node ID [5WUG0hGJTUSaOLZvAjDdKw], cluster name [crate-docker-cluster] cratedb01_1 | [2021-05-13T18:17:33,363][INFO][o.e.n.Node] [cratedb01] version[4.5.0], pid[1], build[fb7a533/2021-03-23T14:12:58Z], OS[Linux/4.15.0-137-generic/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/16/16+36] cratedb01_1 | [2021-05-13T18:17:33,628][INFO][i.c.plugin] [cratedb01] plugins loaded: [lang-js, enterpriseFunctions, jmx-monitoring] cratedb01_1 | SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". cratedb01_1 | SLF4J: Defaulting to no-operation (NOP) logger implementation cratedb01_1 | SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. cratedb03_1 | [2021-05-13T18:17:34,507][INFO][o.e.e.NodeEnvironment] [cratedb03] using [1] data paths, mounts [[/data (/dev/vdc)]], net usable_space [2.7tb], net total_space [2.8tb], types [ext4] cratedb03_1 | [2021-05-13T18:17:34,510][INFO][o.e.e.NodeEnvironment] [cratedb03] heap size [2gb], compressed ordinary object pointers [true] cratedb03_1 | [2021-05-13T18:17:34,563][INFO][o.e.n.Node] [cratedb03] node name [cratedb03], node ID [Tp-wj9ZmSqWCCV3ZVBJ4qQ], cluster name [crate-docker-cluster] cratedb03_1 | [2021-05-13T18:17:34,585][INFO][o.e.n.Node] [cratedb03] version[4.5.0], pid[1], build[fb7a533/2021-03-23T14:12:58Z], OS[Linux/4.15.0-137-generic/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/16/16+36] cratedb01_1 | [2021-05-13T18:17:34,748][INFO][o.e.p.PluginsService] [cratedb01] no modules loaded cratedb01_1 | [2021-05-13T18:17:34,750][INFO][o.e.p.PluginsService] [cratedb01] loaded plugin [crate-azure-discovery] cratedb01_1 | [2021-05-13T18:17:34,750][INFO][o.e.p.PluginsService] [cratedb01] loaded plugin [es-repository-hdfs] cratedb01_1 | [2021-05-13T18:17:34,750][INFO][o.e.p.PluginsService] [cratedb01] loaded plugin [io.crate.plugin.BlobPlugin] cratedb01_1 | [2021-05-13T18:17:34,750][INFO][o.e.p.PluginsService] [cratedb01] loaded plugin [io.crate.plugin.PluginLoaderPlugin] cratedb01_1 | [2021-05-13T18:17:34,751][INFO][o.e.p.PluginsService] [cratedb01] loaded plugin [io.crate.plugin.SrvPlugin] cratedb01_1 | [2021-05-13T18:17:34,751][INFO][o.e.p.PluginsService] [cratedb01] loaded plugin [io.crate.udc.plugin.UDCPlugin] cratedb01_1 | [2021-05-13T18:17:34,751][INFO][o.e.p.PluginsService] [cratedb01] loaded plugin [org.elasticsearch.analysis.common.CommonAnalysisPlugin] cratedb01_1 | [2021-05-13T18:17:34,751][INFO][o.e.p.PluginsService] [cratedb01] loaded plugin [org.elasticsearch.discovery.ec2.Ec2DiscoveryPlugin] cratedb01_1 | [2021-05-13T18:17:34,751][INFO][o.e.p.PluginsService] [cratedb01] loaded plugin [org.elasticsearch.plugin.analysis.AnalysisPhoneticPlugin] cratedb01_1 | [2021-05-13T18:17:34,752][INFO][o.e.p.PluginsService] [cratedb01] loaded plugin [org.elasticsearch.plugin.repository.url.URLRepositoryPlugin] cratedb01_1 | [2021-05-13T18:17:34,752][INFO][o.e.p.PluginsService] [cratedb01] loaded plugin [org.elasticsearch.repositories.azure.AzureRepositoryPlugin] cratedb01_1 | [2021-05-13T18:17:34,752][INFO][o.e.p.PluginsService] [cratedb01] loaded plugin [org.elasticsearch.repositories.s3.S3RepositoryPlugin] cratedb01_1 | [2021-05-13T18:17:34,752][INFO][o.e.p.PluginsService] [cratedb01] loaded plugin [org.elasticsearch.transport.Netty4Plugin] cratedb03_1 | [2021-05-13T18:17:34,963][INFO][i.c.plugin] [cratedb03] plugins loaded: [lang-js, enterpriseFunctions, jmx-monitoring] cratedb02_1 | [2021-05-13T18:17:35,598][INFO][o.e.e.NodeEnvironment] [cratedb02] using [1] data paths, mounts [[/data (/dev/vdc)]], net usable_space [2.7tb], net total_space [2.8tb], types [ext4] cratedb02_1 | [2021-05-13T18:17:35,601][INFO][o.e.e.NodeEnvironment] [cratedb02] heap size [2gb], compressed ordinary object pointers [true] cratedb02_1 | [2021-05-13T18:17:35,646][INFO][o.e.n.Node] [cratedb02] node name [cratedb02], node ID [zsmo2SZvQTezuB9HCAFYKQ], cluster name [crate-docker-cluster] cratedb02_1 | [2021-05-13T18:17:35,662][INFO][o.e.n.Node] [cratedb02] version[4.5.0], pid[1], build[fb7a533/2021-03-23T14:12:58Z], OS[Linux/4.15.0-137-generic/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/16/16+36] cratedb01_1 | [2021-05-13T18:17:35,674][INFO][o.e.d.DiscoveryModule] [cratedb01] using discovery type [zen] and seed hosts providers [settings] cratedb03_1 | SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". cratedb03_1 | SLF4J: Defaulting to no-operation (NOP) logger implementation cratedb03_1 | SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. cratedb02_1 | [2021-05-13T18:17:36,027][INFO][i.c.plugin] [cratedb02] plugins loaded: [lang-js, enterpriseFunctions, jmx-monitoring] cratedb03_1 | [2021-05-13T18:17:36,148][INFO][o.e.p.PluginsService] [cratedb03] no modules loaded cratedb03_1 | [2021-05-13T18:17:36,150][INFO][o.e.p.PluginsService] [cratedb03] loaded plugin [crate-azure-discovery] cratedb03_1 | [2021-05-13T18:17:36,150][INFO][o.e.p.PluginsService] [cratedb03] loaded plugin [es-repository-hdfs] cratedb03_1 | [2021-05-13T18:17:36,150][INFO][o.e.p.PluginsService] [cratedb03] loaded plugin [io.crate.plugin.BlobPlugin] cratedb03_1 | [2021-05-13T18:17:36,151][INFO][o.e.p.PluginsService] [cratedb03] loaded plugin [io.crate.plugin.PluginLoaderPlugin] cratedb03_1 | [2021-05-13T18:17:36,151][INFO][o.e.p.PluginsService] [cratedb03] loaded plugin [io.crate.plugin.SrvPlugin] cratedb03_1 | [2021-05-13T18:17:36,151][INFO][o.e.p.PluginsService] [cratedb03] loaded plugin [io.crate.udc.plugin.UDCPlugin] cratedb03_1 | [2021-05-13T18:17:36,151][INFO][o.e.p.PluginsService] [cratedb03] loaded plugin [org.elasticsearch.analysis.common.CommonAnalysisPlugin] cratedb03_1 | [2021-05-13T18:17:36,151][INFO][o.e.p.PluginsService] [cratedb03] loaded plugin [org.elasticsearch.discovery.ec2.Ec2DiscoveryPlugin] cratedb03_1 | [2021-05-13T18:17:36,151][INFO][o.e.p.PluginsService] [cratedb03] loaded plugin [org.elasticsearch.plugin.analysis.AnalysisPhoneticPlugin] cratedb03_1 | [2021-05-13T18:17:36,151][INFO][o.e.p.PluginsService] [cratedb03] loaded plugin [org.elasticsearch.plugin.repository.url.URLRepositoryPlugin] cratedb03_1 | [2021-05-13T18:17:36,151][INFO][o.e.p.PluginsService] [cratedb03] loaded plugin [org.elasticsearch.repositories.azure.AzureRepositoryPlugin] cratedb03_1 | [2021-05-13T18:17:36,151][INFO][o.e.p.PluginsService] [cratedb03] loaded plugin [org.elasticsearch.repositories.s3.S3RepositoryPlugin] cratedb03_1 | [2021-05-13T18:17:36,152][INFO][o.e.p.PluginsService] [cratedb03] loaded plugin [org.elasticsearch.transport.Netty4Plugin] cratedb01_1 | [2021-05-13T18:17:36,549][INFO][psql] [cratedb01] PSQL SSL support is disabled. cratedb01_1 | [2021-05-13T18:17:36,724][INFO][i.c.p.PipelineRegistry] [cratedb01] HTTP SSL support is disabled. cratedb01_1 | [2021-05-13T18:17:36,774][INFO][o.e.n.Node] [cratedb01] initialized cratedb01_1 | [2021-05-13T18:17:36,775][INFO][o.e.n.Node] [cratedb01] starting ... cratedb02_1 | SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". cratedb02_1 | SLF4J: Defaulting to no-operation (NOP) logger implementation cratedb02_1 | SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. cratedb01_1 | [2021-05-13T18:17:36,928][INFO][psql] [cratedb01] publish_address {192.168.112.3:5432}, bound_addresses {192.168.112.3:5432} cratedb01_1 | [2021-05-13T18:17:36,943][INFO][o.e.h.n.Netty4HttpServerTransport] [cratedb01] publish_address {192.168.112.3:4200}, bound_addresses {192.168.112.3:4200} cratedb01_1 | [2021-05-13T18:17:36,955][INFO][o.e.t.TransportService] [cratedb01] publish_address {192.168.112.3:4300}, bound_addresses {192.168.112.3:4300} cratedb03_1 | [2021-05-13T18:17:36,987][INFO][o.e.d.DiscoveryModule] [cratedb03] using discovery type [zen] and seed hosts providers [settings] cratedb02_1 | [2021-05-13T18:17:37,187][INFO][o.e.p.PluginsService] [cratedb02] no modules loaded cratedb02_1 | [2021-05-13T18:17:37,189][INFO][o.e.p.PluginsService] [cratedb02] loaded plugin [crate-azure-discovery] cratedb02_1 | [2021-05-13T18:17:37,189][INFO][o.e.p.PluginsService] [cratedb02] loaded plugin [es-repository-hdfs] cratedb02_1 | [2021-05-13T18:17:37,189][INFO][o.e.p.PluginsService] [cratedb02] loaded plugin [io.crate.plugin.BlobPlugin] cratedb02_1 | [2021-05-13T18:17:37,189][INFO][o.e.p.PluginsService] [cratedb02] loaded plugin [io.crate.plugin.PluginLoaderPlugin] cratedb02_1 | [2021-05-13T18:17:37,190][INFO][o.e.p.PluginsService] [cratedb02] loaded plugin [io.crate.plugin.SrvPlugin] cratedb02_1 | [2021-05-13T18:17:37,190][INFO][o.e.p.PluginsService] [cratedb02] loaded plugin [io.crate.udc.plugin.UDCPlugin] cratedb02_1 | [2021-05-13T18:17:37,190][INFO][o.e.p.PluginsService] [cratedb02] loaded plugin [org.elasticsearch.analysis.common.CommonAnalysisPlugin] cratedb02_1 | [2021-05-13T18:17:37,191][INFO][o.e.p.PluginsService] [cratedb02] loaded plugin [org.elasticsearch.discovery.ec2.Ec2DiscoveryPlugin] cratedb02_1 | [2021-05-13T18:17:37,191][INFO][o.e.p.PluginsService] [cratedb02] loaded plugin [org.elasticsearch.plugin.analysis.AnalysisPhoneticPlugin] cratedb02_1 | [2021-05-13T18:17:37,191][INFO][o.e.p.PluginsService] [cratedb02] loaded plugin [org.elasticsearch.plugin.repository.url.URLRepositoryPlugin] cratedb02_1 | [2021-05-13T18:17:37,192][INFO][o.e.p.PluginsService] [cratedb02] loaded plugin [org.elasticsearch.repositories.azure.AzureRepositoryPlugin] cratedb02_1 | [2021-05-13T18:17:37,192][INFO][o.e.p.PluginsService] [cratedb02] loaded plugin [org.elasticsearch.repositories.s3.S3RepositoryPlugin] cratedb02_1 | [2021-05-13T18:17:37,192][INFO][o.e.p.PluginsService] [cratedb02] loaded plugin [org.elasticsearch.transport.Netty4Plugin] cratedb01_1 | [2021-05-13T18:17:37,510][INFO][o.e.b.BootstrapChecks] [cratedb01] bound or publishing to a non-loopback address, enforcing bootstrap checks cratedb01_1 | [2021-05-13T18:17:37,512][INFO][o.e.c.c.Coordinator] [cratedb01] cluster UUID [kiTC8eA_S6uxXC0jMcubCw] cratedb03_1 | [2021-05-13T18:17:38,169][INFO][psql] [cratedb03] PSQL SSL support is disabled. cratedb02_1 | [2021-05-13T18:17:38,173][INFO][o.e.d.DiscoveryModule] [cratedb02] using discovery type [zen] and seed hosts providers [settings] cratedb03_1 | [2021-05-13T18:17:38,413][INFO][i.c.p.PipelineRegistry] [cratedb03] HTTP SSL support is disabled. cratedb03_1 | [2021-05-13T18:17:38,482][INFO][o.e.n.Node] [cratedb03] initialized cratedb03_1 | [2021-05-13T18:17:38,482][INFO][o.e.n.Node] [cratedb03] starting ... cratedb03_1 | [2021-05-13T18:17:38,611][INFO][psql] [cratedb03] publish_address {192.168.112.5:5432}, bound_addresses {192.168.112.5:5432} cratedb03_1 | [2021-05-13T18:17:38,624][INFO][o.e.h.n.Netty4HttpServerTransport] [cratedb03] publish_address {192.168.112.5:4200}, bound_addresses {192.168.112.5:4200} cratedb03_1 | [2021-05-13T18:17:38,636][INFO][o.e.t.TransportService] [cratedb03] publish_address {192.168.112.5:4300}, bound_addresses {192.168.112.5:4300} cratedb03_1 | [2021-05-13T18:17:39,172][INFO][o.e.b.BootstrapChecks] [cratedb03] bound or publishing to a non-loopback address, enforcing bootstrap checks cratedb03_1 | [2021-05-13T18:17:39,174][INFO][o.e.c.c.Coordinator] [cratedb03] cluster UUID [kiTC8eA_S6uxXC0jMcubCw] cratedb02_1 | [2021-05-13T18:17:39,176][INFO][psql] [cratedb02] PSQL SSL support is disabled. cratedb02_1 | [2021-05-13T18:17:39,374][INFO][i.c.p.PipelineRegistry] [cratedb02] HTTP SSL support is disabled. cratedb02_1 | [2021-05-13T18:17:39,437][INFO][o.e.n.Node] [cratedb02] initialized cratedb02_1 | [2021-05-13T18:17:39,437][INFO][o.e.n.Node] [cratedb02] starting ... cratedb02_1 | [2021-05-13T18:17:39,593][INFO][psql] [cratedb02] publish_address {192.168.112.6:5432}, bound_addresses {192.168.112.6:5432} cratedb02_1 | [2021-05-13T18:17:39,605][INFO][o.e.h.n.Netty4HttpServerTransport] [cratedb02] publish_address {192.168.112.6:4200}, bound_addresses {192.168.112.6:4200} cratedb02_1 | [2021-05-13T18:17:39,618][INFO][o.e.t.TransportService] [cratedb02] publish_address {192.168.112.6:4300}, bound_addresses {192.168.112.6:4300} cratedb03_1 | [2021-05-13T18:17:39,726][INFO][o.e.c.s.MasterService] [cratedb03] elected-as-master ([2] nodes joined)[{cratedb01}{5WUG0hGJTUSaOLZvAjDdKw}{rN1VxDUYReiq3-b4f99wsw}{192.168.112.3}{192.168.112.3:4300}{http_address=192.168.112.3:4200} elect leader, {cratedb03}{Tp-wj9ZmSqWCCV3ZVBJ4qQ}{YzwXtt2YTnuxjW3iFifjnw}{192.168.112.5}{192.168.112.5:4300}{http_address=192.168.112.5:4200} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 119, version: 4712, reason: master node changed {previous [], current [{cratedb03}{Tp-wj9ZmSqWCCV3ZVBJ4qQ}{YzwXtt2YTnuxjW3iFifjnw}{192.168.112.5}{192.168.112.5:4300}{http_address=192.168.112.5:4200}]}, added {{cratedb01}{5WUG0hGJTUSaOLZvAjDdKw}{rN1VxDUYReiq3-b4f99wsw}{192.168.112.3}{192.168.112.3:4300}{http_address=192.168.112.3:4200},} cratedb02_1 | [2021-05-13T18:17:39,931][INFO][o.e.b.BootstrapChecks] [cratedb02] bound or publishing to a non-loopback address, enforcing bootstrap checks cratedb02_1 | [2021-05-13T18:17:39,933][INFO][o.e.c.c.Coordinator] [cratedb02] cluster UUID [kiTC8eA_S6uxXC0jMcubCw] cratedb01_1 | [2021-05-13T18:17:43,320][INFO][o.e.c.s.ClusterApplierService] [cratedb01] master node changed {previous [], current [{cratedb03}{Tp-wj9ZmSqWCCV3ZVBJ4qQ}{YzwXtt2YTnuxjW3iFifjnw}{192.168.112.5}{192.168.112.5:4300}{http_address=192.168.112.5:4200}]}, added {{cratedb03}{Tp-wj9ZmSqWCCV3ZVBJ4qQ}{YzwXtt2YTnuxjW3iFifjnw}{192.168.112.5}{192.168.112.5:4300}{http_address=192.168.112.5:4200},}, term: 119, version: 4712, reason: ApplyCommitRequest{term=119, version=4712, sourceNode={cratedb03}{Tp-wj9ZmSqWCCV3ZVBJ4qQ}{YzwXtt2YTnuxjW3iFifjnw}{192.168.112.5}{192.168.112.5:4300}{http_address=192.168.112.5:4200}} cratedb01_1 | [2021-05-13T18:17:43,333][INFO][o.e.n.Node] [cratedb01] started cratedb03_1 | [2021-05-13T18:17:43,343][INFO][o.e.c.s.ClusterApplierService] [cratedb03] master node changed {previous [], current [{cratedb03}{Tp-wj9ZmSqWCCV3ZVBJ4qQ}{YzwXtt2YTnuxjW3iFifjnw}{192.168.112.5}{192.168.112.5:4300}{http_address=192.168.112.5:4200}]}, added {{cratedb01}{5WUG0hGJTUSaOLZvAjDdKw}{rN1VxDUYReiq3-b4f99wsw}{192.168.112.3}{192.168.112.3:4300}{http_address=192.168.112.3:4200},}, term: 119, version: 4712, reason: Publication{term=119, version=4712} cratedb03_1 | [2021-05-13T18:17:43,357][INFO][o.e.g.GatewayService] [cratedb03] delaying initial state recovery for [5m]. expecting [3] nodes, but only have [2] cratedb03_1 | [2021-05-13T18:17:43,360][INFO][o.e.n.Node] [cratedb03] started cratedb03_1 | [2021-05-13T18:17:43,370][INFO][o.e.c.s.MasterService] [cratedb03] node-join[{cratedb02}{zsmo2SZvQTezuB9HCAFYKQ}{Y9dzrT-iSsynM_34fn1JJg}{192.168.112.6}{192.168.112.6:4300}{http_address=192.168.112.6:4200} join existing leader], term: 119, version: 4713, reason: added {{cratedb02}{zsmo2SZvQTezuB9HCAFYKQ}{Y9dzrT-iSsynM_34fn1JJg}{192.168.112.6}{192.168.112.6:4300}{http_address=192.168.112.6:4200},} cratedb01_1 | [2021-05-13T18:17:43,650][INFO][o.e.c.s.ClusterApplierService] [cratedb01] added {{cratedb02}{zsmo2SZvQTezuB9HCAFYKQ}{Y9dzrT-iSsynM_34fn1JJg}{192.168.112.6}{192.168.112.6:4300}{http_address=192.168.112.6:4200},}, term: 119, version: 4713, reason: ApplyCommitRequest{term=119, version=4713, sourceNode={cratedb03}{Tp-wj9ZmSqWCCV3ZVBJ4qQ}{YzwXtt2YTnuxjW3iFifjnw}{192.168.112.5}{192.168.112.5:4300}{http_address=192.168.112.5:4200}} cratedb02_1 | [2021-05-13T18:17:46,184][INFO][o.e.c.s.ClusterApplierService] [cratedb02] master node changed {previous [], current [{cratedb03}{Tp-wj9ZmSqWCCV3ZVBJ4qQ}{YzwXtt2YTnuxjW3iFifjnw}{192.168.112.5}{192.168.112.5:4300}{http_address=192.168.112.5:4200}]}, added {{cratedb01}{5WUG0hGJTUSaOLZvAjDdKw}{rN1VxDUYReiq3-b4f99wsw}{192.168.112.3}{192.168.112.3:4300}{http_address=192.168.112.3:4200},{cratedb03}{Tp-wj9ZmSqWCCV3ZVBJ4qQ}{YzwXtt2YTnuxjW3iFifjnw}{192.168.112.5}{192.168.112.5:4300}{http_address=192.168.112.5:4200},}, term: 119, version: 4713, reason: ApplyCommitRequest{term=119, version=4713, sourceNode={cratedb03}{Tp-wj9ZmSqWCCV3ZVBJ4qQ}{YzwXtt2YTnuxjW3iFifjnw}{192.168.112.5}{192.168.112.5:4300}{http_address=192.168.112.5:4200}} cratedb02_1 | [2021-05-13T18:17:46,197][INFO][o.e.n.Node] [cratedb02] started cratedb03_1 | [2021-05-13T18:17:46,198][INFO][o.e.c.s.ClusterApplierService] [cratedb03] added {{cratedb02}{zsmo2SZvQTezuB9HCAFYKQ}{Y9dzrT-iSsynM_34fn1JJg}{192.168.112.6}{192.168.112.6:4300}{http_address=192.168.112.6:4200},}, term: 119, version: 4713, reason: Publication{term=119, version=4713} cratedb03_1 | [2021-05-13T18:17:47,225][INFO][o.e.g.GatewayService] [cratedb03] recovered [28] indices into cluster_state cratedb03_1 | [2021-05-13T18:18:21,241][INFO][o.e.c.r.a.AllocationService] [cratedb03] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[energyeast_users][3], [energyeast_users][1], [energyeast_users][0]]]). ``` I decided to remove a couple of tables and suddenly the cluster is up again (odd). The containers have not been updated nor stopped. ---
**Author:** ![proddata](https://sea2.discourse-cdn.com/flex020/user_avatar/community.cratedb.com/proddata/32/1379_2.png) [@proddata](https://community.cratedb.com/u/proddata)\ **Post date:** [May 14, 2021, 1:02pm UTC](https://community.cratedb.com/t/3-nodes-cluster-suddenly-failing/674/4 "2021-05-14T13:02:10Z")
Ok … the logs just show a pretty normal startup. > [@cande1gut](#): > > I decided to remove a couple of tables and suddenly the cluster is up again (odd). The containers have not been updated nor stopped. How did you remove them, with the cluster down? 🤔 ---
**Author:** ![cande1gut](https://avatars.discourse-cdn.com/v4/letter/c/65b543/32.png) [@cande1gut](https://community.cratedb.com/u/cande1gut)\ **Post date:** [May 15, 2021, 4:36am UTC](https://community.cratedb.com/t/3-nodes-cluster-suddenly-failing/674/5 "2021-05-15T04:36:09Z")
That was a weird situation. The shards were visible in the UI, where I noticed the nodes to be up. Once in the console view, I executed the queries and after the deletion the cluster was up again. ---
**Author:** ![proddata](https://sea2.discourse-cdn.com/flex020/user_avatar/community.cratedb.com/proddata/32/1379_2.png) [@proddata](https://community.cratedb.com/u/proddata)\ **Post date:** [May 15, 2021, 5:57am UTC](https://community.cratedb.com/t/3-nodes-cluster-suddenly-failing/674/6 "2021-05-15T05:57:10Z")
Do you remember what you did before? Was there anything special about the tables you deleted? My suspicion would be, that actually none of the nodes were really down, but there might be a bug with the admin UI. ---
**Author:** ![cande1gut](https://avatars.discourse-cdn.com/v4/letter/c/65b543/32.png) [@cande1gut](https://community.cratedb.com/u/cande1gut)\ **Post date:** [May 15, 2021, 6:01pm UTC](https://community.cratedb.com/t/3-nodes-cluster-suddenly-failing/674/7 "2021-05-15T18:01:43Z")
I checked the UI one day before the problem and everything was working as expected. The next day that I tried to insert new data the cluster was down. I restarted it without luck and by pure chance I thought about deleting a couple of tables. Somehow this triggered something. Nothing special with the tables, they hold around 20 columns and 1k-20k rows. ---
**Author:** ![proddata](https://sea2.discourse-cdn.com/flex020/user_avatar/community.cratedb.com/proddata/32/1379_2.png) [@proddata](https://community.cratedb.com/u/proddata)\ **Post date:** [May 20, 2021, 11:42am UTC](https://community.cratedb.com/t/3-nodes-cluster-suddenly-failing/674/8 "2021-05-20T11:42:48Z")
How did you delete the tables while the cluster wasn’t running 🤔 ? ---
**Author:** ![cande1gut](https://avatars.discourse-cdn.com/v4/letter/c/65b543/32.png) [@cande1gut](https://community.cratedb.com/u/cande1gut)\ **Post date:** [May 20, 2021, 3:09pm UTC](https://community.cratedb.com/t/3-nodes-cluster-suddenly-failing/674/9 "2021-05-20T15:09:34Z")
The UI was still visible, I entered it in the console page. ---
**Author:** ![proddata](https://sea2.discourse-cdn.com/flex020/user_avatar/community.cratedb.com/proddata/32/1379_2.png) [@proddata](https://community.cratedb.com/u/proddata)\ **Post date:** [May 20, 2021, 3:14pm UTC](https://community.cratedb.com/t/3-nodes-cluster-suddenly-failing/674/10 "2021-05-20T15:14:05Z")
I see. So this means the cluster was still running, but there maybe was a problem with the AdminUI. Can you maybe explain what you did, and maybe share the schemas of the tables. You could also DM me, if you don’t want to share them publicly. Did you use any UDFs in the Schemas? ---
**Author:** ![cande1gut](https://avatars.discourse-cdn.com/v4/letter/c/65b543/32.png) [@cande1gut](https://community.cratedb.com/u/cande1gut)\ **Post date:** [July 22, 2021, 4:28pm UTC](https://community.cratedb.com/t/3-nodes-cluster-suddenly-failing/674/11 "2021-07-22T16:28:52Z")
Hey @proddata, sorry for not replying back sooner. I really don’t know what happened, maybe was a server problem while rebooted or space constraint with the docker container, since I have been dealing with some of those problems recently. I didn’t use any UDF over the data, and the schemas are simply strings, integers and array of strings.