Hi, I was having a 3 nodes cluster made from docker and it suddenly stopped working:
If I go and check the shards, they appear alongside the 3 nodes but if I go and check the cluster page, they are not there. Any help is appreciated, thanks!
Are any of the docker containers still running?
Do you have any logs?
This is what the log was looking while trying to reinitialize the cluster:
cratedb01_1 | [2021-05-13T18:17:33,300][INFO ][o.e.e.NodeEnvironment ] [cratedb01] using [1] data paths, mounts [[/data (/dev/vdc)]], net usable_space [2.7tb], net total_space [2.8tb], types [ext4]
cratedb01_1 | [2021-05-13T18:17:33,304][INFO ][o.e.e.NodeEnvironment ] [cratedb01] heap size [2gb], compressed ordinary object pointers [true]
cratedb01_1 | [2021-05-13T18:17:33,347][INFO ][o.e.n.Node ] [cratedb01] node name [cratedb01], node ID [5WUG0hGJTUSaOLZvAjDdKw], cluster name [crate-docker-cluster]
cratedb01_1 | [2021-05-13T18:17:33,363][INFO ][o.e.n.Node ] [cratedb01] version[4.5.0], pid[1], build[fb7a533/2021-03-23T14:12:58Z], OS[Linux/4.15.0-137-generic/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/16/16+36]
cratedb01_1 | [2021-05-13T18:17:33,628][INFO ][i.c.plugin ] [cratedb01] plugins loaded: [lang-js, enterpriseFunctions, jmx-monitoring]
cratedb01_1 | SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
cratedb01_1 | SLF4J: Defaulting to no-operation (NOP) logger implementation
cratedb01_1 | SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
cratedb03_1 | [2021-05-13T18:17:34,507][INFO ][o.e.e.NodeEnvironment ] [cratedb03] using [1] data paths, mounts [[/data (/dev/vdc)]], net usable_space [2.7tb], net total_space [2.8tb], types [ext4]
cratedb03_1 | [2021-05-13T18:17:34,510][INFO ][o.e.e.NodeEnvironment ] [cratedb03] heap size [2gb], compressed ordinary object pointers [true]
cratedb03_1 | [2021-05-13T18:17:34,563][INFO ][o.e.n.Node ] [cratedb03] node name [cratedb03], node ID [Tp-wj9ZmSqWCCV3ZVBJ4qQ], cluster name [crate-docker-cluster]
cratedb03_1 | [2021-05-13T18:17:34,585][INFO ][o.e.n.Node ] [cratedb03] version[4.5.0], pid[1], build[fb7a533/2021-03-23T14:12:58Z], OS[Linux/4.15.0-137-generic/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/16/16+36]
cratedb01_1 | [2021-05-13T18:17:34,748][INFO ][o.e.p.PluginsService ] [cratedb01] no modules loaded
cratedb01_1 | [2021-05-13T18:17:34,750][INFO ][o.e.p.PluginsService ] [cratedb01] loaded plugin [crate-azure-discovery]
cratedb01_1 | [2021-05-13T18:17:34,750][INFO ][o.e.p.PluginsService ] [cratedb01] loaded plugin [es-repository-hdfs]
cratedb01_1 | [2021-05-13T18:17:34,750][INFO ][o.e.p.PluginsService ] [cratedb01] loaded plugin [io.crate.plugin.BlobPlugin]
cratedb01_1 | [2021-05-13T18:17:34,750][INFO ][o.e.p.PluginsService ] [cratedb01] loaded plugin [io.crate.plugin.PluginLoaderPlugin]
cratedb01_1 | [2021-05-13T18:17:34,751][INFO ][o.e.p.PluginsService ] [cratedb01] loaded plugin [io.crate.plugin.SrvPlugin]
cratedb01_1 | [2021-05-13T18:17:34,751][INFO ][o.e.p.PluginsService ] [cratedb01] loaded plugin [io.crate.udc.plugin.UDCPlugin]
cratedb01_1 | [2021-05-13T18:17:34,751][INFO ][o.e.p.PluginsService ] [cratedb01] loaded plugin [org.elasticsearch.analysis.common.CommonAnalysisPlugin]
cratedb01_1 | [2021-05-13T18:17:34,751][INFO ][o.e.p.PluginsService ] [cratedb01] loaded plugin [org.elasticsearch.discovery.ec2.Ec2DiscoveryPlugin]
cratedb01_1 | [2021-05-13T18:17:34,751][INFO ][o.e.p.PluginsService ] [cratedb01] loaded plugin [org.elasticsearch.plugin.analysis.AnalysisPhoneticPlugin]
cratedb01_1 | [2021-05-13T18:17:34,752][INFO ][o.e.p.PluginsService ] [cratedb01] loaded plugin [org.elasticsearch.plugin.repository.url.URLRepositoryPlugin]
cratedb01_1 | [2021-05-13T18:17:34,752][INFO ][o.e.p.PluginsService ] [cratedb01] loaded plugin [org.elasticsearch.repositories.azure.AzureRepositoryPlugin]
cratedb01_1 | [2021-05-13T18:17:34,752][INFO ][o.e.p.PluginsService ] [cratedb01] loaded plugin [org.elasticsearch.repositories.s3.S3RepositoryPlugin]
cratedb01_1 | [2021-05-13T18:17:34,752][INFO ][o.e.p.PluginsService ] [cratedb01] loaded plugin [org.elasticsearch.transport.Netty4Plugin]
cratedb03_1 | [2021-05-13T18:17:34,963][INFO ][i.c.plugin ] [cratedb03] plugins loaded: [lang-js, enterpriseFunctions, jmx-monitoring]
cratedb02_1 | [2021-05-13T18:17:35,598][INFO ][o.e.e.NodeEnvironment ] [cratedb02] using [1] data paths, mounts [[/data (/dev/vdc)]], net usable_space [2.7tb], net total_space [2.8tb], types [ext4]
cratedb02_1 | [2021-05-13T18:17:35,601][INFO ][o.e.e.NodeEnvironment ] [cratedb02] heap size [2gb], compressed ordinary object pointers [true]
cratedb02_1 | [2021-05-13T18:17:35,646][INFO ][o.e.n.Node ] [cratedb02] node name [cratedb02], node ID [zsmo2SZvQTezuB9HCAFYKQ], cluster name [crate-docker-cluster]
cratedb02_1 | [2021-05-13T18:17:35,662][INFO ][o.e.n.Node ] [cratedb02] version[4.5.0], pid[1], build[fb7a533/2021-03-23T14:12:58Z], OS[Linux/4.15.0-137-generic/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/16/16+36]
cratedb01_1 | [2021-05-13T18:17:35,674][INFO ][o.e.d.DiscoveryModule ] [cratedb01] using discovery type [zen] and seed hosts providers [settings]
cratedb03_1 | SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
cratedb03_1 | SLF4J: Defaulting to no-operation (NOP) logger implementation
cratedb03_1 | SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
cratedb02_1 | [2021-05-13T18:17:36,027][INFO ][i.c.plugin ] [cratedb02] plugins loaded: [lang-js, enterpriseFunctions, jmx-monitoring]
cratedb03_1 | [2021-05-13T18:17:36,148][INFO ][o.e.p.PluginsService ] [cratedb03] no modules loaded
cratedb03_1 | [2021-05-13T18:17:36,150][INFO ][o.e.p.PluginsService ] [cratedb03] loaded plugin [crate-azure-discovery]
cratedb03_1 | [2021-05-13T18:17:36,150][INFO ][o.e.p.PluginsService ] [cratedb03] loaded plugin [es-repository-hdfs]
cratedb03_1 | [2021-05-13T18:17:36,150][INFO ][o.e.p.PluginsService ] [cratedb03] loaded plugin [io.crate.plugin.BlobPlugin]
cratedb03_1 | [2021-05-13T18:17:36,151][INFO ][o.e.p.PluginsService ] [cratedb03] loaded plugin [io.crate.plugin.PluginLoaderPlugin]
cratedb03_1 | [2021-05-13T18:17:36,151][INFO ][o.e.p.PluginsService ] [cratedb03] loaded plugin [io.crate.plugin.SrvPlugin]
cratedb03_1 | [2021-05-13T18:17:36,151][INFO ][o.e.p.PluginsService ] [cratedb03] loaded plugin [io.crate.udc.plugin.UDCPlugin]
cratedb03_1 | [2021-05-13T18:17:36,151][INFO ][o.e.p.PluginsService ] [cratedb03] loaded plugin [org.elasticsearch.analysis.common.CommonAnalysisPlugin]
cratedb03_1 | [2021-05-13T18:17:36,151][INFO ][o.e.p.PluginsService ] [cratedb03] loaded plugin [org.elasticsearch.discovery.ec2.Ec2DiscoveryPlugin]
cratedb03_1 | [2021-05-13T18:17:36,151][INFO ][o.e.p.PluginsService ] [cratedb03] loaded plugin [org.elasticsearch.plugin.analysis.AnalysisPhoneticPlugin]
cratedb03_1 | [2021-05-13T18:17:36,151][INFO ][o.e.p.PluginsService ] [cratedb03] loaded plugin [org.elasticsearch.plugin.repository.url.URLRepositoryPlugin]
cratedb03_1 | [2021-05-13T18:17:36,151][INFO ][o.e.p.PluginsService ] [cratedb03] loaded plugin [org.elasticsearch.repositories.azure.AzureRepositoryPlugin]
cratedb03_1 | [2021-05-13T18:17:36,151][INFO ][o.e.p.PluginsService ] [cratedb03] loaded plugin [org.elasticsearch.repositories.s3.S3RepositoryPlugin]
cratedb03_1 | [2021-05-13T18:17:36,152][INFO ][o.e.p.PluginsService ] [cratedb03] loaded plugin [org.elasticsearch.transport.Netty4Plugin]
cratedb01_1 | [2021-05-13T18:17:36,549][INFO ][psql ] [cratedb01] PSQL SSL support is disabled.
cratedb01_1 | [2021-05-13T18:17:36,724][INFO ][i.c.p.PipelineRegistry ] [cratedb01] HTTP SSL support is disabled.
cratedb01_1 | [2021-05-13T18:17:36,774][INFO ][o.e.n.Node ] [cratedb01] initialized
cratedb01_1 | [2021-05-13T18:17:36,775][INFO ][o.e.n.Node ] [cratedb01] starting ...
cratedb02_1 | SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
cratedb02_1 | SLF4J: Defaulting to no-operation (NOP) logger implementation
cratedb02_1 | SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
cratedb01_1 | [2021-05-13T18:17:36,928][INFO ][psql ] [cratedb01] publish_address {192.168.112.3:5432}, bound_addresses {192.168.112.3:5432}
cratedb01_1 | [2021-05-13T18:17:36,943][INFO ][o.e.h.n.Netty4HttpServerTransport] [cratedb01] publish_address {192.168.112.3:4200}, bound_addresses {192.168.112.3:4200}
cratedb01_1 | [2021-05-13T18:17:36,955][INFO ][o.e.t.TransportService ] [cratedb01] publish_address {192.168.112.3:4300}, bound_addresses {192.168.112.3:4300}
cratedb03_1 | [2021-05-13T18:17:36,987][INFO ][o.e.d.DiscoveryModule ] [cratedb03] using discovery type [zen] and seed hosts providers [settings]
cratedb02_1 | [2021-05-13T18:17:37,187][INFO ][o.e.p.PluginsService ] [cratedb02] no modules loaded
cratedb02_1 | [2021-05-13T18:17:37,189][INFO ][o.e.p.PluginsService ] [cratedb02] loaded plugin [crate-azure-discovery]
cratedb02_1 | [2021-05-13T18:17:37,189][INFO ][o.e.p.PluginsService ] [cratedb02] loaded plugin [es-repository-hdfs]
cratedb02_1 | [2021-05-13T18:17:37,189][INFO ][o.e.p.PluginsService ] [cratedb02] loaded plugin [io.crate.plugin.BlobPlugin]
cratedb02_1 | [2021-05-13T18:17:37,189][INFO ][o.e.p.PluginsService ] [cratedb02] loaded plugin [io.crate.plugin.PluginLoaderPlugin]
cratedb02_1 | [2021-05-13T18:17:37,190][INFO ][o.e.p.PluginsService ] [cratedb02] loaded plugin [io.crate.plugin.SrvPlugin]
cratedb02_1 | [2021-05-13T18:17:37,190][INFO ][o.e.p.PluginsService ] [cratedb02] loaded plugin [io.crate.udc.plugin.UDCPlugin]
cratedb02_1 | [2021-05-13T18:17:37,190][INFO ][o.e.p.PluginsService ] [cratedb02] loaded plugin [org.elasticsearch.analysis.common.CommonAnalysisPlugin]
cratedb02_1 | [2021-05-13T18:17:37,191][INFO ][o.e.p.PluginsService ] [cratedb02] loaded plugin [org.elasticsearch.discovery.ec2.Ec2DiscoveryPlugin]
cratedb02_1 | [2021-05-13T18:17:37,191][INFO ][o.e.p.PluginsService ] [cratedb02] loaded plugin [org.elasticsearch.plugin.analysis.AnalysisPhoneticPlugin]
cratedb02_1 | [2021-05-13T18:17:37,191][INFO ][o.e.p.PluginsService ] [cratedb02] loaded plugin [org.elasticsearch.plugin.repository.url.URLRepositoryPlugin]
cratedb02_1 | [2021-05-13T18:17:37,192][INFO ][o.e.p.PluginsService ] [cratedb02] loaded plugin [org.elasticsearch.repositories.azure.AzureRepositoryPlugin]
cratedb02_1 | [2021-05-13T18:17:37,192][INFO ][o.e.p.PluginsService ] [cratedb02] loaded plugin [org.elasticsearch.repositories.s3.S3RepositoryPlugin]
cratedb02_1 | [2021-05-13T18:17:37,192][INFO ][o.e.p.PluginsService ] [cratedb02] loaded plugin [org.elasticsearch.transport.Netty4Plugin]
cratedb01_1 | [2021-05-13T18:17:37,510][INFO ][o.e.b.BootstrapChecks ] [cratedb01] bound or publishing to a non-loopback address, enforcing bootstrap checks
cratedb01_1 | [2021-05-13T18:17:37,512][INFO ][o.e.c.c.Coordinator ] [cratedb01] cluster UUID [kiTC8eA_S6uxXC0jMcubCw]
cratedb03_1 | [2021-05-13T18:17:38,169][INFO ][psql ] [cratedb03] PSQL SSL support is disabled.
cratedb02_1 | [2021-05-13T18:17:38,173][INFO ][o.e.d.DiscoveryModule ] [cratedb02] using discovery type [zen] and seed hosts providers [settings]
cratedb03_1 | [2021-05-13T18:17:38,413][INFO ][i.c.p.PipelineRegistry ] [cratedb03] HTTP SSL support is disabled.
cratedb03_1 | [2021-05-13T18:17:38,482][INFO ][o.e.n.Node ] [cratedb03] initialized
cratedb03_1 | [2021-05-13T18:17:38,482][INFO ][o.e.n.Node ] [cratedb03] starting ...
cratedb03_1 | [2021-05-13T18:17:38,611][INFO ][psql ] [cratedb03] publish_address {192.168.112.5:5432}, bound_addresses {192.168.112.5:5432}
cratedb03_1 | [2021-05-13T18:17:38,624][INFO ][o.e.h.n.Netty4HttpServerTransport] [cratedb03] publish_address {192.168.112.5:4200}, bound_addresses {192.168.112.5:4200}
cratedb03_1 | [2021-05-13T18:17:38,636][INFO ][o.e.t.TransportService ] [cratedb03] publish_address {192.168.112.5:4300}, bound_addresses {192.168.112.5:4300}
cratedb03_1 | [2021-05-13T18:17:39,172][INFO ][o.e.b.BootstrapChecks ] [cratedb03] bound or publishing to a non-loopback address, enforcing bootstrap checks
cratedb03_1 | [2021-05-13T18:17:39,174][INFO ][o.e.c.c.Coordinator ] [cratedb03] cluster UUID [kiTC8eA_S6uxXC0jMcubCw]
cratedb02_1 | [2021-05-13T18:17:39,176][INFO ][psql ] [cratedb02] PSQL SSL support is disabled.
cratedb02_1 | [2021-05-13T18:17:39,374][INFO ][i.c.p.PipelineRegistry ] [cratedb02] HTTP SSL support is disabled.
cratedb02_1 | [2021-05-13T18:17:39,437][INFO ][o.e.n.Node ] [cratedb02] initialized
cratedb02_1 | [2021-05-13T18:17:39,437][INFO ][o.e.n.Node ] [cratedb02] starting ...
cratedb02_1 | [2021-05-13T18:17:39,593][INFO ][psql ] [cratedb02] publish_address {192.168.112.6:5432}, bound_addresses {192.168.112.6:5432}
cratedb02_1 | [2021-05-13T18:17:39,605][INFO ][o.e.h.n.Netty4HttpServerTransport] [cratedb02] publish_address {192.168.112.6:4200}, bound_addresses {192.168.112.6:4200}
cratedb02_1 | [2021-05-13T18:17:39,618][INFO ][o.e.t.TransportService ] [cratedb02] publish_address {192.168.112.6:4300}, bound_addresses {192.168.112.6:4300}
cratedb03_1 | [2021-05-13T18:17:39,726][INFO ][o.e.c.s.MasterService ] [cratedb03] elected-as-master ([2] nodes joined)[{cratedb01}{5WUG0hGJTUSaOLZvAjDdKw}{rN1VxDUYReiq3-b4f99wsw}{192.168.112.3}{192.168.112.3:4300}{http_address=192.168.112.3:4200} elect leader, {cratedb03}{Tp-wj9ZmSqWCCV3ZVBJ4qQ}{YzwXtt2YTnuxjW3iFifjnw}{192.168.112.5}{192.168.112.5:4300}{http_address=192.168.112.5:4200} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 119, version: 4712, reason: master node changed {previous [], current [{cratedb03}{Tp-wj9ZmSqWCCV3ZVBJ4qQ}{YzwXtt2YTnuxjW3iFifjnw}{192.168.112.5}{192.168.112.5:4300}{http_address=192.168.112.5:4200}]}, added {{cratedb01}{5WUG0hGJTUSaOLZvAjDdKw}{rN1VxDUYReiq3-b4f99wsw}{192.168.112.3}{192.168.112.3:4300}{http_address=192.168.112.3:4200},}
cratedb02_1 | [2021-05-13T18:17:39,931][INFO ][o.e.b.BootstrapChecks ] [cratedb02] bound or publishing to a non-loopback address, enforcing bootstrap checks
cratedb02_1 | [2021-05-13T18:17:39,933][INFO ][o.e.c.c.Coordinator ] [cratedb02] cluster UUID [kiTC8eA_S6uxXC0jMcubCw]
cratedb01_1 | [2021-05-13T18:17:43,320][INFO ][o.e.c.s.ClusterApplierService] [cratedb01] master node changed {previous [], current [{cratedb03}{Tp-wj9ZmSqWCCV3ZVBJ4qQ}{YzwXtt2YTnuxjW3iFifjnw}{192.168.112.5}{192.168.112.5:4300}{http_address=192.168.112.5:4200}]}, added {{cratedb03}{Tp-wj9ZmSqWCCV3ZVBJ4qQ}{YzwXtt2YTnuxjW3iFifjnw}{192.168.112.5}{192.168.112.5:4300}{http_address=192.168.112.5:4200},}, term: 119, version: 4712, reason: ApplyCommitRequest{term=119, version=4712, sourceNode={cratedb03}{Tp-wj9ZmSqWCCV3ZVBJ4qQ}{YzwXtt2YTnuxjW3iFifjnw}{192.168.112.5}{192.168.112.5:4300}{http_address=192.168.112.5:4200}}
cratedb01_1 | [2021-05-13T18:17:43,333][INFO ][o.e.n.Node ] [cratedb01] started
cratedb03_1 | [2021-05-13T18:17:43,343][INFO ][o.e.c.s.ClusterApplierService] [cratedb03] master node changed {previous [], current [{cratedb03}{Tp-wj9ZmSqWCCV3ZVBJ4qQ}{YzwXtt2YTnuxjW3iFifjnw}{192.168.112.5}{192.168.112.5:4300}{http_address=192.168.112.5:4200}]}, added {{cratedb01}{5WUG0hGJTUSaOLZvAjDdKw}{rN1VxDUYReiq3-b4f99wsw}{192.168.112.3}{192.168.112.3:4300}{http_address=192.168.112.3:4200},}, term: 119, version: 4712, reason: Publication{term=119, version=4712}
cratedb03_1 | [2021-05-13T18:17:43,357][INFO ][o.e.g.GatewayService ] [cratedb03] delaying initial state recovery for [5m]. expecting [3] nodes, but only have [2]
cratedb03_1 | [2021-05-13T18:17:43,360][INFO ][o.e.n.Node ] [cratedb03] started
cratedb03_1 | [2021-05-13T18:17:43,370][INFO ][o.e.c.s.MasterService ] [cratedb03] node-join[{cratedb02}{zsmo2SZvQTezuB9HCAFYKQ}{Y9dzrT-iSsynM_34fn1JJg}{192.168.112.6}{192.168.112.6:4300}{http_address=192.168.112.6:4200} join existing leader], term: 119, version: 4713, reason: added {{cratedb02}{zsmo2SZvQTezuB9HCAFYKQ}{Y9dzrT-iSsynM_34fn1JJg}{192.168.112.6}{192.168.112.6:4300}{http_address=192.168.112.6:4200},}
cratedb01_1 | [2021-05-13T18:17:43,650][INFO ][o.e.c.s.ClusterApplierService] [cratedb01] added {{cratedb02}{zsmo2SZvQTezuB9HCAFYKQ}{Y9dzrT-iSsynM_34fn1JJg}{192.168.112.6}{192.168.112.6:4300}{http_address=192.168.112.6:4200},}, term: 119, version: 4713, reason: ApplyCommitRequest{term=119, version=4713, sourceNode={cratedb03}{Tp-wj9ZmSqWCCV3ZVBJ4qQ}{YzwXtt2YTnuxjW3iFifjnw}{192.168.112.5}{192.168.112.5:4300}{http_address=192.168.112.5:4200}}
cratedb02_1 | [2021-05-13T18:17:46,184][INFO ][o.e.c.s.ClusterApplierService] [cratedb02] master node changed {previous [], current [{cratedb03}{Tp-wj9ZmSqWCCV3ZVBJ4qQ}{YzwXtt2YTnuxjW3iFifjnw}{192.168.112.5}{192.168.112.5:4300}{http_address=192.168.112.5:4200}]}, added {{cratedb01}{5WUG0hGJTUSaOLZvAjDdKw}{rN1VxDUYReiq3-b4f99wsw}{192.168.112.3}{192.168.112.3:4300}{http_address=192.168.112.3:4200},{cratedb03}{Tp-wj9ZmSqWCCV3ZVBJ4qQ}{YzwXtt2YTnuxjW3iFifjnw}{192.168.112.5}{192.168.112.5:4300}{http_address=192.168.112.5:4200},}, term: 119, version: 4713, reason: ApplyCommitRequest{term=119, version=4713, sourceNode={cratedb03}{Tp-wj9ZmSqWCCV3ZVBJ4qQ}{YzwXtt2YTnuxjW3iFifjnw}{192.168.112.5}{192.168.112.5:4300}{http_address=192.168.112.5:4200}}
cratedb02_1 | [2021-05-13T18:17:46,197][INFO ][o.e.n.Node ] [cratedb02] started
cratedb03_1 | [2021-05-13T18:17:46,198][INFO ][o.e.c.s.ClusterApplierService] [cratedb03] added {{cratedb02}{zsmo2SZvQTezuB9HCAFYKQ}{Y9dzrT-iSsynM_34fn1JJg}{192.168.112.6}{192.168.112.6:4300}{http_address=192.168.112.6:4200},}, term: 119, version: 4713, reason: Publication{term=119, version=4713}
cratedb03_1 | [2021-05-13T18:17:47,225][INFO ][o.e.g.GatewayService ] [cratedb03] recovered [28] indices into cluster_state
cratedb03_1 | [2021-05-13T18:18:21,241][INFO ][o.e.c.r.a.AllocationService] [cratedb03] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[energyeast_users][3], [energyeast_users][1], [energyeast_users][0]]]).
I decided to remove a couple of tables and suddenly the cluster is up again (odd). The containers have not been updated nor stopped.
Ok … the logs just show a pretty normal startup.
How did you remove them, with the cluster down? 
That was a weird situation. The shards were visible in the UI, where I noticed the nodes to be up. Once in the console view, I executed the queries and after the deletion the cluster was up again.
Do you remember what you did before? Was there anything special about the tables you deleted?
My suspicion would be, that actually none of the nodes were really down, but there might be a bug with the admin UI.
I checked the UI one day before the problem and everything was working as expected. The next day that I tried to insert new data the cluster was down. I restarted it without luck and by pure chance I thought about deleting a couple of tables. Somehow this triggered something. Nothing special with the tables, they hold around 20 columns and 1k-20k rows.
How did you delete the tables while the cluster wasn’t running 
?
The UI was still visible, I entered it in the console page.
I see. So this means the cluster was still running, but there maybe was a problem with the AdminUI.
Can you maybe explain what you did, and maybe share the schemas of the tables. You could also DM me, if you don’t want to share them publicly.
Did you use any UDFs in the Schemas?
Hey @proddata, sorry for not replying back sooner. I really don’t know what happened, maybe was a server problem while rebooted or space constraint with the docker container, since I have been dealing with some of those problems recently. I didn’t use any UDF over the data, and the schemas are simply strings, integers and array of strings.