I'm trying to set up a 3-node CrateDB cluster and now I can't even connect with crash

CrateDB
1

Hello,

I’m trying to set up a 3-node CrateDB cluster and now I can’t even connect with crash now on Node 001. Yesterday, I started with the first node and I was able to connect to it via crash, and then I configured all of the nodes as the following:

Node 001:

administrator@crate-dn-001:~$ grep -v "^#" /etc/crate/crate.yml | sed '/^[[:space:]]*$/d'
path.data: /data
gateway.expected_nodes: 3
gateway.recover_after_nodes: 3
network.host: _site_, _local_
discovery.seed_hosts:
    - 192.168.239.30:4300
    - 192.168.239.31:4300
    - 192.168.239.32:4300
cluster.initial_master_nodes:
    - 192.168.239.30
    - 192.168.239.31
    - 192.168.239.32
auth.host_based.enabled: true
auth:
  host_based:
    config:
      0:
        user: crate
        address: _local_
        method: trust
      99:
        method: password
node.name: crate-dn-001

Node 002:

administrator@crate-dn-002:~$ grep -v "^#" /etc/crate/crate.yml | sed '/^[[:space:]]*$/d'
path.data: /data
gateway.expected_nodes: 3
gateway.recover_after_nodes: 3
network.host: _site_, _local_
discovery.seed_hosts:
    - 192.168.239.30:4300
    - 192.168.239.31:4300
    - 192.168.239.32:4300
cluster.initial_master_nodes:
    - 192.168.239.30
    - 192.168.239.31
    - 192.168.239.32
auth.host_based.enabled: true
auth:
  host_based:
    config:
      0:
        user: crate
        address: _local_
        method: trust
      99:
        method: password
node.name: crate-dn-002

Node 003:

administrator@crate-dn-003:~$ grep -v "^#" /etc/crate/crate.yml | sed '/^[[:space:]]*$/d'
path.data: /data
gateway.expected_nodes: 3
gateway.recover_after_nodes: 3
network.host: _site_, _local_
discovery.seed_hosts:
    - 192.168.239.30:4300
    - 192.168.239.31:4300
    - 192.168.239.32:4300
cluster.initial_master_nodes:
    - 192.168.239.30
    - 192.168.239.31
    - 192.168.239.32
auth.host_based.enabled: true
auth:
  host_based:
    config:
      0:
        user: crate
        address: _local_
        method: trust
      99:
        method: password
node.name: crate-dn-003

When I restart the nodes, and check the status, I get the following:

Node 001:

administrator@crate-dn-001:~$ sudo systemctl restart crate
administrator@crate-dn-001:~$ sudo systemctl status crate
● crate.service - CrateDB Server
   Loaded: loaded (/lib/systemd/system/crate.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2021-10-01 10:14:28 UTC; 3s ago
     Docs: https://crate.io/docs/
 Main PID: 12421 (java)
    Tasks: 43 (limit: 36863)
   Memory: 1.0G
   CGroup: /system.slice/crate.service
           └─12421 /usr/share/crate/jdk/bin/java -Xms16G -Xmx16G -Djava.awt.headless=true -XX:+UseG1GC -XX:G1ReservePercent=25 -XX:InitiatingHeapOccu

Oct 01 10:14:30 crate-dn-001 crate[12421]: [2021-10-01T10:14:30,425][INFO ][o.e.p.PluginsService     ] [crate-dn-001] loaded plugin [io.crate.plugin.
Oct 01 10:14:30 crate-dn-001 crate[12421]: [2021-10-01T10:14:30,425][INFO ][o.e.p.PluginsService     ] [crate-dn-001] loaded plugin [io.crate.plugin.
Oct 01 10:14:30 crate-dn-001 crate[12421]: [2021-10-01T10:14:30,425][INFO ][o.e.p.PluginsService     ] [crate-dn-001] loaded plugin [io.crate.udc.plu
Oct 01 10:14:30 crate-dn-001 crate[12421]: [2021-10-01T10:14:30,425][INFO ][o.e.p.PluginsService     ] [crate-dn-001] loaded plugin [org.elasticsearc
Oct 01 10:14:30 crate-dn-001 crate[12421]: [2021-10-01T10:14:30,425][INFO ][o.e.p.PluginsService     ] [crate-dn-001] loaded plugin [org.elasticsearc
Oct 01 10:14:30 crate-dn-001 crate[12421]: [2021-10-01T10:14:30,425][INFO ][o.e.p.PluginsService     ] [crate-dn-001] loaded plugin [org.elasticsearc
Oct 01 10:14:30 crate-dn-001 crate[12421]: [2021-10-01T10:14:30,425][INFO ][o.e.p.PluginsService     ] [crate-dn-001] loaded plugin [org.elasticsearc
Oct 01 10:14:31 crate-dn-001 crate[12421]: [2021-10-01T10:14:31,219][INFO ][o.e.d.DiscoveryModule    ] [crate-dn-001] using discovery type [zen] and
Oct 01 10:14:31 crate-dn-001 crate[12421]: [2021-10-01T10:14:31,857][INFO ][psql                     ] [crate-dn-001] PSQL SSL support is disabled.
Oct 01 10:14:31 crate-dn-001 crate[12421]: [2021-10-01T10:14:31,955][INFO ][i.c.p.PipelineRegistry   ] [crate-dn-001] HTTP SSL support is disabled.

administrator@crate-dn-001:~$ sudo systemctl status crate
● crate.service - CrateDB Server
   Loaded: loaded (/lib/systemd/system/crate.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2021-10-01 10:14:28 UTC; 1min 5s ago
     Docs: https://crate.io/docs/
 Main PID: 12421 (java)
    Tasks: 61 (limit: 36863)
   Memory: 1.7G
   CGroup: /system.slice/crate.service
           └─12421 /usr/share/crate/jdk/bin/java -Xms16G -Xmx16G -Djava.awt.headless=true -XX:+UseG1GC -XX:G1ReservePercent=25 -XX:InitiatingHeapOccu

Oct 01 10:14:32 crate-dn-001 crate[12421]: [2021-10-01T10:14:32,162][INFO ][o.e.n.Node               ] [crate-dn-001] initialized
Oct 01 10:14:32 crate-dn-001 crate[12421]: [2021-10-01T10:14:32,162][INFO ][o.e.n.Node               ] [crate-dn-001] starting ...
Oct 01 10:14:32 crate-dn-001 crate[12421]: [2021-10-01T10:14:32,292][INFO ][psql                     ] [crate-dn-001] publish_address {192.168.239.30
Oct 01 10:14:32 crate-dn-001 crate[12421]: [2021-10-01T10:14:32,304][INFO ][o.e.h.n.Netty4HttpServerTransport] [crate-dn-001] publish_address {192.16
Oct 01 10:14:32 crate-dn-001 crate[12421]: [2021-10-01T10:14:32,315][INFO ][o.e.t.TransportService   ] [crate-dn-001] publish_address {192.168.239.30
Oct 01 10:14:32 crate-dn-001 crate[12421]: [2021-10-01T10:14:32,458][INFO ][o.e.b.BootstrapChecks    ] [crate-dn-001] bound or publishing to a non-lo
Oct 01 10:14:32 crate-dn-001 crate[12421]: [2021-10-01T10:14:32,461][INFO ][o.e.c.c.Coordinator      ] [crate-dn-001] cluster UUID [iXbkexKsQZyDNjMVU
Oct 01 10:14:32 crate-dn-001 crate[12421]: [2021-10-01T10:14:32,577][INFO ][o.e.c.s.MasterService    ] [crate-dn-001] elected-as-master ([1] nodes jo
Oct 01 10:14:32 crate-dn-001 crate[12421]: [2021-10-01T10:14:32,624][INFO ][o.e.c.s.ClusterApplierService] [crate-dn-001] master node changed {previo
Oct 01 10:14:32 crate-dn-001 crate[12421]: [2021-10-01T10:14:32,634][INFO ][o.e.n.Node               ] [crate-dn-001] started

Node 002:

administrator@crate-dn-002:~$ sudo systemctl status crate
● crate.service - CrateDB Server
   Loaded: loaded (/lib/systemd/system/crate.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2021-10-01 10:16:37 UTC; 39s ago
     Docs: https://crate.io/docs/
 Main PID: 4317 (java)
    Tasks: 66 (limit: 36863)
   Memory: 1.4G
   CGroup: /system.slice/crate.service
           └─4317 /usr/share/crate/jdk/bin/java -Xms16G -Xmx16G -Djava.awt.headless=true -XX:+UseG1GC -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -Xlog:

Oct 01 10:16:41 crate-dn-002 crate[4317]: [2021-10-01T10:16:41,348][INFO ][o.e.n.Node               ] [crate-dn-002] initialized
Oct 01 10:16:41 crate-dn-002 crate[4317]: [2021-10-01T10:16:41,348][INFO ][o.e.n.Node               ] [crate-dn-002] starting ...
Oct 01 10:16:41 crate-dn-002 crate[4317]: [2021-10-01T10:16:41,488][INFO ][psql                     ] [crate-dn-002] publish_address {192.168.239.31:5432}, bound_addresse
Oct 01 10:16:41 crate-dn-002 crate[4317]: [2021-10-01T10:16:41,502][INFO ][o.e.h.n.Netty4HttpServerTransport] [crate-dn-002] publish_address {192.168.239.31:4200}, bound_
Oct 01 10:16:41 crate-dn-002 crate[4317]: [2021-10-01T10:16:41,517][INFO ][o.e.t.TransportService   ] [crate-dn-002] publish_address {192.168.239.31:4300}, bound_addresse
Oct 01 10:16:41 crate-dn-002 crate[4317]: [2021-10-01T10:16:41,705][INFO ][o.e.b.BootstrapChecks    ] [crate-dn-002] bound or publishing to a non-loopback address, enforc
Oct 01 10:16:41 crate-dn-002 crate[4317]: [2021-10-01T10:16:41,708][INFO ][o.e.c.c.Coordinator      ] [crate-dn-002] cluster UUID [Eri8ChLNRz23zCUfJOGneA]
Oct 01 10:16:41 crate-dn-002 crate[4317]: [2021-10-01T10:16:41,818][INFO ][o.e.c.s.MasterService    ] [crate-dn-002] elected-as-master ([1] nodes joined)[{crate-dn-002}{z
Oct 01 10:16:41 crate-dn-002 crate[4317]: [2021-10-01T10:16:41,861][INFO ][o.e.c.s.ClusterApplierService] [crate-dn-002] master node changed {previous [], current [{crate
Oct 01 10:16:41 crate-dn-002 crate[4317]: [2021-10-01T10:16:41,870][INFO ][o.e.n.Node               ] [crate-dn-002] started

Node 003:

administrator@crate-dn-003:~$ sudo systemctl restart crate
administrator@crate-dn-003:~$ sudo systemctl status crate
● crate.service - CrateDB Server
   Loaded: loaded (/lib/systemd/system/crate.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2021-10-01 10:18:04 UTC; 6s ago
     Docs: https://crate.io/docs/
 Main PID: 4387 (java)
    Tasks: 67 (limit: 36863)
   Memory: 1.1G
   CGroup: /system.slice/crate.service
           └─4387 /usr/share/crate/jdk/bin/java -Xms16G -Xmx16G -Djava.awt.headless=true -XX:+UseG1GC -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -Xlog:gc*,gc+age=

Oct 01 10:18:07 crate-dn-003 crate[4387]: [2021-10-01T10:18:07,695][INFO ][o.e.n.Node               ] [crate-dn-003] initialized
Oct 01 10:18:07 crate-dn-003 crate[4387]: [2021-10-01T10:18:07,696][INFO ][o.e.n.Node               ] [crate-dn-003] starting ...
Oct 01 10:18:07 crate-dn-003 crate[4387]: [2021-10-01T10:18:07,823][INFO ][psql                     ] [crate-dn-003] publish_address {192.168.239.32:5432}, bound_addresses {[::1]:54
Oct 01 10:18:07 crate-dn-003 crate[4387]: [2021-10-01T10:18:07,835][INFO ][o.e.h.n.Netty4HttpServerTransport] [crate-dn-003] publish_address {192.168.239.32:4200}, bound_addresses {
Oct 01 10:18:07 crate-dn-003 crate[4387]: [2021-10-01T10:18:07,846][INFO ][o.e.t.TransportService   ] [crate-dn-003] publish_address {192.168.239.32:4300}, bound_addresses {[::1]:43
Oct 01 10:18:07 crate-dn-003 crate[4387]: [2021-10-01T10:18:07,995][INFO ][o.e.b.BootstrapChecks    ] [crate-dn-003] bound or publishing to a non-loopback address, enforcing bootstr
Oct 01 10:18:07 crate-dn-003 crate[4387]: [2021-10-01T10:18:07,997][INFO ][o.e.c.c.Coordinator      ] [crate-dn-003] cluster UUID [x7zTdQYtRKG9jOG6Cf4Jdw]
Oct 01 10:18:08 crate-dn-003 crate[4387]: [2021-10-01T10:18:08,058][INFO ][o.e.c.s.MasterService    ] [crate-dn-003] elected-as-master ([1] nodes joined)[{crate-dn-003}{tPM64bSrTeu8
Oct 01 10:18:08 crate-dn-003 crate[4387]: [2021-10-01T10:18:08,104][INFO ][o.e.c.s.ClusterApplierService] [crate-dn-003] master node changed {previous [], current [{crate-dn-003}{tP
Oct 01 10:18:08 crate-dn-003 crate[4387]: [2021-10-01T10:18:08,115][INFO ][o.e.n.Node               ] [crate-dn-003] started

As far as I can see, it seems like each node is selecting itself as the master, is that correct? If so, then why?

Also, on Node 001, I try to connect with crash, but unfortunately:

administrator@crate-dn-001:~$ source /home/administrator/venv/cratedb/bin/activate
(cratedb) administrator@crate-dn-001:~$ crash
CONNECT ERROR
(cratedb) administrator@crate-dn-001:~$ crash --verbose
+-----------------------+-----------+---------+-----------+---------------------------------------+
| server_url            | node_name | version | connected | message                               |
+-----------------------+-----------+---------+-----------+---------------------------------------+
| http://localhost:4200 |      NULL | 0.0.0   | FALSE     | 503 Server Error: Service Unavailable |
+-----------------------+-----------+---------+-----------+---------------------------------------+
CONNECT ERROR

Any ideas why crash stopped being able to connect locally?

And is the cluster formed correctly with 3 nodes?

Am I missing some configuration settings?

2

You installed CrateDB using the apt packages, correct?
If you do that, CrateDB starts automatically ins single-node mode which also prevents adding the nodes to any sort of cluster. This is intentional in a way, that apt packages should start without any additional confiuguration.

There are basically two ways to fix this

  • using the crate-node tool

or (easier)

  • deleting the cluster state by …
    • service crate stop
    • rm -rf /usr/share/crate/data
    • service crate start
1 Like
3

Yes, I’ve installed using apt on all of the 3 Debian GNU/Linux servers.

Based on your recommendation I stopped crate service on all 3 servers and on all of them I ran:

 sudo rm -rf /usr/share/crate/data

Then I started the crate service on first, second and third node respectively, and this is what I get now:

Node 001:

administrator@crate-dn-001:~$ sudo systemctl status crate
● crate.service - CrateDB Server
   Loaded: loaded (/lib/systemd/system/crate.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2021-10-01 10:54:33 UTC; 2min 5s ago
     Docs: https://crate.io/docs/
 Main PID: 12692 (java)
    Tasks: 61 (limit: 36863)
   Memory: 1.9G
   CGroup: /system.slice/crate.service
           └─12692 /usr/share/crate/jdk/bin/java -Xms16G -Xmx16G -Djava.awt.headless=true -XX:+UseG1GC -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -Xlog:gc*,gc+age=trac

Oct 01 10:54:37 crate-dn-001 crate[12692]: [2021-10-01T10:54:37,922][INFO ][o.e.n.Node               ] [crate-dn-001] initialized
Oct 01 10:54:37 crate-dn-001 crate[12692]: [2021-10-01T10:54:37,923][INFO ][o.e.n.Node               ] [crate-dn-001] starting ...
Oct 01 10:54:38 crate-dn-001 crate[12692]: [2021-10-01T10:54:38,070][INFO ][psql                     ] [crate-dn-001] publish_address {192.168.239.30:5432}, bound_addresses {[::1]:5432},
Oct 01 10:54:38 crate-dn-001 crate[12692]: [2021-10-01T10:54:38,082][INFO ][o.e.h.n.Netty4HttpServerTransport] [crate-dn-001] publish_address {192.168.239.30:4200}, bound_addresses {[::1
Oct 01 10:54:38 crate-dn-001 crate[12692]: [2021-10-01T10:54:38,096][INFO ][o.e.t.TransportService   ] [crate-dn-001] publish_address {192.168.239.30:4300}, bound_addresses {[::1]:4300},
Oct 01 10:54:38 crate-dn-001 crate[12692]: [2021-10-01T10:54:38,269][INFO ][o.e.b.BootstrapChecks    ] [crate-dn-001] bound or publishing to a non-loopback address, enforcing bootstrap c
Oct 01 10:54:38 crate-dn-001 crate[12692]: [2021-10-01T10:54:38,270][INFO ][o.e.c.c.Coordinator      ] [crate-dn-001] cluster UUID [iXbkexKsQZyDNjMVU6GD1A]
Oct 01 10:54:38 crate-dn-001 crate[12692]: [2021-10-01T10:54:38,323][INFO ][o.e.c.s.MasterService    ] [crate-dn-001] elected-as-master ([1] nodes joined)[{crate-dn-001}{pp1q4bZ5Tk6m88xu
Oct 01 10:54:38 crate-dn-001 crate[12692]: [2021-10-01T10:54:38,371][INFO ][o.e.c.s.ClusterApplierService] [crate-dn-001] master node changed {previous [], current [{crate-dn-001}{pp1q4b
Oct 01 10:54:38 crate-dn-001 crate[12692]: [2021-10-01T10:54:38,381][INFO ][o.e.n.Node               ] [crate-dn-001] started

Node 002:

administrator@crate-dn-002:~$ sudo systemctl status crate
● crate.service - CrateDB Server
   Loaded: loaded (/lib/systemd/system/crate.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2021-10-01 10:54:40 UTC; 5min ago
     Docs: https://crate.io/docs/
 Main PID: 4538 (java)
    Tasks: 61 (limit: 36863)
   Memory: 3.5G
   CGroup: /system.slice/crate.service
           └─4538 /usr/share/crate/jdk/bin/java -Xms16G -Xmx16G -Djava.awt.headless=true -XX:+UseG1GC -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -Xlog:gc*,gc+

Oct 01 10:54:44 crate-dn-002 crate[4538]: [2021-10-01T10:54:44,441][INFO ][o.e.n.Node               ] [crate-dn-002] initialized
Oct 01 10:54:44 crate-dn-002 crate[4538]: [2021-10-01T10:54:44,441][INFO ][o.e.n.Node               ] [crate-dn-002] starting ...
Oct 01 10:54:44 crate-dn-002 crate[4538]: [2021-10-01T10:54:44,563][INFO ][psql                     ] [crate-dn-002] publish_address {192.168.239.31:5432}, bound_addresses {[::1
Oct 01 10:54:44 crate-dn-002 crate[4538]: [2021-10-01T10:54:44,575][INFO ][o.e.h.n.Netty4HttpServerTransport] [crate-dn-002] publish_address {192.168.239.31:4200}, bound_address
Oct 01 10:54:44 crate-dn-002 crate[4538]: [2021-10-01T10:54:44,588][INFO ][o.e.t.TransportService   ] [crate-dn-002] publish_address {192.168.239.31:4300}, bound_addresses {[::1
Oct 01 10:54:44 crate-dn-002 crate[4538]: [2021-10-01T10:54:44,746][INFO ][o.e.b.BootstrapChecks    ] [crate-dn-002] bound or publishing to a non-loopback address, enforcing boo
Oct 01 10:54:44 crate-dn-002 crate[4538]: [2021-10-01T10:54:44,748][INFO ][o.e.c.c.Coordinator      ] [crate-dn-002] cluster UUID [Eri8ChLNRz23zCUfJOGneA]
Oct 01 10:54:44 crate-dn-002 crate[4538]: [2021-10-01T10:54:44,819][INFO ][o.e.c.s.MasterService    ] [crate-dn-002] elected-as-master ([1] nodes joined)[{crate-dn-002}{zaiVJzVU
Oct 01 10:54:44 crate-dn-002 crate[4538]: [2021-10-01T10:54:44,864][INFO ][o.e.c.s.ClusterApplierService] [crate-dn-002] master node changed {previous [], current [{crate-dn-002
Oct 01 10:54:44 crate-dn-002 crate[4538]: [2021-10-01T10:54:44,874][INFO ][o.e.n.Node               ] [crate-dn-002] started

Node 003:

administrator@crate-dn-003:~$ sudo systemctl status crate
● crate.service - CrateDB Server
   Loaded: loaded (/lib/systemd/system/crate.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2021-10-01 10:54:46 UTC; 6min ago
     Docs: https://crate.io/docs/
 Main PID: 4606 (java)
    Tasks: 62 (limit: 36863)
   Memory: 5.0G
   CGroup: /system.slice/crate.service
           └─4606 /usr/share/crate/jdk/bin/java -Xms16G -Xmx16G -Djava.awt.headless=true -XX:+UseG1GC -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -Xlog:gc*,gc+age=trac

Oct 01 10:54:50 crate-dn-003 crate[4606]: [2021-10-01T10:54:50,033][INFO ][o.e.n.Node               ] [crate-dn-003] initialized
Oct 01 10:54:50 crate-dn-003 crate[4606]: [2021-10-01T10:54:50,033][INFO ][o.e.n.Node               ] [crate-dn-003] starting ...
Oct 01 10:54:50 crate-dn-003 crate[4606]: [2021-10-01T10:54:50,174][INFO ][psql                     ] [crate-dn-003] publish_address {192.168.239.32:5432}, bound_addresses {[::1]:5432},
Oct 01 10:54:50 crate-dn-003 crate[4606]: [2021-10-01T10:54:50,188][INFO ][o.e.h.n.Netty4HttpServerTransport] [crate-dn-003] publish_address {192.168.239.32:4200}, bound_addresses {[::1
Oct 01 10:54:50 crate-dn-003 crate[4606]: [2021-10-01T10:54:50,201][INFO ][o.e.t.TransportService   ] [crate-dn-003] publish_address {192.168.239.32:4300}, bound_addresses {[::1]:4300},
Oct 01 10:54:50 crate-dn-003 crate[4606]: [2021-10-01T10:54:50,372][INFO ][o.e.b.BootstrapChecks    ] [crate-dn-003] bound or publishing to a non-loopback address, enforcing bootstrap c
Oct 01 10:54:50 crate-dn-003 crate[4606]: [2021-10-01T10:54:50,374][INFO ][o.e.c.c.Coordinator      ] [crate-dn-003] cluster UUID [x7zTdQYtRKG9jOG6Cf4Jdw]
Oct 01 10:54:50 crate-dn-003 crate[4606]: [2021-10-01T10:54:50,497][INFO ][o.e.c.s.MasterService    ] [crate-dn-003] elected-as-master ([1] nodes joined)[{crate-dn-003}{tPM64bSrTeu8a0Jm
Oct 01 10:54:50 crate-dn-003 crate[4606]: [2021-10-01T10:54:50,546][INFO ][o.e.c.s.ClusterApplierService] [crate-dn-003] master node changed {previous [], current [{crate-dn-003}{tPM64b
Oct 01 10:54:50 crate-dn-003 crate[4606]: [2021-10-01T10:54:50,555][INFO ][o.e.n.Node               ] [crate-dn-003] started
4

And I still can’t connect with crash locally on the first node (I was able to connect locally before I started setting up configuration for 3-node cluster):

administrator@crate-dn-001:~$ source /home/administrator/venv/cratedb/bin/activate
(cratedb) administrator@crate-dn-001:~$ crash
CONNECT ERROR
(cratedb) administrator@crate-dn-001:~$ crash --verbose
+-----------------------+-----------+---------+-----------+---------------------------------------+
| server_url            | node_name | version | connected | message                               |
+-----------------------+-----------+---------+-----------+---------------------------------------+
| http://localhost:4200 |      NULL | 0.0.0   | FALSE     | 503 Server Error: Service Unavailable |
+-----------------------+-----------+---------+-----------+---------------------------------------+
CONNECT ERROR

The non-commented contents of /etc/crate/crate.yml on that node are:

administrator@crate-dn-001:~$ grep -v "^#" /etc/crate/crate.yml | sed '/^[[:space:]]*$/d'
path.data: /data
gateway.expected_nodes: 3
gateway.recover_after_nodes: 3
network.host: _site_, _local_
discovery.seed_hosts:
    - 192.168.239.30:4300
    - 192.168.239.31:4300
    - 192.168.239.32:4300
cluster.initial_master_nodes:
    - 192.168.239.30
    - 192.168.239.31
    - 192.168.239.32
auth.host_based.enabled: true
auth:
  host_based:
    config:
      0:
        user: crate
        address: _local_
        method: trust
      99:
        method: password
node.name: crate-dn-001
5

I don’t know how this affects forming of the cluster but this morning, I first set cluster.name: crate-energytix in /etc/crate/crate.yml and tried to bring up nodes this way. But then I commented it out, so the default cluster name should be crate again, right?

It would be great if I could see a sample set of YAML files indicating the simplest 3-node cluster set up for an on-premise system that is installed via stable APT packages of CrateDB.

Right now I have 3 powerful Debian GNU/Linux servers up and running, but whatever the configuration problems are with those YAML files on those servers, I end up with the following:

  • It seems like a 3-node CrateDB cluster is not formed.
  • I can’t connect with crash anymore (neither locally with the default crate user, nor remotely with the energytix user I created yesterday).
  • I’ve never been able to connect via a web browser to Admin UI.

Therefore I don’t know what else I should try.

Any ideas?

6

Apparently, I also had to delete whatever cluster state has been created under /data directory. After stopping the service, running rm -rf * in /data, and restarting the CrateDB service on all three nodes, they now see each other and form a cluster.

I’m also able to connect locally with crash.

I created the same energytix user again, because apparently it was deleted when I deleted the cluster state under data directories.

I can also connect remotely via crash.

The only remaining issue I have for now is that I still can’t connect to the Admin UI when I visit http://192.168.239.30:4200 from another machine (an MS Windows machine running Chrome browser, that machine can normally access 192.168.239.30).

7

Thanks for reporting back.

I can also connect remotely via crash .
The only remaining issue I have for now is that I still can’t connect to the Admin UI when I visit http://192.168.239.30:4200 from another machine (an MS Windows machine running Chrome browser, that machine can normally access 192.168.239.30).

If you can connect remotely via crash, then you should also be able to see the Admin UI
crash uses the http-endpoint which serves the AdminUI

GET / → AdminUI
POST /_sql → sql http endpoint (used by both crash and the AdminUI)

8

Thanks for the feedback.

To test this I installed crash on that MS Windows machine, and indeed, it seems like there’s some connectivity problem from that MS Windows machine to the CrateDB machine’s port 4200 because the following command

(cratedb) C:\Users\esevinc>crash --verbose --hosts 192.168.239.30 --username energytix

just hangs without any output. Probably some sort of firewall issue, I’ll talk to our network team.

9

Connection to Admin UI issue has been solved as described at: I can connect via crash remotely, but I can't access Admin UI via web browser, why? - #2 by Emre_Sevinc

1 Like