How can I connect from a remote server to a CrateDB node using crash?

Hello,

I’ve already created another user called “energytix” and can connect with that user on my first CrateDB node:

(cratedb) administrator@crate-dn-001:~$ crash --username energytix
Password:
CONNECT OK
cr> \q
Bye!
(cratedb) administrator@crate-dn-001:~$

But when I try to connect from a remote server, again using crash, I can’t:

(cratedb) loadgenerator001@eris:~$ crash --verbose --hosts 192.168.239.30 --username energytix
+----------------------------+-----------+---------+-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| server_url                 | node_name | version | connected | message                                                                                                                                                                                                                                                                                      |
+----------------------------+-----------+---------+-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| http://192.168.239.30:4200 |      NULL | 0.0.0   | FALSE     | Server not available, exception: HTTPConnectionPool(host='192.168.239.30', port=4200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f76a35450a0>: Failed to establish a new connection: [Errno 111] Connection refused')) |
+----------------------------+-----------+---------+-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
CONNECT ERROR

I read Host-Based Authentication (HBA) — CrateDB: Reference and added the user, as well as the remote server’s IP address to /etc/crate/crate.yml and restarted CrateDB (via systemctl). The relevant part looks like:

# Client access and authentication is then configured via the host based
# config, a map of remote client access entries.
# The following example is a sane configuration that covers a common use case:
# * The predefined superuser `crate` has trusted access from localhost.
# * All other users require username/password authentication from anywhere.
#   Note that this authentication method is only available via Postgres
#   Protocol, meaning that with this configuration, connecting from non-local
#   hosts via HTTP would not be possible!
auth:
  host_based:
    config:
      0:
        user: crate
        address: _local_
        method: trust
      a:
        user: energytix
        address: 192.168.239.20/24
        method: password
        protocol: pg
      99:
        method: password

But maybe I misunderstood something and the YAML configuration above is wrong?

Long story short, how can I connect (using crash ) from a remote server to my CrateDB node?

You typically don’t need to add a user to HBA config.

auth:
  host_based:
    config:
      0:
        user: crate
        address: _local_
        method: trust
      99:
        method: password

Already ensures, that from localhost you can connect with the crate user trust based.
From any other host you have to use password authentication to connect.

Did you setup network.host ?

No, I haven’t touched any other part of /etc/crate/crate.yml yet.

CrateDB by default only binds to the loopback address (do not accidentally expose it anyway)

you need to adjust to e.g.

network.host: _local_, _site_

also see here:

Hello,

I first tried the following in /etc/crate/crate.yml:

network.host: _site_

and restarted CrateDB, and from another, non-CrateDB related server tried to connect using crash, but still got an error:

(cratedb) loadgenerator001@eris:~$ crash --verbose --hosts 192.168.239.30 --username energytix
+----------------------------+-----------+---------+-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| server_url                 | node_name | version | connected | message                                                                                                                                                                                                                                                                                      |
+----------------------------+-----------+---------+-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| http://192.168.239.30:4200 |      NULL | 0.0.0   | FALSE     | Server not available, exception: HTTPConnectionPool(host='192.168.239.30', port=4200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f85efe960a0>: Failed to establish a new connection: [Errno 111] Connection refused')) |
+----------------------------+-----------+---------+-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
CONNECT ERROR
(cratedb) loadgenerator001@eris:~$

Then I tried changing that setting to:

network.host: 192.168.239.30

and restarting, and connecting using crash from another server, still no success.

Finally I tried:

network.host: _global_

But I still can’t connect from a remote server using crash, I get the same error:

(cratedb) loadgenerator001@eris:~$ crash --verbose --hosts 192.168.239.30 --username energytix
+----------------------------+-----------+---------+-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| server_url                 | node_name | version | connected | message                                                                                                                                                                                                                                                                                      |
+----------------------------+-----------+---------+-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| http://192.168.239.30:4200 |      NULL | 0.0.0   | FALSE     | Server not available, exception: HTTPConnectionPool(host='192.168.239.30', port=4200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f35fc8ae0a0>: Failed to establish a new connection: [Errno 111] Connection refused')) |
+----------------------------+-----------+---------+-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
CONNECT ERROR

is there another setting I should check other than network.host? Or should I use another value for that setting?

can you check if the crate service is bound to the address e.g. with netstat -tulpn?

Do you have ufw or another firewall set up that could block access?

just to be sure … can you locally connect with crash?

Hello,

I’ve reverted /etc/crate/crate.yml back to default values and restarted CrateDB on that node.

Let me answer your questions:

administrator@crate-dn-001:~$ sudo netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      733/sshd
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      899/exim4
tcp6       0      0 127.0.0.1:4200          :::*                    LISTEN      8240/java
tcp6       0      0 ::1:4200                :::*                    LISTEN      8240/java
tcp6       0      0 127.0.0.1:4300          :::*                    LISTEN      8240/java
tcp6       0      0 ::1:4300                :::*                    LISTEN      8240/java
tcp6       0      0 :::22                   :::*                    LISTEN      733/sshd
tcp6       0      0 127.0.0.1:5432          :::*                    LISTEN      8240/java
tcp6       0      0 ::1:5432                :::*                    LISTEN      8240/java
tcp6       0      0 ::1:25                  :::*                    LISTEN      899/exim4

This is a default Debian “buster” (10.x) installation, I haven’t enabled any firewall rules:

administrator@crate-dn-001:~$ sudo iptables --list
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

And yes, locally, on that CrateDB node, I can connect via crash both as crate super user and the newly created energytix user:

(cratedb) administrator@crate-dn-001:~$ crash
CONNECT OK
cr> \q
Bye!

(cratedb) administrator@crate-dn-001:~$ crash --username energytix
Password:
CONNECT OK
cr> \q
Bye!

But I also want to connect from a non-CrateDB server via crash, as well as via HTTP (for Admin UI), and of course via PostgreSQL drivers, using this newly created energytix user.

My first test case is crash from a non-CrateDB server, which I haven’t been able to do yet.

ok, thanks.

network.host: _site_, _local_

is indeed the correct setting to be able to connect from the same subnet. Let me check if I can reproduce that.

I’ve copy pasted that line to relevant part in /etc/crate/crate.yml:

administrator@crate-dn-001:~$ grep "network\.host" /etc/crate/crate.yml | grep -v "^#"
network.host: _site_, _local_

But then when I try to restart create via:

administrator@crate-dn-001:~$ sudo systemctl restart crate

and then check the service status, I get an error:

administrator@crate-dn-001:~$ sudo systemctl status crate
● crate.service - CrateDB Server
   Loaded: loaded (/lib/systemd/system/crate.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2021-09-30 14:54:05 UTC; 20s ago
     Docs: https://crate.io/docs/
  Process: 9149 ExecStart=/usr/share/crate/bin/crate -Cpath.conf=${CRATE_PATH_CONF} -Cpath.logs=${CRATE_PATH_LOG} (code=exited, status=78)
 Main PID: 9149 (code=exited, status=78)

Sep 30 14:54:05 crate-dn-001 crate[9149]: [2021-09-30T14:54:05,627][INFO ][o.e.t.TransportService   ] [Pizzo Arera] publish_address {192.168.239.30:4300}, boun
Sep 30 14:54:05 crate-dn-001 crate[9149]: [2021-09-30T14:54:05,791][INFO ][o.e.b.BootstrapChecks    ] [Pizzo Arera] bound or publishing to a non-loopback addre
Sep 30 14:54:05 crate-dn-001 crate[9149]: ERROR: [1] bootstrap checks failed
Sep 30 14:54:05 crate-dn-001 crate[9149]: [1]: initial heap size [268435456] not equal to maximum heap size [16877879296]; this can cause resize pauses and pre
Sep 30 14:54:05 crate-dn-001 crate[9149]: [2021-09-30T14:54:05,796][INFO ][o.e.n.Node               ] [Pizzo Arera] stopping ...
Sep 30 14:54:05 crate-dn-001 crate[9149]: [2021-09-30T14:54:05,808][INFO ][o.e.n.Node               ] [Pizzo Arera] stopped
Sep 30 14:54:05 crate-dn-001 crate[9149]: [2021-09-30T14:54:05,808][INFO ][o.e.n.Node               ] [Pizzo Arera] closing ...
Sep 30 14:54:05 crate-dn-001 crate[9149]: [2021-09-30T14:54:05,815][INFO ][o.e.n.Node               ] [Pizzo Arera] closed
Sep 30 14:54:05 crate-dn-001 systemd[1]: crate.service: Main process exited, code=exited, status=78/CONFIG
Sep 30 14:54:05 crate-dn-001 systemd[1]: crate.service: Failed with result 'exit-code'.
administrator@crate-dn-001:~$

Why is that?

Also, for your information, the IP address info of that CrateDB node:

administrator@crate-dn-001:~$ ip a | grep "inet "
    inet 127.0.0.1/8 scope host lo
    inet 192.168.239.30/24 brd 192.168.239.255 scope global ens3

and the remote, non-CrateDB server from which I want to connect:

loadgenerator001@eris:~$ ip a | grep "inet "
    inet 127.0.0.1/8 scope host lo
    inet 192.168.239.20/24 brd 192.168.239.255 scope global ens6

Further info: for example from that remote server, I can SSH into that CrateDB node:

loadgenerator001@eris:~$ ssh administrator@192.168.239.30
administrator@192.168.239.30's password:
Linux crate-dn-001 4.19.0-17-amd64 #1 SMP Debian 4.19.194-3 (2021-07-18) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Thu Sep 30 11:05:49 2021 from 10.10.113.21
administrator@crate-dn-001:~$  logout
Connection to 192.168.239.30 closed.

mfussenegger
If you change network.host it assumes that you’re using CrateDB for production and enforces some Bootstrap Checks to ensure CrateDB is suitably configured.

When using the package-based setup flavor for Debian or Ubuntu or Red Hat or CentOS, the CrateDB startup script uses Environment variables from the /etc/default/crate file.

or you set it in the console with e.g.

export CRATE_HEAP_SIZE=4g  

I think we might need to adjust the notes a little:


I am sorry that you have such a bad experience. Me and my team are currently working to make the start with CrateDB as easy as possible.

Oh, indeed! I’ve set the heap size to 16 GB (because that node has 64 GB RAM):

administrator@crate-dn-001:~$ grep -i "heap" /etc/default/crate | grep -v "^#"
CRATE_HEAP_SIZE=16G

And then tried _site_, _local_ again:

administrator@crate-dn-001:~$ grep "network\.host" /etc/crate/crate.yml | grep -v "^#"
network.host: _site_, _local_

This time it restarted without any problems, and now I can connect from a remote, non-CrateDB related server with my newly created database user:

(cratedb) loadgenerator001@eris:~$ crash --verbose --hosts 192.168.239.30 --username energytix
Password:
+----------------------------+--------------+---------+-----------+---------+
| server_url                 | node_name    | version | connected | message |
+----------------------------+--------------+---------+-----------+---------+
| http://192.168.239.30:4200 | Gurpitscheck | 4.6.3   | TRUE      | OK      |
+----------------------------+--------------+---------+-----------+---------+
CONNECT OK
CLUSTER CHECK OK
TYPES OF NODE CHECK OK
cr> \q
Bye!

It’ll be good if you put it somewhere visible in the documentation, or some F.A.Q. for pitfalls, etc.

Thanks a lot for your great technical support and solving this issue so quickly. :pray::+1:

1 Like