Unable to enable node-to-node encryption

Hi,

I have a hard time setting up SSL for CrateDB when following the documentation.

Setup

Two nodes with Debian 11 and CrateDB 4.8.0. The nodes are located in different datacenters and have all ports open for each other. One of the nodes is the master.

What I want to do

Make all nodes in the cluster to communicate over SSL/TLS.

What I’ve done (all nodes)

  1. Installed openjdk-17-jdk-headless (to get keytool).
  2. Created a keystore and a truststore owned by crate:crate and chmod 400.
  3. Generated private key and CSR.
  4. Generated a CA key + certificate on my computer.
  5. Downloaded CSRs to my computer and signed them with the CA.
  6. Uploaded CRTs + CA cert. back to the nodes.
  7. Imported the certificates into keystore, and the CA certificate into truststore.
  8. Verified that the keystore contains three entries (private key, signed certificate and root CA certificate).
  9. Verified that the truststore contains one entry (root CA certificate).
  10. Configured the paths and passwords for the keystore and truststore in /etc/crate/crate.yml, as well as added ssl.transport.mode: on.

The problem

The nodes can’t communicate.

Master node says:

exception caught on transport layer [Netty4TcpChannel{localAddress=/X.X.X.X:4300, remoteAddress=/Y.Y.Y.Y:40812}], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown

The other node says:

Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Other notes

  • The nodes can communicate fine without SSL.
  • There is no difference if I disable the separate truststore.

Any help appreciated!

Does the self-signed cert work for https / pgsql tls?
Could you maybe provide the startup logs of both nodes?

Good question - I tested HTTPS now, and after trusting the certificate in the browser the GUI works over HTTPS.

As for the startup logs:

First node (Stockholm / sthlm / X.X.X.X):

May 30 08:35:29 crate-sthlm-01 systemd[1]: Started CrateDB Server.
May 30 08:35:30 crate-sthlm-01 crate[10434]: [2022-05-30T08:35:30,431][INFO ][o.e.e.NodeEnvironment    ] [Stockholm 01] using [1] data paths, mounts [[/ (/dev/sda2)]], net usable_space [42.2gb], net total_space [49.8gb], types [ext4]
May 30 08:35:30 crate-sthlm-01 crate[10434]: [2022-05-30T08:35:30,434][INFO ][o.e.e.NodeEnvironment    ] [Stockholm 01] heap size [1gb], compressed ordinary object pointers [true]
May 30 08:35:30 crate-sthlm-01 crate[10434]: [2022-05-30T08:35:30,521][INFO ][o.e.n.Node               ] [Stockholm 01] node name [Stockholm 01], node ID [0ihwYOjWTGGVt0ByDXPobA], cluster name [my_cluster]
May 30 08:35:30 crate-sthlm-01 crate[10434]: [2022-05-30T08:35:30,536][INFO ][o.e.n.Node               ] [Stockholm 01] version[4.8.0], pid[10434], build[e1dea64/2022-04-28T17:06:46Z], OS[Linux/5.10.0-10-amd64/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.3/17.0.3+7]
May 30 08:35:31 crate-sthlm-01 crate[10434]: [2022-05-30T08:35:31,135][INFO ][o.e.p.PluginsService     ] [Stockholm 01] no modules loaded
May 30 08:35:31 crate-sthlm-01 crate[10434]: [2022-05-30T08:35:31,136][INFO ][o.e.p.PluginsService     ] [Stockholm 01] loaded plugin [cr8-copy-s3]
May 30 08:35:31 crate-sthlm-01 crate[10434]: [2022-05-30T08:35:31,137][INFO ][o.e.p.PluginsService     ] [Stockholm 01] loaded plugin [crate-azure-discovery]
May 30 08:35:31 crate-sthlm-01 crate[10434]: [2022-05-30T08:35:31,137][INFO ][o.e.p.PluginsService     ] [Stockholm 01] loaded plugin [crate-functions]
May 30 08:35:31 crate-sthlm-01 crate[10434]: [2022-05-30T08:35:31,137][INFO ][o.e.p.PluginsService     ] [Stockholm 01] loaded plugin [crate-jmx-monitoring]
May 30 08:35:31 crate-sthlm-01 crate[10434]: [2022-05-30T08:35:31,137][INFO ][o.e.p.PluginsService     ] [Stockholm 01] loaded plugin [crate-lang-js]
May 30 08:35:31 crate-sthlm-01 crate[10434]: [2022-05-30T08:35:31,138][INFO ][o.e.p.PluginsService     ] [Stockholm 01] loaded plugin [es-analysis-common]
May 30 08:35:31 crate-sthlm-01 crate[10434]: [2022-05-30T08:35:31,138][INFO ][o.e.p.PluginsService     ] [Stockholm 01] loaded plugin [es-analysis-phonetic]
May 30 08:35:31 crate-sthlm-01 crate[10434]: [2022-05-30T08:35:31,138][INFO ][o.e.p.PluginsService     ] [Stockholm 01] loaded plugin [es-repository-azure]
May 30 08:35:31 crate-sthlm-01 crate[10434]: [2022-05-30T08:35:31,138][INFO ][o.e.p.PluginsService     ] [Stockholm 01] loaded plugin [io.crate.plugin.SrvPlugin]
May 30 08:35:31 crate-sthlm-01 crate[10434]: [2022-05-30T08:35:31,139][INFO ][o.e.p.PluginsService     ] [Stockholm 01] loaded plugin [io.crate.udc.plugin.UDCPlugin]
May 30 08:35:31 crate-sthlm-01 crate[10434]: [2022-05-30T08:35:31,139][INFO ][o.e.p.PluginsService     ] [Stockholm 01] loaded plugin [org.elasticsearch.discovery.ec2.Ec2DiscoveryPlugin]
May 30 08:35:31 crate-sthlm-01 crate[10434]: [2022-05-30T08:35:31,139][INFO ][o.e.p.PluginsService     ] [Stockholm 01] loaded plugin [org.elasticsearch.plugin.repository.url.URLRepositoryPlugin]
May 30 08:35:31 crate-sthlm-01 crate[10434]: [2022-05-30T08:35:31,139][INFO ][o.e.p.PluginsService     ] [Stockholm 01] loaded plugin [org.elasticsearch.repositories.s3.S3RepositoryPlugin]
May 30 08:35:31 crate-sthlm-01 crate[10434]: [2022-05-30T08:35:31,140][INFO ][o.e.p.PluginsService     ] [Stockholm 01] loaded plugin [org.elasticsearch.transport.Netty4Plugin]
May 30 08:35:31 crate-sthlm-01 crate[10434]: SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
May 30 08:35:31 crate-sthlm-01 crate[10434]: SLF4J: Defaulting to no-operation (NOP) logger implementation
May 30 08:35:31 crate-sthlm-01 crate[10434]: SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
May 30 08:35:31 crate-sthlm-01 crate[10434]: [2022-05-30T08:35:31,548][WARN ][o.e.d.c.s.Settings       ] [Stockholm 01] [gateway.expected_nodes] setting was deprecated in CrateDB and will be removed in a future release! See the breaking changes documentation for the next major version.
May 30 08:35:31 crate-sthlm-01 crate[10434]: [2022-05-30T08:35:31,549][WARN ][o.e.d.c.s.Settings       ] [Stockholm 01] [gateway.recover_after_nodes] setting was deprecated in CrateDB and will be removed in a future release! See the breaking changes documentation for the next major version.
May 30 08:35:31 crate-sthlm-01 crate[10434]: [2022-05-30T08:35:31,732][INFO ][o.e.d.DiscoveryModule    ] [Stockholm 01] using discovery type [zen] and seed hosts providers [settings]
May 30 08:35:32 crate-sthlm-01 crate[10434]: [2022-05-30T08:35:32,374][INFO ][psql                     ] [Stockholm 01] PSQL SSL support is disabled.
May 30 08:35:32 crate-sthlm-01 crate[10434]: [2022-05-30T08:35:32,468][INFO ][i.c.n.c.PipelineRegistry ] [Stockholm 01] HTTP SSL support is enabled.
May 30 08:35:32 crate-sthlm-01 crate[10434]: [2022-05-30T08:35:32,502][INFO ][o.e.n.Node               ] [Stockholm 01] initialized
May 30 08:35:32 crate-sthlm-01 crate[10434]: [2022-05-30T08:35:32,503][INFO ][o.e.n.Node               ] [Stockholm 01] starting ...
May 30 08:35:32 crate-sthlm-01 crate[10434]: [2022-05-30T08:35:32,586][INFO ][psql                     ] [Stockholm 01] publish_address {X.X.X.X:5432}, bound_addresses {X.X.X.X:5432}
May 30 08:35:32 crate-sthlm-01 crate[10434]: [2022-05-30T08:35:32,596][INFO ][o.e.h.n.Netty4HttpServerTransport] [Stockholm 01] publish_address {X.X.X.X:4200}, bound_addresses {X.X.X.X:4200}
May 30 08:35:32 crate-sthlm-01 crate[10434]: [2022-05-30T08:35:32,605][INFO ][o.e.t.TransportService   ] [Stockholm 01] publish_address {X.X.X.X:4300}, bound_addresses {X.X.X.X:4300}
May 30 08:35:32 crate-sthlm-01 crate[10434]: [2022-05-30T08:35:32,751][INFO ][o.e.b.BootstrapChecks    ] [Stockholm 01] bound or publishing to a non-loopback address, enforcing bootstrap checks
May 30 08:35:32 crate-sthlm-01 crate[10434]: [2022-05-30T08:35:32,753][INFO ][o.e.c.c.Coordinator      ] [Stockholm 01] cluster UUID [UFMWQbQ6Qy2oTd9PY09ldw]
May 30 08:35:32 crate-sthlm-01 crate[10434]: [2022-05-30T08:35:32,889][INFO ][o.e.c.s.MasterService    ] [Stockholm 01] elected-as-master ([1] nodes joined)[{Stockholm 01}{0ihwYOjWTGGVt0ByDXPobA}{GmsygQxAR6aWBhsNMMj8fw}{X.X.X.X}{X.X.X.X:4300}{zone=sthlm, http_address=X.X.X.X:4200} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 20, version: 59, reason: master node changed {previous [], current [{Stockholm 01}{0ihwYOjWTGGVt0ByDXPobA}{GmsygQxAR6aWBhsNMMj8fw}{X.X.X.X}{X.X.X.X:4300}{zone=sthlm, http_address=X.X.X.X:4200}]}
May 30 08:35:32 crate-sthlm-01 crate[10434]: [2022-05-30T08:35:32,929][INFO ][o.e.c.s.ClusterApplierService] [Stockholm 01] master node changed {previous [], current [{Stockholm 01}{0ihwYOjWTGGVt0ByDXPobA}{GmsygQxAR6aWBhsNMMj8fw}{X.X.X.X}{X.X.X.X:4300}{zone=sthlm, http_address=X.X.X.X:4200}]}, term: 20, version: 59, reason: Publication{term=20, version=59}
May 30 08:35:32 crate-sthlm-01 crate[10434]: [2022-05-30T08:35:32,951][INFO ][o.e.n.Node               ] [Stockholm 01] started
May 30 08:35:33 crate-sthlm-01 crate[10434]: [2022-05-30T08:35:33,176][WARN ][o.e.t.n.Netty4Transport  ] [Stockholm 01] exception caught on transport layer [Netty4TcpChannel{localAddress=/X.X.X.X:4300, remoteAddress=/Y.Y.Y.Y:55412}], closing connection
May 30 08:35:33 crate-sthlm-01 crate[10434]: io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
May 30 08:35:33 crate-sthlm-01 crate[10434]: #011at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480) ~[netty-codec-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:33 crate-sthlm-01 crate[10434]: #011at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) ~[netty-codec-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:33 crate-sthlm-01 crate[10434]: #011at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:33 crate-sthlm-01 crate[10434]: #011at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:33 crate-sthlm-01 crate[10434]: #011at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:33 crate-sthlm-01 crate[10434]: #011at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:33 crate-sthlm-01 crate[10434]: #011at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:33 crate-sthlm-01 crate[10434]: #011at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:33 crate-sthlm-01 crate[10434]: #011at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:33 crate-sthlm-01 crate[10434]: #011at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800) [netty-transport-classes-epoll-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:33 crate-sthlm-01 crate[10434]: #011at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487) [netty-transport-classes-epoll-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:33 crate-sthlm-01 crate[10434]: #011at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:385) [netty-transport-classes-epoll-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:33 crate-sthlm-01 crate[10434]: #011at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:33 crate-sthlm-01 crate[10434]: #011at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:33 crate-sthlm-01 crate[10434]: #011at java.lang.Thread.run(Thread.java:833) [?:?]
May 30 08:35:33 crate-sthlm-01 crate[10434]: Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
May 30 08:35:33 crate-sthlm-01 crate[10434]: #011at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
May 30 08:35:33 crate-sthlm-01 crate[10434]: #011at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
May 30 08:35:33 crate-sthlm-01 crate[10434]: #011at sun.security.ssl.TransportContext.fatal(TransportContext.java:358) ~[?:?]
May 30 08:35:33 crate-sthlm-01 crate[10434]: #011at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?]
May 30 08:35:33 crate-sthlm-01 crate[10434]: #011at sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) ~[?:?]
May 30 08:35:33 crate-sthlm-01 crate[10434]: #011at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
May 30 08:35:33 crate-sthlm-01 crate[10434]: #011at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]
May 30 08:35:33 crate-sthlm-01 crate[10434]: #011at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]
May 30 08:35:33 crate-sthlm-01 crate[10434]: #011at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]
May 30 08:35:33 crate-sthlm-01 crate[10434]: #011at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]
May 30 08:35:33 crate-sthlm-01 crate[10434]: #011at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]
May 30 08:35:33 crate-sthlm-01 crate[10434]: #011at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:295) ~[netty-handler-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:33 crate-sthlm-01 crate[10434]: #011at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1342) ~[netty-handler-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:33 crate-sthlm-01 crate[10434]: #011at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235) ~[netty-handler-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:33 crate-sthlm-01 crate[10434]: #011at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) ~[netty-handler-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:33 crate-sthlm-01 crate[10434]: #011at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) ~[netty-codec-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:33 crate-sthlm-01 crate[10434]: #011at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) ~[netty-codec-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:33 crate-sthlm-01 crate[10434]: #011... 14 more

Second node (Falkenberg / falk / Y.Y.Y.Y)

May 30 08:35:45 crate-falk-01 systemd[1]: Started CrateDB Server.
May 30 08:35:47 crate-falk-01 crate[9884]: [2022-05-30T08:35:47,332][INFO ][o.e.e.NodeEnvironment    ] [Falkenberg 01] using [1] data paths, mounts [[/ (/dev/sda2)]], net usable_space [37.3gb], net total_space [49.8gb], types [ext4]
May 30 08:35:47 crate-falk-01 crate[9884]: [2022-05-30T08:35:47,335][INFO ][o.e.e.NodeEnvironment    ] [Falkenberg 01] heap size [1gb], compressed ordinary object pointers [true]
May 30 08:35:47 crate-falk-01 crate[9884]: [2022-05-30T08:35:47,422][INFO ][o.e.n.Node               ] [Falkenberg 01] node name [Falkenberg 01], node ID [Q-t-i_70QWW0t-hcaD8_zA], cluster name [my_cluster]
May 30 08:35:47 crate-falk-01 crate[9884]: [2022-05-30T08:35:47,438][INFO ][o.e.n.Node               ] [Falkenberg 01] version[4.8.0], pid[9884], build[e1dea64/2022-04-28T17:06:46Z], OS[Linux/5.10.0-10-amd64/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.3/17.0.3+7]
May 30 08:35:48 crate-falk-01 crate[9884]: [2022-05-30T08:35:48,095][INFO ][o.e.p.PluginsService     ] [Falkenberg 01] no modules loaded
May 30 08:35:48 crate-falk-01 crate[9884]: [2022-05-30T08:35:48,097][INFO ][o.e.p.PluginsService     ] [Falkenberg 01] loaded plugin [cr8-copy-s3]
May 30 08:35:48 crate-falk-01 crate[9884]: [2022-05-30T08:35:48,097][INFO ][o.e.p.PluginsService     ] [Falkenberg 01] loaded plugin [crate-azure-discovery]
May 30 08:35:48 crate-falk-01 crate[9884]: [2022-05-30T08:35:48,098][INFO ][o.e.p.PluginsService     ] [Falkenberg 01] loaded plugin [crate-functions]
May 30 08:35:48 crate-falk-01 crate[9884]: [2022-05-30T08:35:48,098][INFO ][o.e.p.PluginsService     ] [Falkenberg 01] loaded plugin [crate-jmx-monitoring]
May 30 08:35:48 crate-falk-01 crate[9884]: [2022-05-30T08:35:48,098][INFO ][o.e.p.PluginsService     ] [Falkenberg 01] loaded plugin [crate-lang-js]
May 30 08:35:48 crate-falk-01 crate[9884]: [2022-05-30T08:35:48,099][INFO ][o.e.p.PluginsService     ] [Falkenberg 01] loaded plugin [es-analysis-common]
May 30 08:35:48 crate-falk-01 crate[9884]: [2022-05-30T08:35:48,099][INFO ][o.e.p.PluginsService     ] [Falkenberg 01] loaded plugin [es-analysis-phonetic]
May 30 08:35:48 crate-falk-01 crate[9884]: [2022-05-30T08:35:48,100][INFO ][o.e.p.PluginsService     ] [Falkenberg 01] loaded plugin [es-repository-azure]
May 30 08:35:48 crate-falk-01 crate[9884]: [2022-05-30T08:35:48,100][INFO ][o.e.p.PluginsService     ] [Falkenberg 01] loaded plugin [io.crate.plugin.SrvPlugin]
May 30 08:35:48 crate-falk-01 crate[9884]: [2022-05-30T08:35:48,100][INFO ][o.e.p.PluginsService     ] [Falkenberg 01] loaded plugin [io.crate.udc.plugin.UDCPlugin]
May 30 08:35:48 crate-falk-01 crate[9884]: [2022-05-30T08:35:48,100][INFO ][o.e.p.PluginsService     ] [Falkenberg 01] loaded plugin [org.elasticsearch.discovery.ec2.Ec2DiscoveryPlugin]
May 30 08:35:48 crate-falk-01 crate[9884]: [2022-05-30T08:35:48,101][INFO ][o.e.p.PluginsService     ] [Falkenberg 01] loaded plugin [org.elasticsearch.plugin.repository.url.URLRepositoryPlugin]
May 30 08:35:48 crate-falk-01 crate[9884]: [2022-05-30T08:35:48,101][INFO ][o.e.p.PluginsService     ] [Falkenberg 01] loaded plugin [org.elasticsearch.repositories.s3.S3RepositoryPlugin]
May 30 08:35:48 crate-falk-01 crate[9884]: [2022-05-30T08:35:48,102][INFO ][o.e.p.PluginsService     ] [Falkenberg 01] loaded plugin [org.elasticsearch.transport.Netty4Plugin]
May 30 08:35:48 crate-falk-01 crate[9884]: SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
May 30 08:35:48 crate-falk-01 crate[9884]: SLF4J: Defaulting to no-operation (NOP) logger implementation
May 30 08:35:48 crate-falk-01 crate[9884]: SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
May 30 08:35:48 crate-falk-01 crate[9884]: [2022-05-30T08:35:48,624][WARN ][o.e.d.c.s.Settings       ] [Falkenberg 01] [gateway.expected_nodes] setting was deprecated in CrateDB and will be removed in a future release! See the breaking changes documentation for the next major version.
May 30 08:35:48 crate-falk-01 crate[9884]: [2022-05-30T08:35:48,625][WARN ][o.e.d.c.s.Settings       ] [Falkenberg 01] [gateway.recover_after_nodes] setting was deprecated in CrateDB and will be removed in a future release! See the breaking changes documentation for the next major version.
May 30 08:35:48 crate-falk-01 crate[9884]: [2022-05-30T08:35:48,827][INFO ][o.e.d.DiscoveryModule    ] [Falkenberg 01] using discovery type [zen] and seed hosts providers [settings]
May 30 08:35:49 crate-falk-01 crate[9884]: [2022-05-30T08:35:49,524][INFO ][psql                     ] [Falkenberg 01] PSQL SSL support is disabled.
May 30 08:35:49 crate-falk-01 crate[9884]: [2022-05-30T08:35:49,622][INFO ][i.c.n.c.PipelineRegistry ] [Falkenberg 01] HTTP SSL support is enabled.
May 30 08:35:49 crate-falk-01 crate[9884]: [2022-05-30T08:35:49,657][INFO ][o.e.n.Node               ] [Falkenberg 01] initialized
May 30 08:35:49 crate-falk-01 crate[9884]: [2022-05-30T08:35:49,658][INFO ][o.e.n.Node               ] [Falkenberg 01] starting ...
May 30 08:35:49 crate-falk-01 crate[9884]: [2022-05-30T08:35:49,743][INFO ][psql                     ] [Falkenberg 01] publish_address {Y.Y.Y.Y:5432}, bound_addresses {Y.Y.Y.Y:5432}
May 30 08:35:49 crate-falk-01 crate[9884]: [2022-05-30T08:35:49,755][INFO ][o.e.h.n.Netty4HttpServerTransport] [Falkenberg 01] publish_address {Y.Y.Y.Y:4200}, bound_addresses {Y.Y.Y.Y:4200}
May 30 08:35:49 crate-falk-01 crate[9884]: [2022-05-30T08:35:49,764][INFO ][o.e.t.TransportService   ] [Falkenberg 01] publish_address {Y.Y.Y.Y:4300}, bound_addresses {Y.Y.Y.Y:4300}
May 30 08:35:49 crate-falk-01 crate[9884]: [2022-05-30T08:35:49,885][INFO ][o.e.b.BootstrapChecks    ] [Falkenberg 01] bound or publishing to a non-loopback address, enforcing bootstrap checks
May 30 08:35:49 crate-falk-01 crate[9884]: [2022-05-30T08:35:49,889][INFO ][o.e.c.c.Coordinator      ] [Falkenberg 01] cluster UUID [UFMWQbQ6Qy2oTd9PY09ldw]
May 30 08:35:50 crate-falk-01 crate[9884]: [2022-05-30T08:35:50,413][WARN ][o.e.t.OutboundHandler    ] [Falkenberg 01] send message failed [channel: Netty4TcpChannel{localAddress=/Y.Y.Y.Y:55440, remoteAddress=crate-sthlm-01.my.domain/X.X.X.X:4300}]
May 30 08:35:50 crate-falk-01 crate[9884]: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
May 30 08:35:50 crate-falk-01 crate[9884]: #011at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at sun.security.ssl.TransportContext.fatal(TransportContext.java:371) ~[?:?]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at sun.security.ssl.TransportContext.fatal(TransportContext.java:314) ~[?:?]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at sun.security.ssl.TransportContext.fatal(TransportContext.java:309) ~[?:?]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1357) ~[?:?]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1232) ~[?:?]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175) ~[?:?]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277) ~[?:?]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264) ~[?:?]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at java.security.AccessController.doPrivileged(AccessController.java:712) ~[?:?]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209) ~[?:?]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1548) [netty-handler-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1394) [netty-handler-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235) [netty-handler-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) [netty-handler-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) [netty-codec-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) [netty-codec-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) [netty-codec-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800) [netty-transport-classes-epoll-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487) [netty-transport-classes-epoll-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:385) [netty-transport-classes-epoll-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at java.lang.Thread.run(Thread.java:833) [?:?]
May 30 08:35:50 crate-falk-01 crate[9884]: Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
May 30 08:35:50 crate-falk-01 crate[9884]: #011at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439) ~[?:?]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306) ~[?:?]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) ~[?:?]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) ~[?:?]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1335) ~[?:?]
May 30 08:35:50 crate-falk-01 crate[9884]: #011... 28 more
May 30 08:35:50 crate-falk-01 crate[9884]: Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
May 30 08:35:50 crate-falk-01 crate[9884]: #011at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) ~[?:?]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) ~[?:?]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297) ~[?:?]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434) ~[?:?]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306) ~[?:?]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) ~[?:?]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) ~[?:?]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1335) ~[?:?]
May 30 08:35:50 crate-falk-01 crate[9884]: #011... 28 more
May 30 08:35:50 crate-falk-01 crate[9884]: [2022-05-30T08:35:50,422][WARN ][o.e.t.n.Netty4Transport  ] [Falkenberg 01] exception caught on transport layer [Netty4TcpChannel{localAddress=/Y.Y.Y.Y:55440, remoteAddress=crate-sthlm-01.my.domain/X.X.X.X:4300}], closing connection
May 30 08:35:50 crate-falk-01 crate[9884]: io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
May 30 08:35:50 crate-falk-01 crate[9884]: #011at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480) ~[netty-codec-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) ~[netty-codec-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800) [netty-transport-classes-epoll-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487) [netty-transport-classes-epoll-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:385) [netty-transport-classes-epoll-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.76.Final.jar:4.1.76.Final]
May 30 08:35:50 crate-falk-01 crate[9884]: #011at java.lang.Thread.run(Thread.java:833) [?:?]

Hello @Ivar,

Could you please share the common name (CN) used in the step 1
and step 4 (which I expect was done as described here)?

If possible, it would be good to see commands and their outputs used per each step - if they contain sensitive data, feel free to replace domains/names.

1 Like

Hi @Baur,

That’s correct. I didn’t save the output, but these are the subjects of the certificates - is it sufficient?

$ openssl x509 -noout -subject -in rootCA.crt
subject= /C=SE/ST=Stockholm/L=Stockholm/O=My Company, Ltd./CN=*.my.domain/emailAddress=info@my.domain
$ openssl x509 -noout -subject -in crate-sthlm-01.crt
subject= /C=SE/ST=Stockholm/L=Stockholm/O=My Company, Ltd./OU=My Company, Ltd./CN=crate-sthlm-01.my.domain
$ openssl x509 -noout -subject -in crate-falk-01.crt
subject= /C=SE/ST=Stockholm/L=Stockholm/O=My Company, Ltd./OU=My Company, Ltd./CN=crate-falk-01.my.domain

Thanks for providing details.

Subjects look fine - I wanted to check that CN-s satisfy requirement

The common name (CN) should overlap with the CN of the server key generated in the first step

Since you wrote … I’ve done (all nodes) - question to step 4, did you generate one root CA?

I think so (otherwise that’s the issue), in that case would be really great to see commands - the whole flow looks ok, maybe some detail slipped away from attention. Could you try to run them again maybe with replaced domains/names (ideally with saving outputs along the way) and share?

Of course - I’ll do it first-thing tomorrow morning. :slight_smile:

So @Baur - here’s every command and the outputs, step-by-step.

The steps are done one three different machines:

  • Stockholm (FQDN crate-sthlm-01.my.domain, IP X.X.X.X), master
  • Falkenberg (FQDN crate-falk-01.my.domain, IP Y.Y.Y.Y)
  • My computer (as authority)

1. Stockholm - Set up keystore

1.1. Create keystore & generate private key

$ keytool -keystore keystore -genkey -keyalg RSA -alias private_key -validity 36500
Enter keystore password:  
Re-enter new password: 
What is your first and last name?
  [Unknown]:  crate-sthlm-01.my.domain
What is the name of your organizational unit?
  [Unknown]:  My Company
What is the name of your organization?
  [Unknown]:  My Company, Ltd.
What is the name of your City or Locality?
  [Unknown]:  Stockholm
What is the name of your State or Province?
  [Unknown]:  Stockholm
What is the two-letter country code for this unit?
  [Unknown]:  SE
Is CN=crate-sthlm-01.my.domain, OU=My Company, O="My Company, Ltd.", L=Stockholm, ST=Stockholm, C=SE correct?
  [no]:  yes

Generating 2,048 bit RSA key pair and self-signed certificate (SHA256withRSA) with a validity of 36,500 days
	for: CN=crate-sthlm-01.my.domain, OU=My Company, O="My Company, Ltd.", L=Stockholm, ST=Stockholm, C=SE

1.2. Generate signing request (CSR)

$ keytool -keystore keystore -certreq -alias private_key -keyalg RSA -file crate-sthlm-01.csr
Enter keystore password:

2. Falkenberg - Set up keystore

2.1. Create keystore & generate private key

$ keytool -keystore keystore -genkey -keyalg RSA -alias private_key -validity 36500
Enter keystore password:  
Re-enter new password: 
What is your first and last name?
  [Unknown]:  crate-falk-01.my.domain
What is the name of your organizational unit?
  [Unknown]:  My Company
What is the name of your organization?
  [Unknown]:  My Company, Ltd.
What is the name of your City or Locality?
  [Unknown]:  Stockholm
What is the name of your State or Province?
  [Unknown]:  Stockholm
What is the two-letter country code for this unit?
  [Unknown]:  SE
Is CN=crate-falk-01.my.domain, OU=My Company, O="My Company, Ltd.", L=Stockholm, ST=Stockholm, C=SE correct?
  [no]:  yes

Generating 2,048 bit RSA key pair and self-signed certificate (SHA256withRSA) with a validity of 36,500 days
	for: CN=crate-falk-01.my.domain, OU=My Company, O="My Company, Ltd.", L=Stockholm, ST=Stockholm, C=SE

2.2. Generate signing request (CSR)

$ keytool -keystore keystore -certreq -alias private_key -keyalg RSA -file crate-falk-01.csr
Enter keystore password:

3. My computer - Root CA setup

3.1. Generate root CA

$ openssl req -x509 -sha256 -nodes -days 36500 -newkey rsa:2048 \            
    -keyout rootCA.key -out rootCA.crt
Generating a 2048 bit RSA private key
..............+++
.......................+++
writing new private key to 'rootCA.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) []:SE
State or Province Name (full name) []:Stockholm
Locality Name (eg, city) []:Stockholm
Organization Name (eg, company) []:My Company, Ltd.
Organizational Unit Name (eg, section) []:Headquarters
Common Name (eg, fully qualified host name) []:*.my.domain
Email Address []:info@my.domain

3.2. Check content of extension file

$ cat ssl.cnf 
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = @alt_names

[alt_names]
DNS.1 = crate-sthlm-01.my.domain
DNS.2 = crate-falk-01.my.domain
IP.1 = X.X.X.X
IP.2 = Y.Y.Y.Y

4. My computer - Sign certificate for Stockholm

4.1. Download Stockholm CSR

$ scp ivar@crate-sthlm-01.my.domain:/home/crate/crate-sthlm-01.csr .
crate-sthlm-01.csr                          100% 1147   246.0KB/s   00:00

4.2. Generate certificate for Stockholm

$ openssl x509 -req -in crate-sthlm-01.csr -CA rootCA.crt -CAkey rootCA.key \
    -CAcreateserial -out crate-sthlm-01.crt -sha256 -days 36500 -extfile ssl.cnf
Signature ok
subject=/C=SE/ST=Stockholm/L=Stockholm/O=My Company, Ltd./OU=My Company/CN=crate-sthlm-01.my.domain
Getting CA Private Key

4.3. Upload certificate to Stockholm

$ scp crate-sthlm-01.crt ivar@crate-sthlm-01.my.domain:~
crate-sthlm-01.crt                          100% 1765   505.8KB/s   00:00

4.4. Upload root CA certificate to Stockholm

$ scp rootCA.crt ivar@crate-sthlm-01.my.domain:~
rootCA.crt                                  100% 1371   347.0KB/s   00:00

5. My computer - Sign certificate for Falkenberg

5.1. Download Falkenberg CSR

$ scp ivar@crate-falk-01.my.domain:/home/crate/crate-falk-01.csr .
crate-falk-01.csr                           100% 1147    73.6KB/s   00:00

5.2. Generate certificate for Falkenberg

$ openssl x509 -req -in crate-falk-01.csr -CA rootCA.crt -CAkey rootCA.key \
    -CAcreateserial -out crate-falk-01.crt -sha256 -days 36500 -extfile ssl.cnf
Signature ok
subject=/C=SE/ST=Stockholm/L=Stockholm/O=My Company, Ltd./OU=My Company/CN=crate-falk-01.my.domain
Getting CA Private Key

5.3. Upload certificate to Falkenberg

$ scp crate-falk-01.crt ivar@crate-falk-01.my.domain:~
crate-falk-01.crt                           100% 1761   109.1KB/s   00:00

5.4. Upload root CA to Falkenberg

$ scp rootCA.crt ivar@crate-falk-01.my.domain:~
rootCA.crt                                  100% 1371    84.8KB/s   00:00

6. Stockholm - Import certificates

6.1. Move certificates to correct directory and set correct permissions

$ mv /home/ivar/*.crt . && chown crate:crate * && chmod 400 *

6.2. Check files

$ ls -l
total 16
-r-------- 1 crate crate 1765 Jun  2 08:44 crate-sthlm-01.crt
-r-------- 1 crate crate 1147 Jun  2 08:17 crate-sthlm-01.csr
-r-------- 1 crate crate 2834 Jun  2 08:14 keystore
-r-------- 1 crate crate 1371 Jun  2 08:46 rootCA.crt

6.3. Import root CA certificate

$ keytool -import -keystore keystore -file rootCA.crt -alias root_ca
Enter keystore password:  
Owner: EMAILADDRESS=info@my.domain, CN=*.my.domain, OU=Headquarters, O="My Company, Ltd.", L=Stockholm, ST=Stockholm, C=SE
Issuer: EMAILADDRESS=info@my.domain, CN=*.my.domain, OU=Headquarters, O="My Company, Ltd.", L=Stockholm, ST=Stockholm, C=SE
Serial number: 808d1c8e2da5a10f
Valid from: Thu Jun 02 08:37:54 UTC 2022 until: Sat May 09 08:37:54 UTC 2122
Certificate fingerprints:
	 SHA1: A0:6B:FC:21:9A:30:05:38:61:C3:B1:57:70:5A:30:49:1E:D7:6C:4C
	 SHA256: 55:87:C7:29:AE:34:E2:7E:5C:13:6A:23:A4:66:86:B2:78:A6:FF:8B:0B:2B:6D:DC:49:32:15:FF:64:AD:95:C8
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 1
Trust this certificate? [no]:  yes
Certificate was added to keystore

6.4. Import signed certificate

$ keytool -import -keystore keystore -file crate-sthlm-01.crt -alias cert
Enter keystore password:  
Certificate was added to keystore

6.5. Check keystore content

$ keytool -keystore keystore -list
Enter keystore password:  
Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 3 entries

cert, Jun 2, 2022, trustedCertEntry, 
Certificate fingerprint (SHA-256): 22:A9:8E:3E:38:E6:9C:F3:8F:4F:22:7C:10:73:AF:B5:06:E7:5C:40:FD:85:49:8B:E9:A4:F6:97:E7:C9:A3:02
private_key, Jun 2, 2022, PrivateKeyEntry, 
Certificate fingerprint (SHA-256): A8:B2:37:7F:88:26:EC:6D:85:AA:A1:F0:43:46:6E:31:1E:2C:80:01:27:24:FD:C9:8B:4F:24:E2:8F:A8:80:E1
root_ca, Jun 2, 2022, trustedCertEntry, 
Certificate fingerprint (SHA-256): 55:87:C7:29:AE:34:E2:7E:5C:13:6A:23:A4:66:86:B2:78:A6:FF:8B:0B:2B:6D:DC:49:32:15:FF:64:AD:95:C8

6.6. Check SSL config

$ cat /etc/crate/crate.yml

[...]

# Enable encrypted communication for the HTTP endpoints:
ssl.http.enabled: true

# Enable encrypted communication for the PostgreSQL wire protocol:
#ssl.psql.enabled: true

ssl.transport.mode: on

# The full path to the node keystore file
ssl.keystore_filepath: /home/crate/keystore

# The password used to decrypt the keystore_file.jks
ssl.keystore_password: my_password

# The password entered at the end of the keytool -genkey command if different
# than the keystore_password.
ssl.keystore_key_password: my_password

# Optional configuration for truststore

# The full path to the node truststore file
#ssl.truststore_filepath:

# The password used to decrypt the truststore_file.jks
#ssl.truststore_password:

# The frequency at which SSL files are monitored for changes
ssl.resource_poll_interval: 10s

[...]

6.7. Restart Crate

$ service crate restart

7. Falkenberg - Import certificates

7.1. Move certificates to correct directory and set correct permissions

$ mv /home/ivar/*.crt . && chown crate:crate * && chmod 400 *

7.2. Check files

$ ls -l
total 16
-r-------- 1 crate crate 1761 Jun  2 08:50 crate-falk-01.crt
-r-------- 1 crate crate 1147 Jun  2 08:27 crate-falk-01.csr
-r-------- 1 crate crate 2834 Jun  2 08:24 keystore
-r-------- 1 crate crate 1371 Jun  2 08:51 rootCA.crt

7.3. Import root CA certificate

$ keytool -import -keystore keystore -file rootCA.crt -alias root_ca
Enter keystore password:  
Owner: EMAILADDRESS=info@my.domain, CN=*.my.domain, OU=Headquarters, O="My Company, Ltd.", L=Stockholm, ST=Stockholm, C=SE
Issuer: EMAILADDRESS=info@my.domain, CN=*.my.domain, OU=Headquarters, O="My Company, Ltd.", L=Stockholm, ST=Stockholm, C=SE
Serial number: 808d1c8e2da5a10f
Valid from: Thu Jun 02 08:37:54 UTC 2022 until: Sat May 09 08:37:54 UTC 2122
Certificate fingerprints:
	 SHA1: A0:6B:FC:21:9A:30:05:38:61:C3:B1:57:70:5A:30:49:1E:D7:6C:4C
	 SHA256: 55:87:C7:29:AE:34:E2:7E:5C:13:6A:23:A4:66:86:B2:78:A6:FF:8B:0B:2B:6D:DC:49:32:15:FF:64:AD:95:C8
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 1
Trust this certificate? [no]:  yes
Certificate was added to keystore

7.4. Import signed certificate

$ keytool -import -keystore keystore -file crate-falk-01.crt -alias cert
Enter keystore password:  
Certificate was added to keystore

7.5. Check keystore content

$ keytool -keystore keystore -list
Enter keystore password:  
Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 3 entries

cert, Jun 2, 2022, trustedCertEntry, 
Certificate fingerprint (SHA-256): 3A:C0:98:41:B7:9A:52:7F:CF:5C:DB:44:3E:ED:91:1B:8C:1B:2B:B2:3C:65:BE:80:A6:7D:25:6D:7E:F5:80:52
private_key, Jun 2, 2022, PrivateKeyEntry, 
Certificate fingerprint (SHA-256): E7:7E:9C:A7:0D:09:6A:95:4E:F3:9A:84:D3:60:9F:6A:0D:65:FC:45:D5:62:4A:95:48:21:35:AD:2F:DC:96:67
root_ca, Jun 2, 2022, trustedCertEntry, 
Certificate fingerprint (SHA-256): 55:87:C7:29:AE:34:E2:7E:5C:13:6A:23:A4:66:86:B2:78:A6:FF:8B:0B:2B:6D:DC:49:32:15:FF:64:AD:95:C8

7.6. Check SSL config

$ cat /etc/crate/crate.yml

[...]

# Enable encrypted communication for the HTTP endpoints:
ssl.http.enabled: true

# Enable encrypted communication for the PostgreSQL wire protocol:
#ssl.psql.enabled: true

ssl.transport.mode: on

# The full path to the node keystore file
ssl.keystore_filepath: /home/crate/keystore

# The password used to decrypt the keystore_file.jks
ssl.keystore_password: my_password

# The password entered at the end of the keytool -genkey command if different
# than the keystore_password.
ssl.keystore_key_password: my_password

# Optional configuration for truststore

# The full path to the node truststore file
#ssl.truststore_filepath:

# The password used to decrypt the truststore_file.jks
#ssl.truststore_password:

# The frequency at which SSL files are monitored for changes
ssl.resource_poll_interval: 10s

[...]

7.7. Restart Crate

$ service crate restart

8. Result

Same as before.

1 Like

Hi @Ivar

We could reproduce it locally with 4.7.3 and 4.8.1 , but are still unsure if this is a configuration issue with self-signed certificates or maybe a bug. I can definitely say, that we need to improve our documentation around it :grimacing:

Thanks for reporting it :pray:

1 Like

Thanks @proddata for your quick reply!

The documentation could indeed be improved on this (some steps were also missing, e.g. how to add the extension file). If we enjoy the database further on I’d be happy to participate in the documentation.

Okay, so you’re investigating it further? I hate myself for asking this about troubleshooting, but do you have any ETA?

I’d also be fine with using e.g. Let’s Encrypt, would this work better? I’m not sure how that’d work with the keystore though - I’m no Java developer. :sweat_smile:

Okay, so you’re investigating it further?

yes

I hate myself for asking this about troubleshooting, but do you have any ETA?

ETAs are always though to get, but typically the team is rather quick in terms of bug fixes :grimacing:


Are you planning to use logical replication with the node-2-node encryption or a multi-zone setup?

CCR?

Any thoughts regarding Let’s Encrypt? Do you think that would work better than self-signed?

I meant logical replication.


Any thoughts regarding Let’s Encrypt?

It might be worth testing if it is an issue with the CA, but I don’t have to high hopes

Ah - no, we’ll most likely go with a single cluster with multi-zone setup, as described here.

I’ll give Let’s Encrypt a shot, I’m a bit stressed with getting this working.

@proddata - it seems to work good with Let’s Encrypt, I’ll come back with instructions.

1 Like

We figured that it might be a problem with the truststore setup and I planned to give it another try today. The Let’s Encrypt CAs are in the system/java trust store.

For any others who’d like to use Let’s Encrypt with CrateDB, this is what worked for me. Please feel free to add it to your documentation, @proddata.

Paths may vary - this is for Debian 11.

After installing Certbot, you need a post-renewal-hook that loads the certificates into the keystore. Create the file /etc/letsencrypt/renewal-hooks/post/crate.sh with the following content:

#!/bin/sh

# Convert PEM certificate + key into PKCS12 file
openssl pkcs12 -export \
-in {{ABSOLUTE_PATH_TO_FULLCHAIN}} \
-inkey {{ABSOLUTE_PATH_TO_PRIVKEY}} \
-out /tmp/cert.p12 \
-name letsencrypt \
-passout pass:{{TEMPORARY_PASSWORD}} \
> /dev/null 2>&1

# Import PKCS12 file into keystore
keytool -noprompt -importkeystore \
-srckeystore /tmp/cert.p12 \
-srcstoretype PKCS12 \
-srcstorepass {{TEMPORARY_PASSWORD}} \
-deststorepass {{KEYSTORE_PASSWORD}} \
-destkeypass {{PRIVATE_KEY_PASSWORD}} \
-destkeystore {{ABSOLUTE_PATH_TO_KEYSTORE}} \
-alias letsencrypt \
> /dev/null 2>&1

# Delete PKCS12 file
rm /tmp/cert.p12

Ensure to replace all {{VARIABLES}} with the corresponding values and set executable permissions on the file:

$ chmod +x /etc/letsencrypt/renewal-hooks/post/crate.sh

Activate SSL in the CrateDB config (/etc/crate/crate.yml):

# Enable encrypted communication for the HTTP endpoints:
ssl.http.enabled: true

# Enable encrypted communication for the PostgreSQL wire protocol:
ssl.psql.enabled: true

# Enable encrypted node-to-node communication
ssl.transport.mode: on

# The full path to the node keystore file
ssl.keystore_filepath: {{ABSOLUTE_PATH_TO_KEYSTORE}}

# The password used to decrypt the keystore_file.jks
ssl.keystore_password: {{KEYSTORE_PASSWORD}}

# The password entered at the end of the keytool -genkey command if different
# than the keystore_password.
ssl.keystore_key_password: {{PRIVATE_KEY_PASSWORD}}

# The frequency at which SSL files are monitored for changes
ssl.resource_poll_interval: 10s

Finish by restarting CrateDB:

$ service crate restart
2 Likes