CrateDB Prometheus adapter service not starting

BrianS · January 4, 2022, 11:25pm

Hello,

I am trying to use the crate prometheus adapter for long term data retention with prometheus. The problem I am having is with setting up the service where the service is not able to see the config.yml file.

The error I get is the following:

cratedb-prometheus-adapter[222965]: time="2022-01-04T18:02:04-05:00" level=fatal msg="Error loading configuration file \"config.yml\": error reading configuration file: open config.yml: no such file or directory" source="server.go:393"

I was using the instructions a this location: → “GitHub - crate/cratedb-prometheus-adapter: CrateDB Prometheus Adapter”

I’m hoping some has seen this in the past.

Thank you

jayeff · January 5, 2022, 9:10am

Hi @BrianS,

can you paste the command you use to start cratedb-prometheus-adapter? Do you specify the -config.file parameter and point it to your config.yml file?

proddata · January 5, 2022, 12:19pm

I think the Usage instructions aren’t quite clear, as the config.yml is only included with the docker image, but not with the native images. You can create a simple config.yml in the same folder or use the -config.file option as described by @jayeff

cratedb_endpoints:
- host: "localhost"         # Host to connect to (default: "localhost").
  port: 5432                # Port to connect to (default: 5432).
  user: "crate"             # Username to use (default: "crate")
  password: ""              # Password to use (default: "").
  schema: ""                # Schema to use (default: "").
  connect_timeout: 10       # TCP connect timeout (seconds) (default: 10).
  max_connections: 5        # The maximum number of concurrent connections (default: 5).
  enable_tls: false         # Whether to connect using TLS (default: false).
  allow_insecure_tls: false # Whether to allow insecure / invalid TLS certificates (default: false).

BrianS · January 6, 2022, 3:40am

Thanks for the replies.

The location and the content of the file config.yml looks as follows:

cat /etc/cratedb-prometheus-adapter/config.yml 
cratedb_endpoints:
- host: "localhost"         # Host to connect to (default: "localhost").
  port: 5432                # Port to connect to (default: 5432).
  user: "crate"             # Username to use (default: "crate")
  password: ""              # Password to use (default: "").
  schema: ""                # Schema to use (default: "").
  connect_timeout: 10       # TCP connect timeout (seconds) (default: 10).
  max_connections: 5        # The maximum number of concurrent connections (default: 5).
  enable_tls: false         # Whether to connect using TLS (default: false).
  allow_insecure_tls: false # Whether to allow insecure / invalid TLS certificates (default: false).

The start command I used is the following:

systemctl start cratedb-prometheus-adapter.service

The service file content shows the following:

 cat /etc/systemd/system/cratedb-prometheus-adapter.service 
# -*- mode: conf -*-

[Unit]
Description=Prometheus adapter to write to CrateDB
Documentation=https://github.com/crate/cratedb-prometheus-adapter/
After=network.target

[Service]
#EnvironmentFile=-/etc/default/cratedb-prometheus-adapter
#EnvironmentFile=-/etc/cratedb-prometheus-adapter
EnvironmentFile=-/etc/cratedb-prometheus-adapter/config.yml
User=prometheus
ExecStart=/usr/bin/cratedb-prometheus-adapter $CRATEDB_PROMETHEUS_ADAPTER_OPTS
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
LimitNOFILE=65536
LimitNPROC=4096

[Install]
WantedBy=multi-user.target

Thanks again!

jayeff · January 10, 2022, 12:26pm

Hi @BrianS,

EnvironmentFile will put stuff into your env but you cannot directly load the config though it. Instead use it to set a env variable which points to your config file as done in the sample EnvironmentFile. Adjust paths as you need them.

See also the corresponding section in the README: GitHub - crate/cratedb-prometheus-adapter: CrateDB Prometheus Adapter

BrianS · January 14, 2022, 6:13pm

Thanks @jayeff

So I made the following change in the file: /etc/systemd/system/cratedb-prometheus-adapter.service:

Added the Line:
CRATEDB_PROMETHEUS_ADAPTER_OPTS=-config.file=/etc/cratedb-prometheus-adapter/config.yml

Next, this is what the actual file looks like now with the added line:

 # -*- mode: conf -*-

[Unit]
Description=Prometheus adapter to write to CrateDB
Documentation=https://github.com/crate/cratedb-prometheus-adapter/
After=network.target

[Service]
EnvironmentFile=-/etc/cratedb-prometheus-adapter/config.yml
CRATEDB_PROMETHEUS_ADAPTER_OPTS=-config.file=/etc/cratedb-prometheus-adapter/config.yml
User=prometheus
ExecStart=/usr/bin/cratedb-prometheus-adapter $CRATEDB_PROMETHEUS_ADAPTER_OPTS
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
LimitNOFILE=65536
LimitNPROC=4096

Now the file “/etc/cratedb-prometheus-adapter/config.yml” is referenced in the “cat /etc/cratedb-prometheus-adapter/config.yml” variable.

Below is the content of that file:


cratedb_endpoints:
- host: "localhost"         # Host to connect to (default: "localhost").
  port: 5432                # Port to connect to (default: 5432).
  user: "crate"             # Username to use (default: "crate")
  password: ""              # Password to use (default: "").
  schema: ""                # Schema to use (default: "").
  connect_timeout: 10       # TCP connect timeout (seconds) (default: 10).
  max_connections: 5        # The maximum number of concurrent connections (default: 5).
  enable_tls: false         # Whether to connect using TLS (default: false).
  allow_insecure_tls: false # Whether to allow insecure / invalid TLS certificates (default: false).

Finally, I attempt to start the service and then check the status:

systemctl start cratedb-prometheus-adapter


Checking the status:
systemctl status cratedb-prometheus-adapter

● cratedb-prometheus-adapter.service - Prometheus adapter to write to CrateDB
   Loaded: loaded (/etc/systemd/system/cratedb-prometheus-adapter.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2022-01-14 12:50:10 EST; 4s ago
     Docs: https://github.com/crate/cratedb-prometheus-adapter/
  Process: 863958 ExecStart=/usr/bin/cratedb-prometheus-adapter $CRATEDB_PROMETHEUS_ADAPTER_OPTS (code=exited, status=1/FAILURE)
 Main PID: 863958 (code=exited, status=1/FAILURE)

Jan 14 12:50:10 myhostsrv1 systemd[1]: cratedb-prometheus-adapter.service: Service RestartSec=100ms expired, scheduling restart.
Jan 14 12:50:10 myhostsrv1 systemd[1]: cratedb-prometheus-adapter.service: Scheduled restart job, restart counter is at 5.
Jan 14 12:50:10 myhostsrv1 systemd[1]: Stopped Prometheus adapter to write to CrateDB.
Jan 14 12:50:10 myhostsrv1 systemd[1]: cratedb-prometheus-adapter.service: Start request repeated too quickly.
Jan 14 12:50:10 myhostsrv1 systemd[1]: cratedb-prometheus-adapter.service: Failed with result 'exit-code'.
Jan 14 12:50:10 myhostsrv1 systemd[1]: Failed to start Prometheus adapter to write to CrateDB.

Unfortunately it still fails to start the cratedb-protheus service.

When I check the prometheus service, it continues to run just fine as you can see from the output below of the status command:

systemctl status prometheus.service 
● prometheus.service - Prometheus
   Loaded: loaded (/etc/systemd/system/prometheus.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2022-01-14 13:04:35 EST; 30s ago
     Docs: https://prometheus.io/docs/introduction/overview/
 Main PID: 864985 (prometheus)
    Tasks: 14 (limit: 410197)
   Memory: 45.5M
   CGroup: /system.slice/prometheus.service
           └─864985 /usr/local/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus ->

Jan 14 13:04:36 myhostsrv1 prometheus[864985]: ts=2022-01-14T18:04:36.333Z caller=main.go:945 level=info fs_type=EXT4_SUPER_MAGIC
Jan 14 13:04:36 myhostsrv1 prometheus[864985]: ts=2022-01-14T18:04:36.333Z caller=main.go:948 level=info msg="TSDB started"
Jan 14 13:04:36 myhostsrv1 prometheus[864985]: ts=2022-01-14T18:04:36.333Z caller=main.go:1129 level=info msg="Loading configuratio>
Jan 14 13:04:36 myhostsrv1 prometheus[864985]: ts=2022-01-14T18:04:36.334Z caller=dedupe.go:112 component=remote level=info remote_>
Jan 14 13:04:36 myhostsrv1 prometheus[864985]: ts=2022-01-14T18:04:36.334Z caller=dedupe.go:112 component=remote level=info remote_>
Jan 14 13:04:36 myhostsrv1 prometheus[864985]: ts=2022-01-14T18:04:36.334Z caller=dedupe.go:112 component=remote level=info remote_>
Jan 14 13:04:36 myhostsrv1 prometheus[864985]: ts=2022-01-14T18:04:36.341Z caller=main.go:1166 level=info msg="Completed loading of>
Jan 14 13:04:36 myhostsrv1 prometheus[864985]: ts=2022-01-14T18:04:36.341Z caller=main.go:897 level=info msg="Server is ready to re>
Jan 14 13:04:46 myhostsrv1 prometheus[864985]: ts=2022-01-14T18:04:46.165Z caller=dedupe.go:112 component=remote level=info remote_>
Jan 14 13:04:46 myhostsrv1 prometheus[864985]: ts=2022-01-14T18:04:46.196Z caller=dedupe.go:112 component=remote level=warn remote_

Not sure what else may be missing.

Any other suggestion, we be appreciated.

Thanks

amotl · January 14, 2022, 7:01pm

Dear Brian,

thank you for writing in and also thanks for clarifying that you are looking at the systemd-based configuration, which wasn’t clear from your original post. I will try to assist you in configuring this service appropriately.

Introduction

Both configuration settings are clearly incorrect, I will try to elaborate on it.

Regarding EnvironmentFile: The YAML configuration file is specific to the cratedb-prometheus-adapter program and is not an environment variable file which can be loaded by systemd.
Regarding CRATEDB_PROMETHEUS_ADAPTER_OPTS: Plucking environment variables into the systemd unit file like this does not work.

In order to cite @jayeff here:

I admit the instructions at CrateDB Prometheus Adapter » Running as service could be outlined better, but nevertheless I believe they are correct.

Solution

For the systemd-based setup, you need four files to be correctly deployed to your machine.

/usr/bin/cratedb-prometheus-adapter. This is the program itself, extracted from the corresponding tarball distribution package at Index of /downloads/dist/prometheus/.
/etc/cratedb-prometheus-adapter/config.yml. Get it from config.yml and adjust the settings according to your needs.
/etc/systemd/system/cratedb-prometheus-adapter.service. Get it from cratedb-prometheus-adapter.service.
/etc/default/cratedb-prometheus-adapter. Get it from cratedb-prometheus-adapter.default.

I don’t think you will need to make any adjustments to the latter two files. They should work out of the box.

After deploying those files correctly, invoking the following commands should start the service appropriately, given that the settings in /etc/cratedb-prometheus-adapter/config.yml are valid.

systemctl daemon-reload
systemctl restart cratedb-prometheus-adapter

Please let us know about the outcome for you.

With kind regards,
Andreas.

P.S.: Note to self: We might look into bundling this program as Debian and RPM packages in order to make its installation easier. See also Improve installation, configuration, and first invocation · Issue #49 · crate/cratedb-prometheus-adapter · GitHub.

BrianS · January 14, 2022, 8:30pm

Hello Andreas,

Thank you so much for the exact and detail instructions.

It is active!!!

systemctl status cratedb-prometheus-adapter.service 
● cratedb-prometheus-adapter.service - Prometheus adapter to write to CrateDB
   Loaded: loaded (/etc/systemd/system/cratedb-prometheus-adapter.service; disabled; vendor preset: disabled)
   Active: active (running) since Fri 2022-01-14 15:25:27 EST; 8s ago
     Docs: https://github.com/crate/cratedb-prometheus-adapter/
 Main PID: 871719 (cratedb-prometh)
    Tasks: 8 (limit: 410197)
   Memory: 14.3M
   CGroup: /system.slice/cratedb-prometheus-adapter.service
           └─871719 /usr/bin/cratedb-prometheus-adapter -config.file=/etc/cratedb-prometheus-adapter/config.yml

Jan 14 15:25:27 myhostsrv1 systemd[1]: Started Prometheus adapter to write to CrateDB.
Jan 14 15:25:27 myhostsrv1 cratedb-prometheus-adapter[871719]: time="2022-01-14T15:25:27-05:00" level=info msg="Starting CrateDB Pr>
Jan 14 15:25:27 myhostsrv1 cratedb-prometheus-adapter[871719]: time="2022-01-14T15:25:27-05:00" level=info msg="Listening ..." addr>
Jan 14 15:25:27 myhostsrv1 cratedb-prometheus-adapter[871719]: time="2022-01-14T15:25:27-05:00" level=info msg="Connecting ..." end>

Topic		Replies	Views
Storing long-term metrics with Prometheus in CrateDB Tutorials integration , monitoring	1	2887	April 6, 2022
Still new to CrateDB. Have some basic quesions Configuration	5	942	January 18, 2022
CrateDB Prometheus Adapter croaks with err="context deadline exceeded" 3rd Party Tools	16	2636	February 12, 2024
Disk space issues on prometheus integration Integrations	6	85	September 26, 2024
Caller=server.go:349 level=error msg="Failed to write data to CrateDB" err="error closing write batch: timeout: context deadline exceeded" CrateDB prometheus	6	22	July 18, 2024

CrateDB Prometheus adapter service not starting

Introduction

Solution

Related topics