# CrateDB Prometheus Adapter croaks with err="context deadline exceeded" **URL:** \ **Category:** 3rd Party Tools\ **Created:** [April 17, 2020, 8:50am UTC](https://community.cratedb.com/t/cratedb-prometheus-adapter-croaks-with-err-context-deadline-exceeded/376 "2020-04-17T08:50:13Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![RyanWN4](https://avatars.discourse-cdn.com/v4/letter/r/7ab992/32.png) [@RyanWN4](https://community.cratedb.com/u/RyanWN4)\ **Post date:** [April 17, 2020, 8:50am UTC](https://community.cratedb.com/t/cratedb-prometheus-adapter-croaks-with-err-context-deadline-exceeded/376/1 "2020-04-17T08:50:13Z")
Hey guys, So I’m currently running a 3 node CrateDB cluster for Prometheus long term storage and using the CrateDB adapter provided by crateio. I’ve ran into an issue where Prometheus is attempting to POST data to the remote\_write (crate adapter) but it’s unable to. Logs are below from CrateDB adapter ``` time="2020-04-17T08:46:31Z" level=error msg="Failed to POST inserts to Crate." err="context deadline exceeded" source="server.go:332" time="2020-04-17T08:46:31Z" level=error msg="Failed to POST inserts to Crate." err="context deadline exceeded" source="server.go:332" time="2020-04-17T08:46:31Z" level=error msg="Failed to POST inserts to Crate." err="context deadline exceeded" source="server.go:332" time="2020-04-17T08:46:31Z" level=error msg="Failed to POST inserts to Crate." err="context deadline exceeded" source="server.go:332" time="2020-04-17T08:46:31Z" level=error msg="Failed to POST inserts to Crate." err="context deadline exceeded" source="server.go:332" time="2020-04-17T08:46:31Z" level=error msg="Failed to POST inserts to Crate." err="context deadline exceeded" source="server.go:332" ``` Any help would be great 🙂 i’ve tried increasing the scrape\_interval on prometheus and it doesn’t seem to help. I was thinking maybe CrateDB is limiting the number of HTTP connections it can receive but can’t really find any documentation to support that ---
**Author:** ![amotl](https://sea2.discourse-cdn.com/flex020/user_avatar/community.cratedb.com/amotl/32/617_2.png) [@amotl](https://community.cratedb.com/u/amotl)\ **Post date:** [May 4, 2021, 5:35pm UTC](https://community.cratedb.com/t/cratedb-prometheus-adapter-croaks-with-err-context-deadline-exceeded/376/2 "2021-05-04T17:35:05Z")
Dear Ryan, apologies for the very late reply. I answered you recently at [Failed to POST/GET data from CrateDB: Croaks with err="context deadline exceeded" · Issue #33 · crate/cratedb-prometheus-adapter · GitHub](https://github.com/crate/cratedb-prometheus-adapter/issues/33#issuecomment-829445005) : > [@](#): > > `context deadline exceeded` is a very generic error raised from Go which usually indicates that the connection timed out, or that some other networking issue is present, like one communication partner is trying to negotiate a TLS connection while the other one isn’t prepared for that. > > Can you share some more details about your version of CrateDB and the load situation? Attaching to that, I believe you might have been tripped by things like [Networking robustness and resiliency on Azure and beyond (AWS, GCP, AliCloud) · Issue #10779 · crate/crate · GitHub](https://github.com/crate/crate/issues/10779). In this context, may I ask whether you are running CrateDB and Prometheus within a typical cloud environment or, otherwise, how specifically the `cratedb-prometheus-adapter` is connected to CrateDB, network-wise? With kind regards, Andreas. ---
**Author:** ![amotl](https://sea2.discourse-cdn.com/flex020/user_avatar/community.cratedb.com/amotl/32/617_2.png) [@amotl](https://community.cratedb.com/u/amotl)\ **Post date:** [May 4, 2021, 5:37pm UTC](https://community.cratedb.com/t/cratedb-prometheus-adapter-croaks-with-err-context-deadline-exceeded/376/3 "2021-05-04T17:37:45Z")
Dear Ryan, we just added a patch to improve the network behaviour slightly by adjusting the TCP timeout and keepalive settings, see [Network behaviour: Adjust TCP timeout and keepalive settings by amotl · Pull Request #44 · crate/cratedb-prometheus-adapter · GitHub](https://github.com/crate/cratedb-prometheus-adapter/pull/44). Now, those default values are used: - TCP keepalive interval: 30 seconds - TCP connect timeout: 10 seconds The new `-tcp.connect.timeout` command line option can be used to adjust the latter parameter. With kind regards, Andreas. P.S.: We just released _CrateDB Prometheus Adapter 0.4.0_, which is available in form of release archives [1] and a Docker image [2]. [1] [Index of /downloads/dist/prometheus/](https://cdn.crate.io/downloads/dist/prometheus/) [2] [https://ghcr.io/crate/cratedb-prometheus-adapter](https://ghcr.io/crate/cratedb-prometheus-adapter) ---
**Author:** ![Florencia\_Artegoytia](https://sea2.discourse-cdn.com/flex020/user_avatar/community.cratedb.com/florencia_artegoytia/32/552_2.png) [@Florencia\_Artegoytia](https://community.cratedb.com/u/Florencia_Artegoytia)\ **Post date:** [April 5, 2022, 10:07pm UTC](https://community.cratedb.com/t/cratedb-prometheus-adapter-croaks-with-err-context-deadline-exceeded/376/4 "2022-04-05T22:07:43Z")
Hello, I followed the steps in the tutorial at [CrateDB and Prometheus for long-term metrics storage](https://community.cratedb.com/t/cratedb-and-prometheus-for-long-term-metrics-storage/1012) but in the logs for prometheus I’m also receiving this: ![WhatsApp Image 2022-04-05 at 7.05.45 PM](https://us1.discourse-cdn.com/flex020/uploads/crate/original/1X/5d8d4b14aeae08e7ff89091a00a531b3dcea6a79.jpeg) any help? ---
**Author:** ![jayeff](https://sea2.discourse-cdn.com/flex020/user_avatar/community.cratedb.com/jayeff/32/271_2.png) [@jayeff](https://community.cratedb.com/u/jayeff)\ **Post date:** [April 6, 2022, 8:47am UTC](https://community.cratedb.com/t/cratedb-prometheus-adapter-croaks-with-err-context-deadline-exceeded/376/5 "2022-04-06T08:47:13Z")
Hi @Florencia_Artegoytia, can you confirm that all docker containers are running? Is the config file for prometheus-adapter correctly applied? What do the logs output for `cratedb-prometheus-adapter` container? ---
**Author:** ![Florencia\_Artegoytia](https://sea2.discourse-cdn.com/flex020/user_avatar/community.cratedb.com/florencia_artegoytia/32/552_2.png) [@Florencia\_Artegoytia](https://community.cratedb.com/u/Florencia_Artegoytia)\ **Post date:** [April 6, 2022, 12:44pm UTC](https://community.cratedb.com/t/cratedb-prometheus-adapter-croaks-with-err-context-deadline-exceeded/376/6 "2022-04-06T12:44:33Z")
Hi. Thank you for the quick response, Yes, all docker containers are running. I looked into inside the container the file prometheus.yml and I confirmed it was correctly applied. Here are the logs output for cratedb-prometheus-adapter ![image](https://us1.discourse-cdn.com/flex020/uploads/crate/original/1X/f182f3d6ae4fcea44b013c337afa5844baaa54a8.png) ---
**Author:** ![jayeff](https://sea2.discourse-cdn.com/flex020/user_avatar/community.cratedb.com/jayeff/32/271_2.png) [@jayeff](https://community.cratedb.com/u/jayeff)\ **Post date:** [April 6, 2022, 2:37pm UTC](https://community.cratedb.com/t/cratedb-prometheus-adapter-croaks-with-err-context-deadline-exceeded/376/7 "2022-04-06T14:37:56Z")
Hi @Florencia_Artegoytia, `cratedb-prometheus-adapter` looks good. I just followed the tutorial from scratch and it works perfectely for me locally so I’m a bit puzzeled what doesn’t work on your end. Can you check if you use latest version of Prometheus (v2.34.0) and CrateDB (v4.7.1)? Do you start all containers as superuser? Maybe this interferes with the container being able to connect to each other. How much data do you have in Prometheus? Error message `context deadline exceeded` indicates some kind of connection / timeout issue. Maybe the containers can’t see each other. Maybe you transmit too much data and connection times out. ---
**Author:** ![amotl](https://sea2.discourse-cdn.com/flex020/user_avatar/community.cratedb.com/amotl/32/617_2.png) [@amotl](https://community.cratedb.com/u/amotl)\ **Post date:** [April 6, 2022, 3:46pm UTC](https://community.cratedb.com/t/cratedb-prometheus-adapter-croaks-with-err-context-deadline-exceeded/376/8 "2022-04-06T15:46:09Z")
Dear Florencia, thank you for writing in. I deliberately moved your question to this discussion to keep things in line. After Ryan originally reported this issue, we took some actions to adjust the _TCP keepalive interval_ and the _TCP connect timeout_ settings, also making the latter configurable. We outlined the improvements at [Failed to POST/GET data from CrateDB: Croaks with err="context deadline exceeded" · Issue #33 · crate/cratedb-prometheus-adapter · GitHub](https://github.com/crate/cratedb-prometheus-adapter/issues/33), where Ryan also posted the `context deadline exceeded` error he was observing. However, Ryan never reported back if those improvements have been helpful in any way. May I humbly ask you about this, @RyanWN4? @Florencia_Artegoytia: Maybe you can increase the `connect_timeout` setting in your adapter’s `config.yml` as outlined within \[1\], bounce your containers and report back about any improvements you might be able to observe? With kind regards, Andreas. * * * 1. [CrateDB and Prometheus for long-term metrics storage](https://community.cratedb.com/t/cratedb-and-prometheus-for-long-term-metrics-storage/1012#creating-configyml-5) ---
**Author:** ![Florencia\_Artegoytia](https://sea2.discourse-cdn.com/flex020/user_avatar/community.cratedb.com/florencia_artegoytia/32/552_2.png) [@Florencia\_Artegoytia](https://community.cratedb.com/u/Florencia_Artegoytia)\ **Post date:** [April 8, 2022, 6:00pm UTC](https://community.cratedb.com/t/cratedb-prometheus-adapter-croaks-with-err-context-deadline-exceeded/376/9 "2022-04-08T18:00:19Z")
Hi again, > Can you check if you use latest version of Prometheus (v2.34.0) and CrateDB (v4.7.1)? Yes I’m using the latest version. > Do you start all containers as superuser? Yes. I think the problem is, the containers don’t see each other. I tried to ping one of the networks and I get no response. My docker-compose file is the same as the tutorial. ![image](https://us1.discourse-cdn.com/flex020/uploads/crate/original/1X/9639953d9829ff8affcf38f02737abfb2cf93da0.png) ---
**Author:** ![amotl](https://sea2.discourse-cdn.com/flex020/user_avatar/community.cratedb.com/amotl/32/617_2.png) [@amotl](https://community.cratedb.com/u/amotl)\ **Post date:** [April 9, 2022, 9:22pm UTC](https://community.cratedb.com/t/cratedb-prometheus-adapter-croaks-with-err-context-deadline-exceeded/376/10 "2022-04-09T21:22:35Z")
Dear Florencia, > [@Florencia\_Artegoytia](#): > > I think the problem is, the containers don’t see each other. From looking at the console output you’ve shared, I agree. Let’s investigate why that would not work on your machine. In order to assist the procedure, I’ve created a companion repository to the tutorial at [GitHub - crate-workbench/cratedb-prometheus-demo: Companion repository to the »CrateDB and Prometheus for long-term metrics storage« tutorial](https://github.com/crate-workbench/cratedb-prometheus-demo) and, just to be safe, verified it works on both Linux and macOS. The software versions used are: - Debian Linux (Ubuntu 20.04.4 LTS) - macOS (Catalina 10.15.7) - Docker 20.10.14 - Docker Compose v2.3.3 and v2.4.1 While it shouldn’t make much of a difference, can I humbly ask you to try again using the setup shared within the repository and also share the corresponding software versions of your environment with us? With kind regards, Andreas. ---
**Author:** ![CArsten](https://avatars.discourse-cdn.com/v4/letter/c/f14d63/32.png) [@CArsten](https://community.cratedb.com/u/CArsten)\ **Post date:** [August 28, 2023, 8:30am UTC](https://community.cratedb.com/t/cratedb-prometheus-adapter-croaks-with-err-context-deadline-exceeded/376/11 "2023-08-28T08:30:06Z")
Hi, we are currently testing CrateDB as a long term storage backend to Prometheus. At the moment, we are remote\_writing a small subset of metrics via cratedb-prometheus-adapter-0.4.0 into a stand-alone test instance of crate 5.3.4. After a few weeks, the DB holds about 835M records (130GiB) and so far it does not look too bad. However, if we try to run a long query via Prometheus’ data explorer, we end up with a time-out after more or less exactly one minute `remote_read: remote server http://adapter:9268/read returned HTTP status 500 Internal Server Error: context deadline exceeded` Prometheus timeout is set to ```auto remote_read: - url: http://adapter:9268/read remote_timeout: 5m follow_redirects: true enable_http2: true filter_external_labels: true ``` and the adapter’s is `connect_timeout: 300` At the moment, I’m unsure how to proceed/where to look for this specific time-out. Thanks for any pointer Carsten PS: If it matters: CrateDB is a bare metal installation with upstream’s Debian packages `5.3.4-1~bookworm` and the adapter was just downloaded via github and started with a systemd service file. ---
**Author:** ![amotl](https://sea2.discourse-cdn.com/flex020/user_avatar/community.cratedb.com/amotl/32/617_2.png) [@amotl](https://community.cratedb.com/u/amotl)\ **Post date:** [August 28, 2023, 8:59am UTC](https://community.cratedb.com/t/cratedb-prometheus-adapter-croaks-with-err-context-deadline-exceeded/376/12 "2023-08-28T08:59:11Z")
Dear Carsten, thank you for writing in, and your excellent report. After evaluating your observations, and revisiting this topic, I think what is missing in CrateDB Prometheus Adapter, is to be able to properly configure the TCP read timeout. Currently, only the TCP connect timeout is configurable. > It looks like unlocking `SetReadDeadline` and `SetWriteDeadline` on the `net.Conn` object would be right approach for this. Do you agree? With kind regards, Andreas. ---
**Author:** ![CArsten](https://avatars.discourse-cdn.com/v4/letter/c/f14d63/32.png) [@CArsten](https://community.cratedb.com/u/CArsten)\ **Post date:** [August 28, 2023, 9:17am UTC](https://community.cratedb.com/t/cratedb-prometheus-adapter-croaks-with-err-context-deadline-exceeded/376/13 "2023-08-28T09:17:19Z")
Hi Andreas, from what little I understand, that seems to be the correct way, but given that I’ve never written any go code and and programming beyond simple scripting is not my core strength, I would defer that decision to you and/or other experts. That being said, I should be able to quickly test any changes if you could provide a test binary (or I will start looking into how to build go binaries from the git repo). Anyways, thanks a lot for looking into this! Carsten ---
**Author:** ![amotl](https://sea2.discourse-cdn.com/flex020/user_avatar/community.cratedb.com/amotl/32/617_2.png) [@amotl](https://community.cratedb.com/u/amotl)\ **Post date:** [August 28, 2023, 9:18am UTC](https://community.cratedb.com/t/cratedb-prometheus-adapter-croaks-with-err-context-deadline-exceeded/376/14 "2023-08-28T09:18:52Z")
Thanks for your offer to be a canary in this regard ;]. I will get back to you as soon as there will be something to test. I can’t promise anything for this week, but I will try to get into it next week latest. ---
**Author:** ![CArsten](https://avatars.discourse-cdn.com/v4/letter/c/f14d63/32.png) [@CArsten](https://community.cratedb.com/u/CArsten)\ **Post date:** [August 28, 2023, 9:39am UTC](https://community.cratedb.com/t/cratedb-prometheus-adapter-croaks-with-err-context-deadline-exceeded/376/15 "2023-08-28T09:39:17Z")
Well, becoming the canary seems to be the only help I can offer - I’ve quickly browsed [net package - net - Go Packages](https://pkg.go.dev/net#Conn%E2%80%8C), [https://pkg.go.dev/net/http‌](https://pkg.go.dev/net/http%E2%80%8C) and [How to set timeout for http.Get() requests in Golang? - Stack Overflow](https://stackoverflow.com/questions/16895294/how-to-set-timeout-for-http-get-requests-in-golang) but I’m not any wiser now how to add this myself to [CrateDB Prometheus Adapter](https://github.com/crate/cratedb-prometheus-adapter/blob/9c4ed56a8477a52749b825d271d9acc43fc98d37/server.go#L450%E2%80%8C). Thus, I’ll wait patiently 😉 ---
**Author:** ![amotl](https://sea2.discourse-cdn.com/flex020/user_avatar/community.cratedb.com/amotl/32/617_2.png) [@amotl](https://community.cratedb.com/u/amotl)\ **Post date:** [September 1, 2023, 1:30pm UTC](https://community.cratedb.com/t/cratedb-prometheus-adapter-croaks-with-err-context-deadline-exceeded/376/16 "2023-09-01T13:30:35Z")
Dear Carsten, there is no release or pre-release yet, but at least we have been able to [modernize the code base a bit](https://github.com/crate/cratedb-prometheus-adapter/pulls). Other than this, there are two patches specifically addressing performance topics which are probably relevant for you and @RyanWN4. They originate from both your reports, and other investigations on behalf of reports from others, documented at [Investigate how to scale out this service · Issue #72 · crate/cratedb-prometheus-adapter · GitHub](https://github.com/crate/cratedb-prometheus-adapter/issues/72). Thanks, @hernanc. > [@amotl](#): > > Where people may run into problems: > > > [@hernanc](#): > > > > If [we|they] have a lot of things going on in Prometheus, the connections to CrateDB are not freed up quickly enough, so the requests are queued and eventually run into timeout situations. [We|they] assume the bottleneck is this service and not CrateDB behind. > > On this detail, we need to investigate if `cratedb-prometheus-adapter` is just a toy in this regard, or if it would be suitable for working well in high-performance environments. I think both patches are crucial to improve the production-readyness of the application. - [Add query timeouts using context cancellation by amotl · Pull Request #80 · crate/cratedb-prometheus-adapter · GitHub](https://github.com/crate/cratedb-prometheus-adapter/pull/80) > [@Soham Kamani](#): > > In production applications, it is _always_ preferred to have timeouts for all queries: A sudden increase in throughput or a network issue can lead to queries slowing down by orders of magnitude. > > Slow queries block the connections that they are running on, preventing other queries from running on them. We should always set a timeout after which to cancel a running query, to unblock connections in these cases. > > – [Using an SQL Database in Go (With Production Readiness)](https://www.sohamkamani.com/golang/sql-database/#query-timeouts---using-context-cancellation) - [Use a different connection pool for read vs. write operations by amotl · Pull Request #81 · crate/cratedb-prometheus-adapter · GitHub](https://github.com/crate/cratedb-prometheus-adapter/pull/81) > [@](#): > > In this way, excessive/overuse of one communication channel will not impact the other, and different usage characteristics can be tuned individually. We hope to be able to ship a release with those major improvements next week, maybe adding a few more related details. Have a good weekend, and with kind regards, Andreas. ---
**Author:** ![amotl](https://sea2.discourse-cdn.com/flex020/user_avatar/community.cratedb.com/amotl/32/617_2.png) [@amotl](https://community.cratedb.com/u/amotl)\ **Post date:** [February 12, 2024, 6:18pm UTC](https://community.cratedb.com/t/cratedb-prometheus-adapter-croaks-with-err-context-deadline-exceeded/376/17 "2024-02-12T18:18:32Z")
Dear Carsten, we addressed a few of the most prominent performance issues with the most recent release, CrateDB Prometheus Adapter 0.5.0, see also the [release notes](https://github.com/crate/cratedb-prometheus-adapter/releases). If you still want to give it a try, we will be happy to hear about your feedback, and if the improvements will resolve the issues you have been running into. Thank you very much! With kind regards, Andreas.