After upgrading from version 5.10.1 to 5.10.2, some statements could not be completed

CrateDB SQL
1

After the upgrade, there are seventy-one statements related to inserting into the Prometheus table metrics that are still in an active status. I’m not sure if this issue is related to the upgrade, but the start timestamps of these statements coincide with the upgrade time. I didn’t see any logs related to these statements in the logs from the day of the upgrade. Below are our upgrade steps.

Step 1. SET GLOBAL TRANSIENT "cluster.routing.allocation.enable" = 'new_primaries';
Step 2. systemctl stop crate
Step 3. apt-get install crate=5.10.2
Step 4. systemctl start crate
Step 5. After upgrading all nodes.
SET GLOBAL TRANSIENT "cluster.routing.allocation.enable" = 'all';

Below are cratedb-prometheus-adapter log.

/var/log/syslog.1:2025-03-07T09:43:46.387164+08:00 t-infra-tsdb-001 cratedb-prometheus-adapter[1362]: ts=2025-03-07T01:43:46.386Z caller=server.go:349 level=error msg="Failed to write data to CrateDB" err="error closing write batch: timeout: context deadline exceeded"
......
Mar 07 09:47:50 t-infra-tsdb-001 cratedb-prometheus-adapter[1362]: ts=2025-03-07T01:47:50.221Z caller=server.go:349 level=error msg="Failed to write data to CrateDB" err="error closing write batch: timeout: context deadline exceeded"

crate-2025-03-07.log (94.5 KB)
I have also uploaded the upgrade log on March 7th.

+------------------------------------------------------------+--------------------------------------+--------------------------------------------------------------+---------------+------------------------------------------------------------------------------------------------------------------------------------+------------+
| date_format('%Y-%m-%d %H:%i:%S', 'Asia/Shanghai', started) | id                                   | node                                                         |       started | stmt                                                                                                                               | username   |
+------------------------------------------------------------+--------------------------------------+--------------------------------------------------------------+---------------+------------------------------------------------------------------------------------------------------------------------------------+------------+
| 2025-03-07 09:46:15                                        | 90d0678b-3eb8-51b0-c219-024aca9b859a | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311975210 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:46:40                                        | 634bca2d-2e29-744f-75cd-ed641c85be2f | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741312000332 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:43:27                                        | 78dfda97-b8aa-6041-4fcd-d9e2083ac3b3 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311807260 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:45:08                                        | 25ee1b6b-aad9-67e2-2ea0-4c175aecfce0 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311908462 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:46:36                                        | 866e8402-0b79-3929-14d5-9ce7401aaa1e | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311996919 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:45:09                                        | 5f754d0d-3cb2-9ac8-acad-5097c12f8f18 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311909596 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:44:07                                        | 9ba07a05-f50c-4a4d-dc57-333a7af9ac60 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311847422 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:44:29                                        | 93d4d8e4-fa0d-1deb-438d-8f1026fe9da3 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311869340 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:46:13                                        | c6c001ad-323d-d73b-d35c-ea4fe22bf64b | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311973072 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-10 16:49:44                                        | 2202c54a-2fc5-e207-68f9-0858eed28ee3 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741596584248 | select date_format('%Y-%m-%d %H:%i:%S','Asia/Shanghai',started),* from sys.jobs;                                                   | crate      |
| 2025-03-07 09:43:47                                        | 2335bc09-df59-f024-1539-4fffb09e8b04 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311827291 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:44:43                                        | ac566189-fc38-d948-1c51-dd749b46bbd1 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311883837 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:44:27                                        | e3d6a7c4-dd6e-bdd0-03c1-72d840dc3a44 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311867546 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:43:27                                        | 61cd8a82-798d-812d-c641-801496165e83 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311807193 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:45:08                                        | 3de6dca8-2669-fecf-5a36-b6a47dac0ef9 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311908504 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:44:03                                        | 5e44b212-6740-a5cb-5209-1003d91d6dfc | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311843572 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:44:27                                        | 96b65a88-5424-8201-675f-cad4b34f2ae8 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311867571 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:44:23                                        | c6d135a8-163e-4d58-fc7c-5a108ee7fdac | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311863585 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:46:15                                        | 31dde8fc-e893-0892-3ca8-487609870f43 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311975266 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:43:27                                        | 5cab7ee7-5432-8e34-8057-837ea5851da0 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311807242 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:45:24                                        | 609e3afb-d224-00a4-48ae-7c847bd3f45a | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311924668 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:45:07                                        | 84f274c1-fb63-edfc-ac7e-986e4ab24342 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311907780 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:44:07                                        | 2a06aa63-a755-8293-8061-9a6c4595f174 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311847423 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:43:47                                        | de5d6bbb-0dc6-9399-e12e-e6720bf72c44 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311827293 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:43:46                                        | 14094b41-f83d-c0ff-bda2-07fe179ea813 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311826629 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:44:47                                        | e39bae81-fd9c-7711-ccca-b59ab75487f3 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311887877 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:44:27                                        | a632fbe7-f9c2-10bb-9501-fd8614c13f8b | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311867581 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:45:30                                        | 3861d529-5445-50c1-b806-bd77d098b947 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311930180 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:43:26                                        | 26c1989b-3a9b-9e64-688b-bc9cf7826653 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311806585 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:45:29                                        | 65f3b408-7adb-b717-d5fd-a396a98d89b6 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311929429 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:44:06                                        | 5718dd0b-d59c-52e5-26b8-0fbfa1d917d9 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311846741 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:45:29                                        | 0e232b1a-35cd-abc2-93e0-fcd4984a9e26 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311929417 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:45:50                                        | 658809a6-b8be-f31c-39ca-78e4f5c32811 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311950682 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:45:29                                        | b574c14d-c415-8eaf-7239-7abeae8d75fd | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311929538 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:44:47                                        | 0c7d9f41-4c12-ef48-4835-1dd68f2e16d4 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311887873 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:43:49                                        | 9da70eb2-39e4-3329-1fb3-7d0ccc276dc0 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311829240 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:46:39                                        | 99c4c264-3d5a-292a-2ecd-3624d0baf10d | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311999614 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:45:51                                        | e41f80de-d8b6-eba4-2834-dcfaa6e0fe6b | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311951380 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:45:04                                        | a88281fb-e913-b4d4-8021-8c538c4e151c | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311904139 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:45:28                                        | 0a4b4b48-0eea-7d27-89b7-061aaae88a2b | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311928756 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:43:47                                        | e97771ab-ad6c-01de-0951-53c969448f10 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311827328 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:45:45                                        | 33565631-461e-355c-27e9-fd083a14e675 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311945568 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:43:47                                        | e98dc8ad-d099-cb3c-7784-6be6adf02bbd | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311827315 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:46:15                                        | b67ef722-f62b-9729-b981-1cce0e423df9 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311975226 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:44:07                                        | 0203dcc7-2228-4aa3-ad63-d78e53ed27fa | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311847381 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:44:07                                        | 1ea16c33-881a-43cd-031d-09475134c99e | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311847412 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:43:43                                        | 0d9119aa-a168-d1ad-5e1c-8bdb007973fe | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311823474 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:46:40                                        | c1e64357-904e-f5ea-85f1-5422f958d552 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741312000335 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:44:49                                        | df96a385-6c0d-29a8-5c03-d6b0e3e80822 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311889462 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:45:51                                        | aa681bf1-f048-1efe-5eee-adecd8bfec5d | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311951407 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:45:29                                        | a0a15c41-0a8a-350a-af69-9af2704d4c6c | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311929532 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:46:31                                        | 9b4e84cc-a9a2-43dd-d1c3-29102acca138 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311991537 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:46:40                                        | 6fb6929f-a9a3-1048-2bc2-5fa45af3f98c | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741312000334 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:44:47                                        | f52015d3-b83b-68c2-623d-a716a19f1d60 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311887868 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:46:07                                        | 840b4aa8-81d2-242b-b975-480129aac064 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311967627 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:44:09                                        | 2b0ccb41-27a1-4599-1953-6a7d4d7911c1 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311849168 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:44:47                                        | 999d4386-0e28-2aa0-77ef-31c59c5b2d0b | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311887172 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:46:14                                        | 963e3c82-2920-4000-393e-463b1fd239a0 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311974541 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:44:27                                        | 44454c69-f8c1-4d99-044c-2b7ea2623dc9 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311867572 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:44:26                                        | 37e3af3e-1efb-480b-efcf-97450b7bdc31 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311866886 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:45:51                                        | 6fd8198d-aa44-3203-9dd4-e674ea8ea7d2 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311951395 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:46:15                                        | defea75e-93b8-5588-f685-e39b528bb1f4 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311975267 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:45:08                                        | 0eb16a76-87d8-e560-bdf6-394f734e717a | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311908416 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:44:47                                        | e8e6377a-9727-ebc1-cbc5-d53ba15d322a | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311887849 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:43:27                                        | ec709678-2d15-ca1f-1d2f-7be768a93097 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311807239 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:46:40                                        | 3f6b7b7f-aff6-cc72-118e-de86d1f42898 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741312000183 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:45:51                                        | 50c4d64f-9369-403b-7f35-fd1458169a17 | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311951150 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:45:08                                        | 06d9bd9c-b427-1b64-27b9-92c3d22ab4ad | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311908470 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
| 2025-03-07 09:45:51                                        | 05797b78-06f6-ae97-b59b-c64e52d4109e | {"id": "8wGqpC2ITQmOTwp0Ldo_DQ", "name": "t-infra-tsdb-001"} | 1741311951347 | INSERT INTO metrics ("labels", "labels_hash", "timestamp", "value", "valueRaw") VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING | prometheus |
+------------------------------------------------------------+--------------------------------------+--------------------------------------------------------------+---------------+------------------------------------------------------------------------------------------------------------------------------------+------------+
[crate-2025-03-07.log|attachment](upload://3OOHZAqHfBYf6FySCxleuEX8jvs.log) (94.5 KB)
2

Hi @Jun_Zhou, thanks for reporting!

It looks related to a regression in 5.10 which caused transient settings to survive restart. It means that systemctl start crate will still use new_primaries

A fix (Don't write and read (for BWC) transient settings · crate/crate@156e4b5 · GitHub) will be available in the 5.10.3 hotfix release

3

Thanks @Baur, Should I cancel the SQL statements that are currently shown as running in the sys.jobs table, or should I wait for the next upgrade to version 5.10.3 for them to automatically recover?

4

What were the failing statements? Is it only INSERT-s?
What’s the output of select * from sys.health?

Effect of the bug is that cluster wouldn’t allow allocation of replicas after restart but it should be okay at least for reads.

If health shows yellow after


SET GLOBAL TRANSIENT "cluster.routing.allocation.enable" = 'all';

Try manually running


ALTER CLUSTER REROUTE RETRY FAILED

and check health again.

This is BTW something you can try on 5.10.2 because 5.10.3 is not released.

And stuck queries in sys.jobs might be another unrelated issue.