Slow inserts until restart crate

CrateDB
1

Hello Community!

My name is Diego and I’m working as a system engineer in a GPS company from Chile.

We are using Cratedb at work with 2.8 billion Total records with a continuous insert of 400 rows per sec 24/7.

This is a big IOT-database and the engine is working good as expected, but we have a little problem.

When some query reaches the circuit breaker(or some other thing that I can’t view), the inserts goes down to 50 per second until I restart the nodes with systemctl restart crate. After I restart the engine the inserts goes to normal speed: 400/sec.

What can cause and how to fix the slow insert speed behavior without restart the cluster?

Specs:

8 CPU X 32RAM
HEAP: 16GB
Cloud: AWS
Cluster: 3 nodes
Cratedb: 4.1.1
Shards: 10, partitioned by month.
Total rows: 2.8 Billion

Thanks for advance.

2

Hola Diego, que tal todo!?
Sounds like you have setup a very interesting use case!

Sorry to hear that you run into this unexpected behaviour. Could you please share some logs of the circiut breaker that you mentioned (form all of the 3 nodes), and if possible the kind of queries and crate.yml you run in that context. Let’s see where we can go on from there.

In general it sounds like something else is going on, as a circuit breaker is nothing IMHO that would create such a behaviour.

regards,
Walter

1 Like