# CrateDB nodes constantly crashing

**URL:** https://community.cratedb.com/t/cratedb-nodes-constantly-crashing/490
**Category:** CrateDB
**Created:** [September 16, 2020, 9:21pm UTC](https://community.cratedb.com/t/cratedb-nodes-constantly-crashing/490 "2020-09-16T21:21:34Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![Daniel\_Ramos](https://sea2.discourse-cdn.com/flex020/user_avatar/community.cratedb.com/daniel_ramos/32/196_2.png) [@Daniel\_Ramos](https://community.cratedb.com/u/Daniel_Ramos)
#### Post date: [September 16, 2020, 9:21pm UTC](https://community.cratedb.com/t/cratedb-nodes-constantly-crashing/490/1 "2020-09-16T21:21:35Z")

</div>

Hi everyone,

We have a 3 nodes (8 cores - 64 GB RAM each node) cluster in AWS Cloud. Our use case is an IOT platform and our data ingestion rate is about 350 insertions/sec. With a data retention window of about 6 months.

 ![image](https://us1.discourse-cdn.com/flex020/uploads/crate/original/1X/9141718c1b48531121396454e089ba2b6e242b6e.jpeg)

CRATE\_HEAP\_SIZE = 30.5 GB

We are currently getting nodes disconnections every 30 minutes or less. The node completely crashes, restart and a new master node is elected. This happens all day.

Any advice?

Thanks in advance

---

<div class="post-metadata">

### Author: ![proddata](https://sea2.discourse-cdn.com/flex020/user_avatar/community.cratedb.com/proddata/32/1379_2.png) [@proddata](https://community.cratedb.com/u/proddata)
#### Post date: [September 17, 2020, 2:36pm UTC](https://community.cratedb.com/t/cratedb-nodes-constantly-crashing/490/2 "2020-09-17T14:36:45Z")

</div>

Could you share some further information with us?

- Which version of CrateDB are you using?
- Which OS are you running CrateDB on?

Further you might provide:

- DB Schemas
- Config File
- A heap dump
- The crate log of the crashing node

… and …

- Monitoring snapshots (e.g. grafana) of exposed JMX metrix
  - crate\_threadpools queueSize/active/rejected,
  - GC infos:
    - Young + Old Generation avg (jvm\_gc\_collection\_seconds\_sum/jvm\_gc\_collection\_seconds\_count)
    - Survivor space (jvm\_memory\_pool\_bytes\_used)
    - GC rates (jvm\_gc\_collection\_seconds\_count)

  - DirectBuffer memory usage (jvm\_buffer\_pool\_used\_bytes)
  - Queries per second (crate\_query\_total\_count)
  - Query error rate (crate\_query\_failed\_count)
  - CircuitBreaker memory usages (crate\_circuitbreakers)

---

<div class="post-metadata">

### Author: ![Walter\_Behmann](https://sea2.discourse-cdn.com/flex020/user_avatar/community.cratedb.com/walter_behmann/32/117_2.png) [@Walter\_Behmann](https://community.cratedb.com/u/Walter_Behmann)
#### Post date: [September 30, 2020, 5:51am UTC](https://community.cratedb.com/t/cratedb-nodes-constantly-crashing/490/3 "2020-09-30T05:51:43Z")

</div>

Hey Daniel, any update on this one?

I will be glad to look into this. To be sure to be as close to your setup as possible, I appreciate the following information:

- cratedb version
- `crate.yml` / configuration flags
- `show create table` of the ingest column
- how many _data_ do you have in the table/partitions
- os version/image
- EC2 Type you use
- Loadbalancer type
- filesystem/-type/size where the data is stored

Regards,  
Walter