# Insert data to local node only?

**URL:** <https://community.cratedb.com/t/insert-data-to-local-node-only/1066>\
**Category:** CrateDB\
**Created:** [March 28, 2022, 12:20pm UTC](https://community.cratedb.com/t/insert-data-to-local-node-only/1066 "2022-03-28T12:20:45Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![ArnonAyal](https://sea2.discourse-cdn.com/flex020/user_avatar/community.cratedb.com/arnonayal/32/542_2.png) [@ArnonAyal](https://community.cratedb.com/u/ArnonAyal)\
**Post date:** [March 28, 2022, 12:20pm UTC](https://community.cratedb.com/t/insert-data-to-local-node-only/1066/1 "2022-03-28T12:20:45Z")

</div>

Is it possible to overwrite somehow the distribution function configuration, so the data will be stored only at the local node’s shards?  
The idea is to reduce the network’s load since I have lot of data and scalability is not so important.  
Thnaks.

---

<div class="post-metadata">

**Author:** ![proddata](https://sea2.discourse-cdn.com/flex020/user_avatar/community.cratedb.com/proddata/32/1379_2.png) [@proddata](https://community.cratedb.com/u/proddata)\
**Post date:** [March 28, 2022, 12:56pm UTC](https://community.cratedb.com/t/insert-data-to-local-node-only/1066/2 "2022-03-28T12:56:31Z")

</div>

You could use Shard Allocation filtering to only store certain tables / partitions on specified node (types)

> **[Shard allocation filtering](https://crate.io/docs/crate/reference/en/4.7/general/ddl/shard-allocation.html)**
>
> Shard allocation filters allows to configure shard and replicas allocation per table across generic attributes associated with nodes. It is possible to assign certain attributes to a node, see Custom attributes. These attributes can be used with...

---

<div class="post-metadata">

**Author:** ![ArnonAyal](https://sea2.discourse-cdn.com/flex020/user_avatar/community.cratedb.com/arnonayal/32/542_2.png) [@ArnonAyal](https://community.cratedb.com/u/ArnonAyal)\
**Post date:** [March 28, 2022, 1:38pm UTC](https://community.cratedb.com/t/insert-data-to-local-node-only/1066/3 "2022-03-28T13:38:21Z")

</div>

Thanks, that is the closest option I found,  
But my demands are a little different

1. need all tables on all nodes.
2. each table/node will hold only and all the data that was inserted into that node only.
3. I need to run queries that will look into all nodes/tables and join data from all

I know its not the original purpose of crateDB, but need something like that

Thanks.

---

<div class="post-metadata">

**Author:** ![proddata](https://sea2.discourse-cdn.com/flex020/user_avatar/community.cratedb.com/proddata/32/1379_2.png) [@proddata](https://community.cratedb.com/u/proddata)\
**Post date:** [March 31, 2022, 6:36am UTC](https://community.cratedb.com/t/insert-data-to-local-node-only/1066/4 "2022-03-31T06:36:37Z")

</div>

> 1. need all tables on all nodes.  
> They still would be accessible

One option would be to bring a node-identifier into a partition column and then allocate the partitions to the specific nodes. Then you really could have one (partitioned) table that is queryable as single table.

similar but not necessarily the same are hot/cold strategies  
also see [Building a hot and cold storage data retention policy in CrateDB with Apache Airflow](https://community.cratedb.com/t/cratedb-and-apache-airflow-building-a-hot-cold-storage-data-retention-policy/934)

---

<div class="post-metadata">

**Author:** ![jayeff](https://sea2.discourse-cdn.com/flex020/user_avatar/community.cratedb.com/jayeff/32/271_2.png) [@jayeff](https://community.cratedb.com/u/jayeff)\
**Post date:** [March 31, 2022, 6:46am UTC](https://community.cratedb.com/t/insert-data-to-local-node-only/1066/5 "2022-03-31T06:46:35Z")

</div>

@ArnonAyal Can you give us additional insights into your use case? Are you running into limits due to too much network load or is there another reason why you want to rreduce the network load?

---

<div class="post-metadata">

**Author:** ![ArnonAyal](https://sea2.discourse-cdn.com/flex020/user_avatar/community.cratedb.com/arnonayal/32/542_2.png) [@ArnonAyal](https://community.cratedb.com/u/ArnonAyal)\
**Post date:** [March 31, 2022, 7:00am UTC](https://community.cratedb.com/t/insert-data-to-local-node-only/1066/6 "2022-03-31T07:00:00Z")

</div>

@jayeff, yes, that is the exact reason.  
we are dealing with a huge amount of data so in case of replication or data transfer between nodes, we’ll have a problematic network issue, the nodes are distributed all over the world.

---

<div class="post-metadata">

**Author:** ![jayeff](https://sea2.discourse-cdn.com/flex020/user_avatar/community.cratedb.com/jayeff/32/271_2.png) [@jayeff](https://community.cratedb.com/u/jayeff)\
**Post date:** [March 31, 2022, 12:37pm UTC](https://community.cratedb.com/t/insert-data-to-local-node-only/1066/7 "2022-03-31T12:37:50Z")

</div>

Hi @ArnonAyal,

> Is it possible to overwrite somehow the distribution function configuration, so the data will be stored only at the local node’s shards?

sorry but there isn’t an option to configure CrateDB to work in this way.

You potentially have the option to emulate such a behaviour with some additional cluster management.

As @proddata already mentioned you could use [shard allocation filtering](https://crate.io/docs/crate/reference/en/4.7/general/ddl/shard-allocation.html) on a partioned table and assign specific partitions to specific nodes. Alternatively you look into using manual shard allocation to specific nodes with [`ALTER TABLE REROUTE`](https://crate.io/docs/crate/reference/en/4.7/sql/statements/alter-table.html#reroute) (and automatic shard rebalancing disabled).

Additionally you would need to adapt your insert logic so that records are written directly to the CrateDB node where these records should be stored (based on either [partition column](https://crate.io/docs/crate/reference/en/4.7/appendices/glossary.html#gloss-partition-column) value or [routing column](https://crate.io/docs/crate/reference/en/4.7/appendices/glossary.html#gloss-routing-column) value).

Such an apprach is defintively not the easiest solution and also does run a bit counter to what CrateDB is designed to do but maybe something like this or similar to this could help you with decreasing your network load.

Please feel free to reach out if you have additional questions.  
Johannes
