Is there a difference in ingestion speed depending on the end point (Http or Postgres) we use, Is there a benefit in terms of data ingestion over the selection of the endpoint, i.e. to use Http or Postgres end point to communicate with CrateDB.
I have gone through Insert method documentation, my question is more on the choice of the endpoint.
hi Vivek, from the various tests I have run in the past and I explicitly had a look at the endpoint, it is a very small difference, if at all. Assuming you already have batch inserts on the picture, you are good to go.
The implementation on the client side (json en-/decoding, libraries used, etc. ) can have hugh impact. We also published some blueprints recently about ingesting via an Azure Function. Which reaches easily about 100.000 events/sec. on a 5-7 cluster. Side node: insert hits the CPUs, make sure to have a few “arround” … but not too fancy neither. To reach 100k insert/s we typically use 8-12 cpus / node.
It makes sense to use what is the easiest to implement in your case.