Access Logs as streams

Integrations 3rd Party Tools
1

Hi,

We have an application that runs on dozens of servers and each instance generate huge logs. We would like to access logs as streams and parse it and store in crateDB. I want to know the following…

  1. How to access logs as streams from log files?.Please suggest any tool
  2. How to store it in CrateDB…. (bulk insert is the way to go?, Please suggest I’m using python client)
    Thanks & appreciate
2

Regarding 1)
I think you can use any external stream parsing tool once you got the data from the files - Maybe Kafka or Flink?

Regarding 2)
Did you check the documentation for python usage: https://crate.io/docs/clients/python/en/latest/query.html#inserting-data