Apache Spark + Python + Pyspark + Kaola

[Suat Toksöz <st...@gmail.com>]

Hi everyone, I want to ask for guidance for my log analyzer platform idea.
I have an elasticsearch system which collects the logs from different
platforms, and creates alerts. The system writes the alerts to an index on
ES. Also, my alerts are stored in a folder as JSON (multi line format).

The Goals:

   1. Read json folder or ES index as streaming (read in new entry within 5
   min)
   2. Select only alerts that I want to work on ( alert.id = 100 ,
   status=true , ...)
   3. Create a DataFrame + Window for 10 min period
   4. Run a query fro that DataFrame by grupping by IP ( If same IP gets 3
   alerts then show me the result)
   5. All the coding should be in python


The ideas is something like this, my question is how should I proceed to
this task. What are the technologies that I should use?

*Apache Spark + Python + Pyspark + Kaola *can handle this ?

Best regards,

*Suat Toksoz*