[jira] [Comment Edited] (METRON-170) Ability for metron users to author rules (Queries) to generate alerts without deploying code (Batch Rules Engine)

["Zafer Bilaloglu (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/METRON-170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287587#comment-15287587 ] 

Zafer Bilaloglu edited comment on METRON-170 at 5/17/16 9:19 PM:
-----------------------------------------------------------------

Were proposing we tackle the rules engine in two phases 

* Phase 1 - Batch Rule Evaluation 
* Phase 2 - Real Time Rule Evaluation

Phase 1 would be the development of a batch rules engine back-end and the rules engine front-end.  The batch rules engine would fire recurring queries against data at rest (in Elasticsearch, Solr, Hive, or Apache Spark) that will then generate alerts when a query returns data.  This will be easier to develop then the real time rules engine and offer greater flexibility for rules with large time-windows.  Subsequently, the batch rules engine provides the most and immediate value to SIC analysts.

Phase 1 - Components:

*  Rule Authoring - Front End
**  Create Rules
**  View/Edit/Delete Rules
*  Rule Authoring - Back End (Java, Spring Boot)
**  Rest interface (CRUD Rules)
** Rule Repository
**  PostgresDB for repository
*  Batch Rules Engine (Java, Spring, Quartz) - For scheduling, executing batch rules and performing actions:
**  Scheduler
**  Query Executor
**  Action Executor (Send Alerts to various systems, Execute Scripts)
** PostgresDB for scheduler repository
*  New Architecture Components:
** Apache Spark - To either persist data to be queried or push queries to existing data stores and perform joins.
*  Existing Data Stores
** Elasticsearch 
** Solr
** Hive 

----

For the syntax of rule definitions, we propose using SQL.  There are significant advantages as using SQL as our rule syntax, particularly for batch rules.

*Ease of Use:*
* Common knowledge of SQL throughout the enterprise.  Many SIC analysts will already know SQL or due to the ease of SQL, can learn it quickly
* Flexibility to integrate an open source Query Builder UI in the future so analysts can author rules using a UI in addition writing raw SQL.    ** http://querybuilder.js.org/

*Decoupled Interface from various backend systems*
* Using SQL offers good de-coupling from various backend databases and makes them interchangeable.  Rules are authored as SQL but can be run anywhere that accepts SQL statements.  Different Metron deployments are likely to have different stores for where data lives at rest (Hive, Solr, ES, Spark, etc…)
* Apache Spark can be used as a middleman between to convert SQL queries to the NoSQL queries (Solr/Elasticsearch). 
** Spark gives us the capability to do joins and correlate across multiple data sources, something lacking natively in Solr / ES
** ElasticSearch/Spark Integration: https://www.elastic.co/guide/en/elasticsearch/hadoop/current/spark.html
** Solr/Spark Integration: https://github.com/lucidworks/spark-solr
* Another option if we don't use Spark is to leverage other open source libraries, which translate SQL queries to NoSQL
** https://github.com/NLPchina/elasticsearch-sql
** https://cwiki.apache.org/confluence/display/solr/Parallel+SQL+Interface

*Coverage of all rule types:*
* Large enterprises will typically have hundreds of rules.  This makes it impossible to maintain these rules in compiled code.  A very large subset of these rules are simple, and can be expressed as simple SQL statements
* More complex rules can be expressed as more complex SQL statements
* Query results can be handed off to a python script for rules that can’t be expressed in SQL
* Ability to do joins to cross correlate across multiple data sources

----

Influences:

Influences:

Elast Alert (Alerting for Elasticsearch)
http://elastalert.readthedocs.io/en/latest/

Positives: 
        Open Source
	Easy way to author rules
Negatives: 
	Lack of joins across datasets
	No UI, makes maintaining a large set of rules cumbersome
	Deploying new rules requires SSH’ing into the server
	Lack of Failover
	Limited to ElasticSearch





was (Author: zafer.bilaloglu@capitalone.com):
Were proposing we tackle the rules engine in two phases 

* Phase 1 - Batch Rule Evaluation 
* Phase 2 - Real Time Rule Evaluation

Phase 1 would be the development of a batch rules engine back-end and the rules engine front-end.  The batch rules engine would fire recurring queries against data at rest (in Elasticsearch, Solr, Hive, or Apache Spark) that will then generate alerts when a query returns data.  This will be easier to develop then the real time rules engine and offer greater flexibility for rules with large time-windows.  Subsequently, the batch rules engine provides the most and immediate value to SIC analysts.

Phase 1 - Components:

*  Rule Authoring - Front End
**  Create Rules
**  View/Edit/Delete Rules
*  Rule Authoring - Back End (Java, Spring Boot)
**  Rest interface (CRUD Rules)
** Rule Repository
**  PostgresDB for repository
*  Batch Rules Engine (Java, Spring, Quartz) - For scheduling, executing batch rules and performing actions:
**  Scheduler
**  Query Executor
**  Action Executor (Send Alerts to various systems, Execute Scripts)
** PostgresDB for scheduler repository
*  New Architecture Components:
** Apache Spark - To either persist data to be queried or push queries to existing data stores and perform joins.
*  Existing Data Stores
** Elasticsearch 
** Solr
** Hive 

----

For the syntax of rule definitions, we propose using SQL.  There are significant advantages as using SQL as our rule syntax, particularly for batch rules.

*Ease of Use:*
* Common knowledge of SQL throughout the enterprise.  Many SIC analysts will already know SQL or due to the ease of SQL, can learn it quickly
* Flexibility to integrate an open source Query Builder UI in the future so analysts can author rules using a UI in addition writing raw SQL.    ** http://querybuilder.js.org/

*Decoupled Interface from various backend systems*
* Using SQL offers good de-coupling from various backend databases and makes them interchangeable.  Rules are authored as SQL but can be run anywhere that accepts SQL statements.  Different Metron deployments are likely to have different stores for where data lives at rest (Hive, Solr, ES, Spark, etc…)
* Apache Spark can be used as a middleman between to convert SQL queries to the NoSQL queries (Solr/Elasticsearch). 
** Spark gives us the capability to do joins and correlate across multiple data sources, something lacking natively in Solr / ES
** ElasticSearch/Spark Integration: https://www.elastic.co/guide/en/elasticsearch/hadoop/current/spark.html
** Solr/Spark Integration: https://github.com/lucidworks/spark-solr
* Another option if we don't use Spark is to leverage other open source libraries, which translate SQL queries to NoSQL
** https://github.com/NLPchina/elasticsearch-sql
** https://cwiki.apache.org/confluence/display/solr/Parallel+SQL+Interface

*Coverage of all rule types:*
* Large enterprises will typically have hundreds of rules.  This makes it impossible to maintain these rules in compiled code.  A very large subset of these rules are simple, and can be expressed as simple SQL statements
* More complex rules can be expressed as more complex SQL statements
* Query results can be handed off to a python script for rules that can’t be expressed in SQL
* Ability to do joins to cross correlate across multiple data sources

---

Influences:

Influences:

Elast Alert (Alerting for Elasticsearch)
http://elastalert.readthedocs.io/en/latest/

Positives: 
        Open Source
	Easy way to author rules
Negatives: 
	Lack of joins across datasets
	No UI, makes maintaining a large set of rules cumbersome
	Deploying new rules requires SSH’ing into the server
	Lack of Failover
	Limited to ElasticSearch




> Ability for metron users to author rules (Queries) to generate alerts without deploying code (Batch Rules Engine)
> -----------------------------------------------------------------------------------------------------------------
>
>                 Key: METRON-170
>                 URL: https://issues.apache.org/jira/browse/METRON-170
>             Project: Metron
>          Issue Type: New Feature
>            Reporter: Zafer Bilaloglu
>            Priority: Critical
>              Labels: elasticsearch, hive, ruleengine, rules, solr, spark
>   Original Estimate: 2,016h
>  Remaining Estimate: 2,016h
>
> The primary purpose for a rules engine for Apache Metron would be to allow Metron users to author rules that then generate alerts for SIC analysts to investigate. Typical enterprises have hundreds of rules (dozens for each data source) and need the flexibility to alter rules as needed without deploying code to production.  Rules would run on a schedule in batch mode and perform predefined action such as generating an alert.
> Here are some example rules we'd like to be able to run in metron with the rule syntax written in SQL:
> | Rule Description |	Rule Syntax |	Schedule |
> | Mcafee epo log entry notifies us a malware delete failed | 	Select * from mcafee where event_description = “Malware Delete Failed”	| Run every 5 minutes and for new data in the previous 5 minutes. | 
> | Multiple malware events for a single user within a short period of time | 	Select count( * ) as avcount, user from mcafee group by user, dest_ip, os where category like 'av.%' and avcount > 8 | Run every 60 minutes for the previous 60 minutes | 
> Users should have a front end to author rules, decide on a schedule, and configure an alert priority, rule description, and the action to perform(alert, e:
> Here is a sample mockup:
> !http://i.imgur.com/0sbNXPp.png!
> The batch rules engine would fire recurring queries against data at rest in one of the existing Metron datastores (Solr, Hive, Elasticsearch) that will then  perform predefined action such as generating an alert, running a script (python), or kicking off packet capture.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)