[jira] [Commented] (EAGLE-1041) Support policy processing pipeline

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/EAGLE-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16047796#comment-16047796 ] 

ASF GitHub Bot commented on EAGLE-1041:
---------------------------------------

GitHub user qingwen220 opened a pull request:

    https://github.com/apache/eagle/pull/947

    EAGLE-1041: Support policy processing pipeline

    https://issues.apache.org/jira/browse/EAGLE-1041
    
    Two updates:
    * if an inputStream is an intermediate stream (defined by select clause statement), then remove it from inputStream list and outputStream list
    * if an inputStream is an intermediate stream, remove its PartitionSpec 

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/qingwen220/eagle EAGLE-1041

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/eagle/pull/947.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #947
    
----
commit ea47b849700a89e15bc6a2e6b28317ee7a50b315
Author: Zhao, Qingwen <qi...@apache.org>
Date:   2017-06-13T12:22:15Z

    update alertEditCtrl.js

----


> Support policy processing pipeline
> ----------------------------------
>
>                 Key: EAGLE-1041
>                 URL: https://issues.apache.org/jira/browse/EAGLE-1041
>             Project: Eagle
>          Issue Type: Improvement
>    Affects Versions: v0.5.0
>            Reporter: Zhao, Qingwen
>            Assignee: Zhao, Qingwen
>
> In some cases, like increment or decrement pattern, data need to be processed in more than one stage. For example, alert if the metric value increases by N. Two steps to get the right alert.
> 1. sort & filter events which meet the filter conditions.
> 2. define data change pattern. 
> Here is a sample policy
> {code}
> fromHADOOP_JMX_METRIC_STREAM_SANDBOX[metric=="hadoop.namenode.dfs.missingblocks"]#window.externalTime(timestamp,1min) select * group by site,host,component, metric insert into temp;
> from every a=HADOOP_JMX_METRIC_STREAM_SANDBOX[metric=="hadoop.namenode.dfs.missingblocks"],
> b=HADOOP_JMX_METRIC_STREAM_SANDBOX[b.component==a.component and b.metric==a.metric and b.host==a.host and b.value>a.value and a.value>100]
> select b.site,b.host,b.component, b.metric, b.value as newNumOfMissingBlocks, a.value as oldNumOfMissingBlocks, (b.value-a.value) as increastedNumOfMissingBlocks insert into HADOOP_JMX_METRIC_STREAM_SANDBOX_MISS_BLOCKS_LARGER_OUT;
> {code}
> There are two queries in this policy. The first one with the time window condition tells Eagle to sort the original events. The second one defines the data pattern. As the constraint of Siddhi syntax (https://docs.wso2.com/display/CEP420/SiddhiQL+Guide+3.1#SiddhiQLGuide3.1-Pattern), the filtering of input events does not work. 
> Luckily, if we put the output stream of the first query as the input stream of the second query, it works. That's the problem this ticket tries to solve. 
> Ideally, the right policy can be written as 
> {code}
> fromHADOOP_JMX_METRIC_STREAM_SANDBOX[metric=="hadoop.namenode.dfs.missingblocks"]#window.externalTime(timestamp,1min) select * group by site,host,component, metric insert into MISSING_BLOCK_OUT;
> from every a=MISSING_BLOCK_OUT[metric=="hadoop.namenode.dfs.missingblocks"],
> b=MISSING_BLOCK_OUT[b.component==a.component and b.metric==a.metric and b.host==a.host and b.value>a.value and a.value>100]
> select b.site,b.host,b.component, b.metric, b.value as newNumOfMissingBlocks, a.value as oldNumOfMissingBlocks, (b.value-a.value) as increastedNumOfMissingBlocks insert into HADOOP_JMX_METRIC_STREAM_SANDBOX_MISS_BLOCKS_LARGER_OUT;
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)