[jira] [Comment Edited] (EAGLE-21) Balance policy computation in runtime to avoid policy evaluation screw

["Libin, Sun (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/EAGLE-21?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15022314#comment-15022314 ] 

Libin, Sun edited comment on EAGLE-21 at 11/23/15 3:49 PM:
-----------------------------------------------------------

Yes,we have evaluated the PKG solution, the PKG fit for some situation where grouping is not strict like machine learning 

But it don't match our needs for it may partition one key into different bolts which may break our alerts based on slide windows


was (Author: libsun):
Yes,we have evaluated the PKG solutio which fit for some situation where grouping is not strict like machine learning 

But it don't match our needs for it may partition one key into different bolts which may break our alerts based on slide windows

> Balance policy computation in runtime to avoid policy evaluation screw
> ----------------------------------------------------------------------
>
>                 Key: EAGLE-21
>                 URL: https://issues.apache.org/jira/browse/EAGLE-21
>             Project: Eagle
>          Issue Type: New Feature
>            Reporter: Edward Zhang
>            Assignee: Libin, Sun
>
> Today Eagle policy engine provides scalability by partitioning both data and computation. Data partitioning is provided by underlying stream framework for example storm, computation partitioning is provided by Eagle policy framework. 
> Eagle policy evaluation is a critical and heavy operation in Eagle framework. Eagle provides a default policy partitioner class, which evenly splits policies into different physical bolts with each bolt having the same number of policies. But this does not consider weight of policy as different policies may consume very different compute resources. For example a slide window of 1 hour would cost much more resource than a single event evaluation policy. 
> The simple DefaultPolicyPartitioner will cause policy evaluation skew, some bolts are in light load but others are in heavy load in terms of cpu or memory.
> We need develop advanced policy partitioner to balance policy evaluation across a cluster of nodes. It is very nice that Apache Storm provides a very good metric called "Capacity" which represents load in a bolt. Capacity value is equal to (numberOfTuples * avgLatencyForEachTuple)/measurementTime. If capacity is close to 1, then the bolt is doing as fast as it can. Eagle can leverage this capacity value to dynamically balance policies across nodes.
> We should understand that some policy is stateful meaning it is not easy to migrate a heavy policy from one node to another node, for example slide window policy



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)