You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@eagle.apache.org by "Edward Zhang (JIRA)" <ji...@apache.org> on 2015/11/11 02:06:11 UTC
[jira] [Created] (EAGLE-21) Balance policy computation in runtime
to avoid policy evaluation screw
Edward Zhang created EAGLE-21:
---------------------------------
Summary: Balance policy computation in runtime to avoid policy evaluation screw
Key: EAGLE-21
URL: https://issues.apache.org/jira/browse/EAGLE-21
Project: Eagle
Issue Type: New Feature
Reporter: Edward Zhang
Today Eagle policy engine provides scalability by partitioning both data and computation. Data partitioning is provided by underlying stream framework for example storm, computation partitioning is provided by Eagle policy framework.
Eagle policy evaluation is a critical and heavy operation in Eagle framework. Eagle provides a default policy partitioner class, which evenly splits policies into different physical bolts with each bolt having the same number of policies. But this does not consider weight of policy as different policies may consume very different compute resources. For example a slide window of 1 hour would cost much more resource than a single event evaluation policy.
The simple DefaultPolicyPartitioner will cause policy evaluation skew, some bolts are in light load but others are in heavy load in terms of cpu or memory.
We need develop advanced policy partitioner to balance policy evaluation across a cluster of nodes. It is very nice that Apache Storm provides a very good metric called "Capacity" which represents load in a bolt. Capacity value is equal to (numberOfTuples * avgLatencyForEachTuple)/measurementTime. If capacity is close to 1, then the bolt is doing as fast as it can. Eagle can leverage this capacity value to dynamically balance policies across nodes.
We should understand that some policy is stateful meaning it is not easy to migrate a heavy policy from one node to another node, for example slide window policy
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)