[jira] [Created] (EAGLE-348) Alert engtin base on spark streaming

["tangjunjie (JIRA)" <ji...@apache.org>]

tangjunjie created EAGLE-348:
--------------------------------

             Summary: Alert engtin base on spark streaming
                 Key: EAGLE-348
                 URL: https://issues.apache.org/jira/browse/EAGLE-348
             Project: Eagle
          Issue Type: New Feature
    Affects Versions: v0.5.0
            Reporter: tangjunjie
            Assignee: tangjunjie
            Priority: Minor


{noformat}
New alert engine architecture:
1)Metaservice:   a) get meta(policies,topologies,streamdefinitions and so on) 
2)Coordinator:   a) pull policies and figure out if polices are changed by Metaservice and notify config consumer of config changes by ConfigBusService (powered by zk node cache)
                 b) placePolicy depend on policy usage on alertbolt
3)UnitTopology:  a) connect CorrelationSpout(read all topics data from kafka) StreamRouterBolt(find which alertbolt to execute) AlertBolt(execute policy on stream) AlertPublisherBolt (publish execute result to db,email and so on)
                    
Let the alert engine to work on spark streaming,I think :
1) #1 and #2 can be worked on both storm and spark platform, can use broadcast to distribute this service
2) we should rewirte CorrelationSpout,StreamRouterBolt,AlertBolt,AlertPublisherBolt to Dstream and connect them as DAG,then submit into spark platform.
3) spark can dynamic allocation executor without restart 
4) in the future spark streaming can support add or delete kafka topic without restart
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)