[jira] [Created] (EAGLE-2) watch message process backlog in Eagle UI

["Edward Zhang (JIRA)" <ji...@apache.org>]

Edward Zhang created EAGLE-2:
--------------------------------

             Summary: watch message process backlog in Eagle UI
                 Key: EAGLE-2
                 URL: https://issues.apache.org/jira/browse/EAGLE-2
             Project: Eagle
          Issue Type: Improvement
         Environment: production
            Reporter: Edward Zhang


Message latency is a key factor for Eagle to enable realtime security monitoring. For hdfs audit log monitoring, kafka is used as datasource. So there is always some gap between current max offset in kafka and processed offset in eagle. The gap is the backlog which eagle should consume quickly as much as quickly. If the gap can be sampled for every minute or 20 seconds, then we understand if eagle is catching up or is lagging behind more.

The command to get current max offset in kafka is 
bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list xxxx --topic hdfs_audit_log --time -1

and Storm-kafka spout would store processed offset in zookeeper, in the following znode:
/consumers/hdfs_audit_log/eagle.hdfsaudit.consumer/partition_0 

So technically we can get the gap and write that to eagle service then in UI we can watch the backlog



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)