[jira] [Reopened] (EAGLE-2) watch message process backlog in Eagle UI

["Hao Chen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/EAGLE-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hao Chen reopened EAGLE-2:
--------------------------

[~libsun] Reopen the ticket because the feature is actually not finished at all.

TODO:
1) Developer Experience: documentation, API design
2) End-user Experience: helper scripts for setting up, use case sample
3) UI Visualization Integration: it's not for ops/dev, but the end-user of the product



> watch message process backlog in Eagle UI
> -----------------------------------------
>
>                 Key: EAGLE-2
>                 URL: https://issues.apache.org/jira/browse/EAGLE-2
>             Project: Eagle
>          Issue Type: Improvement
>         Environment: production
>            Reporter: Edward Zhang
>            Assignee: Libin, Sun
>   Original Estimate: 96h
>  Remaining Estimate: 96h
>
> Message latency is a key factor for Eagle to enable realtime security monitoring. For hdfs audit log monitoring, kafka is used as datasource. So there is always some gap between current max offset in kafka and processed offset in eagle. The gap is the backlog which eagle should consume quickly as much as quickly. If the gap can be sampled for every minute or 20 seconds, then we understand if eagle is catching up or is lagging behind more.
> The command to get current max offset in kafka is 
> bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list xxxx --topic hdfs_audit_log --time -1
> and Storm-kafka spout would store processed offset in zookeeper, in the following znode:
> /consumers/hdfs_audit_log/eagle.hdfsaudit.consumer/partition_0 
> So technically we can get the gap and write that to eagle service then in UI we can watch the backlog



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)