[jira] [Assigned] (EAGLE-2) watch message process backlog in Eagle UI

["Libin, Sun (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/EAGLE-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Libin, Sun reassigned EAGLE-2:
------------------------------

    Assignee: Libin, Sun  (was: Hao Chen)

> watch message process backlog in Eagle UI
> -----------------------------------------
>
>                 Key: EAGLE-2
>                 URL: https://issues.apache.org/jira/browse/EAGLE-2
>             Project: Eagle
>          Issue Type: Improvement
>         Environment: production
>            Reporter: Edward Zhang
>            Assignee: Libin, Sun
>   Original Estimate: 96h
>  Remaining Estimate: 96h
>
> Message latency is a key factor for Eagle to enable realtime security monitoring. For hdfs audit log monitoring, kafka is used as datasource. So there is always some gap between current max offset in kafka and processed offset in eagle. The gap is the backlog which eagle should consume quickly as much as quickly. If the gap can be sampled for every minute or 20 seconds, then we understand if eagle is catching up or is lagging behind more.
> The command to get current max offset in kafka is 
> bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list xxxx --topic hdfs_audit_log --time -1
> and Storm-kafka spout would store processed offset in zookeeper, in the following znode:
> /consumers/hdfs_audit_log/eagle.hdfsaudit.consumer/partition_0 
> So technically we can get the gap and write that to eagle service then in UI we can watch the backlog



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)