HDFS Data Activity Monitoring with eagle 0.5.0-SNAPSHOT

[Jean Rossier <je...@sqooba.io>]

Hello,

I'm trying to setup the HDFS data activity monitoring stream (as explained
here http://eagle.apache.org/docs/hdfs-data-activity-monitoring.html), but
I don't see any logs flowing into my Kafka topic.

Is there anything to do that is not on the quick start page (mentioned
above) ?

One thing that surprised me, is that I don't have any log4jkafka folder in
/usr/hdp/current/eagle/lib/ (I'm using eagle 0.5.0-SNAPSHOT, from current
master branch). I'm supposed to add the libs from this folder to
the HADOOP_CLASSPATH env variable. I moved them by hand in the proper
folder, but, still, I don't see any logs flowing into my Kafka topic.

I don't see any error regarding kafka in the HDFS audit logs. There are,
though, some solr exceptions but I guess it shouldn't impact the stream I'm
interested in.

Thanks
Jean

Re: HDFS Data Activity Monitoring with eagle 0.5.0-SNAPSHOT

[SUDHA JENSLIN <sj...@gmail.com>]

You can also refer : 
http://eagle.apache.org/docs/import-hdfs-auditLog.html <http://eagle.apache.org/docs/import-hdfs-auditLog.html>

Try having the log stash conf filter and output similar to this:


filter {
 grok {
   match => ["message", "ugi=(?<user>([\w\d\-]+))@|ugi=(?<user>([\w\d\-]+))/[\w\d\-.]+@|ugi=(?<user>([\w\d.\-_]+))[\s(]+"]
  }

}

output {
   stdout {
     codec => rubydebug
   }
   kafka {
     codec => plain {
                 format => "%{message}"
             }
   bootstrap_servers => "127.0.0.1:6667"
   topic_id => "TestTopic"
   retry_backoff_ms => 100
   send_buffer_bytes => 102400



 }
 }


-Sudha Jenslin



> On Mar 21, 2017, at 7:53 PM, Jean Rossier <je...@sqooba.io> wrote:
> 
> Hello,
> 
> I'm trying to setup the HDFS data activity monitoring stream (as explained here http://eagle.apache.org/docs/hdfs-data-activity-monitoring.html <http://eagle.apache.org/docs/hdfs-data-activity-monitoring.html>), but I don't see any logs flowing into my Kafka topic.
> 
> Is there anything to do that is not on the quick start page (mentioned above) ?
> 
> One thing that surprised me, is that I don't have any log4jkafka folder in /usr/hdp/current/eagle/lib/ (I'm using eagle 0.5.0-SNAPSHOT, from current master branch). I'm supposed to add the libs from this folder to the HADOOP_CLASSPATH env variable. I moved them by hand in the proper folder, but, still, I don't see any logs flowing into my Kafka topic.
> 
> I don't see any error regarding kafka in the HDFS audit logs. There are, though, some solr exceptions but I guess it shouldn't impact the stream I'm interested in.
> 
> Thanks
> Jean
>