You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@eagle.apache.org by "Nagalkar, Sanjay Contractor" <Sa...@ssa.gov> on 2016/02/10 21:55:17 UTC
Policy Alerts are not generated in Eagle
Hello
I am having issues generating policy alerts for Apache Eagle.
I have configured Hive and HDFS policies as per the instructions in the Eagle DOCS but I am not seeing any alerts for my Hive and HDFS policy.
I am executing HIVE queries against XADEMO schema to pull PHONE_NUMBER field via HDP sandbox and I was expecting to see some alerts for this data access in eagle. However, I have no policy alerts .
Similar situation with HDFS file access. No alerts for /tmp/private file access.
Please let me know if there is something that I am missing.
Thanks
Sanjay Nagalkar
Re: Policy Alerts are not generated in Eagle
Posted by "Zhang, Edward (GDI Hadoop)" <yo...@ebay.com>.
For HDFS audit log, by default Eagle uses Kafka log4j appender to send
namenode audit log into Kafka.
Could you please try
"/usr/hdp/2.2.4.2-2/kafka/bin/kafka-console-consumer.sh --topic
sandbox_hdfs_audit_log --zookeeper localhost:2181² to see if message is
sent out.
For Hive query log, by default Eagle uses Yarn API to get Hive job and
parse query.
Could you please make sure Hive is using MAP/REDUCE instead of TEZ
scheduler?
In overall, you can double check http://localhost:8744/index.html, and
check if 2 topologies are launched and drill down to each topology to see
if there is any error message or message emitted.
Thanks
Edward
On 2/10/16, 12:55, "Nagalkar, Sanjay Contractor"
<Sa...@ssa.gov> wrote:
>
>
>
>Hello
>
>
>
>I am having issues generating policy alerts for Apache Eagle.
>
>
>
>I have configured Hive and HDFS policies as per the instructions in the
>Eagle DOCS but I am not seeing any alerts for my Hive and HDFS policy.
>
>
>
>I am executing HIVE queries against XADEMO schema to pull PHONE_NUMBER
>field via HDP sandbox and I was expecting to see some alerts for this
>data access in eagle. However, I have no policy alerts .
>
>
>
>Similar situation with HDFS file access. No alerts for /tmp/private file
>access.
>
>
>
>Please let me know if there is something that I am missing.
>
>
>
>Thanks
>
>Sanjay Nagalkar
>
RE: Policy Alerts are not generated in Eagle
Posted by Daniel Zhou <Da...@dataguise.com>.
Hi, Sanjay,
First you can check if hdfs log data are streamed into Kafka by using command like this:
$ /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic sandbox_hdfs_audit_log
If there were error messages and you believe you have the right configuration for kafka, then you can try this:
Stop kafka service and Zookeeper service, after that, first restart zookeeper and then restart Kafka service.
If Kafka works, then you can check log files of Eagle service and Storm service to see more details.
Regards,
Daniel
-----Original Message-----
From: Nagalkar, Sanjay Contractor [mailto:Sanjay.Nagalkar@ssa.gov]
Sent: Wednesday, February 10, 2016 12:55 PM
To: 'dev@eagle.incubator.apache.org' <de...@eagle.incubator.apache.org>
Subject: Policy Alerts are not generated in Eagle
Hello
I am having issues generating policy alerts for Apache Eagle.
I have configured Hive and HDFS policies as per the instructions in the Eagle DOCS but I am not seeing any alerts for my Hive and HDFS policy.
I am executing HIVE queries against XADEMO schema to pull PHONE_NUMBER field via HDP sandbox and I was expecting to see some alerts for this data access in eagle. However, I have no policy alerts .
Similar situation with HDFS file access. No alerts for /tmp/private file access.
Please let me know if there is something that I am missing.
Thanks
Sanjay Nagalkar
RE: Policy Alerts are not generated in Eagle
Posted by Daniel Zhou <Da...@dataguise.com>.
Hi, Nagalkar,
To kill a topology you can log in to Strom UI:
localhost:8744
and click the button ”kill” for that topology, then rerun Eagle’s service.
Regards,
Daniel
From: Nagalkar, Sanjay Contractor [mailto:Sanjay.Nagalkar@ssa.gov]
Sent: Tuesday, February 23, 2016 1:35 PM
To: Daniel Zhou <Da...@dataguise.com>; dev@eagle.incubator.apache.org
Subject: RE: Policy Alerts are not generated in Eagle
Hi Daniel,
I am now seeing the log messages in topic “sandbox_hdfs_audit_log” .. there was some initialization in error in namenode logs.
I am mostly seeing continuous flow logs messages into this kafka topics but still there are no HDFS alerts in eagle for the hdfs actions.
Also, whenever I start the eagle service , I see runtime exception that topology with name ‘sandbox-hdfsAuditLog-topology’ already exist. Below is the detail error screenshot:
[cid:image001.png@01D16E49.E6B2F1D0]
From: Daniel Zhou [mailto:Daniel.Zhou@dataguise.com]
Sent: Thursday, February 18, 2016 5:29 PM
To: Nagalkar, Sanjay Contractor; dev@eagle.incubator.apache.org<ma...@eagle.incubator.apache.org>
Subject: RE: Policy Alerts are not generated in Eagle
Hi Sanjay,
Seems your KAFKA works.
So when you run bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic sandbox_hdfs_audit_log --from-beginning
Can you see the HDFS messages? (not your test messages)
If not, I suggest you to open Storm UI “localhost:8744” and click the topology summary, find the exception/error, and paste them in the email.
And also, double check step3 in this: http://eagle.incubator.apache.org/docs/import-hdfs-auditLog.html
Regards,
Daniel
From: Nagalkar, Sanjay Contractor [mailto:Sanjay.Nagalkar@ssa.gov]
Sent: Thursday, February 18, 2016 2:02 PM
To: Daniel Zhou <Da...@dataguise.com>>; dev@eagle.incubator.apache.org<ma...@eagle.incubator.apache.org>
Subject: RE: Policy Alerts are not generated in Eagle
Hi Daniel,
I did the below test and I was able to receive the messages in consumer. Infact, I also tried below steps with toic ‘sandbox_hdfs_audit_log’ and was able to see my test messages.
Still I don’t see the HDFS policy alerts in eagle
Thanks
Sanjay
From: Daniel Zhou [mailto:Daniel.Zhou@dataguise.com]
Sent: Tuesday, February 16, 2016 5:28 PM
To: dev@eagle.incubator.apache.org<ma...@eagle.incubator.apache.org>; Nagalkar, Sanjay Contractor
Subject: RE: Policy Alerts are not generated in Eagle
Hi, Nagalkar
Could you check if your Kafka works ? Use these steps:
1. Create a topic:
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
2. Check if topic “test” is created:
bin/kafka-topics.sh --list --zookeeper localhost:2181
3. Open a terminal and use producer to send message:
bin/kafka-console-producer.sh --broker-list hostname:6667 --topic test
// then type some messages
4. Open another terminal and see if consumer get the message:
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
// check if messages received
If no messages received, you need to double check the setting of Kafka.
Regards,
Daniel
From: Nagalkar, Sanjay Contractor [mailto:Sanjay.Nagalkar@ssa.gov]
Sent: Tuesday, February 16, 2016 1:58 PM
To: 'dev@eagle.incubator.apache.org' <de...@eagle.incubator.apache.org>>
Subject: FW: Policy Alerts are not generated in Eagle
Hello,
just wondering if there is any update on this issue I reported earlier .
Below is the detailed information about the encountered issue and the configuration made in our eagle setup:
We are planning to implement some tool for sensitive data monitoring /audit and for this purpose , I am exploring Eagle to see if it fits our requirements. My goal is play with eagle in Horton works sandbox environment and once comfortable then move to production.
I have followed the Eagle user guide and had setup Horton works sandbox and eagle service as instructed in the User Docs. (I did not use the Ambari plug-in for eagle as I had some issue with it so I am starting eagle services manually from command line).
I have configured HIVE and HDFS policies but I am not seeing any Alerts generated for data access by these policy.
For HDFS policy –
I have setup policy for /tmp/private file access but here I am receiving error while writing to KAFKA topic.
I am getting "connection Refused" error.
I have a Kafka topic 'Sandbox_hdfs_audit_log' and hdfs action events are not logged into this topic. when I view kafka.out , I see
ERROR Exception emitting metrics - org.apache.hadoop.metrics2.sink.timelineUnableToConnectException:java.net.ConnectException.
I have verified that the Kafka broker is up and configured ( Pls. see below log screenshots )
For HIVE policy -
I have setup Hive policy for PHONE_NUMBER field setup. When I use hive from Horton works Sandbox and execute the HIVE queries against XADEMO schema to select PHONE_NUMBER field, Eagle is not generating any alerts for accessing PHONE_NUMBER field .
Pls. click below link to see document outlining steps for HIVE query and eagle policy.
https://community.hortonworks.com/storage/attachments/1964-dtempsanbox-hive-eagle-issue.pdf
Please let me know if you need more information on this.
Your help with this is very much appreciated.
Thanks
Sanjay Nagalkar
Ph: 410-371-3299
On Wed, Feb 10, 2016 at 12:55 PM, Nagalkar, Sanjay Contractor <Sa...@ssa.gov>> wrote:
Hello
I am having issues generating policy alerts for Apache Eagle.
I have configured Hive and HDFS policies as per the instructions in the Eagle DOCS but I am not seeing any alerts for my Hive and HDFS policy.
I am executing HIVE queries against XADEMO schema to pull PHONE_NUMBER field via HDP sandbox and I was expecting to see some alerts for this data access in eagle. However, I have no policy alerts .
Similar situation with HDFS file access. No alerts for /tmp/private file access.
Please let me know if there is something that I am missing.
Thanks
Sanjay Nagalkar
RE: Policy Alerts are not generated in Eagle
Posted by "Nagalkar, Sanjay Contractor" <Sa...@ssa.gov>.
Hi Daniel,
I am now seeing the log messages in topic “sandbox_hdfs_audit_log” .. there was some initialization in error in namenode logs.
I am mostly seeing continuous flow logs messages into this kafka topics but still there are no HDFS alerts in eagle for the hdfs actions.
Also, whenever I start the eagle service , I see runtime exception that topology with name ‘sandbox-hdfsAuditLog-topology’ already exist. Below is the detail error screenshot:
[cid:image001.png@01D16E58.15AC5D10]
From: Daniel Zhou [mailto:Daniel.Zhou@dataguise.com]
Sent: Thursday, February 18, 2016 5:29 PM
To: Nagalkar, Sanjay Contractor; dev@eagle.incubator.apache.org
Subject: RE: Policy Alerts are not generated in Eagle
Hi Sanjay,
Seems your KAFKA works.
So when you run bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic sandbox_hdfs_audit_log --from-beginning
Can you see the HDFS messages? (not your test messages)
If not, I suggest you to open Storm UI “localhost:8744” and click the topology summary, find the exception/error, and paste them in the email.
And also, double check step3 in this: http://eagle.incubator.apache.org/docs/import-hdfs-auditLog.html
Regards,
Daniel
From: Nagalkar, Sanjay Contractor [mailto:Sanjay.Nagalkar@ssa.gov]
Sent: Thursday, February 18, 2016 2:02 PM
To: Daniel Zhou <Da...@dataguise.com>>; dev@eagle.incubator.apache.org<ma...@eagle.incubator.apache.org>
Subject: RE: Policy Alerts are not generated in Eagle
Hi Daniel,
I did the below test and I was able to receive the messages in consumer. Infact, I also tried below steps with toic ‘sandbox_hdfs_audit_log’ and was able to see my test messages.
Still I don’t see the HDFS policy alerts in eagle
Thanks
Sanjay
From: Daniel Zhou [mailto:Daniel.Zhou@dataguise.com]
Sent: Tuesday, February 16, 2016 5:28 PM
To: dev@eagle.incubator.apache.org<ma...@eagle.incubator.apache.org>; Nagalkar, Sanjay Contractor
Subject: RE: Policy Alerts are not generated in Eagle
Hi, Nagalkar
Could you check if your Kafka works ? Use these steps:
1. Create a topic:
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
2. Check if topic “test” is created:
bin/kafka-topics.sh --list --zookeeper localhost:2181
3. Open a terminal and use producer to send message:
bin/kafka-console-producer.sh --broker-list hostname:6667 --topic test
// then type some messages
4. Open another terminal and see if consumer get the message:
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
// check if messages received
If no messages received, you need to double check the setting of Kafka.
Regards,
Daniel
From: Nagalkar, Sanjay Contractor [mailto:Sanjay.Nagalkar@ssa.gov]
Sent: Tuesday, February 16, 2016 1:58 PM
To: 'dev@eagle.incubator.apache.org' <de...@eagle.incubator.apache.org>>
Subject: FW: Policy Alerts are not generated in Eagle
Hello,
just wondering if there is any update on this issue I reported earlier .
Below is the detailed information about the encountered issue and the configuration made in our eagle setup:
We are planning to implement some tool for sensitive data monitoring /audit and for this purpose , I am exploring Eagle to see if it fits our requirements. My goal is play with eagle in Horton works sandbox environment and once comfortable then move to production.
I have followed the Eagle user guide and had setup Horton works sandbox and eagle service as instructed in the User Docs. (I did not use the Ambari plug-in for eagle as I had some issue with it so I am starting eagle services manually from command line).
I have configured HIVE and HDFS policies but I am not seeing any Alerts generated for data access by these policy.
For HDFS policy –
I have setup policy for /tmp/private file access but here I am receiving error while writing to KAFKA topic.
I am getting "connection Refused" error.
I have a Kafka topic 'Sandbox_hdfs_audit_log' and hdfs action events are not logged into this topic. when I view kafka.out , I see
ERROR Exception emitting metrics - org.apache.hadoop.metrics2.sink.timelineUnableToConnectException:java.net.ConnectException.
I have verified that the Kafka broker is up and configured ( Pls. see below log screenshots )
For HIVE policy -
I have setup Hive policy for PHONE_NUMBER field setup. When I use hive from Horton works Sandbox and execute the HIVE queries against XADEMO schema to select PHONE_NUMBER field, Eagle is not generating any alerts for accessing PHONE_NUMBER field .
Pls. click below link to see document outlining steps for HIVE query and eagle policy.
https://community.hortonworks.com/storage/attachments/1964-dtempsanbox-hive-eagle-issue.pdf
Please let me know if you need more information on this.
Your help with this is very much appreciated.
Thanks
Sanjay Nagalkar
Ph: 410-371-3299
On Wed, Feb 10, 2016 at 12:55 PM, Nagalkar, Sanjay Contractor <Sa...@ssa.gov>> wrote:
Hello
I am having issues generating policy alerts for Apache Eagle.
I have configured Hive and HDFS policies as per the instructions in the Eagle DOCS but I am not seeing any alerts for my Hive and HDFS policy.
I am executing HIVE queries against XADEMO schema to pull PHONE_NUMBER field via HDP sandbox and I was expecting to see some alerts for this data access in eagle. However, I have no policy alerts .
Similar situation with HDFS file access. No alerts for /tmp/private file access.
Please let me know if there is something that I am missing.
Thanks
Sanjay Nagalkar
RE: Policy Alerts are not generated in Eagle
Posted by Daniel Zhou <Da...@dataguise.com>.
Hi Sanjay,
Seems your KAFKA works.
So when you run bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic sandbox_hdfs_audit_log --from-beginning
Can you see the HDFS messages? (not your test messages)
If not, I suggest you to open Storm UI “localhost:8744” and click the topology summary, find the exception/error, and paste them in the email.
And also, double check step3 in this: http://eagle.incubator.apache.org/docs/import-hdfs-auditLog.html
Regards,
Daniel
From: Nagalkar, Sanjay Contractor [mailto:Sanjay.Nagalkar@ssa.gov]
Sent: Thursday, February 18, 2016 2:02 PM
To: Daniel Zhou <Da...@dataguise.com>; dev@eagle.incubator.apache.org
Subject: RE: Policy Alerts are not generated in Eagle
Hi Daniel,
I did the below test and I was able to receive the messages in consumer. Infact, I also tried below steps with toic ‘sandbox_hdfs_audit_log’ and was able to see my test messages.
Still I don’t see the HDFS policy alerts in eagle
Thanks
Sanjay
From: Daniel Zhou [mailto:Daniel.Zhou@dataguise.com]
Sent: Tuesday, February 16, 2016 5:28 PM
To: dev@eagle.incubator.apache.org<ma...@eagle.incubator.apache.org>; Nagalkar, Sanjay Contractor
Subject: RE: Policy Alerts are not generated in Eagle
Hi, Nagalkar
Could you check if your Kafka works ? Use these steps:
1. Create a topic:
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
2. Check if topic “test” is created:
bin/kafka-topics.sh --list --zookeeper localhost:2181
3. Open a terminal and use producer to send message:
bin/kafka-console-producer.sh --broker-list hostname:6667 --topic test
// then type some messages
4. Open another terminal and see if consumer get the message:
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
// check if messages received
If no messages received, you need to double check the setting of Kafka.
Regards,
Daniel
From: Nagalkar, Sanjay Contractor [mailto:Sanjay.Nagalkar@ssa.gov]
Sent: Tuesday, February 16, 2016 1:58 PM
To: 'dev@eagle.incubator.apache.org' <de...@eagle.incubator.apache.org>>
Subject: FW: Policy Alerts are not generated in Eagle
Hello,
just wondering if there is any update on this issue I reported earlier .
Below is the detailed information about the encountered issue and the configuration made in our eagle setup:
We are planning to implement some tool for sensitive data monitoring /audit and for this purpose , I am exploring Eagle to see if it fits our requirements. My goal is play with eagle in Horton works sandbox environment and once comfortable then move to production.
I have followed the Eagle user guide and had setup Horton works sandbox and eagle service as instructed in the User Docs. (I did not use the Ambari plug-in for eagle as I had some issue with it so I am starting eagle services manually from command line).
I have configured HIVE and HDFS policies but I am not seeing any Alerts generated for data access by these policy.
For HDFS policy –
I have setup policy for /tmp/private file access but here I am receiving error while writing to KAFKA topic.
I am getting "connection Refused" error.
I have a Kafka topic 'Sandbox_hdfs_audit_log' and hdfs action events are not logged into this topic. when I view kafka.out , I see
ERROR Exception emitting metrics - org.apache.hadoop.metrics2.sink.timelineUnableToConnectException:java.net.ConnectException.
I have verified that the Kafka broker is up and configured ( Pls. see below log screenshots )
For HIVE policy -
I have setup Hive policy for PHONE_NUMBER field setup. When I use hive from Horton works Sandbox and execute the HIVE queries against XADEMO schema to select PHONE_NUMBER field, Eagle is not generating any alerts for accessing PHONE_NUMBER field .
Pls. click below link to see document outlining steps for HIVE query and eagle policy.
https://community.hortonworks.com/storage/attachments/1964-dtempsanbox-hive-eagle-issue.pdf
Please let me know if you need more information on this.
Your help with this is very much appreciated.
Thanks
Sanjay Nagalkar
Ph: 410-371-3299
On Wed, Feb 10, 2016 at 12:55 PM, Nagalkar, Sanjay Contractor <Sa...@ssa.gov>> wrote:
Hello
I am having issues generating policy alerts for Apache Eagle.
I have configured Hive and HDFS policies as per the instructions in the Eagle DOCS but I am not seeing any alerts for my Hive and HDFS policy.
I am executing HIVE queries against XADEMO schema to pull PHONE_NUMBER field via HDP sandbox and I was expecting to see some alerts for this data access in eagle. However, I have no policy alerts .
Similar situation with HDFS file access. No alerts for /tmp/private file access.
Please let me know if there is something that I am missing.
Thanks
Sanjay Nagalkar
RE: Policy Alerts are not generated in Eagle
Posted by "Nagalkar, Sanjay Contractor" <Sa...@ssa.gov>.
Hi Daniel,
I did the below test and I was able to receive the messages in consumer. Infact, I also tried below steps with toic ‘sandbox_hdfs_audit_log’ and was able to see my test messages.
Still I don’t see the HDFS policy alerts in eagle
Thanks
Sanjay
From: Daniel Zhou [mailto:Daniel.Zhou@dataguise.com]
Sent: Tuesday, February 16, 2016 5:28 PM
To: dev@eagle.incubator.apache.org; Nagalkar, Sanjay Contractor
Subject: RE: Policy Alerts are not generated in Eagle
Hi, Nagalkar
Could you check if your Kafka works ? Use these steps:
1. Create a topic:
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
2. Check if topic “test” is created:
bin/kafka-topics.sh --list --zookeeper localhost:2181
3. Open a terminal and use producer to send message:
bin/kafka-console-producer.sh --broker-list hostname:6667 --topic test
// then type some messages
4. Open another terminal and see if consumer get the message:
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
// check if messages received
If no messages received, you need to double check the setting of Kafka.
Regards,
Daniel
From: Nagalkar, Sanjay Contractor [mailto:Sanjay.Nagalkar@ssa.gov]
Sent: Tuesday, February 16, 2016 1:58 PM
To: 'dev@eagle.incubator.apache.org' <de...@eagle.incubator.apache.org>>
Subject: FW: Policy Alerts are not generated in Eagle
Hello,
just wondering if there is any update on this issue I reported earlier .
Below is the detailed information about the encountered issue and the configuration made in our eagle setup:
We are planning to implement some tool for sensitive data monitoring /audit and for this purpose , I am exploring Eagle to see if it fits our requirements. My goal is play with eagle in Horton works sandbox environment and once comfortable then move to production.
I have followed the Eagle user guide and had setup Horton works sandbox and eagle service as instructed in the User Docs. (I did not use the Ambari plug-in for eagle as I had some issue with it so I am starting eagle services manually from command line).
I have configured HIVE and HDFS policies but I am not seeing any Alerts generated for data access by these policy.
For HDFS policy –
I have setup policy for /tmp/private file access but here I am receiving error while writing to KAFKA topic.
I am getting "connection Refused" error.
I have a Kafka topic 'Sandbox_hdfs_audit_log' and hdfs action events are not logged into this topic. when I view kafka.out , I see
ERROR Exception emitting metrics - org.apache.hadoop.metrics2.sink.timelineUnableToConnectException:java.net.ConnectException.
I have verified that the Kafka broker is up and configured ( Pls. see below log screenshots )
For HIVE policy -
I have setup Hive policy for PHONE_NUMBER field setup. When I use hive from Horton works Sandbox and execute the HIVE queries against XADEMO schema to select PHONE_NUMBER field, Eagle is not generating any alerts for accessing PHONE_NUMBER field .
Pls. click below link to see document outlining steps for HIVE query and eagle policy.
https://community.hortonworks.com/storage/attachments/1964-dtempsanbox-hive-eagle-issue.pdf
Please let me know if you need more information on this.
Your help with this is very much appreciated.
Thanks
Sanjay Nagalkar
Ph: 410-371-3299
On Wed, Feb 10, 2016 at 12:55 PM, Nagalkar, Sanjay Contractor <Sa...@ssa.gov>> wrote:
Hello
I am having issues generating policy alerts for Apache Eagle.
I have configured Hive and HDFS policies as per the instructions in the Eagle DOCS but I am not seeing any alerts for my Hive and HDFS policy.
I am executing HIVE queries against XADEMO schema to pull PHONE_NUMBER field via HDP sandbox and I was expecting to see some alerts for this data access in eagle. However, I have no policy alerts .
Similar situation with HDFS file access. No alerts for /tmp/private file access.
Please let me know if there is something that I am missing.
Thanks
Sanjay Nagalkar
RE: Policy Alerts are not generated in Eagle
Posted by Daniel Zhou <Da...@dataguise.com>.
Hi, Nagalkar
Could you check if your Kafka works ? Use these steps:
1. Create a topic:
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
2. Check if topic “test” is created:
bin/kafka-topics.sh --list --zookeeper localhost:2181
3. Open a terminal and use producer to send message:
bin/kafka-console-producer.sh --broker-list hostname:6667 --topic test
// then type some messages
4. Open another terminal and see if consumer get the message:
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
// check if messages received
If no messages received, you need to double check the setting of Kafka.
Regards,
Daniel
From: Nagalkar, Sanjay Contractor [mailto:Sanjay.Nagalkar@ssa.gov]
Sent: Tuesday, February 16, 2016 1:58 PM
To: 'dev@eagle.incubator.apache.org' <de...@eagle.incubator.apache.org>
Subject: FW: Policy Alerts are not generated in Eagle
Hello,
just wondering if there is any update on this issue I reported earlier .
Below is the detailed information about the encountered issue and the configuration made in our eagle setup:
We are planning to implement some tool for sensitive data monitoring /audit and for this purpose , I am exploring Eagle to see if it fits our requirements. My goal is play with eagle in Horton works sandbox environment and once comfortable then move to production.
I have followed the Eagle user guide and had setup Horton works sandbox and eagle service as instructed in the User Docs. (I did not use the Ambari plug-in for eagle as I had some issue with it so I am starting eagle services manually from command line).
I have configured HIVE and HDFS policies but I am not seeing any Alerts generated for data access by these policy.
For HDFS policy –
I have setup policy for /tmp/private file access but here I am receiving error while writing to KAFKA topic.
I am getting "connection Refused" error.
I have a Kafka topic 'Sandbox_hdfs_audit_log' and hdfs action events are not logged into this topic. when I view kafka.out , I see
ERROR Exception emitting metrics - org.apache.hadoop.metrics2.sink.timelineUnableToConnectException:java.net.ConnectException.
I have verified that the Kafka broker is up and configured ( Pls. see below log screenshots )
For HIVE policy -
I have setup Hive policy for PHONE_NUMBER field setup. When I use hive from Horton works Sandbox and execute the HIVE queries against XADEMO schema to select PHONE_NUMBER field, Eagle is not generating any alerts for accessing PHONE_NUMBER field .
Pls. click below link to see document outlining steps for HIVE query and eagle policy.
https://community.hortonworks.com/storage/attachments/1964-dtempsanbox-hive-eagle-issue.pdf
Please let me know if you need more information on this.
Your help with this is very much appreciated.
Thanks
Sanjay Nagalkar
Ph: 410-371-3299
On Wed, Feb 10, 2016 at 12:55 PM, Nagalkar, Sanjay Contractor <Sa...@ssa.gov>> wrote:
Hello
I am having issues generating policy alerts for Apache Eagle.
I have configured Hive and HDFS policies as per the instructions in the Eagle DOCS but I am not seeing any alerts for my Hive and HDFS policy.
I am executing HIVE queries against XADEMO schema to pull PHONE_NUMBER field via HDP sandbox and I was expecting to see some alerts for this data access in eagle. However, I have no policy alerts .
Similar situation with HDFS file access. No alerts for /tmp/private file access.
Please let me know if there is something that I am missing.
Thanks
Sanjay Nagalkar
Re: FW: Policy Alerts are not generated in Eagle
Posted by Edward Zhang <yo...@gmail.com>.
Hi Sanjay,
I think we did respond your question from 2 developers immediately after
you raised question last time. Did you receive that?
Looks when we reply, your email is not in the TO list by default :-) let me
forward that to you again.
By the way, did you subscribe to Eagle dev mail list? You can follow this
link to subscribe http://eagle.incubator.apache.org/docs/community.html
(send email to dev-subscribe@eagle.incubabator.apache.org, and following
the steps from the response)
Thanks
Edward
On Tue, Feb 16, 2016 at 1:57 PM, Nagalkar, Sanjay Contractor <
Sanjay.Nagalkar@ssa.gov> wrote:
> Hello,
>
>
>
> just wondering if there is any update on this issue I reported earlier .
>
>
>
> Below is the detailed information about the encountered issue and the
> configuration made in our eagle setup:
>
>
>
> We are planning to implement some tool for sensitive data monitoring
> /audit and for this purpose , I am exploring Eagle to see if it fits our
> requirements. My goal is play with eagle in Horton works sandbox
> environment and once comfortable then move to production.
>
>
>
> I have followed the Eagle user guide and had setup Horton works sandbox
> and eagle service as instructed in the User Docs. (I did not use the
> Ambari plug-in for eagle as I had some issue with it so I am starting eagle
> services manually from command line).
>
>
>
>
>
> I have configured HIVE and HDFS policies but I am not seeing any Alerts
> generated for data access by these policy.
>
>
>
>
>
> *For HDFS policy* –
>
> I have setup policy for /tmp/private file access but here I am receiving
> error while writing to KAFKA topic.
>
>
>
> I am getting "connection Refused" error.
>
>
>
> I have a Kafka topic 'Sandbox_hdfs_audit_log' and hdfs action events are
> not logged into this topic. when I view kafka.out , I see
>
>
>
> ERROR Exception emitting metrics -
> org.apache.hadoop.metrics2.sink.timelineUnableToConnectException:java.net.ConnectException.
>
>
>
>
>
>
> I have verified that the Kafka broker is up and configured ( Pls. see
> below log screenshots )
>
>
>
>
>
>
>
> *For HIVE policy -*
>
> I have setup Hive policy for PHONE_NUMBER field setup. When I use hive
> from Horton works Sandbox and execute the HIVE queries against XADEMO
> schema to select PHONE_NUMBER field, Eagle is not generating any alerts for
> accessing PHONE_NUMBER field .
>
>
>
> Pls. click below link to see document outlining steps for HIVE query and
> eagle policy.
>
>
>
>
> https://community.hortonworks.com/storage/attachments/1964-dtempsanbox-hive-eagle-issue.pdf
>
>
>
>
>
> Please let me know if you need more information on this.
>
>
>
> Your help with this is very much appreciated.
>
>
>
> Thanks
>
> Sanjay Nagalkar
>
> Ph: 410-371-3299
>
>
>
>
>
>
>
>
>
> On Wed, Feb 10, 2016 at 12:55 PM, Nagalkar, Sanjay Contractor <
> Sanjay.Nagalkar@ssa.gov> wrote:
>
>
>
>
> Hello
>
>
>
> I am having issues generating policy alerts for Apache Eagle.
>
>
>
> I have configured Hive and HDFS policies as per the instructions in the
> Eagle DOCS but I am not seeing any alerts for my Hive and HDFS policy.
>
>
>
> I am executing HIVE queries against XADEMO schema to pull PHONE_NUMBER
> field via HDP sandbox and I was expecting to see some alerts for this data
> access in eagle. However, I have no policy alerts .
>
>
>
> Similar situation with HDFS file access. No alerts for /tmp/private file
> access.
>
>
>
> Please let me know if there is something that I am missing.
>
>
>
> Thanks
>
> Sanjay Nagalkar
>
>
>
FW: Policy Alerts are not generated in Eagle
Posted by "Nagalkar, Sanjay Contractor" <Sa...@ssa.gov>.
Hello,
just wondering if there is any update on this issue I reported earlier .
Below is the detailed information about the encountered issue and the configuration made in our eagle setup:
We are planning to implement some tool for sensitive data monitoring /audit and for this purpose , I am exploring Eagle to see if it fits our requirements. My goal is play with eagle in Horton works sandbox environment and once comfortable then move to production.
I have followed the Eagle user guide and had setup Horton works sandbox and eagle service as instructed in the User Docs. (I did not use the Ambari plug-in for eagle as I had some issue with it so I am starting eagle services manually from command line).
I have configured HIVE and HDFS policies but I am not seeing any Alerts generated for data access by these policy.
For HDFS policy –
I have setup policy for /tmp/private file access but here I am receiving error while writing to KAFKA topic.
I am getting "connection Refused" error.
I have a Kafka topic 'Sandbox_hdfs_audit_log' and hdfs action events are not logged into this topic. when I view kafka.out , I see
ERROR Exception emitting metrics - org.apache.hadoop.metrics2.sink.timelineUnableToConnectException:java.net.ConnectException.
I have verified that the Kafka broker is up and configured ( Pls. see below log screenshots )
For HIVE policy -
I have setup Hive policy for PHONE_NUMBER field setup. When I use hive from Horton works Sandbox and execute the HIVE queries against XADEMO schema to select PHONE_NUMBER field, Eagle is not generating any alerts for accessing PHONE_NUMBER field .
Pls. click below link to see document outlining steps for HIVE query and eagle policy.
https://community.hortonworks.com/storage/attachments/1964-dtempsanbox-hive-eagle-issue.pdf
Please let me know if you need more information on this.
Your help with this is very much appreciated.
Thanks
Sanjay Nagalkar
Ph: 410-371-3299
On Wed, Feb 10, 2016 at 12:55 PM, Nagalkar, Sanjay Contractor <Sa...@ssa.gov>> wrote:
Hello
I am having issues generating policy alerts for Apache Eagle.
I have configured Hive and HDFS policies as per the instructions in the Eagle DOCS but I am not seeing any alerts for my Hive and HDFS policy.
I am executing HIVE queries against XADEMO schema to pull PHONE_NUMBER field via HDP sandbox and I was expecting to see some alerts for this data access in eagle. However, I have no policy alerts .
Similar situation with HDFS file access. No alerts for /tmp/private file access.
Please let me know if there is something that I am missing.
Thanks
Sanjay Nagalkar