You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@eagle.apache.org by qi...@apache.org on 2016/10/19 03:19:01 UTC
[02/28] incubator-eagle git commit: [EAGLE-279] Create Documentation
for Sandbox Quick starter and JMX monitring
[EAGLE-279] Create Documentation for Sandbox Quick starter and JMX monitring
Create Documentation for Sandbox Quick starter and JMX monitring
Author: hdendukuri <hd...@ebay.com>
Closes #159 from hdendukuri/master.
Project: http://git-wip-us.apache.org/repos/asf/incubator-eagle/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-eagle/commit/df576452
Tree: http://git-wip-us.apache.org/repos/asf/incubator-eagle/tree/df576452
Diff: http://git-wip-us.apache.org/repos/asf/incubator-eagle/diff/df576452
Branch: refs/heads/master
Commit: df576452f174166407dfaa7a705b7f263f932f33
Parents: 9849303
Author: hdendukuri <hd...@ebay.com>
Authored: Fri May 27 19:36:26 2016 +0800
Committer: Hao Chen <ha...@apache.org>
Committed: Fri May 27 19:36:26 2016 +0800
----------------------------------------------------------------------
eagle-docs/images/eagle-service.png | Bin 0 -> 449283 bytes
eagle-docs/images/hbase-superuser.png | Bin 0 -> 147292 bytes
eagle-docs/images/hdfs-env-conf.png | Bin 0 -> 460829 bytes
eagle-docs/images/hdfs-env-conf2.png | Bin 0 -> 265131 bytes
eagle-docs/images/hdfs-log4j-conf.png | Bin 0 -> 359551 bytes
eagle-docs/images/new-site.png | Bin 0 -> 61032 bytes
eagle-docs/images/nn-restart.png | Bin 0 -> 79477 bytes
eagle-docs/images/start-storm.png | Bin 0 -> 500714 bytes
eagle-docs/userguide/jmx-metric-monitoring.md | 77 ++++++++++++
eagle-docs/userguide/sandbox-quick-start.md | 140 +++++++++++++++++++++
10 files changed, 217 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/df576452/eagle-docs/images/eagle-service.png
----------------------------------------------------------------------
diff --git a/eagle-docs/images/eagle-service.png b/eagle-docs/images/eagle-service.png
new file mode 100644
index 0000000..c649e36
Binary files /dev/null and b/eagle-docs/images/eagle-service.png differ
http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/df576452/eagle-docs/images/hbase-superuser.png
----------------------------------------------------------------------
diff --git a/eagle-docs/images/hbase-superuser.png b/eagle-docs/images/hbase-superuser.png
new file mode 100644
index 0000000..71b4a04
Binary files /dev/null and b/eagle-docs/images/hbase-superuser.png differ
http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/df576452/eagle-docs/images/hdfs-env-conf.png
----------------------------------------------------------------------
diff --git a/eagle-docs/images/hdfs-env-conf.png b/eagle-docs/images/hdfs-env-conf.png
new file mode 100644
index 0000000..b1ba80e
Binary files /dev/null and b/eagle-docs/images/hdfs-env-conf.png differ
http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/df576452/eagle-docs/images/hdfs-env-conf2.png
----------------------------------------------------------------------
diff --git a/eagle-docs/images/hdfs-env-conf2.png b/eagle-docs/images/hdfs-env-conf2.png
new file mode 100644
index 0000000..069acee
Binary files /dev/null and b/eagle-docs/images/hdfs-env-conf2.png differ
http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/df576452/eagle-docs/images/hdfs-log4j-conf.png
----------------------------------------------------------------------
diff --git a/eagle-docs/images/hdfs-log4j-conf.png b/eagle-docs/images/hdfs-log4j-conf.png
new file mode 100644
index 0000000..0eade68
Binary files /dev/null and b/eagle-docs/images/hdfs-log4j-conf.png differ
http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/df576452/eagle-docs/images/new-site.png
----------------------------------------------------------------------
diff --git a/eagle-docs/images/new-site.png b/eagle-docs/images/new-site.png
new file mode 100644
index 0000000..793aa18
Binary files /dev/null and b/eagle-docs/images/new-site.png differ
http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/df576452/eagle-docs/images/nn-restart.png
----------------------------------------------------------------------
diff --git a/eagle-docs/images/nn-restart.png b/eagle-docs/images/nn-restart.png
new file mode 100644
index 0000000..562641f
Binary files /dev/null and b/eagle-docs/images/nn-restart.png differ
http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/df576452/eagle-docs/images/start-storm.png
----------------------------------------------------------------------
diff --git a/eagle-docs/images/start-storm.png b/eagle-docs/images/start-storm.png
new file mode 100644
index 0000000..705b60e
Binary files /dev/null and b/eagle-docs/images/start-storm.png differ
http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/df576452/eagle-docs/userguide/jmx-metric-monitoring.md
----------------------------------------------------------------------
diff --git a/eagle-docs/userguide/jmx-metric-monitoring.md b/eagle-docs/userguide/jmx-metric-monitoring.md
new file mode 100644
index 0000000..1a31d63
--- /dev/null
+++ b/eagle-docs/userguide/jmx-metric-monitoring.md
@@ -0,0 +1,77 @@
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+
+
+---
+layout: doc
+title: "JMX Metric Monitoring"
+permalink: /docs/jmx-metric-monitoring.html
+---
+-->
+
+JMX metric for hadoop namenode url [http://127.0.0.1:50070/jmx](http://127.0.0.1:50070/jmx) can be monitored using Eagle. Follow below steps to enable this feature in Eagle.
+
+1. Install Python script (To populate JMX metric values to Kafka message topic periodically.)
+2. Deploy "hadoopjmx" storm topology.
+3. Create new site and policy in UI
+4. Validate policy alert.
+
+<br/>
+
+
+### **Prerequisite**
+* Complete the setup from [sandbox-quick-start.md](https://github.com/hdendukuri/incubator-eagle/blob/master/eagle-docs/userguide/sandbox-quick-start.md)
+
+<br/>
+
+
+### **Setup**
+From Hortonworks sandbox just run below setup script to Install Pyton JMX script, Create Kafka topic, update Hbase tables and deploy "hadoopjmx" storm topology.
+
+ $ /usr/hdp/current/eagle/examples/hadoop-metric-sandbox-starter.sh
+ $ /usr/hdp/current/eagle/examples/hadoop-metric-policy-create.sh
+
+<br/>
+
+
+### **Application Setup in UI**
+1. Login to Eagle UI [http://localhost:9099/eagle-service/](http://localhost:9099/eagle-service/) username and password as "admin" and "secret"
+2. Click on "Admin" from top right and click "Management" button.
+3. On Admin management page add "New Site" name "hadoopJmxMetricDataSource" by clicking on "New Site" link.
+
+4. Save the changes.
+5. On eagle home page you should see new tab called "METRIC", besides "DAM".
+6. Click on "JmxMetricMonitor" under "METRIC".
+
+You should see safeModePolicy policy.
+
+<br/>
+
+
+### **Demo**
+
+* First make sure that Kafka topic "nn_jmx_metric_sandbox" is populated with JMX metric data periodically.(To make sure that python script is running)
+
+ $ /usr/hdp/2.2.4.2-2/kafka/bin/kafka-console-consumer.sh --zookeeper sandbox.hortonworks.com:2181 --topic nn_jmx_metric_sandbox
+
+* Genrate Alert by producing alert triggering message in Kafka topic.
+
+
+ $ /usr/hdp/2.2.4.2-2/kafka/bin/kafka-console-producer.sh --broker-list sandbox.hortonworks.com:6667 --topic nn_jmx_metric_sandbox
+ $ {"host": "localhost", "timestamp": 1457033916718, "metric": "hadoop.namenode.fsnamesystemstate.fsstate", "component": "namenode", "site": "sandbox", "value": 1.0}
+
http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/df576452/eagle-docs/userguide/sandbox-quick-start.md
----------------------------------------------------------------------
diff --git a/eagle-docs/userguide/sandbox-quick-start.md b/eagle-docs/userguide/sandbox-quick-start.md
new file mode 100644
index 0000000..8ad41d3
--- /dev/null
+++ b/eagle-docs/userguide/sandbox-quick-start.md
@@ -0,0 +1,140 @@
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+Guide To Install Eagle Hortonworks sandbox.
+===============================================
+* Prerequisite
+* Download + Patch + Build
+* Setup Hadoop Environment.
+* Stream HDFS audit logs into Kafka.
+* Install Eagle.
+* Demos "HDFS File System" & "Hive" monitoring
+
+### **Prerequisite**
+* To install Eagle on a sandbox you need to run a HDP sandbox image in a virtual machine with 8GB memory recommended.
+ 1. Get Virtual Box or VMware [Virtualization environment](http://hortonworks.com/products/hortonworks-sandbox/#install)
+ 2. Get [Hortonworks Sandbox v 2.2.4](http://hortonworks.com/products/hortonworks-sandbox/#archive)
+* JDK 1.7
+* NPM (On MAC OS try "brew install node")
+<br/>
+
+### **Download + Patch + Build**
+* Download latest Eagle source released From Apache [[Tar]](http://www-us.apache.org/dist/incubator/eagle/apache-eagle-0.3.0-incubating/apache-eagle-0.3.0-incubating-src.tar.gz) , [[MD5]](http://www-us.apache.org/dist/incubator/eagle/apache-eagle-0.3.0-incubating/apache-eagle-0.3.0-incubating-src.tar.gz.md5)
+* Build manually with [Apache Maven](https://maven.apache.org/):
+
+ >$ tar -zxvf apache-eagle-0.3.0-incubating-src.tar.gz <br/>
+ >$ cd incubator-eagle-release-0.3.0-rc3 <br/>
+ >$ curl -O https://patch-diff.githubusercontent.com/raw/apache/incubator-eagle/pull/150.patch <br/>
+ >$ git apply 150.patch <br/>
+ >$ mvn clean package -DskipTests <br/>
+
+* After building successfully, you will get the tarball under `eagle-assembly/target/` named as `eagle-0.3.0-incubating-bin.tar.gz`
+<br/>
+
+### **Setup Hadoop Environment In Sandbox**
+1. Launch Ambari to manage the Hadoop environment
+ * Enable Ambari in sandbox http://127.0.0.1:8000 (Click on Enable Button)
+ * Login to Ambari UI http://127.0.0.1:8080/ with username and password as "admin"
+2. Grant root as HBase superuser via Ambari
+
+3. Start HBase,Storm & Kafka from Ambari. Showing Storm as an example below.
+
+4. If the NAT network is used in a virtual machine, it's required to add port 9099 to forwarding ports
+ 
+
+<br/>
+
+### **Stream HDFS audit logs into Kafka**
+
+**Note**: This section is only needed for "HDFS File System" Monitoring.
+
+* **Step 1**: Configure Advanced hadoop-log4j via <a href="http://localhost:8080/#/main/services/HDFS/configs" target="_blank">Ambari UI</a>, and add below "KAFKA_HDFS_AUDIT" log4j appender to hdfs audit logging.
+
+ log4j.appender.KAFKA_HDFS_AUDIT=org.apache.eagle.log4j.kafka.KafkaLog4jAppender
+ log4j.appender.KAFKA_HDFS_AUDIT.Topic=sandbox_hdfs_audit_log
+ log4j.appender.KAFKA_HDFS_AUDIT.BrokerList=sandbox.hortonworks.com:6667
+ log4j.appender.KAFKA_HDFS_AUDIT.KeyClass=org.apache.eagle.log4j.kafka.hadoop.AuditLogKeyer
+ log4j.appender.KAFKA_HDFS_AUDIT.Layout=org.apache.log4j.PatternLayout
+ log4j.appender.KAFKA_HDFS_AUDIT.Layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n
+ log4j.appender.KAFKA_HDFS_AUDIT.ProducerType=async
+
+ 
+
+* **Step 2**: Edit Advanced hadoop-env via <a href="http://localhost:8080/#/main/services/HDFS/configs" target="_blank">Ambari UI</a>, and add the reference to KAFKA_HDFS_AUDIT to HADOOP_NAMENODE_OPTS.
+
+ -Dhdfs.audit.logger=INFO,DRFAAUDIT,KAFKA_HDFS_AUDIT
+
+ 
+
+* **Step 3**: Edit Advanced hadoop-env via <a href="http://localhost:8080/#/main/services/HDFS/configs" target="_blank">Ambari UI</a>, and append the following command to it.
+
+ export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:/usr/hdp/current/eagle/lib/log4jkafka/lib/*
+
+ 
+
+* **Step 4**: save the changes
+
+* **Step 5**: "Restart All" Storm & Kafka from Ambari. (Similar to starting server in pervious step except make sure to click on "Restart All")
+
+* **Step 6**: Restart name node
+
+
+* **Step 7**: Check whether logs from are flowing into topic `sandbox_hdfs_audit_log`
+
+ > /usr/hdp/2.2.4.2-2/kafka/bin/kafka-console-consumer.sh --zookeeper sandbox.hortonworks.com:2181 --topic sandbox_hdfs_audit_log
+
+<br/>
+
+### **Install Eagle**
+The following installation actually contains installing and setting up a sandbox site with HdfsAuditLog & HiveQueryLog data sources.
+
+ >$ scp -P 2222 /eagle-assembly/target/eagle-0.3.0-incubating-bin.tar.gz root@127.0.0.1:/root/ <br/>
+ >$ ssh root@127.0.0.1 -p 2222 <br/>
+ >$ tar -zxvf eagle-0.3.0-incubating-bin.tar.gz <br/>
+ >$ mv eagle-0.3.0-incubating eagle <br/>
+ >$ mv eagle /usr/hdp/current/ <br/>
+ >$ cd /usr/hdp/current/eagle <br/>
+ >$ examples/eagle-sandbox-starter.sh <br/>
+
+<br/>
+
+### **Demos**
+* Login to Eagle UI [http://localhost:9099/eagle-service/](http://localhost:9099/eagle-service/) username and password as "admin" and "secret"
+* **HDFS**:
+ 1. Click on menu "DAM" and select "HDFS" to view HDFS policy
+ 2. You should see policy with name "viewPrivate". This Policy generates alert when any user reads HDFS file name "private" under "tmp" folder.
+ 3. In sandbox read restricted HDFS file "/tmp/private" by using command
+
+ > hadoop fs -cat "/tmp/private"
+
+ From UI click on alert tab and you should see alert for the attempt to read restricted file.
+* **Hive**:
+ 1. Click on menu "DAM" and select "Hive" to view Hive policy
+ 2. You should see policy with name "queryPhoneNumber". This Policy generates alert when hive table with sensitivity(Phone_Number) information is queried.
+ 3. In sandbox read restricted sensitive HIVE column.
+
+ >$ su hive <br/>
+ >$ hive <br/>
+ >$ set hive.execution.engine=mr; <br/>
+ >$ use xademo; <br/>
+ >$ select a.phone_number from customer_details a, call_detail_records b where a.phone_number=b.phone_number; <br/>
+
+ From UI click on alert tab and you should see alert for your attempt to dfsf read restricted column.
+
+<br/>