You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@eagle.apache.org by ha...@apache.org on 2017/11/22 06:00:06 UTC
[2/3] eagle git commit: Update site
http://git-wip-us.apache.org/repos/asf/eagle/blob/0094010b/_site/docs/installation.html
----------------------------------------------------------------------
diff --git a/_site/docs/installation.html b/_site/docs/installation.html
index adf5c4c..6d3b8b7 100644
--- a/_site/docs/installation.html
+++ b/_site/docs/installation.html
@@ -180,16 +180,19 @@
<h4 id="install-eagle">Install Eagle</h4>
<ul>
- <li>
- <p><strong>Step 1</strong>: Clone stable version from <a href="https://github.com/apache/eagle/releases/tag/v0.4.0-incubating">eagle github</a>
-> Build project mvn clean install -DskipTests=true</p>
+ <li><strong>Step 1</strong>: Clone stable version from <a href="https://github.com/apache/eagle/releases/tag/v0.4.0-incubating">eagle github</a>
+ <blockquote>
+ <div class="highlighter-rouge"><pre class="highlight"><code> Build project mvn clean install -DskipTests=true
+</code></pre>
+ </div>
+ </blockquote>
</li>
<li>
<p><strong>Step 2</strong>: Download eagle-bin-0.1.0.tar.gz package from successful build into your HDP sandbox.</p>
<ul>
<li>
- <p>Option 1: <code>scp -P 2222 eagle/eagle-assembly/target/eagle-0.1.0-bin.tar.gz root@127.0.0.1:/usr/hdp/current/</code></p>
+ <p>Option 1: <code class="highlighter-rouge">scp -P 2222 eagle/eagle-assembly/target/eagle-0.1.0-bin.tar.gz root@127.0.0.1:/usr/hdp/current/</code></p>
</li>
<li>
<p>Option 2: Create shared directory between host and Sandbox, and restart Sandbox. Then you can find the shared directory under /media in Sandbox.</p>
@@ -199,27 +202,30 @@
<li>
<p><strong>Step 3</strong>: Extract eagle tarball package</p>
- <pre><code>$ cd /usr/hdp/current
+ <div class="highlighter-rouge"><pre class="highlight"><code>$ cd /usr/hdp/current
$ tar -zxvf eagle-0.1.0-bin.tar.gz
$ mv eagle-0.1.0 eagle
</code></pre>
+ </div>
</li>
<li>
<p><strong>Step 4</strong>: Add root as a HBase<sup id="fnref:HBASE"><a href="#fn:HBASE" class="footnote">1</a></sup> superuser via <a href="http://127.0.0.1:8080/#/main/services/HBASE/configs">Ambari</a> (Optional, a user can operate HBase by sudo su hbase, as an alternative).</p>
</li>
- <li>
- <p><strong>Step 5</strong>: Install Eagle Ambari<sup id="fnref:AMBARI"><a href="#fn:AMBARI" class="footnote">2</a></sup> service
->
- /usr/hdp/current/eagle/bin/eagle-ambari.sh install.</p>
+ <li><strong>Step 5</strong>: Install Eagle Ambari<sup id="fnref:AMBARI"><a href="#fn:AMBARI" class="footnote">2</a></sup> service
+ <blockquote>
+
+ <p>/usr/hdp/current/eagle/bin/eagle-ambari.sh install.</p>
+ </blockquote>
</li>
<li>
<p><strong>Step 6</strong>: Restart <a href="http://127.0.0.1:8000/">Ambari</a> click on disable and enable Ambari back.</p>
</li>
- <li>
- <p><strong>Step 7</strong>: Start HBase & Storm<sup id="fnref:STORM"><a href="#fn:STORM" class="footnote">3</a></sup> & Kafka<sup id="fnref:KAFKA"><a href="#fn:KAFKA" class="footnote">4</a></sup>
-From Ambari UI, restart any suggested components(“Restart button on top”) & Start Storm (Start “Nimbus” ,”Supervisor” & “Storm UI Server”), Kafka (Start “Kafka Broker”) , HBase (Start “RegionServer” and “ HBase Master”)
->
-<img src="/images/docs/Services.png" alt="Restart Services" title="Services" /></p>
+ <li><strong>Step 7</strong>: Start HBase & Storm<sup id="fnref:STORM"><a href="#fn:STORM" class="footnote">3</a></sup> & Kafka<sup id="fnref:KAFKA"><a href="#fn:KAFKA" class="footnote">4</a></sup>
+From Ambari UI, restart any suggested components(“Restart button on top”) & Start Storm (Start “Nimbus” ,”Supervisor” & “Storm UI Server”), Kafka (Start “Kafka Broker”) , HBase (Start “RegionServer” and “ HBase Master”)
+ <blockquote>
+
+ <p><img src="/images/docs/Services.png" alt="Restart Services" title="Services" /></p>
+ </blockquote>
</li>
<li>
<p><strong>Step 8</strong>: Add Eagle Service To Ambari. (Click For Video)</p>
@@ -241,9 +247,10 @@ EagleServiceSuccess</p>
<li>
<p><strong>Step 9</strong>: Add Policies and meta data required by running below script.</p>
- <pre><code>$ /usr/hdp/current/eagle/examples/sample-sensitivity-resource-create.sh
+ <div class="highlighter-rouge"><pre class="highlight"><code>$ /usr/hdp/current/eagle/examples/sample-sensitivity-resource-create.sh
$ /usr/hdp/current/eagle/examples/sample-policy-create.sh
</code></pre>
+ </div>
</li>
</ul>
@@ -254,16 +261,16 @@ $ /usr/hdp/current/eagle/examples/sample-policy-create.sh
<div class="footnotes">
<ol>
<li id="fn:HBASE">
- <p><em>All mentions of “hbase” on this page represent Apache HBase.</em> <a href="#fnref:HBASE" class="reversefootnote">↩</a></p>
+ <p><em>All mentions of “hbase” on this page represent Apache HBase.</em> <a href="#fnref:HBASE" class="reversefootnote">↩</a></p>
</li>
<li id="fn:AMBARI">
- <p><em>All mentions of “ambari” on this page represent Apache Ambari.</em> <a href="#fnref:AMBARI" class="reversefootnote">↩</a></p>
+ <p><em>All mentions of “ambari” on this page represent Apache Ambari.</em> <a href="#fnref:AMBARI" class="reversefootnote">↩</a></p>
</li>
<li id="fn:STORM">
- <p><em>All mentions of “storm” on this page represent Apache Storm.</em> <a href="#fnref:STORM" class="reversefootnote">↩</a></p>
+ <p><em>All mentions of “storm” on this page represent Apache Storm.</em> <a href="#fnref:STORM" class="reversefootnote">↩</a></p>
</li>
<li id="fn:KAFKA">
- <p><em>All mentions of “kafka” on this page represent Apache Kafka.</em> <a href="#fnref:KAFKA" class="reversefootnote">↩</a></p>
+ <p><em>All mentions of “kafka” on this page represent Apache Kafka.</em> <a href="#fnref:KAFKA" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
http://git-wip-us.apache.org/repos/asf/eagle/blob/0094010b/_site/docs/jmx-metric-monitoring.html
----------------------------------------------------------------------
diff --git a/_site/docs/jmx-metric-monitoring.html b/_site/docs/jmx-metric-monitoring.html
index bd6b2d8..9574f54 100644
--- a/_site/docs/jmx-metric-monitoring.html
+++ b/_site/docs/jmx-metric-monitoring.html
@@ -177,9 +177,10 @@
<h3 id="setup"><strong>Setup</strong></h3>
<p>From Hortonworks sandbox just run below setup script to Install Pyton JMX script, Create Kafka topic, update Apache Hbase tables and deploy “hadoopjmx” Storm topology.</p>
-<pre><code>$ /usr/hdp/current/eagle/examples/hadoop-metric-sandbox-starter.sh
+<div class="highlighter-rouge"><pre class="highlight"><code>$ /usr/hdp/current/eagle/examples/hadoop-metric-sandbox-starter.sh
$ /usr/hdp/current/eagle/examples/hadoop-metric-policy-create.sh
</code></pre>
+</div>
<p><br /></p>
@@ -204,15 +205,17 @@ $ /usr/hdp/current/eagle/examples/hadoop-metric-policy-create.sh
<li>
<p>First make sure that Kafka topic “nn_jmx_metric_sandbox” is populated with JMX metric data periodically.(To make sure that python script is running)</p>
- <pre><code> $ /usr/hdp/2.2.4.2-2/kafka/bin/kafka-console-consumer.sh --zookeeper sandbox.hortonworks.com:2181 --topic nn_jmx_metric_sandbox
+ <div class="highlighter-rouge"><pre class="highlight"><code> $ /usr/hdp/2.2.4.2-2/kafka/bin/kafka-console-consumer.sh --zookeeper sandbox.hortonworks.com:2181 --topic nn_jmx_metric_sandbox
</code></pre>
+ </div>
</li>
<li>
<p>Genrate Alert by producing alert triggering message into Kafka topic.</p>
- <pre><code> $ /usr/hdp/2.2.4.2-2/kafka/bin/kafka-console-producer.sh --broker-list sandbox.hortonworks.com:6667 --topic nn_jmx_metric_sandbox
+ <div class="highlighter-rouge"><pre class="highlight"><code> $ /usr/hdp/2.2.4.2-2/kafka/bin/kafka-console-producer.sh --broker-list sandbox.hortonworks.com:6667 --topic nn_jmx_metric_sandbox
$ {"host": "localhost", "timestamp": 1457033916718, "metric": "hadoop.namenode.fsnamesystemstate.fsstate", "component": "namenode", "site": "sandbox", "value": 1.0}
</code></pre>
+ </div>
</li>
</ul>
@@ -223,10 +226,10 @@ $ /usr/hdp/current/eagle/examples/hadoop-metric-policy-create.sh
<div class="footnotes">
<ol>
<li id="fn:KAFKA">
- <p><em>All mentions of “kafka” on this page represent Apache Kafka.</em> <a href="#fnref:KAFKA" class="reversefootnote">↩</a></p>
+ <p><em>All mentions of “kafka” on this page represent Apache Kafka.</em> <a href="#fnref:KAFKA" class="reversefootnote">↩</a></p>
</li>
<li id="fn:STORM">
- <p><em>All mentions of “storm” on this page represent Apache Storm.</em> <a href="#fnref:STORM" class="reversefootnote">↩</a></p>
+ <p><em>All mentions of “storm” on this page represent Apache Storm.</em> <a href="#fnref:STORM" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
http://git-wip-us.apache.org/repos/asf/eagle/blob/0094010b/_site/docs/mapr-integration.html
----------------------------------------------------------------------
diff --git a/_site/docs/mapr-integration.html b/_site/docs/mapr-integration.html
index bb0a9dd..116d119 100644
--- a/_site/docs/mapr-integration.html
+++ b/_site/docs/mapr-integration.html
@@ -171,84 +171,97 @@
<p>Here are the steps to follow:</p>
<h4 id="step1-enable-audit-logs-for-filesystem-operations-and-table-operations-in-mapr">Step1: Enable audit logs for FileSystem Operations and Table Operations in MapR</h4>
-<p>First we need to enable data auditing at all three levels: cluster level, volume level and directory,file or table level.
-##### Cluster level:</p>
+<p>First we need to enable data auditing at all three levels: cluster level, volume level and directory,file or table level.</p>
+<h5 id="cluster-level">Cluster level:</h5>
-<pre><code> $ maprcli audit data -cluster <cluster name> -enabled true
+<div class="highlighter-rouge"><pre class="highlight"><code> $ maprcli audit data -cluster <cluster name> -enabled true
[ -maxsize <GB, defaut value is 32. When size of audit logs exceed this number, an alarm will be sent to the dashboard in the MapR Control Service > ]
[ -retention <number of Days> ]
</code></pre>
+</div>
<p>Example:</p>
-<pre><code> $ maprcli audit data -cluster mapr.cluster.com -enabled true -maxsize 30 -retention 30
+<div class="highlighter-rouge"><pre class="highlight"><code> $ maprcli audit data -cluster mapr.cluster.com -enabled true -maxsize 30 -retention 30
</code></pre>
+</div>
<h5 id="volume-level">Volume level:</h5>
-<pre><code> $ maprcli volume audit -cluster <cluster name> -enabled true
+<div class="highlighter-rouge"><pre class="highlight"><code> $ maprcli volume audit -cluster <cluster name> -enabled true
-name <volume name>
[ -coalesce <interval in minutes, the interval of time during which READ, WRITE, or GETATTR operations on one file from one client IP address are logged only once, if auditing is enabled> ]
</code></pre>
+</div>
<p>Example:</p>
-<pre><code> $ maprcli volume audit -cluster mapr.cluster.com -name mapr.tmp -enabled true
+<div class="highlighter-rouge"><pre class="highlight"><code> $ maprcli volume audit -cluster mapr.cluster.com -name mapr.tmp -enabled true
</code></pre>
+</div>
<p>To verify that auditing is enabled for a particular volume, use this command:</p>
-<pre><code> $ maprcli volume info -name <volume name> -json | grep -i 'audited\|coalesce'
+<div class="highlighter-rouge"><pre class="highlight"><code> $ maprcli volume info -name <volume name> -json | grep -i 'audited\|coalesce'
</code></pre>
+</div>
<p>and you should see something like this:</p>
-<pre><code> "audited":1,
+<div class="highlighter-rouge"><pre class="highlight"><code> "audited":1,
"coalesceInterval":60
</code></pre>
+</div>
<p>If “audited” is ‘1’ then auditing is enabled for this volume.</p>
<h5 id="directory-file-or-mapr-db-table-level">Directory, file, or MapR-DB table level:</h5>
-<pre><code> $ hadoop mfs -setaudit on <directory|file|table>
+<div class="highlighter-rouge"><pre class="highlight"><code> $ hadoop mfs -setaudit on <directory|file|table>
</code></pre>
+</div>
-<p>To check whether Auditing is Enabled for a Directory, File, or MapR-DB Table, use <code>$ hadoop mfs -ls</code>
+<p>To check whether Auditing is Enabled for a Directory, File, or MapR-DB Table, use <code class="highlighter-rouge">$ hadoop mfs -ls</code>
Example:
-Before enable the audit log on file <code>/tmp/dir</code>, try <code>$ hadoop mfs -ls /tmp/dir</code>, you should see something like this:</p>
+Before enable the audit log on file <code class="highlighter-rouge">/tmp/dir</code>, try <code class="highlighter-rouge">$ hadoop mfs -ls /tmp/dir</code>, you should see something like this:</p>
-<pre><code>drwxr-xr-x Z U U - root root 0 2016-03-02 15:02 268435456 /tmp/dir
+<div class="highlighter-rouge"><pre class="highlight"><code>drwxr-xr-x Z U U - root root 0 2016-03-02 15:02 268435456 /tmp/dir
p 2050.32.131328 mapr2.da.dg:5660 mapr1.da.dg:5660
</code></pre>
+</div>
-<p>The second <code>U</code> means auditing on this file is not enabled.
+<p>The second <code class="highlighter-rouge">U</code> means auditing on this file is not enabled.
Enable auditing with this command:</p>
-<pre><code>$ hadoop mfs -setaudit on /tmp/dir
+<div class="highlighter-rouge"><pre class="highlight"><code>$ hadoop mfs -setaudit on /tmp/dir
</code></pre>
+</div>
<p>Then check the auditing bit with :</p>
-<pre><code>$ hadoop mfs -ls /tmp/dir
+<div class="highlighter-rouge"><pre class="highlight"><code>$ hadoop mfs -ls /tmp/dir
</code></pre>
+</div>
<p>you should see something like this:</p>
-<pre><code>drwxr-xr-x Z U A - root root 0 2016-03-02 15:02 268435456 /tmp/dir
+<div class="highlighter-rouge"><pre class="highlight"><code>drwxr-xr-x Z U A - root root 0 2016-03-02 15:02 268435456 /tmp/dir
p 2050.32.131328 mapr2.da.dg:5660 mapr1.da.dg:5660
</code></pre>
+</div>
-<p>We can see the previous <code>U</code> has been changed to <code>A</code> which indicates auditing on this file is enabled.</p>
+<p>We can see the previous <code class="highlighter-rouge">U</code> has been changed to <code class="highlighter-rouge">A</code> which indicates auditing on this file is enabled.</p>
-<p><code>Important</code>:
+<p><code class="highlighter-rouge">Important</code>:
When a directory has been enabled auditing, directories/files located in this dir won’t inherit auditing, but a newly created file/dir (after enabling the auditing on this dir) in this directory will.</p>
<h4 id="step2-stream-log-data-into-kafka-by-using-logstash">Step2: Stream log data into Kafka by using Logstash</h4>
-<p>As MapR do not have name node, instead it use CLDB service, we have to use logstash to stream log data into Kafka.
-- First find out the nodes that have CLDB service
-- Then find out the location of audit log files, eg: <code>/mapr/mapr.cluster.com/var/mapr/local/mapr1.da.dg/audit/</code>, file names should be in this format: <code>FSAudit.log-2016-05-04-001.json</code>
-- Created a logstash conf file and run it, following this doc<a href="https://github.com/apache/eagle/blob/master/eagle-assembly/src/main/docs/logstash-kafka-conf.md">Logstash-kafka</a></p>
+<p>As MapR do not have name node, instead it use CLDB service, we have to use logstash to stream log data into Kafka.</p>
+<ul>
+ <li>First find out the nodes that have CLDB service</li>
+ <li>Then find out the location of audit log files, eg: <code class="highlighter-rouge">/mapr/mapr.cluster.com/var/mapr/local/mapr1.da.dg/audit/</code>, file names should be in this format: <code class="highlighter-rouge">FSAudit.log-2016-05-04-001.json</code></li>
+ <li>Created a logstash conf file and run it, following this doc<a href="https://github.com/apache/eagle/blob/master/eagle-assembly/src/main/docs/logstash-kafka-conf.md">Logstash-kafka</a></li>
+</ul>
<h4 id="step3-set-up-maprfsauditlog-applicaiton-in-eagle-service">Step3: Set up maprFSAuditLog applicaiton in Eagle Service</h4>
-<p>After Eagle Service gets started, create mapFSAuditLog application using: <code>$ ./maprFSAuditLog-init.sh</code>. By default it will create maprFSAuditLog in site “sandbox”, you may need to change it to your own site.
+<p>After Eagle Service gets started, create mapFSAuditLog application using: <code class="highlighter-rouge">$ ./maprFSAuditLog-init.sh</code>. By default it will create maprFSAuditLog in site “sandbox”, you may need to change it to your own site.
After these steps you are good to go.</p>
<p>Have fun!!! :)</p>
@@ -266,7 +279,7 @@ After these steps you are good to go.</p>
<div class="footnotes">
<ol>
<li id="fn:KAFKA">
- <p><em>All mentions of “kafka” on this page represent Apache Kafka.</em> <a href="#fnref:KAFKA" class="reversefootnote">↩</a></p>
+ <p><em>All mentions of “kafka” on this page represent Apache Kafka.</em> <a href="#fnref:KAFKA" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
http://git-wip-us.apache.org/repos/asf/eagle/blob/0094010b/_site/docs/quick-start-0.3.0.html
----------------------------------------------------------------------
diff --git a/_site/docs/quick-start-0.3.0.html b/_site/docs/quick-start-0.3.0.html
index 4d048fc..1991131 100644
--- a/_site/docs/quick-start-0.3.0.html
+++ b/_site/docs/quick-start-0.3.0.html
@@ -178,21 +178,22 @@
<li>
<p>Build manually with <a href="https://maven.apache.org/">Apache Maven</a>:</p>
- <pre><code>$ tar -zxvf apache-eagle-0.3.0-incubating-src.tar.gz
+ <div class="highlighter-rouge"><pre class="highlight"><code>$ tar -zxvf apache-eagle-0.3.0-incubating-src.tar.gz
$ cd incubator-eagle-release-0.3.0-rc3
$ curl -O https://patch-diff.githubusercontent.com/raw/apache/eagle/pull/180.patch
$ git apply 180.patch
$ mvn clean package -DskipTests
</code></pre>
+ </div>
- <p>After building successfully, you will get tarball under <code>eagle-assembly/target/</code> named as <code>eagle-0.3.0-incubating-bin.tar.gz</code>
+ <p>After building successfully, you will get tarball under <code class="highlighter-rouge">eagle-assembly/target/</code> named as <code class="highlighter-rouge">eagle-0.3.0-incubating-bin.tar.gz</code>
<br /></p>
</li>
</ul>
<h3 id="install-eagle"><strong>Install Eagle</strong></h3>
-<pre><code> $ scp -P 2222 eagle-assembly/target/eagle-0.3.0-incubating-bin.tar.gz root@127.0.0.1:/root/
+<div class="highlighter-rouge"><pre class="highlight"><code> $ scp -P 2222 eagle-assembly/target/eagle-0.3.0-incubating-bin.tar.gz root@127.0.0.1:/root/
$ ssh root@127.0.0.1 -p 2222 (password is hadoop)
$ tar -zxvf eagle-0.3.0-incubating-bin.tar.gz
$ mv eagle-0.3.0-incubating eagle
@@ -200,6 +201,7 @@ $ mvn clean package -DskipTests
$ cd /usr/hdp/current/eagle
$ examples/eagle-sandbox-starter.sh
</code></pre>
+</div>
<p><br /></p>
@@ -217,7 +219,7 @@ $ mvn clean package -DskipTests
<div class="footnotes">
<ol>
<li id="fn:HADOOP">
- <p><em>All mentions of “hadoop” on this page represent Apache Hadoop.</em> <a href="#fnref:HADOOP" class="reversefootnote">↩</a></p>
+ <p><em>All mentions of “hadoop” on this page represent Apache Hadoop.</em> <a href="#fnref:HADOOP" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
http://git-wip-us.apache.org/repos/asf/eagle/blob/0094010b/_site/docs/quick-start.html
----------------------------------------------------------------------
diff --git a/_site/docs/quick-start.html b/_site/docs/quick-start.html
index 6575409..18ae834 100644
--- a/_site/docs/quick-start.html
+++ b/_site/docs/quick-start.html
@@ -179,21 +179,22 @@
<li>
<p>Build manually with <a href="https://maven.apache.org/">Apache Maven</a>:</p>
- <pre><code>$ tar -zxvf apache-eagle-0.4.0-incubating-src.tar.gz
+ <div class="highlighter-rouge"><pre class="highlight"><code>$ tar -zxvf apache-eagle-0.4.0-incubating-src.tar.gz
$ cd apache-eagle-0.4.0-incubating-src
$ curl -O https://patch-diff.githubusercontent.com/raw/apache/eagle/pull/268.patch
$ git apply 268.patch
$ mvn clean package -DskipTests
</code></pre>
+ </div>
- <p>After building successfully, you will get a tarball under <code>eagle-assembly/target/</code> named <code>apache-eagle-0.4.0-incubating-bin.tar.gz</code>
+ <p>After building successfully, you will get a tarball under <code class="highlighter-rouge">eagle-assembly/target/</code> named <code class="highlighter-rouge">apache-eagle-0.4.0-incubating-bin.tar.gz</code>
<br /></p>
</li>
</ul>
<h3 id="install-eagle"><strong>Install Eagle</strong></h3>
-<pre><code> $ scp -P 2222 eagle-assembly/target/apache-eagle-0.4.0-incubating-bin.tar.gz root@127.0.0.1:/root/
+<div class="highlighter-rouge"><pre class="highlight"><code> $ scp -P 2222 eagle-assembly/target/apache-eagle-0.4.0-incubating-bin.tar.gz root@127.0.0.1:/root/
$ ssh root@127.0.0.1 -p 2222 (password is hadoop)
$ tar -zxvf apache-eagle-0.4.0-incubating-bin.tar.gz
$ mv apache-eagle-0.4.0-incubating eagle
@@ -201,20 +202,22 @@ $ mvn clean package -DskipTests
$ cd /usr/hdp/current/eagle
$ examples/eagle-sandbox-starter.sh
</code></pre>
+</div>
<p><br /></p>
<h3 id="sample-application-hive-query-activity-monitoring-in-sandbox"><strong>Sample Application: Hive query activity monitoring in sandbox</strong></h3>
-<p>After executing <code>examples/eagle-sandbox-starter.sh</code>, you have a sample application (topology) running on the Apache Storm (check with <a href="http://sandbox.hortonworks.com:8744/index.html">storm ui</a>), and a sample policy of Hive activity monitoring defined.</p>
+<p>After executing <code class="highlighter-rouge">examples/eagle-sandbox-starter.sh</code>, you have a sample application (topology) running on the Apache Storm (check with <a href="http://sandbox.hortonworks.com:8744/index.html">storm ui</a>), and a sample policy of Hive activity monitoring defined.</p>
<p>Next you can trigger an alert by running a Hive query.</p>
-<pre><code>$ su hive
+<div class="highlighter-rouge"><pre class="highlight"><code>$ su hive
$ hive
$ set hive.execution.engine=mr;
$ use xademo;
$ select a.phone_number from customer_details a, call_detail_records b where a.phone_number=b.phone_number;
</code></pre>
+</div>
<p><br /></p>
<hr />
@@ -224,10 +227,10 @@ $ select a.phone_number from customer_details a, call_detail_records b where a.p
<div class="footnotes">
<ol>
<li id="fn:HADOOP">
- <p><em>Apache Hadoop.</em> <a href="#fnref:HADOOP" class="reversefootnote">↩</a></p>
+ <p><em>Apache Hadoop.</em> <a href="#fnref:HADOOP" class="reversefootnote">↩</a></p>
</li>
<li id="fn:HIVE">
- <p><em>All mentions of “hive” on this page represent Apache Hive.</em> <a href="#fnref:HIVE" class="reversefootnote">↩</a></p>
+ <p><em>All mentions of “hive” on this page represent Apache Hive.</em> <a href="#fnref:HIVE" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
http://git-wip-us.apache.org/repos/asf/eagle/blob/0094010b/_site/docs/security.html
----------------------------------------------------------------------
diff --git a/_site/docs/security.html b/_site/docs/security.html
index 0fc1599..34794a1 100644
--- a/_site/docs/security.html
+++ b/_site/docs/security.html
@@ -158,7 +158,7 @@
<h1 class="page-header" style="margin-top: 0px">Apache Eagle Security</h1>
<p>The Apache Software Foundation takes a very active stance in eliminating security problems in its software products. Apache Eagle is also responsive to such issues around its features.</p>
-<p>If you have any concern regarding to Eagle’s Security or you believe a vulnerability is discovered, don’t hesitate to get connected with Aapche Security Team by sending emails to <a href="mailto:security@apache.org">security@apache.org</a>. In the message, you can indicate the project name is Eagle, provide a description of the issue, and you are recommended to give the way of reproducing it. The security team and eagle community will get back to you after assessing the findings.</p>
+<p>If you have any concern regarding to Eagle’s Security or you believe a vulnerability is discovered, don’t hesitate to get connected with Aapche Security Team by sending emails to <a href="mailto:security@apache.org">security@apache.org</a>. In the message, you can indicate the project name is Eagle, provide a description of the issue, and you are recommended to give the way of reproducing it. The security team and eagle community will get back to you after assessing the findings.</p>
<blockquote>
<p><strong>PLEASE PAY ATTENTION</strong> to report any security problem to the security email address before disclosing it publicly.</p>
http://git-wip-us.apache.org/repos/asf/eagle/blob/0094010b/_site/docs/serviceconfiguration.html
----------------------------------------------------------------------
diff --git a/_site/docs/serviceconfiguration.html b/_site/docs/serviceconfiguration.html
index 2519ae8..3ecd741 100644
--- a/_site/docs/serviceconfiguration.html
+++ b/_site/docs/serviceconfiguration.html
@@ -171,7 +171,7 @@ description of Eagle Service configuration.</p>
<li>for hbase</li>
</ul>
-<pre><code>eagle {
+<div class="highlighter-rouge"><pre class="highlight"><code>eagle {
service{
storage-type="hbase"
hbase-zookeeper-quorum="sandbox.hortonworks.com"
@@ -182,12 +182,13 @@ description of Eagle Service configuration.</p>
}
}
</code></pre>
+</div>
<ul>
<li>for mysql</li>
</ul>
-<pre><code>eagle {
+<div class="highlighter-rouge"><pre class="highlight"><code>eagle {
service {
storage-type="jdbc"
storage-adapter="mysql"
@@ -201,12 +202,13 @@ description of Eagle Service configuration.</p>
}
}
</code></pre>
+</div>
<ul>
<li>for derby</li>
</ul>
-<pre><code>eagle {
+<div class="highlighter-rouge"><pre class="highlight"><code>eagle {
service {
storage-type="jdbc"
storage-adapter="derby"
@@ -220,6 +222,7 @@ description of Eagle Service configuration.</p>
}
}
</code></pre>
+</div>
<p><br /></p>
</div><!--end of loadcontent-->
http://git-wip-us.apache.org/repos/asf/eagle/blob/0094010b/_site/docs/terminology.html
----------------------------------------------------------------------
diff --git a/_site/docs/terminology.html b/_site/docs/terminology.html
index 1ee3156..27c99ca 100644
--- a/_site/docs/terminology.html
+++ b/_site/docs/terminology.html
@@ -193,10 +193,10 @@ They are basic knowledge of Eagle which also will help to well understand Eagle.
<div class="footnotes">
<ol>
<li id="fn:HADOOP">
- <p><em>All mentions of “hadoop” on this page represent Apache Hadoop.</em> <a href="#fnref:HADOOP" class="reversefootnote">↩</a></p>
+ <p><em>All mentions of “hadoop” on this page represent Apache Hadoop.</em> <a href="#fnref:HADOOP" class="reversefootnote">↩</a></p>
</li>
<li id="fn:HIVE">
- <p><em>Apache Hive.</em> <a href="#fnref:HIVE" class="reversefootnote">↩</a></p>
+ <p><em>Apache Hive.</em> <a href="#fnref:HIVE" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
http://git-wip-us.apache.org/repos/asf/eagle/blob/0094010b/_site/docs/tutorial/classification.html
----------------------------------------------------------------------
diff --git a/_site/docs/tutorial/classification.html b/_site/docs/tutorial/classification.html
index e79dc14..64e6c49 100644
--- a/_site/docs/tutorial/classification.html
+++ b/_site/docs/tutorial/classification.html
@@ -182,30 +182,33 @@ Currently this feature is available ONLY for applications monitoring HDFS, Hive<
<p>You may configure the default path for Apache Hadoop clients to connect remote hdfs namenode.</p>
- <pre><code> classification.fs.defaultFS=hdfs://sandbox.hortonworks.com:8020
+ <div class="highlighter-rouge"><pre class="highlight"><code> classification.fs.defaultFS=hdfs://sandbox.hortonworks.com:8020
</code></pre>
+ </div>
</li>
<li>
<p>HA case</p>
<p>Basically, you point your fs.defaultFS at your nameservice and let the client know how its configured (the backing namenodes) and how to fail over between them under the HA mode</p>
- <pre><code> classification.fs.defaultFS=hdfs://nameservice1
+ <div class="highlighter-rouge"><pre class="highlight"><code> classification.fs.defaultFS=hdfs://nameservice1
classification.dfs.nameservices=nameservice1
classification.dfs.ha.namenodes.nameservice1=namenode1,namenode2
classification.dfs.namenode.rpc-address.nameservice1.namenode1=hadoopnamenode01:8020
classification.dfs.namenode.rpc-address.nameservice1.namenode2=hadoopnamenode02:8020
classification.dfs.client.failover.proxy.provider.nameservice1=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
</code></pre>
+ </div>
</li>
<li>
<p>Kerberos-secured cluster</p>
<p>For Kerberos-secured cluster, you need to get a keytab file and the principal from your admin, and configure “eagle.keytab.file” and “eagle.kerberos.principal” to authenticate its access.</p>
- <pre><code> classification.eagle.keytab.file=/EAGLE-HOME/.keytab/eagle.keytab
+ <div class="highlighter-rouge"><pre class="highlight"><code> classification.eagle.keytab.file=/EAGLE-HOME/.keytab/eagle.keytab
classification.eagle.kerberos.principal=eagle@SOMEWHERE.COM
</code></pre>
+ </div>
<p>If there is an exception about “invalid server principal name”, you may need to check the DNS resolver, or the data transfer , such as “dfs.encrypt.data.transfer”, “dfs.encrypt.data.transfer.algorithm”, “dfs.trustedchannel.resolver.class”, “dfs.datatransfer.client.encrypt”.</p>
</li>
@@ -216,12 +219,13 @@ Currently this feature is available ONLY for applications monitoring HDFS, Hive<
<li>
<p>Basic</p>
- <pre><code> classification.accessType=metastoredb_jdbc
+ <div class="highlighter-rouge"><pre class="highlight"><code> classification.accessType=metastoredb_jdbc
classification.password=hive
classification.user=hive
classification.jdbcDriverClassName=com.mysql.jdbc.Driver
classification.jdbcUrl=jdbc:mysql://sandbox.hortonworks.com/hive?createDatabaseIfNotExist=true
</code></pre>
+ </div>
</li>
</ul>
</li>
@@ -234,16 +238,17 @@ Currently this feature is available ONLY for applications monitoring HDFS, Hive<
<p>You need to sett “hbase.zookeeper.quorum”:”localhost” property and “hbase.zookeeper.property.clientPort” property.</p>
- <pre><code> classification.hbase.zookeeper.property.clientPort=2181
+ <div class="highlighter-rouge"><pre class="highlight"><code> classification.hbase.zookeeper.property.clientPort=2181
classification.hbase.zookeeper.quorum=localhost
</code></pre>
+ </div>
</li>
<li>
<p>Kerberos-secured cluster</p>
<p>According to your environment, you can add or remove some of the following properties. Here is the reference.</p>
- <pre><code> classification.hbase.zookeeper.property.clientPort=2181
+ <div class="highlighter-rouge"><pre class="highlight"><code> classification.hbase.zookeeper.property.clientPort=2181
classification.hbase.zookeeper.quorum=localhost
classification.hbase.security.authentication=kerberos
classification.hbase.master.kerberos.principal=hadoop/_HOST@EXAMPLE.COM
@@ -251,6 +256,7 @@ Currently this feature is available ONLY for applications monitoring HDFS, Hive<
classification.eagle.keytab.file=/EAGLE-HOME/.keytab/eagle.keytab
classification.eagle.kerberos.principal=eagle@EXAMPLE.COM
</code></pre>
+ </div>
</li>
</ul>
</li>
@@ -321,10 +327,10 @@ Currently this feature is available ONLY for applications monitoring HDFS, Hive<
<div class="footnotes">
<ol>
<li id="fn:HIVE">
- <p><em>All mentions of “hive” on this page represent Apache Hive.</em> <a href="#fnref:HIVE" class="reversefootnote">↩</a></p>
+ <p><em>All mentions of “hive” on this page represent Apache Hive.</em> <a href="#fnref:HIVE" class="reversefootnote">↩</a></p>
</li>
<li id="fn:HBASE">
- <p><em>All mentions of “hbase” on this page represent Apache HBase.</em> <a href="#fnref:HBASE" class="reversefootnote">↩</a></p>
+ <p><em>All mentions of “hbase” on this page represent Apache HBase.</em> <a href="#fnref:HBASE" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
http://git-wip-us.apache.org/repos/asf/eagle/blob/0094010b/_site/docs/tutorial/ldap.html
----------------------------------------------------------------------
diff --git a/_site/docs/tutorial/ldap.html b/_site/docs/tutorial/ldap.html
index 7cc814e..918fe8c 100644
--- a/_site/docs/tutorial/ldap.html
+++ b/_site/docs/tutorial/ldap.html
@@ -160,7 +160,7 @@
<p>Step 1: edit configuration under conf/ldap.properties.</p>
-<pre><code>ldap.server=ldap://localhost:10389
+<div class="highlighter-rouge"><pre class="highlight"><code>ldap.server=ldap://localhost:10389
ldap.username=uid=admin,ou=system
ldap.password=secret
ldap.user.searchBase=ou=Users,o=mojo
@@ -169,12 +169,13 @@ ldap.user.groupSearchBase=ou=groups,o=mojo
acl.adminRole=
acl.defaultRole=ROLE_USER
</code></pre>
+</div>
<p>acl.adminRole and acl.defaultRole are two customized properties for Eagle. Eagle manages admin users with groups. If you set acl.adminRole as ROLE_{EAGLE-ADMIN-GROUP-NAME}, members in this group have the admin privilege. acl.defaultRole is ROLE_USER.</p>
<p>Step 2: edit conf/eagle-service.conf, and add springActiveProfile=”default”</p>
-<pre><code>eagle{
+<div class="highlighter-rouge"><pre class="highlight"><code>eagle{
service{
storage-type="hbase"
hbase-zookeeper-quorum="localhost"
@@ -184,6 +185,7 @@ acl.defaultRole=ROLE_USER
}
}
</code></pre>
+</div>
</div><!--end of loadcontent-->
http://git-wip-us.apache.org/repos/asf/eagle/blob/0094010b/_site/docs/tutorial/notificationplugin.html
----------------------------------------------------------------------
diff --git a/_site/docs/tutorial/notificationplugin.html b/_site/docs/tutorial/notificationplugin.html
index 627b7b1..db26f69 100644
--- a/_site/docs/tutorial/notificationplugin.html
+++ b/_site/docs/tutorial/notificationplugin.html
@@ -183,12 +183,12 @@
</li>
</ul>
-<p><img src="/images/notificationPlugin.png" alt="notificationPlugin" />
-### Customized Notification Plugin</p>
+<p><img src="/images/notificationPlugin.png" alt="notificationPlugin" /></p>
+<h3 id="customized-notification-plugin">Customized Notification Plugin</h3>
<p>To integrate a customized notification plugin, we must implement an interface</p>
-<pre><code>public interface NotificationPlugin {
+<div class="highlighter-rouge"><pre class="highlight"><code>public interface NotificationPlugin {
/**
* for initialization
* @throws Exception
@@ -218,24 +218,26 @@ void onAlert(AlertAPIEntity alertEntity) throws Exception;
List<NotificationStatus> getStatusList();
} Examples: AlertKafkaPlugin, AlertEmailPlugin, and AlertEagleStorePlugin.
</code></pre>
+</div>
<p>The second and crucial step is to register the configurations of the customized plugin. In other words, we need persist the configuration template into database in order to expose the configurations to users in the front end.</p>
<p>Examples:</p>
-<pre><code>{
- "prefix": "alertNotifications",
- "tags": {
- "notificationType": "kafka"
- },
- "className": "org.apache.eagle.notification.plugin.AlertKafkaPlugin",
- "description": "send alert to kafka bus",
- "enabled":true,
- "fields": "[{\"name\":\"kafka_broker\",\"value\":\"sandbox.hortonworks.com:6667\"},{\"name\":\"topic\"}]"
-}
-</code></pre>
+<div class="highlighter-rouge"><pre class="highlight"><code><span class="p">{</span><span class="w">
+ </span><span class="nt">"prefix"</span><span class="p">:</span><span class="w"> </span><span class="s2">"alertNotifications"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"tags"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"notificationType"</span><span class="p">:</span><span class="w"> </span><span class="s2">"kafka"</span><span class="w">
+ </span><span class="p">},</span><span class="w">
+ </span><span class="nt">"className"</span><span class="p">:</span><span class="w"> </span><span class="s2">"org.apache.eagle.notification.plugin.AlertKafkaPlugin"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"description"</span><span class="p">:</span><span class="w"> </span><span class="s2">"send alert to kafka bus"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"enabled"</span><span class="p">:</span><span class="kc">true</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"fields"</span><span class="p">:</span><span class="w"> </span><span class="s2">"[{\"name\":\"kafka_broker\",\"value\":\"sandbox.hortonworks.com:6667\"},{\"name\":\"topic\"}]"</span><span class="w">
+</span><span class="p">}</span><span class="w">
+</span></code></pre>
+</div>
-<p><strong>Note</strong>: <code>fields</code> is the configuration for notification type <code>kafka</code></p>
+<p><strong>Note</strong>: <code class="highlighter-rouge">fields</code> is the configuration for notification type <code class="highlighter-rouge">kafka</code></p>
<p>How can we do that? <a href="https://github.com/apache/eagle/blob/master/eagle-assembly/src/main/bin/eagle-topology-init.sh">Here</a> are Eagle other notification plugin configurations. Just append yours to it, and run this script when Eagle service is up.</p>
@@ -246,7 +248,7 @@ List<NotificationStatus> getStatusList();
<div class="footnotes">
<ol>
<li id="fn:KAFKA">
- <p><em>All mentions of “kafka” on this page represent Apache Kafka.</em> <a href="#fnref:KAFKA" class="reversefootnote">↩</a></p>
+ <p><em>All mentions of “kafka” on this page represent Apache Kafka.</em> <a href="#fnref:KAFKA" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
http://git-wip-us.apache.org/repos/asf/eagle/blob/0094010b/_site/docs/tutorial/policy.html
----------------------------------------------------------------------
diff --git a/_site/docs/tutorial/policy.html b/_site/docs/tutorial/policy.html
index e732eb0..657cf2d 100644
--- a/_site/docs/tutorial/policy.html
+++ b/_site/docs/tutorial/policy.html
@@ -183,12 +183,13 @@
<li>
<p><strong>Step 2</strong>: Eagle supports a variety of properties for match critera where users can set different values. Eagle also supports window functions to extend policies with time functions.</p>
- <pre><code>command = delete
+ <div class="highlighter-rouge"><pre class="highlight"><code>command = delete
(Eagle currently supports the following commands open, delete, copy, append, copy from local, get, move, mkdir, create, list, change permissions)
source = /tmp/private
(Eagle supports wildcarding for property values for example /tmp/*)
</code></pre>
+ </div>
<p><img src="/images/docs/hdfs-policy2.png" alt="HDFS Policies" /></p>
</li>
@@ -215,12 +216,13 @@ source = /tmp/private
<li>
<p><strong>Step 2</strong>: Eagle support a variety of properties for match critera where users can set different values. Eagle also supports window functions to extend policies with time functions.</p>
- <pre><code>command = Select
+ <div class="highlighter-rouge"><pre class="highlight"><code>command = Select
(Eagle currently supports the following commands DDL statements Create, Drop, Alter, Truncate, Show)
sensitivity type = PHONE_NUMBER
(Eagle supports classifying data in Hive with different sensitivity types. Users can use these sensitivity types to create policies)
</code></pre>
+ </div>
<p><img src="/images/docs/hive-policy2.png" alt="Hive Policies" /></p>
</li>
@@ -238,7 +240,7 @@ sensitivity type = PHONE_NUMBER
<div class="footnotes">
<ol>
<li id="fn:HIVE">
- <p><em>All mentions of “hive” on this page represent Apache Hive.</em> <a href="#fnref:HIVE" class="reversefootnote">↩</a></p>
+ <p><em>All mentions of “hive” on this page represent Apache Hive.</em> <a href="#fnref:HIVE" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
http://git-wip-us.apache.org/repos/asf/eagle/blob/0094010b/_site/docs/tutorial/site-0.3.0.html
----------------------------------------------------------------------
diff --git a/_site/docs/tutorial/site-0.3.0.html b/_site/docs/tutorial/site-0.3.0.html
index a542128..45784d9 100644
--- a/_site/docs/tutorial/site-0.3.0.html
+++ b/_site/docs/tutorial/site-0.3.0.html
@@ -180,32 +180,35 @@ Here we give configuration examples for HDFS, HBASE, and Hive.</p>
<p>You may configure the default path for Hadoop clients to connect remote hdfs namenode.</p>
- <pre><code> {"fs.defaultFS":"hdfs://sandbox.hortonworks.com:8020"}
-</code></pre>
+ <div class="highlighter-rouge"><pre class="highlight"><code><span class="w"> </span><span class="p">{</span><span class="nt">"fs.defaultFS"</span><span class="p">:</span><span class="s2">"hdfs://sandbox.hortonworks.com:8020"</span><span class="p">}</span><span class="w">
+</span></code></pre>
+ </div>
</li>
<li>
<p>HA case</p>
<p>Basically, you point your fs.defaultFS at your nameservice and let the client know how its configured (the backing namenodes) and how to fail over between them under the HA mode</p>
- <pre><code> {"fs.defaultFS":"hdfs://nameservice1",
- "dfs.nameservices": "nameservice1",
- "dfs.ha.namenodes.nameservice1":"namenode1,namenode2",
- "dfs.namenode.rpc-address.nameservice1.namenode1": "hadoopnamenode01:8020",
- "dfs.namenode.rpc-address.nameservice1.namenode2": "hadoopnamenode02:8020",
- "dfs.client.failover.proxy.provider.nameservice1": "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider"
- }
-</code></pre>
+ <div class="highlighter-rouge"><pre class="highlight"><code><span class="w"> </span><span class="p">{</span><span class="nt">"fs.defaultFS"</span><span class="p">:</span><span class="s2">"hdfs://nameservice1"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"dfs.nameservices"</span><span class="p">:</span><span class="w"> </span><span class="s2">"nameservice1"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"dfs.ha.namenodes.nameservice1"</span><span class="p">:</span><span class="s2">"namenode1,namenode2"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"dfs.namenode.rpc-address.nameservice1.namenode1"</span><span class="p">:</span><span class="w"> </span><span class="s2">"hadoopnamenode01:8020"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"dfs.namenode.rpc-address.nameservice1.namenode2"</span><span class="p">:</span><span class="w"> </span><span class="s2">"hadoopnamenode02:8020"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"dfs.client.failover.proxy.provider.nameservice1"</span><span class="p">:</span><span class="w"> </span><span class="s2">"org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider"</span><span class="w">
+ </span><span class="p">}</span><span class="w">
+</span></code></pre>
+ </div>
</li>
<li>
<p>Kerberos-secured cluster</p>
<p>For Kerberos-secured cluster, you need to get a keytab file and the principal from your admin, and configure “eagle.keytab.file” and “eagle.kerberos.principal” to authenticate its access.</p>
- <pre><code> { "eagle.keytab.file":"/EAGLE-HOME/.keytab/eagle.keytab",
- "eagle.kerberos.principal":"eagle@SOMEWHERE.COM"
- }
-</code></pre>
+ <div class="highlighter-rouge"><pre class="highlight"><code><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"eagle.keytab.file"</span><span class="p">:</span><span class="s2">"/EAGLE-HOME/.keytab/eagle.keytab"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"eagle.kerberos.principal"</span><span class="p">:</span><span class="s2">"eagle@SOMEWHERE.COM"</span><span class="w">
+ </span><span class="p">}</span><span class="w">
+</span></code></pre>
+ </div>
<p>If there is an exception about “invalid server principal name”, you may need to check the DNS resolver, or the data transfer , such as “dfs.encrypt.data.transfer”, “dfs.encrypt.data.transfer.algorithm”, “dfs.trustedchannel.resolver.class”, “dfs.datatransfer.client.encrypt”.</p>
</li>
@@ -216,14 +219,15 @@ Here we give configuration examples for HDFS, HBASE, and Hive.</p>
<li>
<p>Basic</p>
- <pre><code> {
- "accessType": "metastoredb_jdbc",
- "password": "hive",
- "user": "hive",
- "jdbcDriverClassName": "com.mysql.jdbc.Driver",
- "jdbcUrl": "jdbc:mysql://sandbox.hortonworks.com/hive?createDatabaseIfNotExist=true"
- }
-</code></pre>
+ <div class="highlighter-rouge"><pre class="highlight"><code><span class="w"> </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"accessType"</span><span class="p">:</span><span class="w"> </span><span class="s2">"metastoredb_jdbc"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"password"</span><span class="p">:</span><span class="w"> </span><span class="s2">"hive"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"user"</span><span class="p">:</span><span class="w"> </span><span class="s2">"hive"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"jdbcDriverClassName"</span><span class="p">:</span><span class="w"> </span><span class="s2">"com.mysql.jdbc.Driver"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"jdbcUrl"</span><span class="p">:</span><span class="w"> </span><span class="s2">"jdbc:mysql://sandbox.hortonworks.com/hive?createDatabaseIfNotExist=true"</span><span class="w">
+ </span><span class="p">}</span><span class="w">
+</span></code></pre>
+ </div>
</li>
</ul>
</li>
@@ -236,27 +240,29 @@ Here we give configuration examples for HDFS, HBASE, and Hive.</p>
<p>You need to sett “hbase.zookeeper.quorum”:”localhost” property and “hbase.zookeeper.property.clientPort” property.</p>
- <pre><code> {
- "hbase.zookeeper.property.clientPort":"2181",
- "hbase.zookeeper.quorum":"localhost"
- }
-</code></pre>
+ <div class="highlighter-rouge"><pre class="highlight"><code><span class="w"> </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"hbase.zookeeper.property.clientPort"</span><span class="p">:</span><span class="s2">"2181"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"hbase.zookeeper.quorum"</span><span class="p">:</span><span class="s2">"localhost"</span><span class="w">
+ </span><span class="p">}</span><span class="w">
+</span></code></pre>
+ </div>
</li>
<li>
<p>Kerberos-secured cluster</p>
<p>According to your environment, you can add or remove some of the following properties. Here is the reference.</p>
- <pre><code> {
- "hbase.zookeeper.property.clientPort":"2181",
- "hbase.zookeeper.quorum":"localhost",
- "hbase.security.authentication":"kerberos",
- "hbase.master.kerberos.principal":"hadoop/_HOST@EXAMPLE.COM",
- "zookeeper.znode.parent":"/hbase",
- "eagle.keytab.file":"/EAGLE-HOME/.keytab/eagle.keytab",
- "eagle.kerberos.principal":"eagle@EXAMPLE.COM"
- }
-</code></pre>
+ <div class="highlighter-rouge"><pre class="highlight"><code><span class="w"> </span><span class="p">{</span><span class="w">
+ </span><span class="nt">"hbase.zookeeper.property.clientPort"</span><span class="p">:</span><span class="s2">"2181"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"hbase.zookeeper.quorum"</span><span class="p">:</span><span class="s2">"localhost"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"hbase.security.authentication"</span><span class="p">:</span><span class="s2">"kerberos"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"hbase.master.kerberos.principal"</span><span class="p">:</span><span class="s2">"hadoop/_HOST@EXAMPLE.COM"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"zookeeper.znode.parent"</span><span class="p">:</span><span class="s2">"/hbase"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"eagle.keytab.file"</span><span class="p">:</span><span class="s2">"/EAGLE-HOME/.keytab/eagle.keytab"</span><span class="p">,</span><span class="w">
+ </span><span class="nt">"eagle.kerberos.principal"</span><span class="p">:</span><span class="s2">"eagle@EXAMPLE.COM"</span><span class="w">
+ </span><span class="p">}</span><span class="w">
+</span></code></pre>
+ </div>
</li>
</ul>
</li>
@@ -274,13 +280,13 @@ Here we give configuration examples for HDFS, HBASE, and Hive.</p>
<div class="footnotes">
<ol>
<li id="fn:HADOOP">
- <p><em>All mentions of “hadoop” on this page represent Apache Hadoop.</em> <a href="#fnref:HADOOP" class="reversefootnote">↩</a></p>
+ <p><em>All mentions of “hadoop” on this page represent Apache Hadoop.</em> <a href="#fnref:HADOOP" class="reversefootnote">↩</a></p>
</li>
<li id="fn:HIVE">
- <p><em>Apache Hive.</em> <a href="#fnref:HIVE" class="reversefootnote">↩</a></p>
+ <p><em>Apache Hive.</em> <a href="#fnref:HIVE" class="reversefootnote">↩</a></p>
</li>
<li id="fn:HBASE">
- <p><em>Apache HBase.</em> <a href="#fnref:HBASE" class="reversefootnote">↩</a></p>
+ <p><em>Apache HBase.</em> <a href="#fnref:HBASE" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
http://git-wip-us.apache.org/repos/asf/eagle/blob/0094010b/_site/docs/tutorial/topologymanagement.html
----------------------------------------------------------------------
diff --git a/_site/docs/tutorial/topologymanagement.html b/_site/docs/tutorial/topologymanagement.html
index 2a95034..d635def 100644
--- a/_site/docs/tutorial/topologymanagement.html
+++ b/_site/docs/tutorial/topologymanagement.html
@@ -168,7 +168,7 @@
<p>Application manager consists of a daemon scheduler and an execution module. The scheduler periodically loads user operations(start/stop) from database, and the execution module executes these operations. For more details, please refer to <a href="https://cwiki.apache.org/confluence/display/EAG/Application+Management">here</a>.</p>
<h3 id="configurations">Configurations</h3>
-<p>The configuration file <code>eagle-scheduler.conf</code> defines scheduler parameters, execution platform settings and parts of default topology configuration.</p>
+<p>The configuration file <code class="highlighter-rouge">eagle-scheduler.conf</code> defines scheduler parameters, execution platform settings and parts of default topology configuration.</p>
<ul>
<li>
@@ -262,7 +262,7 @@
<li>
<p>Editing eagle-scheduler.conf, and start Eagle service</p>
- <pre><code> # enable application manager
+ <div class="highlighter-rouge"><pre class="highlight"><code> # enable application manager
appCommandLoaderEnabled = true
# provide jar path
@@ -272,9 +272,10 @@
envContextConfig.url = "http://sandbox.hortonworks.com:8744"
envContextConfig.nimbusHost = "sandbox.hortonworks.com"
</code></pre>
+ </div>
<p>For more configurations, please back to <a href="/docs/configuration.html">Application Configuration</a>. <br />
- After the configuration is ready, start Eagle service <code>bin/eagle-service.sh start</code>.</p>
+ After the configuration is ready, start Eagle service <code class="highlighter-rouge">bin/eagle-service.sh start</code>.</p>
</li>
<li>
<p>Go to admin page
@@ -296,11 +297,11 @@
<li>
<p>Go to site page, and add topology configurations.</p>
- <p><strong>NOTICE</strong> topology configurations defined here are REQUIRED an extra prefix <code>.app</code></p>
+ <p><strong>NOTICE</strong> topology configurations defined here are REQUIRED an extra prefix <code class="highlighter-rouge">.app</code></p>
<p>Blow are some example configurations for [site=sandbox, applicatoin=hbaseSecurityLog].</p>
- <pre><code> classification.hbase.zookeeper.property.clientPort=2181
+ <div class="highlighter-rouge"><pre class="highlight"><code> classification.hbase.zookeeper.property.clientPort=2181
classification.hbase.zookeeper.quorum=sandbox.hortonworks.com
app.envContextConfig.env=storm
@@ -329,6 +330,7 @@
app.eagleProps.eagleService.username=admin
app.eagleProps.eagleService.password=secret
</code></pre>
+ </div>
<p><img src="/images/appManager/topology-configuration-1.png" alt="topology-configuration-1" />
<img src="/images/appManager/topology-configuration-2.png" alt="topology-configuration-2" /></p>
@@ -351,7 +353,7 @@
<div class="footnotes">
<ol>
<li id="fn:STORM">
- <p><em>All mentions of “storm” on this page represent Apache Storm.</em> <a href="#fnref:STORM" class="reversefootnote">↩</a></p>
+ <p><em>All mentions of “storm” on this page represent Apache Storm.</em> <a href="#fnref:STORM" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
http://git-wip-us.apache.org/repos/asf/eagle/blob/0094010b/_site/docs/tutorial/userprofile.html
----------------------------------------------------------------------
diff --git a/_site/docs/tutorial/userprofile.html b/_site/docs/tutorial/userprofile.html
index fbf55e6..5b442ab 100644
--- a/_site/docs/tutorial/userprofile.html
+++ b/_site/docs/tutorial/userprofile.html
@@ -173,9 +173,10 @@ is started.</p>
<li>
<p>Option 1: command line</p>
- <pre><code>$ cd <eagle-home>/bin
+ <div class="highlighter-rouge"><pre class="highlight"><code>$ cd <eagle-home>/bin
$ bin/eagle-userprofile-scheduler.sh --site sandbox start
</code></pre>
+ </div>
</li>
<li>
<p>Option 2: start via Apache Ambari
@@ -203,8 +204,9 @@ $ bin/eagle-userprofile-scheduler.sh --site sandbox start
<p>submit userProfiles topology if it’s not on <a href="http://sandbox.hortonworks.com:8744">topology UI</a></p>
- <pre><code>$ bin/eagle-topology.sh --main org.apache.eagle.security.userprofile.UserProfileDetectionMain --config conf/sandbox-userprofile-topology.conf start
+ <div class="highlighter-rouge"><pre class="highlight"><code>$ bin/eagle-topology.sh --main org.apache.eagle.security.userprofile.UserProfileDetectionMain --config conf/sandbox-userprofile-topology.conf start
</code></pre>
+ </div>
</li>
<li>
<p><strong>Option 2</strong>: Apache Ambari</p>
@@ -219,23 +221,26 @@ $ bin/eagle-userprofile-scheduler.sh --site sandbox start
<li>Prepare sample data for ML training and validation sample data
<ul>
<li>a. Download following sample data to be used for training</li>
- <li><a href="/data/user1.hdfs-audit.2015-10-11-00.txt"><code>user1.hdfs-audit.2015-10-11-00.txt</code></a></li>
- <li><a href="/data/user1.hdfs-audit.2015-10-11-01.txt"><code>user1.hdfs-audit.2015-10-11-01.txt</code></a></li>
- <li>b. Downlaod <a href="/data/userprofile-validate.txt"><code>userprofile-validate.txt</code></a>file which contains data points that you can try to test the models</li>
+ </ul>
+ <ul>
+ <li><a href="/data/user1.hdfs-audit.2015-10-11-00.txt"><code class="highlighter-rouge">user1.hdfs-audit.2015-10-11-00.txt</code></a></li>
+ <li><a href="/data/user1.hdfs-audit.2015-10-11-01.txt"><code class="highlighter-rouge">user1.hdfs-audit.2015-10-11-01.txt</code></a>
+ * b. Downlaod <a href="/data/userprofile-validate.txt"><code class="highlighter-rouge">userprofile-validate.txt</code></a>file which contains data points that you can try to test the models</li>
</ul>
</li>
<li>Copy the files (downloaded in the previous step) into a location in sandbox
-For example: <code>/usr/hdp/current/eagle/lib/userprofile/data/</code></li>
- <li>Modify <code><Eagle-home>/conf/sandbox-userprofile-scheduler.conf </code>
-update <code>training-audit-path</code> to set to the path for training data sample (the path you used for Step 1.a)
+For example: <code class="highlighter-rouge">/usr/hdp/current/eagle/lib/userprofile/data/</code></li>
+ <li>Modify <code class="highlighter-rouge"><Eagle-home>/conf/sandbox-userprofile-scheduler.conf </code>
+update <code class="highlighter-rouge">training-audit-path</code> to set to the path for training data sample (the path you used for Step 1.a)
update detection-audit-path to set to the path for validation (the path you used for Step 1.b)</li>
<li>Run ML training program from eagle UI</li>
<li>
<p>Produce Apache Kafka data using the contents from validate file (Step 1.b)
-Run the command (assuming the eagle configuration uses Kafka topic <code>sandbox_hdfs_audit_log</code>)</p>
+Run the command (assuming the eagle configuration uses Kafka topic <code class="highlighter-rouge">sandbox_hdfs_audit_log</code>)</p>
- <pre><code> ./kafka-console-producer.sh --broker-list sandbox.hortonworks.com:6667 --topic sandbox_hdfs_audit_log
+ <div class="highlighter-rouge"><pre class="highlight"><code> ./kafka-console-producer.sh --broker-list sandbox.hortonworks.com:6667 --topic sandbox_hdfs_audit_log
</code></pre>
+ </div>
</li>
<li>Paste few lines of data from file validate onto kafka-console-producer
Check <a href="http://localhost:9099/eagle-service/#/dam/alertList">http://localhost:9099/eagle-service/#/dam/alertList</a> for generated alerts</li>
http://git-wip-us.apache.org/repos/asf/eagle/blob/0094010b/_site/docs/usecases.html
----------------------------------------------------------------------
diff --git a/_site/docs/usecases.html b/_site/docs/usecases.html
index 056c0ce..ebb0a23 100644
--- a/_site/docs/usecases.html
+++ b/_site/docs/usecases.html
@@ -218,16 +218,16 @@
<div class="footnotes">
<ol>
<li id="fn:HADOOP">
- <p><em>All mentions of “hadoop” on this page represent Apache Hadoop.</em> <a href="#fnref:HADOOP" class="reversefootnote">↩</a></p>
+ <p><em>All mentions of “hadoop” on this page represent Apache Hadoop.</em> <a href="#fnref:HADOOP" class="reversefootnote">↩</a></p>
</li>
<li id="fn:HIVE">
- <p><em>All mentions of “hive” on this page represent Apache Hive.</em> <a href="#fnref:HIVE" class="reversefootnote">↩</a></p>
+ <p><em>All mentions of “hive” on this page represent Apache Hive.</em> <a href="#fnref:HIVE" class="reversefootnote">↩</a></p>
</li>
<li id="fn:SPARK">
- <p><em>All mentions of “spark” on this page represent Apache Spark.</em> <a href="#fnref:SPARK" class="reversefootnote">↩</a></p>
+ <p><em>All mentions of “spark” on this page represent Apache Spark.</em> <a href="#fnref:SPARK" class="reversefootnote">↩</a></p>
</li>
<li id="fn:CASSANDRA">
- <p><em>Apache Cassandra.</em> <a href="#fnref:CASSANDRA" class="reversefootnote">↩</a></p>
+ <p><em>Apache Cassandra.</em> <a href="#fnref:CASSANDRA" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
http://git-wip-us.apache.org/repos/asf/eagle/blob/0094010b/_site/feed.xml
----------------------------------------------------------------------
diff --git a/_site/feed.xml b/_site/feed.xml
index bd59283..a2720f7 100644
--- a/_site/feed.xml
+++ b/_site/feed.xml
@@ -5,9 +5,9 @@
<description>Eagle - Analyze Big Data Platforms for Security and Performance</description>
<link>http://goeagle.io/</link>
<atom:link href="http://goeagle.io/feed.xml" rel="self" type="application/rss+xml"/>
- <pubDate>Mon, 03 Apr 2017 19:55:40 +0800</pubDate>
- <lastBuildDate>Mon, 03 Apr 2017 19:55:40 +0800</lastBuildDate>
- <generator>Jekyll v2.5.3</generator>
+ <pubDate>Wed, 22 Nov 2017 13:52:35 +0800</pubDate>
+ <lastBuildDate>Wed, 22 Nov 2017 13:52:35 +0800</lastBuildDate>
+ <generator>Jekyll v3.4.3</generator>
<item>
<title>Apache Eagle 正式发布：分布式实时Hadoop数据安全方案</title>
@@ -17,7 +17,7 @@
<p>日前，eBay公司隆重宣布正式向开源业界推出分布式实时安全监控方案 － Apache Eagle (http://goeagle.io)，该项目已于2015年10月26日正式加入Apache 成为孵化器项目。Apache Eagle提供一套高效分布式的流式策略引擎，具有高实时、可伸缩、易扩展、交互友好等特点，同时集成机器学习对用户行为建立Profile以实现智能实时地保护Hadoop生态系统中大数据的安全。</p>
-<h2 id="section">背景</h2>
+<h2 id="背景">背景</h2>
<p>随着大数据的发展，越来越多的成功企业或者组织开始采取数据驱动商业的运作模式。在eBay，我们拥有数万名工程师、分析师和数据科学家，他们每天访问分析数PB级的数据，以为我们的用户带来无与伦比的体验。在全球业务中，我们也广泛地利用海量大数据来连接我们数以亿计的用户。</p>
<p>近年来，Hadoop已经逐渐成为大数据分析领域最受欢迎的解决方案，eBay也一直在使用Hadoop技术从数据中挖掘价值，例如，我们通过大数据提高用户的搜索体验，识别和优化精准广告投放，充实我们的产品目录，以及通过点击流分析以理解用户如何使用我们的在线市场平台等。</p>
@@ -54,20 +54,20 @@
<li><strong>开源</strong>：Eagle一直根据开源的标准开发，并构建于诸多大数据领域的开源产品之上，因此我们决定以Apache许可证开源Eagle，以回馈社区，同时也期待获得社区的反馈、协作与支持。</li>
</ul>
-<h2 id="eagle">Eagle概览</h2>
+<h2 id="eagle概览">Eagle概览</h2>
<p><img src="/images/posts/eagle-group.png" alt="" /></p>
-<h4 id="data-collection-and-storage">数据流接入和存储（Data Collection and Storage）</h4>
+<h4 id="数据流接入和存储data-collection-and-storage">数据流接入和存储（Data Collection and Storage）</h4>
<p>Eagle提供高度可扩展的编程API，可以支持将任何类型的数据源集成到Eagle的策略执行引擎中。例如，在Eagle HDFS 审计事件（Audit）监控模块中，通过Kafka来实时接收来自Namenode Log4j Appender 或者 Logstash Agent 收集的数据；在Eagle Hive 监控模块中，通过YARN API 收集正在运行Job的Hive 查询日志，并保证比较高的可伸缩性和容错性。</p>
-<h4 id="data-processing">数据实时处理（Data Processing）</h4>
+<h4 id="数据实时处理data-processing">数据实时处理（Data Processing）</h4>
<p><strong>流处理API（Stream Processing API）Eagle</strong> 提供独立于物理平台而高度抽象的流处理API，目前默认支持Apache Storm，但是也允许扩展到其他任意流处理引擎，比如Flink 或者 Samza等。该层抽象允许开发者在定义监控数据处理逻辑时，无需在物理执行层绑定任何特定流处理平台，而只需通过复用、拼接和组装例如数据转换、过滤、外部数据Join等组件，以实现满足需求的DAG（有向无环图），同时，开发者也可以很容易地以编程地方式将业务逻辑流程和Eagle 策略引擎框架集成起来。Eagle框架内部会将描述业务逻辑的DAG编译成底层流处理架构的原生应用，例如Apache Storm Topology 等，从事实现平台的独立。</p>
<p><strong>以下是一个Eagle如何处理事件和告警的示例：</strong></p>
-<pre><code>StormExecutionEnvironment env = ExecutionEnvironmentFactory.getStorm(config); // storm env
+<div class="highlighter-rouge"><pre class="highlight"><code>StormExecutionEnvironment env = ExecutionEnvironmentFactory.getStorm(config); // storm env
StreamProducer producer = env.newSource(new KafkaSourcedSpoutProvider().getSpout(config)).renameOutputFields(1) // declare kafka source
.flatMap(new AuditLogTransformer()) // transform event
.groupBy(Arrays.asList(0)) // group by 1st field
@@ -75,6 +75,7 @@ StreamProducer producer = env.newSource(new KafkaSourcedSpoutProvider().getSpout
.alertWithConsumer(“userActivity“,”userProfileExecutor“) // ML policy evaluation
env.execute(); // execute stream processing and alert
</code></pre>
+</div>
<p><strong>告警框架（Alerting Framework）Eagle</strong>告警框架由流元数据API、策略引擎服务提供API、策略Partitioner API 以及预警去重框架等组成:</p>
@@ -84,7 +85,7 @@ env.execute(); // execute stream processing and alert
<li>
<p><strong>扩展性</strong> Eagle的策略引擎服务提供API允许你插入新的策略引擎</p>
- <pre><code> public interface PolicyEvaluatorServiceProvider {
+ <div class="highlighter-rouge"><pre class="highlight"><code> public interface PolicyEvaluatorServiceProvider {
public String getPolicyType(); // literal string to identify one type of policy
public Class&lt;? extends PolicyEvaluator&gt; getPolicyEvaluator(); // get policy evaluator implementation
public List&lt;Module&gt; getBindingModules(); // policy text with json format to object mapping
@@ -95,15 +96,17 @@ env.execute(); // execute stream processing and alert
public void onPolicyDelete(); // invoked when policy is deleted
}
</code></pre>
+ </div>
</li>
<li><strong>策略Partitioner API</strong> 允许策略在不同的物理节点上并行执行。也允许你自定义策略Partitioner类。这些功能使得策略和事件完全以分布式的方式执行。</li>
<li>
<p><strong>可伸缩性</strong> Eagle 通过支持策略的分区接口来实现大量的策略可伸缩并发地运行</p>
- <pre><code> public interface PolicyPartitioner extends Serializable {
+ <div class="highlighter-rouge"><pre class="highlight"><code> public interface PolicyPartitioner extends Serializable {
int partition(int numTotalPartitions, String policyType, String policyId); // method to distribute policies
}
</code></pre>
+ </div>
<p><img src="/images/posts/policy-partition.png" alt="" /></p>
@@ -160,26 +163,29 @@ Eagle 支持根据用户在Hadoop平台上历史使用行为习惯来定义行
<li>
<p>单一事件执行策略（用户访问Hive中的敏感数据列）</p>
- <pre><code> from hiveAccessLogStream[sensitivityType=='PHONE_NUMBER'] select * insert into outputStream;
+ <div class="highlighter-rouge"><pre class="highlight"><code> from hiveAccessLogStream[sensitivityType=='PHONE_NUMBER'] select * insert into outputStream;
</code></pre>
+ </div>
</li>
<li>
<p>基于窗口的策略（用户在10分钟内访问目录 /tmp/private 多余 5次）</p>
- <pre><code> hdfsAuditLogEventStream[(src == '/tmp/private')]#window.externalTime(timestamp,10 min) select user, count(timestamp) as aggValue group by user having aggValue &gt;= 5 insert into outputStream;
+ <div class="highlighter-rouge"><pre class="highlight"><code> hdfsAuditLogEventStream[(src == '/tmp/private')]#window.externalTime(timestamp,10 min) select user, count(timestamp) as aggValue group by user having aggValue &gt;= 5 insert into outputStream;
</code></pre>
+ </div>
</li>
</ul>
<p><strong>查询服务（Query Service）</strong> Eagle 提供类SQL的REST API用来实现针对海量数据集的综合计算、查询和分析的能力，支持例如过滤、聚合、直方运算、排序、top、算术表达式以及分页等。Eagle优先支持HBase 作为其默认数据存储，但是同时也支持基JDBC的关系型数据库。特别是当选择以HBase作为存储时，Eagle便原生拥有了HBase存储和查询海量监控数据的能力，Eagle 查询框架会将用户提供的类SQL查询语法最终编译成为HBase 原生的Filter 对象，并支持通过HBase Coprocessor进一步提升响应速度。</p>
-<pre><code>query=AlertDefinitionService[@dataSource="hiveQueryLog"]{@policyDef}&amp;pageSize=100000
+<div class="highlighter-rouge"><pre class="highlight"><code>query=AlertDefinitionService[@dataSource="hiveQueryLog"]{@policyDef}&amp;pageSize=100000
</code></pre>
+</div>
-<h2 id="eagleebay">Eagle在eBay的使用场景</h2>
+<h2 id="eagle在ebay的使用场景">Eagle在eBay的使用场景</h2>
<p>目前，Eagle的数据行为监控系统已经部署到一个拥有2500多个节点的Hadoop集群之上，用以保护数百PB数据的安全，并正计划于今年年底之前扩展到其他上十个Hadoop集群上，从而覆盖eBay 所有主要Hadoop的10000多台节点。在我们的生产环境中，我们已针对HDFS、Hive 等集群中的数据配置了一些基础的安全策略，并将于年底之前不断引入更多的策略，以确保重要数据的绝对安全。目前，Eagle的策略涵盖多种模式，包括从访问模式、频繁访问数据集，预定义查询类型、Hive 表和列、HBase 表以及基于机器学习模型生成的用户Profile相关的所有策略等。同时，我们也有广泛的策略来防止数据的丢失、数据被拷贝到不安全地点、敏感数据被未授权区域访问等。Eagle策略定义上极大的灵活性和扩展性使得我们未来可以轻易地继续扩展更多更复杂的策略以支持更多多元
化的用例场景。</p>
-<h2 id="section-1">后续计划</h2>
+<h2 id="后续计划">后续计划</h2>
<p>过去两年中，在eBay 除了被用于数据行为监控以外，Eagle 核心框架还被广泛用于监控节点健康状况、Hadoop应用性能指标、Hadoop 核心服务以及整个Hadoop集群的健康状况等诸多领域。我们还建立一系列的自动化机制，例如节点修复等，帮助我们平台部门极大得节省了我们人工劳力，并有效地提升了整个集群资源地利用率。</p>
<p>以下是我们目前正在开发中地一些特性：</p>
@@ -196,7 +202,7 @@ Eagle 支持根据用户在Hadoop平台上历史使用行为习惯来定义行
</li>
</ul>
-<h2 id="section-2">关于作者</h2>
+<h2 id="关于作者">关于作者</h2>
<p><a href="https://github.com/haoch">陈浩</a>，Apache Eagle Committer 和 PMC 成员，eBay 分析平台基础架构部门高级软件工程师，负责Eagle的产品设计、技术架构、核心实现以及开源社区推广等。</p>
<p>感谢以下来自Apache Eagle社区和eBay公司的联合作者们对本文的贡献：</p>
@@ -210,7 +216,7 @@ Eagle 支持根据用户在Hadoop平台上历史使用行为习惯来定义行
<p>eBay 分析平台基础架构部（Analytics Data Infrastructure）是eBay的全球数据及分析基础架构部门，负责eBay在数据库、数据仓库、Hadoop、商务智能以及机器学习等各个数据平台开发、管理等,支持eBay全球各部门运用高端的数据分析解决方案作出及时有效的作业决策，为遍布全球的业务用户提供数据分析解决方案。</p>
-<h2 id="section-3">参考资料</h2>
+<h2 id="参考资料">参考资料</h2>
<ul>
<li>Apache Eagle 文档：<a href="http://goeagle.io">http://goeagle.io</a></li>
@@ -218,7 +224,7 @@ Eagle 支持根据用户在Hadoop平台上历史使用行为习惯来定义行
<li>Apache Eagle 项目：<a href="http://incubator.apache.org/projects/eagle.html">http://incubator.apache.org/projects/eagle.html</a></li>
</ul>
-<h2 id="section-4">引用链接</h2>
+<h2 id="引用链接">引用链接</h2>
<ul>
<li><strong>CSDN</strong>: <a href="http://www.csdn.net/article/2015-10-29/2826076">http://www.csdn.net/article/2015-10-29/2826076</a></li>
<li><strong>OSCHINA</strong>: <a href="http://www.oschina.net/news/67515/apache-eagle">http://www.oschina.net/news/67515/apache-eagle</a></li>
http://git-wip-us.apache.org/repos/asf/eagle/blob/0094010b/_site/post/2015/10/27/apache-eagle-announce-cn.html
----------------------------------------------------------------------
diff --git a/_site/post/2015/10/27/apache-eagle-announce-cn.html b/_site/post/2015/10/27/apache-eagle-announce-cn.html
index 5d87315..088e22d 100644
--- a/_site/post/2015/10/27/apache-eagle-announce-cn.html
+++ b/_site/post/2015/10/27/apache-eagle-announce-cn.html
@@ -93,7 +93,7 @@
<p>日前，eBay公司隆重宣布正式向开源业界推出分布式实时安全监控方案 － Apache Eagle (http://goeagle.io)，该项目已于2015年10月26日正式加入Apache 成为孵化器项目。Apache Eagle提供一套高效分布式的流式策略引擎，具有高实时、可伸缩、易扩展、交互友好等特点，同时集成机器学习对用户行为建立Profile以实现智能实时地保护Hadoop生态系统中大数据的安全。</p>
-<h2 id="section">背景</h2>
+<h2 id="背景">背景</h2>
<p>随着大数据的发展，越来越多的成功企业或者组织开始采取数据驱动商业的运作模式。在eBay，我们拥有数万名工程师、分析师和数据科学家，他们每天访问分析数PB级的数据，以为我们的用户带来无与伦比的体验。在全球业务中，我们也广泛地利用海量大数据来连接我们数以亿计的用户。</p>
<p>近年来，Hadoop已经逐渐成为大数据分析领域最受欢迎的解决方案，eBay也一直在使用Hadoop技术从数据中挖掘价值，例如，我们通过大数据提高用户的搜索体验，识别和优化精准广告投放，充实我们的产品目录，以及通过点击流分析以理解用户如何使用我们的在线市场平台等。</p>
@@ -130,20 +130,20 @@
<li><strong>开源</strong>：Eagle一直根据开源的标准开发，并构建于诸多大数据领域的开源产品之上，因此我们决定以Apache许可证开源Eagle，以回馈社区，同时也期待获得社区的反馈、协作与支持。</li>
</ul>
-<h2 id="eagle">Eagle概览</h2>
+<h2 id="eagle概览">Eagle概览</h2>
<p><img src="/images/posts/eagle-group.png" alt="" /></p>
-<h4 id="data-collection-and-storage">数据流接入和存储（Data Collection and Storage）</h4>
+<h4 id="数据流接入和存储data-collection-and-storage">数据流接入和存储（Data Collection and Storage）</h4>
<p>Eagle提供高度可扩展的编程API，可以支持将任何类型的数据源集成到Eagle的策略执行引擎中。例如，在Eagle HDFS 审计事件（Audit）监控模块中，通过Kafka来实时接收来自Namenode Log4j Appender 或者 Logstash Agent 收集的数据；在Eagle Hive 监控模块中，通过YARN API 收集正在运行Job的Hive 查询日志，并保证比较高的可伸缩性和容错性。</p>
-<h4 id="data-processing">数据实时处理（Data Processing）</h4>
+<h4 id="数据实时处理data-processing">数据实时处理（Data Processing）</h4>
<p><strong>流处理API（Stream Processing API）Eagle</strong> 提供独立于物理平台而高度抽象的流处理API，目前默认支持Apache Storm，但是也允许扩展到其他任意流处理引擎，比如Flink 或者 Samza等。该层抽象允许开发者在定义监控数据处理逻辑时，无需在物理执行层绑定任何特定流处理平台，而只需通过复用、拼接和组装例如数据转换、过滤、外部数据Join等组件，以实现满足需求的DAG（有向无环图），同时，开发者也可以很容易地以编程地方式将业务逻辑流程和Eagle 策略引擎框架集成起来。Eagle框架内部会将描述业务逻辑的DAG编译成底层流处理架构的原生应用，例如Apache Storm Topology 等，从事实现平台的独立。</p>
<p><strong>以下是一个Eagle如何处理事件和告警的示例：</strong></p>
-<pre><code>StormExecutionEnvironment env = ExecutionEnvironmentFactory.getStorm(config); // storm env
+<div class="highlighter-rouge"><pre class="highlight"><code>StormExecutionEnvironment env = ExecutionEnvironmentFactory.getStorm(config); // storm env
StreamProducer producer = env.newSource(new KafkaSourcedSpoutProvider().getSpout(config)).renameOutputFields(1) // declare kafka source
.flatMap(new AuditLogTransformer()) // transform event
.groupBy(Arrays.asList(0)) // group by 1st field
@@ -151,6 +151,7 @@ StreamProducer producer = env.newSource(new KafkaSourcedSpoutProvider().getSpout
.alertWithConsumer(“userActivity“,”userProfileExecutor“) // ML policy evaluation
env.execute(); // execute stream processing and alert
</code></pre>
+</div>
<p><strong>告警框架（Alerting Framework）Eagle</strong>告警框架由流元数据API、策略引擎服务提供API、策略Partitioner API 以及预警去重框架等组成:</p>
@@ -160,7 +161,7 @@ env.execute(); // execute stream processing and alert
<li>
<p><strong>扩展性</strong> Eagle的策略引擎服务提供API允许你插入新的策略引擎</p>
- <pre><code> public interface PolicyEvaluatorServiceProvider {
+ <div class="highlighter-rouge"><pre class="highlight"><code> public interface PolicyEvaluatorServiceProvider {
public String getPolicyType(); // literal string to identify one type of policy
public Class<? extends PolicyEvaluator> getPolicyEvaluator(); // get policy evaluator implementation
public List<Module> getBindingModules(); // policy text with json format to object mapping
@@ -171,15 +172,17 @@ env.execute(); // execute stream processing and alert
public void onPolicyDelete(); // invoked when policy is deleted
}
</code></pre>
+ </div>
</li>
<li><strong>策略Partitioner API</strong> 允许策略在不同的物理节点上并行执行。也允许你自定义策略Partitioner类。这些功能使得策略和事件完全以分布式的方式执行。</li>
<li>
<p><strong>可伸缩性</strong> Eagle 通过支持策略的分区接口来实现大量的策略可伸缩并发地运行</p>
- <pre><code> public interface PolicyPartitioner extends Serializable {
+ <div class="highlighter-rouge"><pre class="highlight"><code> public interface PolicyPartitioner extends Serializable {
int partition(int numTotalPartitions, String policyType, String policyId); // method to distribute policies
}
</code></pre>
+ </div>
<p><img src="/images/posts/policy-partition.png" alt="" /></p>
@@ -236,26 +239,29 @@ Eagle 支持根据用户在Hadoop平台上历史使用行为习惯来定义行
<li>
<p>单一事件执行策略（用户访问Hive中的敏感数据列）</p>
- <pre><code> from hiveAccessLogStream[sensitivityType=='PHONE_NUMBER'] select * insert into outputStream;
+ <div class="highlighter-rouge"><pre class="highlight"><code> from hiveAccessLogStream[sensitivityType=='PHONE_NUMBER'] select * insert into outputStream;
</code></pre>
+ </div>
</li>
<li>
<p>基于窗口的策略（用户在10分钟内访问目录 /tmp/private 多余 5次）</p>
- <pre><code> hdfsAuditLogEventStream[(src == '/tmp/private')]#window.externalTime(timestamp,10 min) select user, count(timestamp) as aggValue group by user having aggValue >= 5 insert into outputStream;
+ <div class="highlighter-rouge"><pre class="highlight"><code> hdfsAuditLogEventStream[(src == '/tmp/private')]#window.externalTime(timestamp,10 min) select user, count(timestamp) as aggValue group by user having aggValue >= 5 insert into outputStream;
</code></pre>
+ </div>
</li>
</ul>
<p><strong>查询服务（Query Service）</strong> Eagle 提供类SQL的REST API用来实现针对海量数据集的综合计算、查询和分析的能力，支持例如过滤、聚合、直方运算、排序、top、算术表达式以及分页等。Eagle优先支持HBase 作为其默认数据存储，但是同时也支持基JDBC的关系型数据库。特别是当选择以HBase作为存储时，Eagle便原生拥有了HBase存储和查询海量监控数据的能力，Eagle 查询框架会将用户提供的类SQL查询语法最终编译成为HBase 原生的Filter 对象，并支持通过HBase Coprocessor进一步提升响应速度。</p>
-<pre><code>query=AlertDefinitionService[@dataSource="hiveQueryLog"]{@policyDef}&pageSize=100000
+<div class="highlighter-rouge"><pre class="highlight"><code>query=AlertDefinitionService[@dataSource="hiveQueryLog"]{@policyDef}&pageSize=100000
</code></pre>
+</div>
-<h2 id="eagleebay">Eagle在eBay的使用场景</h2>
+<h2 id="eagle在ebay的使用场景">Eagle在eBay的使用场景</h2>
<p>目前，Eagle的数据行为监控系统已经部署到一个拥有2500多个节点的Hadoop集群之上，用以保护数百PB数据的安全，并正计划于今年年底之前扩展到其他上十个Hadoop集群上，从而覆盖eBay 所有主要Hadoop的10000多台节点。在我们的生产环境中，我们已针对HDFS、Hive 等集群中的数据配置了一些基础的安全策略，并将于年底之前不断引入更多的策略，以确保重要数据的绝对安全。目前，Eagle的策略涵盖多种模式，包括从访问模式、频繁访问数据集，预定义查询类型、Hive 表和列、HBase 表以及基于机器学习模型生成的用户Profile相关的所有策略等。同时，我们也有广泛的策略来防止数据的丢失、数据被拷贝到不安全地点、敏感数据被未授权区域访问等。Eagle策略定义上极大的灵活性和扩展性使得我们未来可以轻易地继续扩展更多更复杂的策略以支持更多多元化的
用例场景。</p>
-<h2 id="section-1">后续计划</h2>
+<h2 id="后续计划">后续计划</h2>
<p>过去两年中，在eBay 除了被用于数据行为监控以外，Eagle 核心框架还被广泛用于监控节点健康状况、Hadoop应用性能指标、Hadoop 核心服务以及整个Hadoop集群的健康状况等诸多领域。我们还建立一系列的自动化机制，例如节点修复等，帮助我们平台部门极大得节省了我们人工劳力，并有效地提升了整个集群资源地利用率。</p>
<p>以下是我们目前正在开发中地一些特性：</p>
@@ -272,7 +278,7 @@ Eagle 支持根据用户在Hadoop平台上历史使用行为习惯来定义行
</li>
</ul>
-<h2 id="section-2">关于作者</h2>
+<h2 id="关于作者">关于作者</h2>
<p><a href="https://github.com/haoch">陈浩</a>，Apache Eagle Committer 和 PMC 成员，eBay 分析平台基础架构部门高级软件工程师，负责Eagle的产品设计、技术架构、核心实现以及开源社区推广等。</p>
<p>感谢以下来自Apache Eagle社区和eBay公司的联合作者们对本文的贡献：</p>
@@ -286,7 +292,7 @@ Eagle 支持根据用户在Hadoop平台上历史使用行为习惯来定义行
<p>eBay 分析平台基础架构部（Analytics Data Infrastructure）是eBay的全球数据及分析基础架构部门，负责eBay在数据库、数据仓库、Hadoop、商务智能以及机器学习等各个数据平台开发、管理等,支持eBay全球各部门运用高端的数据分析解决方案作出及时有效的作业决策，为遍布全球的业务用户提供数据分析解决方案。</p>
-<h2 id="section-3">参考资料</h2>
+<h2 id="参考资料">参考资料</h2>
<ul>
<li>Apache Eagle 文档：<a href="http://goeagle.io">http://goeagle.io</a></li>
@@ -294,7 +300,7 @@ Eagle 支持根据用户在Hadoop平台上历史使用行为习惯来定义行
<li>Apache Eagle 项目：<a href="http://incubator.apache.org/projects/eagle.html">http://incubator.apache.org/projects/eagle.html</a></li>
</ul>
-<h2 id="section-4">引用链接</h2>
+<h2 id="引用链接">引用链接</h2>
<ul>
<li><strong>CSDN</strong>: <a href="http://www.csdn.net/article/2015-10-29/2826076">http://www.csdn.net/article/2015-10-29/2826076</a></li>
<li><strong>OSCHINA</strong>: <a href="http://www.oschina.net/news/67515/apache-eagle">http://www.oschina.net/news/67515/apache-eagle</a></li>