You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@eagle.apache.org by ha...@apache.org on 2015/11/19 14:47:13 UTC
svn commit: r1715179 [8/14] - in /incubator/eagle/site: ./ about/ css/ data/
development/ docs/ docs/tutorial/ fonts/ images/ images/docs/ images/posts/
images/slider/ images/usecases/ js/ post/ post/2015/ post/2015/10/
post/2015/10/27/
Added: incubator/eagle/site/docs/tutorial/setup.html
URL: http://svn.apache.org/viewvc/incubator/eagle/site/docs/tutorial/setup.html?rev=1715179&view=auto
==============================================================================
--- incubator/eagle/site/docs/tutorial/setup.html (added)
+++ incubator/eagle/site/docs/tutorial/setup.html Thu Nov 19 13:47:10 2015
@@ -0,0 +1,273 @@
+<!DOCTYPE html>
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+
+ <title>Eagle - Site Management</title>
+ <meta name="description" content="Eagle - Secure Hadoop in Real Time">
+
+ <meta name="keywords" content="Eagle, Hadoop, Security, Real Time">
+ <meta name="author" content="eBay Inc.">
+
+ <meta charset="utf-8">
+ <meta name="viewport" content="initial-scale=1">
+
+ <link rel="stylesheet" href="/css/animate.css">
+ <link rel="stylesheet" href="/css/bootstrap.min.css">
+
+ <link rel="stylesheet" href="/css/font-awesome.min.css">
+
+ <link rel="stylesheet" href="/css/misc.css">
+ <link rel="stylesheet" href="/css/style.css">
+ <link rel="stylesheet" href="/css/styles.css">
+ <link rel="stylesheet" href="/css/main.css">
+ <link rel="alternate" type="application/rss+xml" title="Eagle" href="http://goeagle.io/feed.xml" />
+ <link rel="shortcut icon" href="/images/favicon.png">
+
+ <!-- Baidu Analytics Tracking-->
+ <script>
+ var _hmt = _hmt || [];
+ (function() {
+ var hm = document.createElement("script");
+ hm.src = "//hm.baidu.com/hm.js?fedc55df2ea52777a679192e8f849ece";
+ var s = document.getElementsByTagName("script")[0];
+ s.parentNode.insertBefore(hm, s);
+ })();
+ </script>
+
+ <!-- Google Analytics Tracking -->
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+ ga('create', 'UA-68929805-1', 'auto');
+ ga('send', 'pageview');
+ </script>
+</head>
+<body>
+<!-- header start -->
+<div id="home_page">
+ <div class="topbar">
+ <div class="container">
+ <div class="row" >
+ <nav class="navbar navbar-default">
+ <div class="container-fluid">
+ <!-- Brand and toggle get grouped for better mobile display -->
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1"> <span class="sr-only">Toggle navigation</span> <span class="icon-bar"></span> <span class="icon-bar"></span> <span class="icon-bar"></span> </button>
+ <a class="navbar-brand" href="/"><img src="/images/logo2.png" height="44px" style="margin-top:-7px"></a> </div>
+
+ <!-- Collect the nav links, forms, and other content for toggling -->
+ <!-- <div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
+ <ul class="nav navbar-nav navbar-right" id="top-menu">
+ <li><a class="menu" href="/#home_page">HOME</a></li>
+ <li><a class="menu" href="/docs/">DOCS</a></li>
+ <li><a class="menu" href="/#about_page">ABOUT</a></li>
+ <li><a class="menu" href="/#diagram_page">ARCHITECTURE</a></li>
+ <li><a class="menu" href="/#modules_page">MODULES</a></li>
+ <li><a class="menu" href="/#usecase_page">USE CASES</a></li>
+ <li>
+ </li>
+ </ul> -->
+ </div>
+ </div>
+ <!-- /.container-fluid -->
+ </nav>
+ </div>
+ </div>
+ </div>
+ <div class="headerimage">
+ <div class="flexslider">
+ <ul class="slides">
+ <li><img src="/images/slider/4.jpg" alt="Slide 1"></li>
+ </ul>
+ </div>
+ </div>
+ <div class="particles" style="height:40%"> </div><!---particles-->
+ <div class="slider-caption" style="top:80px;">
+ <div class="homewrapper">
+ <div class="hometitle">
+ <a href="/">
+ <img src="/images/feather.png" height="80px">
+ </a>
+ </div>
+ <div class="hometext">
+ <h2>Secure Hadoop in Real Time</h2>
+ <div class="social-buttons">
+ <!-- <a href="https://github.com/ebay/eagle"><i class="fa fa-github"></i></a>
+ <a href="http://twitter.com/TheApacheEagle"><i class="fa fa-twitter"></i></a>
+ <a href="https://www.facebook.com/TheApacheEagle/"><i class="fa fa-facebook"></i></a>
+ <a href="#"><i class="fa fa-weixin"></i></a> -->
+ <!-- <a href="https://www.weibo.com/ApacheEagle/"><i class="fa fa-weibo"></i></a> -->
+ </div>
+ </div>
+ </div>
+ </div>
+</div>
+<!-- header end -->
+<div class="container-fluid page-content">
+ <a href="https://github.com/ebay/eagle" target="_blank"><img style="z-index:10000;position: fixed; top: 0; right: 0; border: 0;" src="https://camo.githubusercontent.com/365986a132ccd6a44c23a9169022c0b5c890c387/68747470733a2f2f73332e616d617a6f6e6177732e636f6d2f6769746875622f726962626f6e732f666f726b6d655f72696768745f7265645f6161303030302e706e67" alt="Fork eagle on GitHub" data-canonical-src="https://s3.amazonaws.com/github/ribbons/forkme_right_red_aa0000.png"></a>
+
+ <div class="row">
+ <div class="col-md-10 col-md-offset-1">
+ <!-- sidebar -->
+ <div class="col-xs-6 col-sm-3" id="sidebar" role="navigation">
+ <ul class="nav" id="adminnav">
+
+ <li class="heading">Getting Started</li>
+
+ <li class="sidenavli "><a href="/docs/index.html" data-permalink="/docs/tutorial/setup.html" id="">Introduction</a></li>
+
+ <li class="sidenavli "><a href="/docs/quick-start.html" data-permalink="/docs/tutorial/setup.html" id="">Quick Start</a></li>
+
+ <li class="sidenavli "><a href="/docs/terminology.html" data-permalink="/docs/tutorial/setup.html" id="">Terminology</a></li>
+
+ <li class="sidenavli "><a href="/docs/ecosystem.html" data-permalink="/docs/tutorial/setup.html" id="">Ecosystem</a></li>
+
+ <li class="sidenavli "><a href="/docs/community.html" data-permalink="/docs/tutorial/setup.html" id="">Community</a></li>
+
+ <li class="sidenavli "><a href="/docs/FAQ.html" data-permalink="/docs/tutorial/setup.html" id="">FAQ</a></li>
+
+ <li class="divider"></li>
+
+ <li class="heading">Installation</li>
+
+ <li class="sidenavli "><a href="/docs/deployment-in-sandbox.html" data-permalink="/docs/tutorial/setup.html" id="">Deploy Eagle in Sandbox</a></li>
+
+ <li class="sidenavli "><a href="/docs/deployment-in-production.html" data-permalink="/docs/tutorial/setup.html" id="">Deploy Eagle in the Production</a></li>
+
+ <li class="sidenavli "><a href="/docs/import-hdfs-auditLog.html" data-permalink="/docs/tutorial/setup.html" id="">Stream Log Data into Kafka</a></li>
+
+ <li class="sidenavli "><a href="/docs/ambari-plugin-install.html" data-permalink="/docs/tutorial/setup.html" id="">Install Eagle Ambari Plugin</a></li>
+
+ <li class="sidenavli "><a href="/docs/configuration.html" data-permalink="/docs/tutorial/setup.html" id="">Configuration</a></li>
+
+ <li class="divider"></li>
+
+ <li class="heading">Tutorial</li>
+
+ <li class="sidenavli current"><a href="/docs/tutorial/setup.html" data-permalink="/docs/tutorial/setup.html" id="">Site Management Tutorial</a></li>
+
+ <li class="sidenavli "><a href="/docs/tutorial/policy.html" data-permalink="/docs/tutorial/setup.html" id="">Eagle Policy Tutorial</a></li>
+
+ <li class="sidenavli "><a href="/docs/tutorial/userprofile.html" data-permalink="/docs/tutorial/setup.html" id="">User Profile Tutorial</a></li>
+
+ <li class="sidenavli "><a href="/docs/tutorial/classification.html" data-permalink="/docs/tutorial/setup.html" id="">Data Classification Tutorial</a></li>
+
+ <li class="divider"></li>
+
+ <li class="heading">Advanced</li>
+
+ <li class="sidenavli "><a href="/docs/architecture-highlights.html" data-permalink="/docs/tutorial/setup.html" id="">Eagle Architecture Highlights</a></li>
+
+ <li class="sidenavli "><a href="/docs/policy-api.html" data-permalink="/docs/tutorial/setup.html" id="">Policy API Design</a></li>
+
+ <li class="sidenavli "><a href="/docs/user-profile-ml.html" data-permalink="/docs/tutorial/setup.html" id="">User Profile Machine Learning</a></li>
+
+ <li class="divider"></li>
+
+ <li class="sidenavli">
+ <a href="https://github.com/ebay/eagle/issues" target="_blank">Need Help?</a>
+ </li>
+ </ul>
+ </div>
+ <div class="col-xs-6 col-sm-9 page-main-content" style="margin-left: -15px" id="loadcontent">
+ <h1 class="page-header" style="margin-top: 0px">Site Management</h1>
+ <p>Eagle identifies different Hadoop environments as different sites, such as sandbox, datacenter1, datacenter2. In each site,
+a user can add different data sources as the monitoring targets. For each data source, a connection configuration is required.</p>
+
+<p>This page shows how to setup a site completely. It has two parts. The first part is about how to add a new site on the web;
+the second part shows how to configure each data source in this site.</p>
+
+<blockquote>
+ <p><strong>Notice</strong>: the following configuration part, we uses Hortonworks sandbox as examples</p>
+</blockquote>
+
+<h3 id="part-1-add-site"><strong>Part 1: Add Site</strong></h3>
+
+<p>The following is an example which creates a new site âDemoâ, and add two data sources as its monitoring targets.
+<img src="/images/docs/new-site.png" alt="setup a site" /></p>
+
+<h3 id="part-2-add-configuration"><strong>Part 2: Add Configuration</strong></h3>
+
+<p>After create a new site, we need to configure the connection string for each data source in this site.</p>
+
+<ul>
+ <li>
+ <p>HDFS Audit Log</p>
+
+ <pre><code> {"hdfsEndpoint":"hdfs://sandbox.hortonworks.com:8020"}
+</code></pre>
+
+ <p><img src="/images/docs/hdfs-setup.png" alt="hdfs setup" /></p>
+ </li>
+ <li>
+ <p>Hive Query Log</p>
+
+ <pre><code> {
+ "accessType": "metastoredb_jdbc",
+ "password": "hive",
+ "user": "hive",
+ "jdbcDriverClassName": "com.mysql.jdbc.Driver",
+ "jdbcUrl": "jdbc:mysql://sandbox.hortonworks.com/hive?createDatabaseIfNotExist=true"
+ }
+</code></pre>
+ </li>
+</ul>
+
+<p><img src="/images/docs/hive-setup.png" alt="hdfs setup" /></p>
+
+<ul>
+ <li>
+ <p>UserProfile</p>
+
+ <pre><code> {
+ "features": "getfileinfo,open,listStatus,setTimes,setPermission,rename,mkdirs,create,setReplication,contentSummary,delete,setOwner,fsck"
+ }
+</code></pre>
+ </li>
+</ul>
+
+<p><img src="/images/docs/userprofile-setup.png" alt="hdfs setup" /></p>
+
+ </div><!--end of loadcontent-->
+ </div>
+ <!--end of centered content-->
+ </div>
+</div>
+<!--end of container-->
+
+<!-- footer start -->
+<div class="footerwrapper">
+ <div class="container">
+ <div class="row">
+ <div class="col-md-12"><div style="margin-left:auto; margin-right:auto; text-align:center">Copyright © 2015 <a href="#">eBay</a> | 2065 Hamilton Ave, San Jose, CA </div></div>
+ </div>
+ </div>
+</div>
+<!-- footer end -->
+
+<!-- JavaScripts -->
+<script src="/js/jquery-1.11.1.min.js"></script>
+<script src="/js/jquery.singlePageNav.js"></script>
+<script src="/js/jquery.flexslider.js"></script>
+<script src="/js/modernizr.min.js"></script>
+<script src="/js/svg.js"></script>
+<script>
+ /************** FlexSlider *********************/
+ $('.flexslider').flexslider({
+ animation: "fade",
+ directionNav: false
+ });
+</script>
+
+<script>
+ /************** FlexSlider *********************/
+ $('.flexslider').flexslider({
+ animation: "fade",
+ directionNav: false
+ });
+</script>
+</body>
+</html>
\ No newline at end of file
Added: incubator/eagle/site/docs/tutorial/userprofile.html
URL: http://svn.apache.org/viewvc/incubator/eagle/site/docs/tutorial/userprofile.html?rev=1715179&view=auto
==============================================================================
--- incubator/eagle/site/docs/tutorial/userprofile.html (added)
+++ incubator/eagle/site/docs/tutorial/userprofile.html Thu Nov 19 13:47:10 2015
@@ -0,0 +1,301 @@
+<!DOCTYPE html>
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+
+ <title>Eagle - User Profile Tutorial</title>
+ <meta name="description" content="Eagle - Secure Hadoop in Real Time">
+
+ <meta name="keywords" content="Eagle, Hadoop, Security, Real Time">
+ <meta name="author" content="eBay Inc.">
+
+ <meta charset="utf-8">
+ <meta name="viewport" content="initial-scale=1">
+
+ <link rel="stylesheet" href="/css/animate.css">
+ <link rel="stylesheet" href="/css/bootstrap.min.css">
+
+ <link rel="stylesheet" href="/css/font-awesome.min.css">
+
+ <link rel="stylesheet" href="/css/misc.css">
+ <link rel="stylesheet" href="/css/style.css">
+ <link rel="stylesheet" href="/css/styles.css">
+ <link rel="stylesheet" href="/css/main.css">
+ <link rel="alternate" type="application/rss+xml" title="Eagle" href="http://goeagle.io/feed.xml" />
+ <link rel="shortcut icon" href="/images/favicon.png">
+
+ <!-- Baidu Analytics Tracking-->
+ <script>
+ var _hmt = _hmt || [];
+ (function() {
+ var hm = document.createElement("script");
+ hm.src = "//hm.baidu.com/hm.js?fedc55df2ea52777a679192e8f849ece";
+ var s = document.getElementsByTagName("script")[0];
+ s.parentNode.insertBefore(hm, s);
+ })();
+ </script>
+
+ <!-- Google Analytics Tracking -->
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+ ga('create', 'UA-68929805-1', 'auto');
+ ga('send', 'pageview');
+ </script>
+</head>
+<body>
+<!-- header start -->
+<div id="home_page">
+ <div class="topbar">
+ <div class="container">
+ <div class="row" >
+ <nav class="navbar navbar-default">
+ <div class="container-fluid">
+ <!-- Brand and toggle get grouped for better mobile display -->
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1"> <span class="sr-only">Toggle navigation</span> <span class="icon-bar"></span> <span class="icon-bar"></span> <span class="icon-bar"></span> </button>
+ <a class="navbar-brand" href="/"><img src="/images/logo2.png" height="44px" style="margin-top:-7px"></a> </div>
+
+ <!-- Collect the nav links, forms, and other content for toggling -->
+ <!-- <div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
+ <ul class="nav navbar-nav navbar-right" id="top-menu">
+ <li><a class="menu" href="/#home_page">HOME</a></li>
+ <li><a class="menu" href="/docs/">DOCS</a></li>
+ <li><a class="menu" href="/#about_page">ABOUT</a></li>
+ <li><a class="menu" href="/#diagram_page">ARCHITECTURE</a></li>
+ <li><a class="menu" href="/#modules_page">MODULES</a></li>
+ <li><a class="menu" href="/#usecase_page">USE CASES</a></li>
+ <li>
+ </li>
+ </ul> -->
+ </div>
+ </div>
+ <!-- /.container-fluid -->
+ </nav>
+ </div>
+ </div>
+ </div>
+ <div class="headerimage">
+ <div class="flexslider">
+ <ul class="slides">
+ <li><img src="/images/slider/4.jpg" alt="Slide 1"></li>
+ </ul>
+ </div>
+ </div>
+ <div class="particles" style="height:40%"> </div><!---particles-->
+ <div class="slider-caption" style="top:80px;">
+ <div class="homewrapper">
+ <div class="hometitle">
+ <a href="/">
+ <img src="/images/feather.png" height="80px">
+ </a>
+ </div>
+ <div class="hometext">
+ <h2>Secure Hadoop in Real Time</h2>
+ <div class="social-buttons">
+ <!-- <a href="https://github.com/ebay/eagle"><i class="fa fa-github"></i></a>
+ <a href="http://twitter.com/TheApacheEagle"><i class="fa fa-twitter"></i></a>
+ <a href="https://www.facebook.com/TheApacheEagle/"><i class="fa fa-facebook"></i></a>
+ <a href="#"><i class="fa fa-weixin"></i></a> -->
+ <!-- <a href="https://www.weibo.com/ApacheEagle/"><i class="fa fa-weibo"></i></a> -->
+ </div>
+ </div>
+ </div>
+ </div>
+</div>
+<!-- header end -->
+<div class="container-fluid page-content">
+ <a href="https://github.com/ebay/eagle" target="_blank"><img style="z-index:10000;position: fixed; top: 0; right: 0; border: 0;" src="https://camo.githubusercontent.com/365986a132ccd6a44c23a9169022c0b5c890c387/68747470733a2f2f73332e616d617a6f6e6177732e636f6d2f6769746875622f726962626f6e732f666f726b6d655f72696768745f7265645f6161303030302e706e67" alt="Fork eagle on GitHub" data-canonical-src="https://s3.amazonaws.com/github/ribbons/forkme_right_red_aa0000.png"></a>
+
+ <div class="row">
+ <div class="col-md-10 col-md-offset-1">
+ <!-- sidebar -->
+ <div class="col-xs-6 col-sm-3" id="sidebar" role="navigation">
+ <ul class="nav" id="adminnav">
+
+ <li class="heading">Getting Started</li>
+
+ <li class="sidenavli "><a href="/docs/index.html" data-permalink="/docs/tutorial/userprofile.html" id="">Introduction</a></li>
+
+ <li class="sidenavli "><a href="/docs/quick-start.html" data-permalink="/docs/tutorial/userprofile.html" id="">Quick Start</a></li>
+
+ <li class="sidenavli "><a href="/docs/terminology.html" data-permalink="/docs/tutorial/userprofile.html" id="">Terminology</a></li>
+
+ <li class="sidenavli "><a href="/docs/ecosystem.html" data-permalink="/docs/tutorial/userprofile.html" id="">Ecosystem</a></li>
+
+ <li class="sidenavli "><a href="/docs/community.html" data-permalink="/docs/tutorial/userprofile.html" id="">Community</a></li>
+
+ <li class="sidenavli "><a href="/docs/FAQ.html" data-permalink="/docs/tutorial/userprofile.html" id="">FAQ</a></li>
+
+ <li class="divider"></li>
+
+ <li class="heading">Installation</li>
+
+ <li class="sidenavli "><a href="/docs/deployment-in-sandbox.html" data-permalink="/docs/tutorial/userprofile.html" id="">Deploy Eagle in Sandbox</a></li>
+
+ <li class="sidenavli "><a href="/docs/deployment-in-production.html" data-permalink="/docs/tutorial/userprofile.html" id="">Deploy Eagle in the Production</a></li>
+
+ <li class="sidenavli "><a href="/docs/import-hdfs-auditLog.html" data-permalink="/docs/tutorial/userprofile.html" id="">Stream Log Data into Kafka</a></li>
+
+ <li class="sidenavli "><a href="/docs/ambari-plugin-install.html" data-permalink="/docs/tutorial/userprofile.html" id="">Install Eagle Ambari Plugin</a></li>
+
+ <li class="sidenavli "><a href="/docs/configuration.html" data-permalink="/docs/tutorial/userprofile.html" id="">Configuration</a></li>
+
+ <li class="divider"></li>
+
+ <li class="heading">Tutorial</li>
+
+ <li class="sidenavli "><a href="/docs/tutorial/setup.html" data-permalink="/docs/tutorial/userprofile.html" id="">Site Management Tutorial</a></li>
+
+ <li class="sidenavli "><a href="/docs/tutorial/policy.html" data-permalink="/docs/tutorial/userprofile.html" id="">Eagle Policy Tutorial</a></li>
+
+ <li class="sidenavli current"><a href="/docs/tutorial/userprofile.html" data-permalink="/docs/tutorial/userprofile.html" id="">User Profile Tutorial</a></li>
+
+ <li class="sidenavli "><a href="/docs/tutorial/classification.html" data-permalink="/docs/tutorial/userprofile.html" id="">Data Classification Tutorial</a></li>
+
+ <li class="divider"></li>
+
+ <li class="heading">Advanced</li>
+
+ <li class="sidenavli "><a href="/docs/architecture-highlights.html" data-permalink="/docs/tutorial/userprofile.html" id="">Eagle Architecture Highlights</a></li>
+
+ <li class="sidenavli "><a href="/docs/policy-api.html" data-permalink="/docs/tutorial/userprofile.html" id="">Policy API Design</a></li>
+
+ <li class="sidenavli "><a href="/docs/user-profile-ml.html" data-permalink="/docs/tutorial/userprofile.html" id="">User Profile Machine Learning</a></li>
+
+ <li class="divider"></li>
+
+ <li class="sidenavli">
+ <a href="https://github.com/ebay/eagle/issues" target="_blank">Need Help?</a>
+ </li>
+ </ul>
+ </div>
+ <div class="col-xs-6 col-sm-9 page-main-content" style="margin-left: -15px" id="loadcontent">
+ <h1 class="page-header" style="margin-top: 0px">User Profile Tutorial</h1>
+ <p>This document will introduce how to start the online processing on user profiles. Assume Eagle has been installed and <a href="http://sandbox.hortonworks.com:9099/eagle-service">Eagle service</a>
+is started.</p>
+
+<h3 id="user-profile-offline-training">User Profile Offline Training</h3>
+
+<ul>
+ <li>
+ <p><strong>Step 1</strong>: Start Spark if not started
+<img src="/images/docs/start-spark.png" alt="Start Spark" /></p>
+ </li>
+ <li>
+ <p><strong>Step 2</strong>: start offline scheduler</p>
+
+ <ul>
+ <li>
+ <p>Option 1: command line</p>
+
+ <pre><code>$ cd <eagle-home>/bin
+$ bin/eagle-userprofile-scheduler.sh --site sandbox start
+</code></pre>
+ </li>
+ <li>
+ <p>Option 2: start via Ambari
+ <img src="/images/docs/offline-userprofile.png" alt="Click "ops"" /></p>
+ </li>
+ </ul>
+ </li>
+ <li>
+ <p><strong>Step 3</strong>: generate a model</p>
+
+ <p><img src="/images/docs/userProfile1.png" alt="Click "ops"" />
+ <img src="/images/docs/userProfile2.png" alt="Click "Update Now"" />
+ <img src="/images/docs/userProfile3.png" alt="Click "Confirm"" />
+ <img src="/images/docs/userprofile4.png" alt="Check" /></p>
+ </li>
+</ul>
+
+<h3 id="user-profile-online-detection">User Profile Online Detection</h3>
+
+<p>Two options to start the topology are provided.</p>
+
+<ul>
+ <li>
+ <p><strong>Option 1</strong>: command line</p>
+
+ <p>submit userProfiles topology if itâs not on <a href="http://sandbox.hortonworks.com:8744">topology UI</a></p>
+
+ <pre><code>$ bin/eagle-topology.sh --main eagle.security.userprofile.UserProfileDetectionMain --config conf/sandbox-userprofile-topology.conf start
+</code></pre>
+ </li>
+ <li>
+ <p><strong>Option 2</strong>: Ambari</p>
+
+ <p><img src="/images/docs/online-userprofile.png" alt="Online userProfiles" /></p>
+ </li>
+</ul>
+
+<h3 id="evaluate-user-profile-in-sandbox">Evaluate User Profile in Sandbox</h3>
+
+<ol>
+ <li>Prepare sample data for ML training and validation sample data
+ <ul>
+ <li>a. Download following sample data to be used for training</li>
+ <li><a href="/data/user1.hdfs-audit.2015-10-11-00.txt"><code>user1.hdfs-audit.2015-10-11-00.txt</code></a></li>
+ <li><a href="/data/user1.hdfs-audit.2015-10-11-01.txt"><code>user1.hdfs-audit.2015-10-11-01.txt</code></a></li>
+ <li>b. Downlaod <a href="/data/userprofile-validate.txt"><code>userprofile-validate.txt</code></a>file which contains data points that you can try to test the models</li>
+ </ul>
+ </li>
+ <li>Copy the files (downloaded in the previous step) into a location in sandbox
+For example: <code>/usr/hdp/current/eagle/lib/userprofile/data/</code></li>
+ <li>Modify <code><Eagle-home>/conf/sandbox-userprofile-scheduler.conf </code>
+update <code>training-audit-path</code> to set to the path for training data sample (the path you used for Step 1.a)
+update detection-audit-path to set to the path for validation (the path you used for Step 1.b)</li>
+ <li>Run ML training program from eagle UI</li>
+ <li>
+ <p>Produce kafka data using the contents from validate file (Step 1.b)
+Run the command (assuming the eagle configuration uses kafka topic <code>sandbox_hdfs_audit_log</code>)</p>
+
+ <pre><code> ./kafka-console-producer.sh --broker-list sandbox.hortonworks.com:6667 --topic sandbox_hdfs_audit_log
+</code></pre>
+ </li>
+ <li>Paste few lines of data from file validate onto kafka-console-producer
+Check <a href="http://localhost:9099/eagle-service/#/dam/alertList">http://localhost:9099/eagle-service/#/dam/alertList</a> for generated alerts</li>
+</ol>
+
+ </div><!--end of loadcontent-->
+ </div>
+ <!--end of centered content-->
+ </div>
+</div>
+<!--end of container-->
+
+<!-- footer start -->
+<div class="footerwrapper">
+ <div class="container">
+ <div class="row">
+ <div class="col-md-12"><div style="margin-left:auto; margin-right:auto; text-align:center">Copyright © 2015 <a href="#">eBay</a> | 2065 Hamilton Ave, San Jose, CA </div></div>
+ </div>
+ </div>
+</div>
+<!-- footer end -->
+
+<!-- JavaScripts -->
+<script src="/js/jquery-1.11.1.min.js"></script>
+<script src="/js/jquery.singlePageNav.js"></script>
+<script src="/js/jquery.flexslider.js"></script>
+<script src="/js/modernizr.min.js"></script>
+<script src="/js/svg.js"></script>
+<script>
+ /************** FlexSlider *********************/
+ $('.flexslider').flexslider({
+ animation: "fade",
+ directionNav: false
+ });
+</script>
+
+<script>
+ /************** FlexSlider *********************/
+ $('.flexslider').flexslider({
+ animation: "fade",
+ directionNav: false
+ });
+</script>
+</body>
+</html>
\ No newline at end of file
Added: incubator/eagle/site/docs/usecase.html
URL: http://svn.apache.org/viewvc/incubator/eagle/site/docs/usecase.html?rev=1715179&view=auto
==============================================================================
--- incubator/eagle/site/docs/usecase.html (added)
+++ incubator/eagle/site/docs/usecase.html Thu Nov 19 13:47:10 2015
@@ -0,0 +1,227 @@
+<!DOCTYPE html>
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+
+ <title>Eagle - Use Cases</title>
+ <meta name="description" content="Eagle - Secure Hadoop in Real Time">
+
+ <meta name="keywords" content="Eagle, Hadoop, Security, Real Time">
+ <meta name="author" content="eBay Inc.">
+
+ <meta charset="utf-8">
+ <meta name="viewport" content="initial-scale=1">
+
+ <link rel="stylesheet" href="/css/animate.css">
+ <link rel="stylesheet" href="/css/bootstrap.min.css">
+
+ <link rel="stylesheet" href="/css/font-awesome.min.css">
+
+ <link rel="stylesheet" href="/css/misc.css">
+ <link rel="stylesheet" href="/css/style.css">
+ <link rel="stylesheet" href="/css/styles.css">
+ <link rel="stylesheet" href="/css/main.css">
+ <link rel="alternate" type="application/rss+xml" title="Eagle" href="http://goeagle.io/feed.xml" />
+ <link rel="shortcut icon" href="/images/favicon.png">
+
+ <!-- Baidu Analytics Tracking-->
+ <script>
+ var _hmt = _hmt || [];
+ (function() {
+ var hm = document.createElement("script");
+ hm.src = "//hm.baidu.com/hm.js?fedc55df2ea52777a679192e8f849ece";
+ var s = document.getElementsByTagName("script")[0];
+ s.parentNode.insertBefore(hm, s);
+ })();
+ </script>
+
+ <!-- Google Analytics Tracking -->
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+ ga('create', 'UA-68929805-1', 'auto');
+ ga('send', 'pageview');
+ </script>
+</head>
+<body>
+<!-- header start -->
+<div id="home_page">
+ <div class="topbar">
+ <div class="container">
+ <div class="row" >
+ <nav class="navbar navbar-default">
+ <div class="container-fluid">
+ <!-- Brand and toggle get grouped for better mobile display -->
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1"> <span class="sr-only">Toggle navigation</span> <span class="icon-bar"></span> <span class="icon-bar"></span> <span class="icon-bar"></span> </button>
+ <a class="navbar-brand" href="/"><img src="/images/logo2.png" height="44px" style="margin-top:-7px"></a> </div>
+
+ <!-- Collect the nav links, forms, and other content for toggling -->
+ <!-- <div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
+ <ul class="nav navbar-nav navbar-right" id="top-menu">
+ <li><a class="menu" href="/#home_page">HOME</a></li>
+ <li><a class="menu" href="/docs/">DOCS</a></li>
+ <li><a class="menu" href="/#about_page">ABOUT</a></li>
+ <li><a class="menu" href="/#diagram_page">ARCHITECTURE</a></li>
+ <li><a class="menu" href="/#modules_page">MODULES</a></li>
+ <li><a class="menu" href="/#usecase_page">USE CASES</a></li>
+ <li>
+ </li>
+ </ul> -->
+ </div>
+ </div>
+ <!-- /.container-fluid -->
+ </nav>
+ </div>
+ </div>
+ </div>
+ <div class="headerimage">
+ <div class="flexslider">
+ <ul class="slides">
+ <li><img src="/images/slider/4.jpg" alt="Slide 1"></li>
+ </ul>
+ </div>
+ </div>
+ <div class="particles" style="height:40%"> </div><!---particles-->
+ <div class="slider-caption" style="top:80px;">
+ <div class="homewrapper">
+ <div class="hometitle">
+ <a href="/">
+ <img src="/images/feather.png" height="80px">
+ </a>
+ </div>
+ <div class="hometext">
+ <h2>Secure Hadoop in Real Time</h2>
+ <div class="social-buttons">
+ <!-- <a href="https://github.com/ebay/eagle"><i class="fa fa-github"></i></a>
+ <a href="http://twitter.com/TheApacheEagle"><i class="fa fa-twitter"></i></a>
+ <a href="https://www.facebook.com/TheApacheEagle/"><i class="fa fa-facebook"></i></a>
+ <a href="#"><i class="fa fa-weixin"></i></a> -->
+ <!-- <a href="https://www.weibo.com/ApacheEagle/"><i class="fa fa-weibo"></i></a> -->
+ </div>
+ </div>
+ </div>
+ </div>
+</div>
+<!-- header end -->
+<div class="container-fluid page-content">
+ <a href="https://github.com/ebay/eagle" target="_blank"><img style="z-index:10000;position: fixed; top: 0; right: 0; border: 0;" src="https://camo.githubusercontent.com/365986a132ccd6a44c23a9169022c0b5c890c387/68747470733a2f2f73332e616d617a6f6e6177732e636f6d2f6769746875622f726962626f6e732f666f726b6d655f72696768745f7265645f6161303030302e706e67" alt="Fork eagle on GitHub" data-canonical-src="https://s3.amazonaws.com/github/ribbons/forkme_right_red_aa0000.png"></a>
+
+ <div class="row">
+ <div class="col-md-10 col-md-offset-1">
+ <!-- sidebar -->
+ <div class="col-xs-6 col-sm-3" id="sidebar" role="navigation">
+ <ul class="nav" id="adminnav">
+
+ <li class="heading">Getting Started</li>
+
+ <li class="sidenavli "><a href="/docs/index.html" data-permalink="/docs/usecase.html" id="">Introduction</a></li>
+
+ <li class="sidenavli "><a href="/docs/quick-start.html" data-permalink="/docs/usecase.html" id="">Quick Start</a></li>
+
+ <li class="sidenavli "><a href="/docs/terminology.html" data-permalink="/docs/usecase.html" id="">Terminology</a></li>
+
+ <li class="sidenavli "><a href="/docs/ecosystem.html" data-permalink="/docs/usecase.html" id="">Ecosystem</a></li>
+
+ <li class="sidenavli "><a href="/docs/community.html" data-permalink="/docs/usecase.html" id="">Community</a></li>
+
+ <li class="sidenavli "><a href="/docs/FAQ.html" data-permalink="/docs/usecase.html" id="">FAQ</a></li>
+
+ <li class="divider"></li>
+
+ <li class="heading">Installation</li>
+
+ <li class="sidenavli "><a href="/docs/deployment-in-sandbox.html" data-permalink="/docs/usecase.html" id="">Deploy Eagle in Sandbox</a></li>
+
+ <li class="sidenavli "><a href="/docs/deployment-in-production.html" data-permalink="/docs/usecase.html" id="">Deploy Eagle in the Production</a></li>
+
+ <li class="sidenavli "><a href="/docs/import-hdfs-auditLog.html" data-permalink="/docs/usecase.html" id="">Stream Log Data into Kafka</a></li>
+
+ <li class="sidenavli "><a href="/docs/ambari-plugin-install.html" data-permalink="/docs/usecase.html" id="">Install Eagle Ambari Plugin</a></li>
+
+ <li class="sidenavli "><a href="/docs/configuration.html" data-permalink="/docs/usecase.html" id="">Configuration</a></li>
+
+ <li class="divider"></li>
+
+ <li class="heading">Tutorial</li>
+
+ <li class="sidenavli "><a href="/docs/tutorial/setup.html" data-permalink="/docs/usecase.html" id="">Site Management Tutorial</a></li>
+
+ <li class="sidenavli "><a href="/docs/tutorial/policy.html" data-permalink="/docs/usecase.html" id="">Eagle Policy Tutorial</a></li>
+
+ <li class="sidenavli "><a href="/docs/tutorial/userprofile.html" data-permalink="/docs/usecase.html" id="">User Profile Tutorial</a></li>
+
+ <li class="sidenavli "><a href="/docs/tutorial/classification.html" data-permalink="/docs/usecase.html" id="">Data Classification Tutorial</a></li>
+
+ <li class="divider"></li>
+
+ <li class="heading">Advanced</li>
+
+ <li class="sidenavli "><a href="/docs/architecture-highlights.html" data-permalink="/docs/usecase.html" id="">Eagle Architecture Highlights</a></li>
+
+ <li class="sidenavli "><a href="/docs/policy-api.html" data-permalink="/docs/usecase.html" id="">Policy API Design</a></li>
+
+ <li class="sidenavli "><a href="/docs/user-profile-ml.html" data-permalink="/docs/usecase.html" id="">User Profile Machine Learning</a></li>
+
+ <li class="divider"></li>
+
+ <li class="sidenavli">
+ <a href="https://github.com/ebay/eagle/issues" target="_blank">Need Help?</a>
+ </li>
+ </ul>
+ </div>
+ <div class="col-xs-6 col-sm-9 page-main-content" style="margin-left: -15px" id="loadcontent">
+ <h1 class="page-header" style="margin-top: 0px">Use Cases</h1>
+ <p>Here are some of the hadoop eagle data activity monitoring use cases:</p>
+
+<ul>
+ <li>Anomalous access detection.</li>
+ <li>Monitor data access traffic.</li>
+ <li>Discover intrusions and security breach.</li>
+ <li>Discover and prevent sensitive data loss and leaks.</li>
+ <li>Monitor data access based on user behaviour patterns</li>
+ <li>Ability to analyze access logs for audit purpose</li>
+</ul>
+
+ </div><!--end of loadcontent-->
+ </div>
+ <!--end of centered content-->
+ </div>
+</div>
+<!--end of container-->
+
+<!-- footer start -->
+<div class="footerwrapper">
+ <div class="container">
+ <div class="row">
+ <div class="col-md-12"><div style="margin-left:auto; margin-right:auto; text-align:center">Copyright © 2015 <a href="#">eBay</a> | 2065 Hamilton Ave, San Jose, CA </div></div>
+ </div>
+ </div>
+</div>
+<!-- footer end -->
+
+<!-- JavaScripts -->
+<script src="/js/jquery-1.11.1.min.js"></script>
+<script src="/js/jquery.singlePageNav.js"></script>
+<script src="/js/jquery.flexslider.js"></script>
+<script src="/js/modernizr.min.js"></script>
+<script src="/js/svg.js"></script>
+<script>
+ /************** FlexSlider *********************/
+ $('.flexslider').flexslider({
+ animation: "fade",
+ directionNav: false
+ });
+</script>
+
+<script>
+ /************** FlexSlider *********************/
+ $('.flexslider').flexslider({
+ animation: "fade",
+ directionNav: false
+ });
+</script>
+</body>
+</html>
\ No newline at end of file
Added: incubator/eagle/site/docs/user-profile-ml.html
URL: http://svn.apache.org/viewvc/incubator/eagle/site/docs/user-profile-ml.html?rev=1715179&view=auto
==============================================================================
--- incubator/eagle/site/docs/user-profile-ml.html (added)
+++ incubator/eagle/site/docs/user-profile-ml.html Thu Nov 19 13:47:10 2015
@@ -0,0 +1,236 @@
+<!DOCTYPE html>
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+
+ <title>Eagle - User Profile Machine Learning</title>
+ <meta name="description" content="Eagle - Secure Hadoop in Real Time">
+
+ <meta name="keywords" content="Eagle, Hadoop, Security, Real Time">
+ <meta name="author" content="eBay Inc.">
+
+ <meta charset="utf-8">
+ <meta name="viewport" content="initial-scale=1">
+
+ <link rel="stylesheet" href="/css/animate.css">
+ <link rel="stylesheet" href="/css/bootstrap.min.css">
+
+ <link rel="stylesheet" href="/css/font-awesome.min.css">
+
+ <link rel="stylesheet" href="/css/misc.css">
+ <link rel="stylesheet" href="/css/style.css">
+ <link rel="stylesheet" href="/css/styles.css">
+ <link rel="stylesheet" href="/css/main.css">
+ <link rel="alternate" type="application/rss+xml" title="Eagle" href="http://goeagle.io/feed.xml" />
+ <link rel="shortcut icon" href="/images/favicon.png">
+
+ <!-- Baidu Analytics Tracking-->
+ <script>
+ var _hmt = _hmt || [];
+ (function() {
+ var hm = document.createElement("script");
+ hm.src = "//hm.baidu.com/hm.js?fedc55df2ea52777a679192e8f849ece";
+ var s = document.getElementsByTagName("script")[0];
+ s.parentNode.insertBefore(hm, s);
+ })();
+ </script>
+
+ <!-- Google Analytics Tracking -->
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+ ga('create', 'UA-68929805-1', 'auto');
+ ga('send', 'pageview');
+ </script>
+</head>
+<body>
+<!-- header start -->
+<div id="home_page">
+ <div class="topbar">
+ <div class="container">
+ <div class="row" >
+ <nav class="navbar navbar-default">
+ <div class="container-fluid">
+ <!-- Brand and toggle get grouped for better mobile display -->
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1"> <span class="sr-only">Toggle navigation</span> <span class="icon-bar"></span> <span class="icon-bar"></span> <span class="icon-bar"></span> </button>
+ <a class="navbar-brand" href="/"><img src="/images/logo2.png" height="44px" style="margin-top:-7px"></a> </div>
+
+ <!-- Collect the nav links, forms, and other content for toggling -->
+ <!-- <div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
+ <ul class="nav navbar-nav navbar-right" id="top-menu">
+ <li><a class="menu" href="/#home_page">HOME</a></li>
+ <li><a class="menu" href="/docs/">DOCS</a></li>
+ <li><a class="menu" href="/#about_page">ABOUT</a></li>
+ <li><a class="menu" href="/#diagram_page">ARCHITECTURE</a></li>
+ <li><a class="menu" href="/#modules_page">MODULES</a></li>
+ <li><a class="menu" href="/#usecase_page">USE CASES</a></li>
+ <li>
+ </li>
+ </ul> -->
+ </div>
+ </div>
+ <!-- /.container-fluid -->
+ </nav>
+ </div>
+ </div>
+ </div>
+ <div class="headerimage">
+ <div class="flexslider">
+ <ul class="slides">
+ <li><img src="/images/slider/4.jpg" alt="Slide 1"></li>
+ </ul>
+ </div>
+ </div>
+ <div class="particles" style="height:40%"> </div><!---particles-->
+ <div class="slider-caption" style="top:80px;">
+ <div class="homewrapper">
+ <div class="hometitle">
+ <a href="/">
+ <img src="/images/feather.png" height="80px">
+ </a>
+ </div>
+ <div class="hometext">
+ <h2>Secure Hadoop in Real Time</h2>
+ <div class="social-buttons">
+ <!-- <a href="https://github.com/ebay/eagle"><i class="fa fa-github"></i></a>
+ <a href="http://twitter.com/TheApacheEagle"><i class="fa fa-twitter"></i></a>
+ <a href="https://www.facebook.com/TheApacheEagle/"><i class="fa fa-facebook"></i></a>
+ <a href="#"><i class="fa fa-weixin"></i></a> -->
+ <!-- <a href="https://www.weibo.com/ApacheEagle/"><i class="fa fa-weibo"></i></a> -->
+ </div>
+ </div>
+ </div>
+ </div>
+</div>
+<!-- header end -->
+<div class="container-fluid page-content">
+ <a href="https://github.com/ebay/eagle" target="_blank"><img style="z-index:10000;position: fixed; top: 0; right: 0; border: 0;" src="https://camo.githubusercontent.com/365986a132ccd6a44c23a9169022c0b5c890c387/68747470733a2f2f73332e616d617a6f6e6177732e636f6d2f6769746875622f726962626f6e732f666f726b6d655f72696768745f7265645f6161303030302e706e67" alt="Fork eagle on GitHub" data-canonical-src="https://s3.amazonaws.com/github/ribbons/forkme_right_red_aa0000.png"></a>
+
+ <div class="row">
+ <div class="col-md-10 col-md-offset-1">
+ <!-- sidebar -->
+ <div class="col-xs-6 col-sm-3" id="sidebar" role="navigation">
+ <ul class="nav" id="adminnav">
+
+ <li class="heading">Getting Started</li>
+
+ <li class="sidenavli "><a href="/docs/index.html" data-permalink="/docs/user-profile-ml.html" id="">Introduction</a></li>
+
+ <li class="sidenavli "><a href="/docs/quick-start.html" data-permalink="/docs/user-profile-ml.html" id="">Quick Start</a></li>
+
+ <li class="sidenavli "><a href="/docs/terminology.html" data-permalink="/docs/user-profile-ml.html" id="">Terminology</a></li>
+
+ <li class="sidenavli "><a href="/docs/ecosystem.html" data-permalink="/docs/user-profile-ml.html" id="">Ecosystem</a></li>
+
+ <li class="sidenavli "><a href="/docs/community.html" data-permalink="/docs/user-profile-ml.html" id="">Community</a></li>
+
+ <li class="sidenavli "><a href="/docs/FAQ.html" data-permalink="/docs/user-profile-ml.html" id="">FAQ</a></li>
+
+ <li class="divider"></li>
+
+ <li class="heading">Installation</li>
+
+ <li class="sidenavli "><a href="/docs/deployment-in-sandbox.html" data-permalink="/docs/user-profile-ml.html" id="">Deploy Eagle in Sandbox</a></li>
+
+ <li class="sidenavli "><a href="/docs/deployment-in-production.html" data-permalink="/docs/user-profile-ml.html" id="">Deploy Eagle in the Production</a></li>
+
+ <li class="sidenavli "><a href="/docs/import-hdfs-auditLog.html" data-permalink="/docs/user-profile-ml.html" id="">Stream Log Data into Kafka</a></li>
+
+ <li class="sidenavli "><a href="/docs/ambari-plugin-install.html" data-permalink="/docs/user-profile-ml.html" id="">Install Eagle Ambari Plugin</a></li>
+
+ <li class="sidenavli "><a href="/docs/configuration.html" data-permalink="/docs/user-profile-ml.html" id="">Configuration</a></li>
+
+ <li class="divider"></li>
+
+ <li class="heading">Tutorial</li>
+
+ <li class="sidenavli "><a href="/docs/tutorial/setup.html" data-permalink="/docs/user-profile-ml.html" id="">Site Management Tutorial</a></li>
+
+ <li class="sidenavli "><a href="/docs/tutorial/policy.html" data-permalink="/docs/user-profile-ml.html" id="">Eagle Policy Tutorial</a></li>
+
+ <li class="sidenavli "><a href="/docs/tutorial/userprofile.html" data-permalink="/docs/user-profile-ml.html" id="">User Profile Tutorial</a></li>
+
+ <li class="sidenavli "><a href="/docs/tutorial/classification.html" data-permalink="/docs/user-profile-ml.html" id="">Data Classification Tutorial</a></li>
+
+ <li class="divider"></li>
+
+ <li class="heading">Advanced</li>
+
+ <li class="sidenavli "><a href="/docs/architecture-highlights.html" data-permalink="/docs/user-profile-ml.html" id="">Eagle Architecture Highlights</a></li>
+
+ <li class="sidenavli "><a href="/docs/policy-api.html" data-permalink="/docs/user-profile-ml.html" id="">Policy API Design</a></li>
+
+ <li class="sidenavli current"><a href="/docs/user-profile-ml.html" data-permalink="/docs/user-profile-ml.html" id="">User Profile Machine Learning</a></li>
+
+ <li class="divider"></li>
+
+ <li class="sidenavli">
+ <a href="https://github.com/ebay/eagle/issues" target="_blank">Need Help?</a>
+ </li>
+ </ul>
+ </div>
+ <div class="col-xs-6 col-sm-9 page-main-content" style="margin-left: -15px" id="loadcontent">
+ <h1 class="page-header" style="margin-top: 0px">User Profile Machine Learning</h1>
+ <p>Eagle provides capabilities to define user activity patterns or user profiles for Hadoop users based on the user behavior in the platform. The idea is to provide anomaly detection capability without setting hard thresholds in the system. The user profiles generated by our system are modeled using machine-learning algorithms and used for detection of anomalous user activities, where usersâ activity pattern differs from their pattern history. Currently Eagle uses two algorithms for anomaly detection: Eigen-Value Decomposition and Density Estimation. The algorithms read data from HDFS audit logs, slice and dice data, and generate models for each user in the system. Once models are generated, Eagle uses the Storm framework for near-real-time anomaly detection to determine if current user activities are suspicious or not with respect to their model. The block diagram below shows the current pipeline for user profile training and online detection.</p>
+
+<p><img src="/images/docs/userprofile-arch.png" alt="" /></p>
+
+<p>Eagle online anomaly detection uses the Eagle policy framework, and the user profile is defined as one of the policies in the system. The user profile policy is evaluated by a machine-learning evaluator extended from the Eagle policy evaluator. Policy definition includes the features that are needed for anomaly detection (same as the ones used for training purposes).</p>
+
+<p>A scheduler runs a Spark-based offline training program (to generate user profiles or models) at a configurable time interval; currently, the training program generates new models once every month.</p>
+
+<p>The following are some details on the algorithms.</p>
+
+<ul>
+ <li><strong>Density Estimation</strong>: In this algorithm, the idea is to evaluate, for each user, a probability density function from the observed training data sample. We mean-normalize a training dataset for each feature. Normalization allows datasets to be on the same scale. In our probability density estimation, we use a Gaussian distribution function as the method for computing probability density. Features are conditionally independent of one another; therefore, the final Gaussian probability density can be computed by factorizing each featureâs probability density. During the online detection phase, we compute the probability of a userâs activity. If the probability of the user performing the activity is below threshold (determined from the training program, using a method called Mathews Correlation Coefficient), we signal anomaly alerts.</li>
+ <li><strong>Eigen-Value Decomposition</strong>: Our goal in user profile generation is to find interesting behavioral patterns for users. One way to achieve that goal is to consider a combination of features and see how each one influences the others. When the data volume is large, which is generally the case for us, abnormal patterns among features may go unnoticed due to the huge number of normal patterns. As normal behavioral patterns can lie within very low-dimensional subspace, we can potentially reduce the dimension of the dataset to better understand the user behavior pattern. This method also reduces noise, if any, in the training dataset. Based on the amount of variance of the data we maintain for a user, which is usually 95% for our case, we seek to find the number of principal components k that represents 95% variance. We consider first k principal components as normal subspace for the user. The remaining (n-k) principal components are considered as abnormal subspace.</
li>
+</ul>
+
+<p>During online anomaly detection, if the user behavior lies near normal subspace, we consider the behavior to be normal. On the other hand, if the user behavior lies near the abnormal subspace, we raise an alarm as we believe usual user behavior should generally fall within normal subspace. We use the Euclidian distance method to compute whether a userâs current activity is near normal or abnormal subspace.</p>
+
+<p><img src="/images/docs/userprofile-model.png" alt="" /></p>
+
+
+ </div><!--end of loadcontent-->
+ </div>
+ <!--end of centered content-->
+ </div>
+</div>
+<!--end of container-->
+
+<!-- footer start -->
+<div class="footerwrapper">
+ <div class="container">
+ <div class="row">
+ <div class="col-md-12"><div style="margin-left:auto; margin-right:auto; text-align:center">Copyright © 2015 <a href="#">eBay</a> | 2065 Hamilton Ave, San Jose, CA </div></div>
+ </div>
+ </div>
+</div>
+<!-- footer end -->
+
+<!-- JavaScripts -->
+<script src="/js/jquery-1.11.1.min.js"></script>
+<script src="/js/jquery.singlePageNav.js"></script>
+<script src="/js/jquery.flexslider.js"></script>
+<script src="/js/modernizr.min.js"></script>
+<script src="/js/svg.js"></script>
+<script>
+ /************** FlexSlider *********************/
+ $('.flexslider').flexslider({
+ animation: "fade",
+ directionNav: false
+ });
+</script>
+
+<script>
+ /************** FlexSlider *********************/
+ $('.flexslider').flexslider({
+ animation: "fade",
+ directionNav: false
+ });
+</script>
+</body>
+</html>
\ No newline at end of file