You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spot.apache.org by le...@apache.org on 2019/09/11 01:39:34 UTC
[incubator-spot] 05/45: Updating website
This is an automated email from the ASF dual-hosted git repository.
leahy pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/incubator-spot.git
commit 263c2647c6cb9ee93e355038382db9cc4bafbf1c
Author: Mark Grover <ma...@apache.org>
AuthorDate: Wed Oct 12 09:55:42 2016 -0700
Updating website
---
2016/03/index.html | 38 +--
2016/08/index.html | 40 +--
2016/{03 => 09}/index.html | 111 +++----
apache-spot-3-most-asked-questions/feed/index.html | 18 ++
.../index.html | 150 +++++-----
.../feed/index.html | 18 ++
.../index.html | 153 +++++-----
blog/index.html | 79 +++--
category/cybersecurity/feed/index.html | 10 +-
category/cybersecurity/index.html | 35 +--
category/data-science/feed/index.html | 62 ++++
{2016/03 => category/data-science}/index.html | 97 +++---
category/ipython-notebooks/feed/index.html | 62 ++++
{2016/03 => category/ipython-notebooks}/index.html | 95 +++---
category/security-analytics/feed/index.html | 10 +-
category/security-analytics/index.html | 46 ++-
category/threat-analysis-tools/feed/index.html | 62 ++++
.../threat-analysis-tools}/index.html | 97 +++---
category/uncategorized/feed/index.html | 56 +++-
category/uncategorized/index.html | 45 ++-
feed/index.html | 84 ++++--
.../feed/index.html | 18 ++
.../index.html | 135 ++++-----
index.html | 99 +-----
.../feed/index.html | 18 ++
jupyter-notebooks-for-data-analysis/index.html | 333 +++++++++++++++++++++
tag/github/feed/index.html | 12 +-
tag/github/index.html | 46 ++-
tag/open-network-insight/feed/index.html | 16 +-
tag/open-network-insight/index.html | 50 ++--
tag/open-source/feed/index.html | 2 +-
tag/open-source/index.html | 44 +--
wp-content/themes/oni/favicon.ico | Bin 0 -> 1150 bytes
wp-content/themes/oni/favicon.png | Bin 564 -> 2787 bytes
wp-content/themes/oni/library/css/style.css | 2 +
.../themes/oni/library/images/add-to-slack.png | Bin 0 -> 3789 bytes
.../themes/oni/library/images/apple-touch-icon.png | Bin 5971 -> 11173 bytes
.../themes/oni/library/images/win8-tile-icon.png | Bin 0 -> 14766 bytes
wp-content/uploads/2016/03/advantages-1.png | Bin 35716 -> 35751 bytes
wp-content/uploads/2016/03/how-it-works.png | Bin 47916 -> 52599 bytes
.../2016/09/ONI_Architecture-Diagram_1300_v4.png | Bin 431626 -> 449035 bytes
...ng => ONI_Architecture-Diagram_1300_v4_old.png} | Bin
wp-content/uploads/2016/10/iPython-1-300x89.png | Bin 0 -> 13403 bytes
wp-content/uploads/2016/10/iPython-1.png | Bin 0 -> 12682 bytes
44 files changed, 1252 insertions(+), 891 deletions(-)
diff --git a/2016/03/index.html b/2016/03/index.html
index 4971dc0..e1ee89e 100644
--- a/2016/03/index.html
+++ b/2016/03/index.html
@@ -125,24 +125,6 @@
<script type='text/javascript' src='../../wp-content/themes/oni/library/js/jquery.meanmenu.js'></script>
<link rel='https://api.w.org/' href='../../wp-json/' />
- <script>
- (function(i, s, o, g, r, a, m) {
- i['GoogleAnalyticsObject'] = r;
- i[r] = i[r] ||
- function() {
- (i[r].q = i[r].q || []).push(arguments)
- }, i[r].l = 1 * new Date();
- a = s.createElement(o),
- m = s.getElementsByTagName(o)[0];
- a.async = 1;
- a.src = g;
- m.parentNode.insertBefore(a, m)
- })(window, document, 'script', '//www.google-analytics.com/analytics.js', 'ga');
-
- ga('create', 'UA-75955621-1', 'auto');
- ga('send', 'pageview');
-
- </script>
</head>
<body class="archive date" itemscope itemtype="http://schema.org/WebPage">
@@ -195,7 +177,7 @@
<header class="entry-header article-header">
- <h3 class="h2 entry-title"><a href="../../open-network-insight-3-most-asked-questions/" rel="bookmark" title="Apache Spot: Three Most-Asked Questions">Apache Spot: Three Most-Asked Questions</a></h3>
+ <h3 class="h2 entry-title"><a href="../../apache-spot-3-most-asked-questions/" rel="bookmark" title="Apache Spot: Three Most-Asked Questions">Apache Spot (Incubating): Three Most-Asked Questions</a></h3>
<p class="byline entry-meta vcard">
<time class="updated entry-time" datetime="2016-03-29" itemprop="datePublished">
March 29, 2016
@@ -208,7 +190,7 @@
<section class="entry-content cf">
<p>
- While this is not the first blog post about Apache Spot, it is the first one by a creator of the solution. As a security data scientist in Intel’s Data Center Group, I joined a small team to start thinking about solving really hard problems in cloud analytics. The team grew, and out of that… <a class="excerpt-read-more" href="../../open-network-insight-3-most-asked-questions/" title="Read Apache Spot: Three Most-Asked Questions">Read more & [...]
+ While this is not the first blog post about Apache Spot, it is the first one by a creator of the solution. As a security data scientist in Intel’s Data Center Group, I joined a small team to start thinking about solving really hard problems in cloud analytics. The team grew, and out of that… <a class="excerpt-read-more" href="../../apache-spot-3-most-asked-questions/" title="Read Apache Spot: Three Most-Asked Questions">Read more »</a>
</p>
</section>
@@ -227,13 +209,16 @@
<h4 class="widgettitle">Recent Posts</h4>
<ul>
<li>
- <a href="../../open-network-insight-oni-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/">Apache Spot and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a>
+ <a href="../jupyter-notebooks-for-data-analysis/">Jupyter Notebooks for Data Analysis</a>
+ </li>
+ <li>
+ <a href="../apache-spot-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/">Apache Spot (Incubating) and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a>
</li>
<li>
- <a href="../../how-open-network-insight-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/">How Apache Spot Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a>
+ <a href="../how-apache-spot-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/">How Apache Spot (Incubating) Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a>
</li>
<li>
- <a href="../../open-network-insight-3-most-asked-questions/">Apache Spot: Three Most-Asked Questions</a>
+ <a href="../apache-spot-3-most-asked-questions/">Apache Spot (Incubating): 3 Most-Asked Questions</a>
</li>
</ul>
</div>
@@ -241,10 +226,13 @@
<h4 class="widgettitle">Archives</h4>
<ul>
<li>
- <a href='../../2016/08/'>August 2016</a>
+ <a href='../09/'>September 2016</a>
+ </li>
+ <li>
+ <a href='../08/'>August 2016</a>
</li>
<li>
- <a href='../../2016/03/'>March 2016</a>
+ <a href='../03/'>March 2016</a>
</li>
</ul>
</div>
diff --git a/2016/08/index.html b/2016/08/index.html
index 455c751..a1eddf4 100644
--- a/2016/08/index.html
+++ b/2016/08/index.html
@@ -125,24 +125,6 @@
<script type='text/javascript' src='../../wp-content/themes/oni/library/js/jquery.meanmenu.js'></script>
<link rel='https://api.w.org/' href='../../wp-json/' />
- <script>
- (function(i, s, o, g, r, a, m) {
- i['GoogleAnalyticsObject'] = r;
- i[r] = i[r] ||
- function() {
- (i[r].q = i[r].q || []).push(arguments)
- }, i[r].l = 1 * new Date();
- a = s.createElement(o),
- m = s.getElementsByTagName(o)[0];
- a.async = 1;
- a.src = g;
- m.parentNode.insertBefore(a, m)
- })(window, document, 'script', '//www.google-analytics.com/analytics.js', 'ga');
-
- ga('create', 'UA-75955621-1', 'auto');
- ga('send', 'pageview');
-
- </script>
</head>
<body class="archive date" itemscope itemtype="http://schema.org/WebPage">
@@ -195,7 +177,7 @@
<header class="entry-header article-header">
- <h3 class="h2 entry-title"><a href="../../open-network-insight-oni-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/" rel="bookmark" title="Apache Spot and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure">Apache Spot and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a></h3>
+ <h3 class="h2 entry-title"><a href="../../apache-spot-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/" rel="bookmark" title="Apache Spot and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure">Apache Spot (Incubating) and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a></h3>
<p class="byline entry-meta vcard">
<time class="updated entry-time" datetime="2016-08-08" itemprop="datePublished">
August 8, 2016
@@ -223,7 +205,7 @@
<header class="entry-header article-header">
- <h3 class="h2 entry-title"><a href="../../how-open-network-insight-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/" rel="bookmark" title="How Apache Spot Helps Create Well-Stocked Data Lakes and Catch Powerful Insights">How Apache Spot Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a></h3>
+ <h3 class="h2 entry-title"><a href="../../how-apache-spot-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/" rel="bookmark" title="How Apache Spot Helps Create Well-Stocked Data Lakes and Catch Powerful Insights">How Apache Spot (Incubating) Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a></h3>
<p class="byline entry-meta vcard">
<time class="updated entry-time" datetime="2016-08-08" itemprop="datePublished">
August 8, 2016
@@ -236,7 +218,7 @@
<section class="entry-content cf">
<p>
- About four years ago, the era of the Big Data analytics began. Paired with advanced analytics, massive volumes of data can be culled to not only inform critical decisions, but also to simulate sophisticated “what if” scenarios that allow companies to gain competitive advantages by generating and predicting different scenarios. For example, a financial services… <a class="excerpt-read-more" href="../../how-open-network-insight-helps-create-well-st [...]
+ About four years ago, the era of the Big Data analytics began. Paired with advanced analytics, massive volumes of data can be culled to not only inform critical decisions, but also to simulate sophisticated “what if” scenarios that allow companies to gain competitive advantages by generating and predicting different scenarios. For example, a financial services… <a class="excerpt-read-more" href="../../how-apache-spot-helps-create-well-stocked-dat [...]
</p>
</section>
@@ -255,13 +237,16 @@
<h4 class="widgettitle">Recent Posts</h4>
<ul>
<li>
- <a href="../../open-network-insight-oni-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/">Apache Spot and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a>
+ <a href="../jupyter-notebooks-for-data-analysis/">Jupyter Notebooks for Data Analysis</a>
+ </li>
+ <li>
+ <a href="../apache-spot-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/">Apache Spot (Incubating) and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a>
</li>
<li>
- <a href="../../how-open-network-insight-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/">How Apache Spot Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a>
+ <a href="../how-apache-spot-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/">How Apache Spot (Incubating) Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a>
</li>
<li>
- <a href="../../open-network-insight-3-most-asked-questions/">Apache Spot: Three Most-Asked Questions</a>
+ <a href="../apache-spot-3-most-asked-questions/">Apache Spot (Incubating): 3 Most-Asked Questions</a>
</li>
</ul>
</div>
@@ -269,10 +254,13 @@
<h4 class="widgettitle">Archives</h4>
<ul>
<li>
- <a href='../../2016/08/'>August 2016</a>
+ <a href='../09/'>September 2016</a>
+ </li>
+ <li>
+ <a href='../08/'>August 2016</a>
</li>
<li>
- <a href='../../2016/03/'>March 2016</a>
+ <a href='../03/'>March 2016</a>
</li>
</ul>
</div>
diff --git a/2016/03/index.html b/2016/09/index.html
similarity index 63%
copy from 2016/03/index.html
copy to 2016/09/index.html
index 4971dc0..006c8f3 100644
--- a/2016/03/index.html
+++ b/2016/09/index.html
@@ -12,33 +12,29 @@
<meta http-equiv="X-UA-Compatible" content="IE=edge">
- <title>Apache Spot 2016 March</title>
+ <title>Apache Spot 2016 September</title>
<meta name="HandheldFriendly" content="True">
<meta name="MobileOptimized" content="320">
<meta name="viewport" content="width=device-width, initial-scale=1"/>
- <link rel="apple-touch-icon" href="../../wp-content/themes/oni/library/images/apple-touch-icon.png">
- <link rel="icon" href="../../wp-content/themes/oni/favicon.png">
+ <link rel="apple-touch-icon" href="../../../../wp-content/themes/oni/library/images/apple-touch-icon.png">
+ <link rel="icon" href="../../../../wp-content/themes/oni/favicon.png">
<!--[if IE]>
- <link rel="shortcut icon" href="../../wp-content/themes/oni/favicon.ico">
+ <link rel="shortcut icon" href="http://spot.incubator.apache.org/wp-content/themes/oni/favicon.ico">
<![endif]-->
<meta name="msapplication-TileColor" content="#f01d4f">
- <meta name="msapplication-TileImage" content="../../wp-content/themes/oni/library/images/win8-tile-icon.png">
+ <meta name="msapplication-TileImage" content="../../../../wp-content/themes/oni/library/images/win8-tile-icon.png">
<meta name="theme-color" content="#121212">
- <link rel="pingback" href="../../xmlrpc.php">
+ <link rel="pingback" href="http://spot.incubator.apache.org/xmlrpc.php">
- <link rel='dns-prefetch' href='//fonts.googleapis.com' />
- <link rel='dns-prefetch' href='//s.w.org' />
- <link rel="alternate" type="application/rss+xml" title="Apache Spot » Feed" href="../../feed/" />
- <link rel="alternate" type="application/rss+xml" title="Apache Spot » Comments Feed" href="../../comments/feed/" />
+ <link rel="alternate" type="application/rss+xml" title="Apache Spot » Feed" href="http://spot.incubator.apache.org/feed/" />
+ <link rel="alternate" type="application/rss+xml" title="Apache Spot » Comments Feed" href="http://spot.incubator.apache.org/comments/feed/" />
<script type="text/javascript">
window._wpemojiSettings = {
- "baseUrl" : "https:\/\/s.w.org\/images\/core\/emoji\/2\/72x72\/",
+ "baseUrl" : "https:\/\/s.w.org\/images\/core\/emoji\/72x72\/",
"ext" : ".png",
- "svgUrl" : "https:\/\/s.w.org\/images\/core\/emoji\/2\/svg\/",
- "svgExt" : ".svg",
"source" : {
"concatemoji" : "http:\/\/spot.incubator.apache.org\/wp-includes\/js\/wp-emoji-release.min.js"
}
@@ -47,30 +43,24 @@
var c,
d,
e,
- f,
- g,
- h = b.createElement("canvas"),
- i = h.getContext && h.getContext("2d"),
- j = String.fromCharCode;
- if (!i || !i.fillText)
+ f = b.createElement("canvas"),
+ g = f.getContext && f.getContext("2d"),
+ h = String.fromCharCode;
+ if (!g || !g.fillText)
return !1;
- switch(i.textBaseline="top",i.font="600 32px Arial",a) {
+ switch(g.textBaseline="top",g.font="600 32px Arial",a) {
case"flag":
- return i.fillText(j(55356, 56806, 55356, 56826), 0, 0), !(h.toDataURL().length < 3e3) && (i.clearRect(0, 0, h.width, h.height), i.fillText(j(55356, 57331, 65039, 8205, 55356, 57096), 0, 0),
- c = h.toDataURL(), i.clearRect(0, 0, h.width, h.height), i.fillText(j(55356, 57331, 55356, 57096), 0, 0),
- d = h.toDataURL(), c !== d);
+ return g.fillText(h(55356, 56806, 55356, 56826), 0, 0), f.toDataURL().length > 3e3;
case"diversity":
- return i.fillText(j(55356, 57221), 0, 0),
- e = i.getImageData(16, 16, 1, 1).data,
- f = e[0] + "," + e[1] + "," + e[2] + "," + e[3], i.fillText(j(55356, 57221, 55356, 57343), 0, 0),
- e = i.getImageData(16, 16, 1, 1).data,
- g = e[0] + "," + e[1] + "," + e[2] + "," + e[3], f !== g;
+ return g.fillText(h(55356, 57221), 0, 0),
+ c = g.getImageData(16, 16, 1, 1).data,
+ d = c[0] + "," + c[1] + "," + c[2] + "," + c[3], g.fillText(h(55356, 57221, 55356, 57343), 0, 0),
+ c = g.getImageData(16, 16, 1, 1).data,
+ e = c[0] + "," + c[1] + "," + c[2] + "," + c[3], d !== e;
case"simple":
- return i.fillText(j(55357, 56835), 0, 0), 0 !== i.getImageData(16,16,1,1).data[0];
+ return g.fillText(h(55357, 56835), 0, 0), 0 !== g.getImageData(16,16,1,1).data[0];
case"unicode8":
- return i.fillText(j(55356, 57135), 0, 0), 0 !== i.getImageData(16,16,1,1).data[0];
- case"unicode9":
- return i.fillText(j(55358, 56631), 0, 0), 0 !== i.getImageData(16,16,1,1).data[0]
+ return g.fillText(h(55356, 57135), 0, 0), 0 !== g.getImageData(16,16,1,1).data[0]
}
return !1
}
@@ -84,7 +74,7 @@
g,
h,
i;
- for ( i = Array("simple", "flag", "unicode8", "diversity", "unicode9"), c.supports = {
+ for ( i = Array("simple", "flag", "unicode8", "diversity"), c.supports = {
everything : !0,
everythingExceptFlag : !0
},
@@ -116,7 +106,7 @@
<link rel='stylesheet' id='googleFonts-css' href='http://fonts.googleapis.com/css?family=Lato%3A400%2C700%2C400italic%2C700italic' type='text/css' media='all' />
<link rel='stylesheet' id='bones-stylesheet-css' href='../../wp-content/themes/oni/library/css/style.css' type='text/css' media='all' />
<!--[if lt IE 9]>
- <link rel='stylesheet' id='bones-ie-only-css' href='../../wp-content/themes/oni/library/css/ie.css' type='text/css' media='all' />
+ <link rel='stylesheet' id='bones-ie-only-css' href='http://spot.incubator.apache.org/wp-content/themes/oni/library/css/ie.css' type='text/css' media='all' />
<![endif]-->
<link rel='stylesheet' id='mm-css-css' href='../../wp-content/themes/oni/library/css/meanmenu.css' type='text/css' media='all' />
<script type='text/javascript' src='../../wp-content/themes/oni/library/js/libs/modernizr.custom.min.js'></script>
@@ -125,24 +115,6 @@
<script type='text/javascript' src='../../wp-content/themes/oni/library/js/jquery.meanmenu.js'></script>
<link rel='https://api.w.org/' href='../../wp-json/' />
- <script>
- (function(i, s, o, g, r, a, m) {
- i['GoogleAnalyticsObject'] = r;
- i[r] = i[r] ||
- function() {
- (i[r].q = i[r].q || []).push(arguments)
- }, i[r].l = 1 * new Date();
- a = s.createElement(o),
- m = s.getElementsByTagName(o)[0];
- a.async = 1;
- a.src = g;
- m.parentNode.insertBefore(a, m)
- })(window, document, 'script', '//www.google-analytics.com/analytics.js', 'ga');
-
- ga('create', 'UA-75955621-1', 'auto');
- ga('send', 'pageview');
-
- </script>
</head>
<body class="archive date" itemscope itemtype="http://schema.org/WebPage">
@@ -159,16 +131,16 @@
<nav role="navigation" itemscope itemtype="http://schema.org/SiteNavigationElement">
<ul id="menu-main-menu" class="nav top-nav cf">
- <li id="menu-item-129" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-129">
+ <li id="menu-item-4" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-4">
<a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight">Get Started</a>
</li>
<li id="menu-item-5" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-5">
<a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight#if-you-want-all-of-the-oni-code-at-once-just-clone-it">Download</a>
</li>
- <li id="menu-item-130" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-130">
+ <li id="menu-item-6" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-6">
<a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight#contributing-to-oni">Contribute</a>
</li>
- <li id="menu-item-106" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-106">
+ <li id="menu-item-7" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-7">
<a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight/wiki">Wiki</a>
</li>
<li id="menu-item-13" class="menu-item menu-item-type-post_type menu-item-object-page current_page_parent menu-item-13">
@@ -189,16 +161,16 @@
<main id="main" class="m-all t-2of3 d-5of7 cf" role="main" itemscope itemprop="mainContentOfPage" itemtype="http://schema.org/Blog">
- <h1 class="page-title">Month: March 2016</h1>
+ <h1 class="page-title">Month: September 2016</h1>
- <article id="post-62" class="cf post-62 post type-post status-publish format-standard hentry category-security-analytics tag-github tag-open-network-insight tag-open-source" role="article">
+ <article id="post-136" class="cf post-136 post type-post status-publish format-standard hentry category-uncategorized" role="article">
<header class="entry-header article-header">
- <h3 class="h2 entry-title"><a href="../../open-network-insight-3-most-asked-questions/" rel="bookmark" title="Apache Spot: Three Most-Asked Questions">Apache Spot: Three Most-Asked Questions</a></h3>
+ <h3 class="h2 entry-title"><a href="../../jupyter-notebooks-for-data-analysis/" rel="bookmark" title="Jupyter Notebooks for Data Analysis">Jupyter Notebooks for Data Analysis</a></h3>
<p class="byline entry-meta vcard">
- <time class="updated entry-time" datetime="2016-03-29" itemprop="datePublished">
- March 29, 2016
+ <time class="updated entry-time" datetime="2016-09-22" itemprop="datePublished">
+ September 22, 2016
</time>
</span>
</p>
@@ -208,7 +180,7 @@
<section class="entry-content cf">
<p>
- While this is not the first blog post about Apache Spot, it is the first one by a creator of the solution. As a security data scientist in Intel’s Data Center Group, I joined a small team to start thinking about solving really hard problems in cloud analytics. The team grew, and out of that… <a class="excerpt-read-more" href="../../open-network-insight-3-most-asked-questions/" title="Read Apache Spot: Three Most-Asked Questions">Read more & [...]
+ Why Does Apache Spot Include iPython notebooks? The project team wants Apache Spot needs to be a versatile tool that can be used by anyone. This means that data scientists and developers need to be able to query and handle the source data to find all the information they need for their decision making. The… <a class="excerpt-read-more" href="../../jupyter-notebooks-for-data-analysis/" title="Read Jupyter Notebooks for Data Analysis">Read more &ra [...]
</p>
</section>
@@ -221,19 +193,23 @@
</main>
+
<div id="sidebar1" class="sidebar m-all t-1of3 d-2of7 last-col cf" role="complementary">
<div id="recent-posts-2" class="widget widget_recent_entries">
<h4 class="widgettitle">Recent Posts</h4>
<ul>
<li>
- <a href="../../open-network-insight-oni-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/">Apache Spot and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a>
+ <a href="../jupyter-notebooks-for-data-analysis/">Jupyter Notebooks for Data Analysis</a>
</li>
<li>
- <a href="../../how-open-network-insight-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/">How Apache Spot Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a>
+ <a href="../apache-spot-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/">Apache Spot (Incubating) and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a>
</li>
<li>
- <a href="../../open-network-insight-3-most-asked-questions/">Apache Spot: Three Most-Asked Questions</a>
+ <a href="../how-apache-spot-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/">How Apache Spot (Incubating) Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a>
+ </li>
+ <li>
+ <a href="../apache-spot-3-most-asked-questions/">Apache Spot (Incubating): 3 Most-Asked Questions</a>
</li>
</ul>
</div>
@@ -241,10 +217,13 @@
<h4 class="widgettitle">Archives</h4>
<ul>
<li>
- <a href='../../2016/08/'>August 2016</a>
+ <a href='../09/'>September 2016</a>
+ </li>
+ <li>
+ <a href='../08/'>August 2016</a>
</li>
<li>
- <a href='../../2016/03/'>March 2016</a>
+ <a href='../03/'>March 2016</a>
</li>
</ul>
</div>
diff --git a/apache-spot-3-most-asked-questions/feed/index.html b/apache-spot-3-most-asked-questions/feed/index.html
new file mode 100644
index 0000000..766b30f
--- /dev/null
+++ b/apache-spot-3-most-asked-questions/feed/index.html
@@ -0,0 +1,18 @@
+<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
+ xmlns:content="http://purl.org/rss/1.0/modules/content/"
+ xmlns:dc="http://purl.org/dc/elements/1.1/"
+ xmlns:atom="http://www.w3.org/2005/Atom"
+ xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
+
+ >
+<channel>
+ <title>Comments on: Apache Spot (Incubating): Three Most-Asked Questions</title>
+ <atom:link href="http://spot.incubator.apache.org/apache-spot-3-most-asked-questions/feed/" rel="self" type="application/rss+xml" />
+ <link>http://spot.incubator.apache.org/apache-spot-3-most-asked-questions/</link>
+ <description></description>
+ <lastBuildDate></lastBuildDate>
+ <sy:updatePeriod>hourly</sy:updatePeriod>
+ <sy:updateFrequency>1</sy:updateFrequency>
+
+</channel>
+</rss>
diff --git a/blog/index.html b/apache-spot-3-most-asked-questions/index.html
similarity index 56%
copy from blog/index.html
copy to apache-spot-3-most-asked-questions/index.html
index 81a376e..e0cbedd 100644
--- a/blog/index.html
+++ b/apache-spot-3-most-asked-questions/index.html
@@ -12,7 +12,7 @@
<meta http-equiv="X-UA-Compatible" content="IE=edge">
- <title>Apache Spot Blog</title>
+ <title>Apache Spot (Incubating): Three Most-Asked Questions</title>
<meta name="HandheldFriendly" content="True">
<meta name="MobileOptimized" content="320">
@@ -33,6 +33,7 @@
<link rel='dns-prefetch' href='//s.w.org' />
<link rel="alternate" type="application/rss+xml" title="Apache Spot » Feed" href="../feed/" />
<link rel="alternate" type="application/rss+xml" title="Apache Spot » Comments Feed" href="../comments/feed/" />
+ <link rel="alternate" type="application/rss+xml" title="Apache Spot » Apache Spot: Three Most-Asked Questions Comments Feed" href="../open-network-insight-3-most-asked-questions/feed/" />
<script type="text/javascript">
window._wpemojiSettings = {
"baseUrl" : "https:\/\/s.w.org\/images\/core\/emoji\/2\/72x72\/",
@@ -124,28 +125,14 @@
<script type='text/javascript' src='../wp-includes/js/jquery/jquery-migrate.min.js'></script>
<script type='text/javascript' src='../wp-content/themes/oni/library/js/jquery.meanmenu.js'></script>
<link rel='https://api.w.org/' href='../wp-json/' />
+ <link rel="canonical" href="../open-network-insight-3-most-asked-questions/" />
+ <link rel='shortlink' href='../?p=62' />
+ <link rel="alternate" type="application/json+oembed" href="../wp-json/oembed/1.0/embed?url=http%3A%2F%2Fnolamarketing.com%2Fclient%2Foni%2Fopen-network-insight-3-most-asked-questions%2F" />
+ <link rel="alternate" type="text/xml+oembed" href="../wp-json/oembed/1.0/embed?url=http%3A%2F%2Fnolamarketing.com%2Fclient%2Foni%2Fopen-network-insight-3-most-asked-questions%2F&format=xml" />
- <script>
- (function(i, s, o, g, r, a, m) {
- i['GoogleAnalyticsObject'] = r;
- i[r] = i[r] ||
- function() {
- (i[r].q = i[r].q || []).push(arguments)
- }, i[r].l = 1 * new Date();
- a = s.createElement(o),
- m = s.getElementsByTagName(o)[0];
- a.async = 1;
- a.src = g;
- m.parentNode.insertBefore(a, m)
- })(window, document, 'script', '//www.google-analytics.com/analytics.js', 'ga');
-
- ga('create', 'UA-75955621-1', 'auto');
- ga('send', 'pageview');
-
- </script>
</head>
- <body class="blog" itemscope itemtype="http://schema.org/WebPage">
+ <body class="single single-post postid-62 single-format-standard" itemscope itemtype="http://schema.org/WebPage">
<div id="container">
@@ -171,7 +158,7 @@
<li id="menu-item-106" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-106">
<a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight/wiki">Wiki</a>
</li>
- <li id="menu-item-13" class="menu-item menu-item-type-post_type menu-item-object-page current-menu-item page_item page-item-10 current_page_item current_page_parent menu-item-13">
+ <li id="menu-item-13" class="menu-item menu-item-type-post_type menu-item-object-page current_page_parent menu-item-13">
<a href="../blog/">Blog</a>
</li>
</ul>
@@ -186,64 +173,17 @@
<div id="content">
<div id="inner-content" class="wrap cf">
- <main id="main" class="m-all t-2of3 d-5of7 cf" role="main" itemscope itemprop="mainContentOfPage" itemtype="http://schema.org/Blog">
-
- <article id="post-117" class="cf post-117 post type-post status-publish format-standard hentry category-cybersecurity" role="article">
-
- <header class="article-header">
-
- <h1 class="h2 entry-title"><a href="../open-network-insight-oni-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/" rel="bookmark" title="Apache Spot and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure">Apache Spot and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a></h1>
- <p class="byline entry-meta vcard">
- <time class="updated entry-time" datetime="2016-08-08" itemprop="datePublished">
- August 8, 2016
- </time>
- </span>
- </p>
-
- </header>
- <section class="entry-content cf">
- <div class="m-all t-1of4 d-2of7"></div>
- <div class="m-all t-3of4 d-5of7">
- <p>
- The “first” documented cybersecurity case was the worm replication, which was initiated by Robert T. Morris on November 2, 1988. Wow! Here we are in 2016, 28 years later, with viruses and worms giving way to Trojan horses and polymorphic code. Nowadays, we are also fighting against DDoS, phishing, spear phishing attacks, command and controls… <a class="excerpt-read-more" href="../open-network-insight-oni-and-cybersecurity-using-netflows-to-de [...]
- </p>
- </div>
- </section>
+ <main id="main" class="m-all t-2of3 d-5of7 cf" role="main" itemscope itemprop="mainContentOfPage" itemtype="http://schema.org/Blog">
- </article>
+ <article id="post-62" class="cf post-62 post type-post status-publish format-standard hentry category-security-analytics tag-github tag-open-network-insight tag-open-source" role="article" itemscope itemprop="blogPost" itemtype="http://schema.org/BlogPosting">
- <article id="post-113" class="cf post-113 post type-post status-publish format-standard hentry category-uncategorized" role="article">
+ <header class="article-header entry-header">
- <header class="article-header">
+ <h1 class="entry-title single-title" itemprop="headline" rel="bookmark">Apache Spot (Incubating): Three Most-Asked Questions</h1>
- <h1 class="h2 entry-title"><a href="../how-open-network-insight-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/" rel="bookmark" title="How Apache Spot Helps Create Well-Stocked Data Lakes and Catch Powerful Insights">How Apache Spot Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a></h1>
<p class="byline entry-meta vcard">
- <time class="updated entry-time" datetime="2016-08-08" itemprop="datePublished">
- August 8, 2016
- </time>
- </span>
- </p>
- </header>
-
- <section class="entry-content cf">
- <div class="m-all t-1of4 d-2of7"></div>
- <div class="m-all t-3of4 d-5of7">
- <p>
- About four years ago, the era of the Big Data analytics began. Paired with advanced analytics, massive volumes of data can be culled to not only inform critical decisions, but also to simulate sophisticated “what if” scenarios that allow companies to gain competitive advantages by generating and predicting different scenarios. For example, a financial services… <a class="excerpt-read-more" href="../how-open-network-insight-helps-create-well-s [...]
- </p>
- </div>
- </section>
-
- </article>
-
- <article id="post-62" class="cf post-62 post type-post status-publish format-standard hentry category-security-analytics tag-github tag-open-network-insight tag-open-source" role="article">
-
- <header class="article-header">
-
- <h1 class="h2 entry-title"><a href="../open-network-insight-3-most-asked-questions/" rel="bookmark" title="Apache Spot: Three Most-Asked Questions">Apache Spot: Three Most-Asked Questions</a></h1>
- <p class="byline entry-meta vcard">
<time class="updated entry-time" datetime="2016-03-29" itemprop="datePublished">
March 29, 2016
</time>
@@ -251,15 +191,52 @@
</p>
</header>
-
- <section class="entry-content cf">
- <div class="m-all t-1of4 d-2of7"></div>
- <div class="m-all t-3of4 d-5of7">
- <p>
- While this is not the first blog post about Apache Spot, it is the first one by a creator of the solution. As a security data scientist in Intel’s Data Center Group, I joined a small team to start thinking about solving really hard problems in cloud analytics. The team grew, and out of that… <a class="excerpt-read-more" href="../open-network-insight-3-most-asked-questions/" title="Read Apache Spot: Three Most-Asked Questions">Read more [...]
- </p>
- </div>
+ <section class="entry-content cf" itemprop="articleBody">
+ <p>
+ While this is not the first blog post about Apache Spot, it is the first one by a creator of the solution. As a security data scientist in Intel’s Data Center Group, I joined a small team to start thinking about solving really hard problems in cloud analytics. The team grew, and out of that effort, came Apache Spot. Since we started talking about the project, these are the three questions I am asked the most.
+ </p>
+ <p>
+ <strong>What Is Apache Spot?</strong>
+ <br />
+ Apache Spot is an open source, flow and packet analytics solution built on Hadoop. It combines big data processing, at-scale machine learning, and unique security analytics to put potential threats in front of defenders. While I am a data scientist today, I was a security investigator just a few years ago. I wanted to develop a solution that would put new tools and technology in play for defenders, but without requiring them to walk away from security [...]
+ </p>
+ <p>
+ We wanted to start with the hard problems, so we looked at the emerging need to analyze data that was produced at a scale outside what a lot of security solutions could handle. The data is being created today, and lack of visibility into that data gives attackers a profound advantage. Also, in this new era of security, many defenders (public and private sector) have to answer to their citizens and customers when these threats occur. In other words, an [...]
+ <br />
+ <a href="../wp-content/uploads/2016/03/FloCon2015.png" rel="attachment wp-att-66"><img class="aligncenter size-full wp-image-66" src="../wp-content/uploads/2016/03/FloCon2015.png" alt="open source packet and flow analytics" width="960" height="720" srcset="../wp-content/uploads/2016/03/FloCon2015.png 960w, ../wp-content/uploads/2016/03/FloCon2015-300x225.png 300w, ../wp-content/uploads/2016/03/FloCon2015-768x576.png 768w" sizes="(max-width: 960px) 100v [...]
+ </p>
+ <p>
+ The gist is that while processing is a challenge at higher scales, the amount of insight gained is higher when analyzing flows and packets from key protocols (like DNS). And that’s how we got here.
+ </p>
+ <p>
+ <strong>Why Intel?</strong>
+ </p>
+ <p>
+ At Intel, I have worked in IT, for a security product company (McAfee), and in the Data Center Group. Intel IT was an early pioneer of the concept of proactive investigations to protect intellectual property. McAfee (now Intel Security Group) has a broad customer base in the realms of network, endpoint, and content security, to name only a few. And the Intel Data Center group has strategic partnerships with Cloudera and Accenture, as well as some prett [...]
+ </p>
+ <p>
+ <strong>Why Open Source?</strong>
+ </p>
+ <p>
+ I learned from my earlier efforts in security analytics, that to invite collaboration from academia, the public sector, and the private sector, open source software is an excellent choice. We are now seeking to build a community of developers, data scientists, and security enthusiasts to grow Apache Spot into something we can all be proud of. We have also chosen an Apache software license, so that it can enrich commercial software offerings as well.
+ </p>
+ <p>
+ The greatest thing for me since we announced at RSA is to hear OTHER people talk about Apache Spot (formerly Open Network Insight or ONI), here are some of my favorites, from <a href="http://vision.cloudera.com/open-network-insight-changing-infosec-data-science-forever/" target="_blank">a Data Scientist @ eBay </a>, <a href="https://newsroom.accenture.com/news/accenture-introduces-the-accenture-cyber-intelligence-platform-to-help-organizations-continuo [...]
+ </p>
+ <p>
+ Fork us on Github!
+ </p>
+ <p>
+ Grant Babb
+ </p>
</section>
+ <footer class="article-footer">
+
+ filed under: <a href="../category/security-analytics/" rel="category tag">Security Analytics</a>
+ <p class="tags">
+ <span class="tags-title">Tags:</span><a href="../tag/github/" rel="tag">github</a>, <a href="../tag/open-network-insight/" rel="tag">open network insight</a>, <a href="../tag/open-source/" rel="tag">open source</a>
+ </p>
+ </footer>
</article>
@@ -271,13 +248,16 @@
<h4 class="widgettitle">Recent Posts</h4>
<ul>
<li>
- <a href="../open-network-insight-oni-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/">Apache Spot and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a>
+ <a href="../jupyter-notebooks-for-data-analysis/">Jupyter Notebooks for Data Analysis</a>
+ </li>
+ <li>
+ <a href="../apache-spot-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/">Apache Spot (Incubating) and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a>
</li>
<li>
- <a href="../how-open-network-insight-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/">How Apache Spot Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a>
+ <a href="../how-apache-spot-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/">How Apache Spot (Incubating) Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a>
</li>
<li>
- <a href="../open-network-insight-3-most-asked-questions/">Apache Spot: Three Most-Asked Questions</a>
+ <a href="../apache-spot-3-most-asked-questions/">Apache Spot (Incubating): 3 Most-Asked Questions</a>
</li>
</ul>
</div>
@@ -285,6 +265,9 @@
<h4 class="widgettitle">Archives</h4>
<ul>
<li>
+ <a href='../2016/09/'>September 2016</a>
+ </li>
+ <li>
<a href='../2016/08/'>August 2016</a>
</li>
<li>
@@ -314,6 +297,7 @@
</div>
+ <script type='text/javascript' src='../wp-includes/js/comment-reply.min.js'></script>
<script type='text/javascript' src='../wp-content/themes/oni/library/js/scripts.js'></script>
<script type='text/javascript' src='../wp-includes/js/wp-embed.min.js'></script>
diff --git a/apache-spot-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/feed/index.html b/apache-spot-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/feed/index.html
new file mode 100644
index 0000000..0a44b5d
--- /dev/null
+++ b/apache-spot-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/feed/index.html
@@ -0,0 +1,18 @@
+<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
+ xmlns:content="http://purl.org/rss/1.0/modules/content/"
+ xmlns:dc="http://purl.org/dc/elements/1.1/"
+ xmlns:atom="http://www.w3.org/2005/Atom"
+ xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
+
+ >
+<channel>
+ <title>Comments on: Apache Spot (Incubating) and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</title>
+ <atom:link href="http://spot.incubator.apache.org/apache-spot-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/feed/" rel="self" type="application/rss+xml" />
+ <link>http://spot.incubator.apache.org/apache-spot-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/</link>
+ <description></description>
+ <lastBuildDate></lastBuildDate>
+ <sy:updatePeriod>hourly</sy:updatePeriod>
+ <sy:updateFrequency>1</sy:updateFrequency>
+
+</channel>
+</rss>
diff --git a/blog/index.html b/apache-spot-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/index.html
similarity index 56%
copy from blog/index.html
copy to apache-spot-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/index.html
index 81a376e..11fb459 100644
--- a/blog/index.html
+++ b/apache-spot-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/index.html
@@ -12,7 +12,7 @@
<meta http-equiv="X-UA-Compatible" content="IE=edge">
- <title>Apache Spot Blog</title>
+ <title>Apache Spot (Incubating) and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</title>
<meta name="HandheldFriendly" content="True">
<meta name="MobileOptimized" content="320">
@@ -33,6 +33,7 @@
<link rel='dns-prefetch' href='//s.w.org' />
<link rel="alternate" type="application/rss+xml" title="Apache Spot » Feed" href="../feed/" />
<link rel="alternate" type="application/rss+xml" title="Apache Spot » Comments Feed" href="../comments/feed/" />
+ <link rel="alternate" type="application/rss+xml" title="Apache Spot » Apache Spot and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure Comments Feed" href="../open-network-insight-oni-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/feed/" />
<script type="text/javascript">
window._wpemojiSettings = {
"baseUrl" : "https:\/\/s.w.org\/images\/core\/emoji\/2\/72x72\/",
@@ -124,28 +125,14 @@
<script type='text/javascript' src='../wp-includes/js/jquery/jquery-migrate.min.js'></script>
<script type='text/javascript' src='../wp-content/themes/oni/library/js/jquery.meanmenu.js'></script>
<link rel='https://api.w.org/' href='../wp-json/' />
+ <link rel="canonical" href="../open-network-insight-oni-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/" />
+ <link rel='shortlink' href='../?p=117' />
+ <link rel="alternate" type="application/json+oembed" href="../wp-json/oembed/1.0/embed?url=http%3A%2F%2Fnolamarketing.com%2Fclient%2Foni%2Fopen-network-insight-oni-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure%2F" />
+ <link rel="alternate" type="text/xml+oembed" href="../wp-json/oembed/1.0/embed?url=http%3A%2F%2Fnolamarketing.com%2Fclient%2Foni%2Fopen-network-insight-oni-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure%2F&format=xml" />
- <script>
- (function(i, s, o, g, r, a, m) {
- i['GoogleAnalyticsObject'] = r;
- i[r] = i[r] ||
- function() {
- (i[r].q = i[r].q || []).push(arguments)
- }, i[r].l = 1 * new Date();
- a = s.createElement(o),
- m = s.getElementsByTagName(o)[0];
- a.async = 1;
- a.src = g;
- m.parentNode.insertBefore(a, m)
- })(window, document, 'script', '//www.google-analytics.com/analytics.js', 'ga');
-
- ga('create', 'UA-75955621-1', 'auto');
- ga('send', 'pageview');
-
- </script>
</head>
- <body class="blog" itemscope itemtype="http://schema.org/WebPage">
+ <body class="single single-post postid-117 single-format-standard" itemscope itemtype="http://schema.org/WebPage">
<div id="container">
@@ -171,7 +158,7 @@
<li id="menu-item-106" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-106">
<a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight/wiki">Wiki</a>
</li>
- <li id="menu-item-13" class="menu-item menu-item-type-post_type menu-item-object-page current-menu-item page_item page-item-10 current_page_item current_page_parent menu-item-13">
+ <li id="menu-item-13" class="menu-item menu-item-type-post_type menu-item-object-page current_page_parent menu-item-13">
<a href="../blog/">Blog</a>
</li>
</ul>
@@ -186,39 +173,17 @@
<div id="content">
<div id="inner-content" class="wrap cf">
- <main id="main" class="m-all t-2of3 d-5of7 cf" role="main" itemscope itemprop="mainContentOfPage" itemtype="http://schema.org/Blog">
-
- <article id="post-117" class="cf post-117 post type-post status-publish format-standard hentry category-cybersecurity" role="article">
-
- <header class="article-header">
-
- <h1 class="h2 entry-title"><a href="../open-network-insight-oni-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/" rel="bookmark" title="Apache Spot and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure">Apache Spot and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a></h1>
- <p class="byline entry-meta vcard">
- <time class="updated entry-time" datetime="2016-08-08" itemprop="datePublished">
- August 8, 2016
- </time>
- </span>
- </p>
- </header>
-
- <section class="entry-content cf">
- <div class="m-all t-1of4 d-2of7"></div>
- <div class="m-all t-3of4 d-5of7">
- <p>
- The “first” documented cybersecurity case was the worm replication, which was initiated by Robert T. Morris on November 2, 1988. Wow! Here we are in 2016, 28 years later, with viruses and worms giving way to Trojan horses and polymorphic code. Nowadays, we are also fighting against DDoS, phishing, spear phishing attacks, command and controls… <a class="excerpt-read-more" href="../open-network-insight-oni-and-cybersecurity-using-netflows-to-de [...]
- </p>
- </div>
- </section>
+ <main id="main" class="m-all t-2of3 d-5of7 cf" role="main" itemscope itemprop="mainContentOfPage" itemtype="http://schema.org/Blog">
- </article>
+ <article id="post-117" class="cf post-117 post type-post status-publish format-standard hentry category-cybersecurity" role="article" itemscope itemprop="blogPost" itemtype="http://schema.org/BlogPosting">
- <article id="post-113" class="cf post-113 post type-post status-publish format-standard hentry category-uncategorized" role="article">
+ <header class="article-header entry-header">
- <header class="article-header">
+ <h1 class="entry-title single-title" itemprop="headline" rel="bookmark">Apache Spot (Incubating) and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</h1>
- <h1 class="h2 entry-title"><a href="../how-open-network-insight-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/" rel="bookmark" title="How Apache Spot Helps Create Well-Stocked Data Lakes and Catch Powerful Insights">How Apache Spot Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a></h1>
<p class="byline entry-meta vcard">
+
<time class="updated entry-time" datetime="2016-08-08" itemprop="datePublished">
August 8, 2016
</time>
@@ -226,40 +191,57 @@
</p>
</header>
-
- <section class="entry-content cf">
- <div class="m-all t-1of4 d-2of7"></div>
- <div class="m-all t-3of4 d-5of7">
- <p>
- About four years ago, the era of the Big Data analytics began. Paired with advanced analytics, massive volumes of data can be culled to not only inform critical decisions, but also to simulate sophisticated “what if” scenarios that allow companies to gain competitive advantages by generating and predicting different scenarios. For example, a financial services… <a class="excerpt-read-more" href="../how-open-network-insight-helps-create-well-s [...]
- </p>
- </div>
- </section>
-
- </article>
-
- <article id="post-62" class="cf post-62 post type-post status-publish format-standard hentry category-security-analytics tag-github tag-open-network-insight tag-open-source" role="article">
-
- <header class="article-header">
-
- <h1 class="h2 entry-title"><a href="../open-network-insight-3-most-asked-questions/" rel="bookmark" title="Apache Spot: Three Most-Asked Questions">Apache Spot: Three Most-Asked Questions</a></h1>
- <p class="byline entry-meta vcard">
- <time class="updated entry-time" datetime="2016-03-29" itemprop="datePublished">
- March 29, 2016
- </time>
- </span>
+ <section class="entry-content cf" itemprop="articleBody">
+ <p>
+ The “first” documented cybersecurity case was the worm replication, which was initiated by Robert T. Morris on November 2, 1988. Wow! Here we are in 2016, 28 years later, with viruses and worms giving way to Trojan horses and polymorphic code. Nowadays, we are also fighting against DDoS, phishing, spear phishing attacks, command and controls along with APTs such as Aurora, Zeus, Red October and Stuxnet. What happened with our security controls on each attack?
+ </p>
+ <p>
+ Despite heroic efforts, internal and external security controls, no matter if they are preventive, detective or corrective, can be bypassed by different situations or misconfigurations. Capabilities to detect bugs or vulnerabilities in code, protocol, etc. are still limited. When we consider how to close these gaps, we have two options:
+ </p>
+ <ol>
+ <li>
+ Collect information from each device that is part of the environment.
+ </li>
+ <li>
+ Collect information from the critical infrastructure that is used for most, if not all, of the systems.
+ </li>
+ </ol>
+ <p>
+ This blog will describe the second approach.
+ </p>
+ <p>
+ Critical infrastructure, including nationally significant infrastructure, can be broadly defined as the systems, assets, facilities and networks that provide essential services. For nations, this means protecting the national security, economic security and prosperity as well as the health and safety of their citizenry. If we extrapolate this definition on the IT enterprise environments, we can define the critical infrastructure as the service, or serv [...]
+ </p>
+ <p>
+ Effectively protecting critical infrastructure means that millions, if not billions, of different scenarios must be identified and monitored. To do this, the cybersecurity problem must be broken into small pieces.
+ </p>
+ <p>
+ Let’s consider DNS — your communications to the Web. How can you know which communications are being established by your critical servers? And, what if you want to do it for most of your infrastructure?
+ </p>
+ <p>
+ First idea: Use NetFlow, which is a network protocol that helps us collect IP traffic information and monitor network traffic. NetFlow has the details on the communications of all of your network traffic. However, the normal data on an enterprise environment includes billions of NetFlow events per day. To use this data to identify issues, it must be stored and analyzed. Storage alone is costly. Analyzing what amount Big Data stores is an entire other c [...]
+ </p>
+ <p>
+ Apache Spot offers a solution. It was designed to gather, store and analyze Big Data. In fact, Apache Spot is an ideal solution for this cybersecurity challenge. Apache Spot can integrate many different data sources in a data lake then add operational context to the data by linking configuration, inventory, service databases and other data stores. This helps you to prioritize the actions to take under different attack, malware, APT and hacking scenario [...]
</p>
+ <p>
+ Not only can Apache Spot collect, store and analyze billions of NetFlow packets, but it can also be adapted to meet the unique requirements of your organization. How? Apache Spot is an open-source project.
+ </p>
+ <p>
+ <strong>But Wait, There’s More</strong>
+ </p>
+ <p>
+ Check out “<a href="../how-apache-spot-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/"><u>How Apache Spot Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</u></a>” to learn more about the underlying Apache Spot architecture.
+ </p>
+ <p>
+ This is the first of a series of blogs that we will be writing about cybersecurity, so check back to read more.
+ </p>
+ </section>
+ <footer class="article-footer">
- </header>
+ filed under: <a href="../category/cybersecurity/" rel="category tag">Cybersecurity</a>
- <section class="entry-content cf">
- <div class="m-all t-1of4 d-2of7"></div>
- <div class="m-all t-3of4 d-5of7">
- <p>
- While this is not the first blog post about Apache Spot, it is the first one by a creator of the solution. As a security data scientist in Intel’s Data Center Group, I joined a small team to start thinking about solving really hard problems in cloud analytics. The team grew, and out of that… <a class="excerpt-read-more" href="../open-network-insight-3-most-asked-questions/" title="Read Apache Spot: Three Most-Asked Questions">Read more [...]
- </p>
- </div>
- </section>
+ </footer>
</article>
@@ -271,13 +253,16 @@
<h4 class="widgettitle">Recent Posts</h4>
<ul>
<li>
- <a href="../open-network-insight-oni-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/">Apache Spot and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a>
+ <a href="../jupyter-notebooks-for-data-analysis/">Jupyter Notebooks for Data Analysis</a>
+ </li>
+ <li>
+ <a href="../apache-spot-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/">Apache Spot (Incubating) and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a>
</li>
<li>
- <a href="../how-open-network-insight-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/">How Apache Spot Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a>
+ <a href="../how-apache-spot-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/">How Apache Spot (Incubating) Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a>
</li>
<li>
- <a href="../open-network-insight-3-most-asked-questions/">Apache Spot: Three Most-Asked Questions</a>
+ <a href="../apache-spot-3-most-asked-questions/">Apache Spot (Incubating): 3 Most-Asked Questions</a>
</li>
</ul>
</div>
@@ -285,6 +270,9 @@
<h4 class="widgettitle">Archives</h4>
<ul>
<li>
+ <a href='../2016/09/'>September 2016</a>
+ </li>
+ <li>
<a href='../2016/08/'>August 2016</a>
</li>
<li>
@@ -314,6 +302,7 @@
</div>
+ <script type='text/javascript' src='../wp-includes/js/comment-reply.min.js'></script>
<script type='text/javascript' src='../wp-content/themes/oni/library/js/scripts.js'></script>
<script type='text/javascript' src='../wp-includes/js/wp-embed.min.js'></script>
diff --git a/blog/index.html b/blog/index.html
index 81a376e..859e75e 100644
--- a/blog/index.html
+++ b/blog/index.html
@@ -42,7 +42,8 @@
"source" : {
"concatemoji" : "http:\/\/spot.incubator.apache.org\/wp-includes\/js\/wp-emoji-release.min.js"
}
- }; ! function(a, b, c) {
+ };
+ ! function(a, b, c) {
function d(a) {
var c,
d,
@@ -125,24 +126,7 @@
<script type='text/javascript' src='../wp-content/themes/oni/library/js/jquery.meanmenu.js'></script>
<link rel='https://api.w.org/' href='../wp-json/' />
- <script>
- (function(i, s, o, g, r, a, m) {
- i['GoogleAnalyticsObject'] = r;
- i[r] = i[r] ||
- function() {
- (i[r].q = i[r].q || []).push(arguments)
- }, i[r].l = 1 * new Date();
- a = s.createElement(o),
- m = s.getElementsByTagName(o)[0];
- a.async = 1;
- a.src = g;
- m.parentNode.insertBefore(a, m)
- })(window, document, 'script', '//www.google-analytics.com/analytics.js', 'ga');
-
- ga('create', 'UA-75955621-1', 'auto');
- ga('send', 'pageview');
- </script>
</head>
<body class="blog" itemscope itemtype="http://schema.org/WebPage">
@@ -160,16 +144,16 @@
<nav role="navigation" itemscope itemtype="http://schema.org/SiteNavigationElement">
<ul id="menu-main-menu" class="nav top-nav cf">
<li id="menu-item-129" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-129">
- <a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight">Get Started</a>
+ <a target="_blank" href="https://github.com/Open-Network-Insight/apache-spot">Get Started</a>
</li>
<li id="menu-item-5" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-5">
- <a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight#if-you-want-all-of-the-oni-code-at-once-just-clone-it">Download</a>
+ <a target="_blank" href="https://github.com/Open-Network-Insight/apache-spot#if-you-want-all-of-the-oni-code-at-once-just-clone-it">Download</a>
</li>
<li id="menu-item-130" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-130">
- <a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight#contributing-to-oni">Contribute</a>
+ <a target="_blank" href="https://github.com/Open-Network-Insight/apache-spot#contributing-to-oni">Contribute</a>
</li>
<li id="menu-item-106" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-106">
- <a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight/wiki">Wiki</a>
+ <a target="_blank" href="https://github.com/Open-Network-Insight/apache-spot/wiki">Wiki</a>
</li>
<li id="menu-item-13" class="menu-item menu-item-type-post_type menu-item-object-page current-menu-item page_item page-item-10 current_page_item current_page_parent menu-item-13">
<a href="../blog/">Blog</a>
@@ -188,11 +172,36 @@
<div id="inner-content" class="wrap cf">
<main id="main" class="m-all t-2of3 d-5of7 cf" role="main" itemscope itemprop="mainContentOfPage" itemtype="http://schema.org/Blog">
+ <article id="post-136" class="cf post-136 post type-post status-publish format-standard hentry category-uncategorized" role="article">
+
+ <header class="article-header">
+
+ <h1 class="h2 entry-title"><a href="../jupyter-notebooks-for-data-analysis/" rel="bookmark" title="Jupyter Notebooks for Data Analysis">Jupyter Notebooks for Data Analysis</a></h1>
+ <p class="byline entry-meta vcard">
+ <time class="updated entry-time" datetime="2016-09-22" itemprop="datePublished">
+ September 22, 2016
+ </time>
+ </span>
+ </p>
+
+ </header>
+
+ <section class="entry-content cf">
+ <div class="m-all t-1of4 d-2of7"></div>
+ <div class="m-all t-3of4 d-5of7">
+ <p>
+ Why Does Apache Spot Include iPython notebooks? The project team wants Apache Spot to be a versatile tool that can be used by anyone. This means that data scientists and developers need to be able to query and handle the source data to find all the information they need for their decision making. The… <a class="excerpt-read-more" href="../jupyter-notebooks-for-data-analysis/" title="Read Jupyter Notebooks for Data Analysis">Read more »</a>
+ </p>
+ </div>
+ </section>
+
+ </article>
+
<article id="post-117" class="cf post-117 post type-post status-publish format-standard hentry category-cybersecurity" role="article">
<header class="article-header">
- <h1 class="h2 entry-title"><a href="../open-network-insight-oni-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/" rel="bookmark" title="Apache Spot and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure">Apache Spot and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a></h1>
+ <h1 class="h2 entry-title"><a href="../apache-spot-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/" rel="bookmark" title="Apache Spot and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure">Apache Spot (Incubating) and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a></h1>
<p class="byline entry-meta vcard">
<time class="updated entry-time" datetime="2016-08-08" itemprop="datePublished">
August 8, 2016
@@ -206,7 +215,7 @@
<div class="m-all t-1of4 d-2of7"></div>
<div class="m-all t-3of4 d-5of7">
<p>
- The “first” documented cybersecurity case was the worm replication, which was initiated by Robert T. Morris on November 2, 1988. Wow! Here we are in 2016, 28 years later, with viruses and worms giving way to Trojan horses and polymorphic code. Nowadays, we are also fighting against DDoS, phishing, spear phishing attacks, command and controls… <a class="excerpt-read-more" href="../open-network-insight-oni-and-cybersecurity-using-netflows-to-de [...]
+ The “first” documented cybersecurity case was the worm replication, which was initiated by Robert T. Morris on November 2, 1988. Wow! Here we are in 2016, 28 years later, with viruses and worms giving way to Trojan horses and polymorphic code. Nowadays, we are also fighting against DDoS, phishing, spear phishing attacks, command and controls… <a class="excerpt-read-more" href="../apache-spot-and-cybersecurity-using-netflows-to-detect-threats- [...]
</p>
</div>
</section>
@@ -217,7 +226,7 @@
<header class="article-header">
- <h1 class="h2 entry-title"><a href="../how-open-network-insight-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/" rel="bookmark" title="How Apache Spot Helps Create Well-Stocked Data Lakes and Catch Powerful Insights">How Apache Spot Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a></h1>
+ <h1 class="h2 entry-title"><a href="../how-apache-spot-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/" rel="bookmark" title="How Apache Spot Helps Create Well-Stocked Data Lakes and Catch Powerful Insights">How Apache Spot (Incubating) Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a></h1>
<p class="byline entry-meta vcard">
<time class="updated entry-time" datetime="2016-08-08" itemprop="datePublished">
August 8, 2016
@@ -231,18 +240,18 @@
<div class="m-all t-1of4 d-2of7"></div>
<div class="m-all t-3of4 d-5of7">
<p>
- About four years ago, the era of the Big Data analytics began. Paired with advanced analytics, massive volumes of data can be culled to not only inform critical decisions, but also to simulate sophisticated “what if” scenarios that allow companies to gain competitive advantages by generating and predicting different scenarios. For example, a financial services… <a class="excerpt-read-more" href="../how-open-network-insight-helps-create-well-s [...]
+ About four years ago, the era of the Big Data analytics began. Paired with advanced analytics, massive volumes of data can be culled to not only inform critical decisions, but also to simulate sophisticated “what if” scenarios that allow companies to gain competitive advantages by generating and predicting different scenarios. For example, a financial services… <a class="excerpt-read-more" href="../how-apache-spot-helps-create-well-stocked-da [...]
</p>
</div>
</section>
</article>
- <article id="post-62" class="cf post-62 post type-post status-publish format-standard hentry category-security-analytics tag-github tag-open-network-insight tag-open-source" role="article">
+ <article id="post-62" class="cf post-62 post type-post status-publish format-standard hentry category-security-analytics tag-github tag-apache-spot tag-open-source" role="article">
<header class="article-header">
- <h1 class="h2 entry-title"><a href="../open-network-insight-3-most-asked-questions/" rel="bookmark" title="Apache Spot: Three Most-Asked Questions">Apache Spot: Three Most-Asked Questions</a></h1>
+ <h1 class="h2 entry-title"><a href="../apache-spot-3-most-asked-questions/" rel="bookmark" title="Apache Spot: Three Most-Asked Questions">Apache Spot (Incubating): Three Most-Asked Questions</a></h1>
<p class="byline entry-meta vcard">
<time class="updated entry-time" datetime="2016-03-29" itemprop="datePublished">
March 29, 2016
@@ -256,7 +265,7 @@
<div class="m-all t-1of4 d-2of7"></div>
<div class="m-all t-3of4 d-5of7">
<p>
- While this is not the first blog post about Apache Spot, it is the first one by a creator of the solution. As a security data scientist in Intel’s Data Center Group, I joined a small team to start thinking about solving really hard problems in cloud analytics. The team grew, and out of that… <a class="excerpt-read-more" href="../open-network-insight-3-most-asked-questions/" title="Read Apache Spot: Three Most-Asked Questions">Read more [...]
+ While this is not the first blog post about Apache Spot, it is the first one by a creator of the solution. As a security data scientist in Intel’s Data Center Group, I joined a small team to start thinking about solving really hard problems in cloud analytics. The team grew, and out of that… <a class="excerpt-read-more" href="../apache-spot-3-most-asked-questions/" title="Read Apache Spot: Three Most-Asked Questions">Read more »</a>
</p>
</div>
</section>
@@ -271,13 +280,16 @@
<h4 class="widgettitle">Recent Posts</h4>
<ul>
<li>
- <a href="../open-network-insight-oni-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/">Apache Spot and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a>
+ <a href="../jupyter-notebooks-for-data-analysis/">Jupyter Notebooks for Data Analysis</a>
+ </li>
+ <li>
+ <a href="../apache-spot-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/">Apache Spot (Incubating) and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a>
</li>
<li>
- <a href="../how-open-network-insight-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/">How Apache Spot Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a>
+ <a href="../how-apache-spot-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/">How Apache Spot (Incubating) Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a>
</li>
<li>
- <a href="../open-network-insight-3-most-asked-questions/">Apache Spot: Three Most-Asked Questions</a>
+ <a href="../apache-spot-3-most-asked-questions/">Apache Spot (Incubating): 3 Most-Asked Questions</a>
</li>
</ul>
</div>
@@ -285,6 +297,9 @@
<h4 class="widgettitle">Archives</h4>
<ul>
<li>
+ <a href='../2016/09/'>September 2016</a>
+ </li>
+ <li>
<a href='../2016/08/'>August 2016</a>
</li>
<li>
diff --git a/category/cybersecurity/feed/index.html b/category/cybersecurity/feed/index.html
index 01c43db..fac6f3a 100644
--- a/category/cybersecurity/feed/index.html
+++ b/category/cybersecurity/feed/index.html
@@ -19,14 +19,14 @@
<item>
<title>Apache Spot and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</title>
- <link>http://spot.incubator.apache.org/open-network-insight-oni-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/</link>
- <comments>http://spot.incubator.apache.org/open-network-insight-oni-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/#respond</comments>
+ <link>http://spot.incubator.apache.org/apache-spot-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/</link>
+ <comments>http://spot.incubator.apache.org/apache-spot-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/#respond</comments>
<pubDate>Mon, 08 Aug 2016 08:00:32 +0000</pubDate>
<dc:creator><![CDATA[oni-admin]]></dc:creator>
<category><![CDATA[Cybersecurity]]></category>
<guid isPermaLink="false">http://spot.incubator.apache.org/?p=117</guid>
- <description><![CDATA[The “first” documented cybersecurity case was the worm replication, which was initiated by Robert T. Morris on November 2, 1988. Wow! Here we are in 2016, 28 years later, with viruses and worms giving way to Trojan horses and polymorphic code. Nowadays, we are also fighting against DDoS, phishing, spear phishing attacks, command and controls... <a class="excerpt-read-more" href="http://spot.incubator.apache.org/open-network-insight-oni-and-cybersecurity-using-net [...]
+ <description><![CDATA[The “first” documented cybersecurity case was the worm replication, which was initiated by Robert T. Morris on November 2, 1988. Wow! Here we are in 2016, 28 years later, with viruses and worms giving way to Trojan horses and polymorphic code. Nowadays, we are also fighting against DDoS, phishing, spear phishing attacks, command and controls... <a class="excerpt-read-more" href="http://spot.incubator.apache.org/apache-spot-and-cybersecurity-using-netflows-to-dete [...]
<content:encoded><![CDATA[<p>The “first” documented cybersecurity case was the worm replication, which was initiated by Robert T. Morris on November 2, 1988. Wow! Here we are in 2016, 28 years later, with viruses and worms giving way to Trojan horses and polymorphic code. Nowadays, we are also fighting against DDoS, phishing, spear phishing attacks, command and controls along with APTs such as Aurora, Zeus, Red October and Stuxnet. What happened with our security controls on each att [...]
<p>Despite heroic efforts, internal and external security controls, no matter if they are preventive, detective or corrective, can be bypassed by different situations or misconfigurations. Capabilities to detect bugs or vulnerabilities in code, protocol, etc. are still limited. When we consider how to close these gaps, we have two options:</p>
<ol>
@@ -41,10 +41,10 @@
<p>Apache Spot offers a solution. It was designed to gather, store and analyze Big Data. In fact, Apache Spot is an ideal solution for this cybersecurity challenge. Apache Spot can integrate many different data sources in a data lake then add operational context to the data by linking configuration, inventory, service databases and other data stores. This helps you to prioritize the actions to take under different attack, malware, APT and hacking scenarios. With Apache Spot, attacks that [...]
<p>Not only can Apache Spot collect, store and analyze billions of NetFlow packets, but it can also be adapted to meet the unique requirements of your organization. How? Apache Spot is an open-source project.</p>
<p><strong>But Wait, There’s More</strong></p>
-<p>Check out “<a href="http://spot.incubator.apache.org/how-open-network-insight-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/"><u>How Apache Spot Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</u></a>” to learn more about the underlying Apache Spot architecture.</p>
+<p>Check out “<a href="http://spot.incubator.apache.org/how-apache-spot-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/"><u>How Apache Spot Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</u></a>” to learn more about the underlying Apache Spot architecture.</p>
<p>This is the first of a series of blogs that we will be writing about cybersecurity, so check back to read more.</p>
]]></content:encoded>
- <wfw:commentRss>http://spot.incubator.apache.org/open-network-insight-oni-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/feed/</wfw:commentRss>
+ <wfw:commentRss>http://spot.incubator.apache.org/apache-spot-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/feed/</wfw:commentRss>
<slash:comments>0</slash:comments>
</item>
</channel>
diff --git a/category/cybersecurity/index.html b/category/cybersecurity/index.html
index 676bc20..7b440f1 100644
--- a/category/cybersecurity/index.html
+++ b/category/cybersecurity/index.html
@@ -12,7 +12,7 @@
<meta http-equiv="X-UA-Compatible" content="IE=edge">
- <title>Apache Spot Cybersecurity</title>
+ <title>Cybersecurity</title>
<meta name="HandheldFriendly" content="True">
<meta name="MobileOptimized" content="320">
@@ -126,24 +126,7 @@
<script type='text/javascript' src='../../wp-content/themes/oni/library/js/jquery.meanmenu.js'></script>
<link rel='https://api.w.org/' href='../../wp-json/' />
- <script>
- (function(i, s, o, g, r, a, m) {
- i['GoogleAnalyticsObject'] = r;
- i[r] = i[r] ||
- function() {
- (i[r].q = i[r].q || []).push(arguments)
- }, i[r].l = 1 * new Date();
- a = s.createElement(o),
- m = s.getElementsByTagName(o)[0];
- a.async = 1;
- a.src = g;
- m.parentNode.insertBefore(a, m)
- })(window, document, 'script', '//www.google-analytics.com/analytics.js', 'ga');
-
- ga('create', 'UA-75955621-1', 'auto');
- ga('send', 'pageview');
- </script>
</head>
<body class="archive category category-cybersecurity category-9" itemscope itemtype="http://schema.org/WebPage">
@@ -196,7 +179,7 @@
<header class="entry-header article-header">
- <h3 class="h2 entry-title"><a href="../../open-network-insight-oni-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/" rel="bookmark" title="Apache Spot and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure">Apache Spot and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a></h3>
+ <h3 class="h2 entry-title"><a href="../../apache-spot-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/" rel="bookmark" title="Apache Spot and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure">Apache Spot (Incubating) and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a></h3>
<p class="byline entry-meta vcard">
<time class="updated entry-time" datetime="2016-08-08" itemprop="datePublished">
August 8, 2016
@@ -209,7 +192,7 @@
<section class="entry-content cf">
<p>
- The “first” documented cybersecurity case was the worm replication, which was initiated by Robert T. Morris on November 2, 1988. Wow! Here we are in 2016, 28 years later, with viruses and worms giving way to Trojan horses and polymorphic code. Nowadays, we are also fighting against DDoS, phishing, spear phishing attacks, command and controls… <a class="excerpt-read-more" href="../../open-network-insight-oni-and-cybersecurity-using-netflows-to-det [...]
+ The “first” documented cybersecurity case was the worm replication, which was initiated by Robert T. Morris on November 2, 1988. Wow! Here we are in 2016, 28 years later, with viruses and worms giving way to Trojan horses and polymorphic code. Nowadays, we are also fighting against DDoS, phishing, spear phishing attacks, command and controls… <a class="excerpt-read-more" href="../../apache-spot-and-cybersecurity-using-netflows-to-detect-threats-t [...]
</p>
</section>
@@ -228,13 +211,16 @@
<h4 class="widgettitle">Recent Posts</h4>
<ul>
<li>
- <a href="../../open-network-insight-oni-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/">Apache Spot and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a>
+ <a href="../../jupyter-notebooks-for-data-analysis/">Jupyter Notebooks for Data Analysis</a>
+ </li>
+ <li>
+ <a href="../../apache-spot-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/">Apache Spot (Incubating) and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a>
</li>
<li>
- <a href="../../how-open-network-insight-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/">How Apache Spot Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a>
+ <a href="../../how-apache-spot-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/">How Apache Spot (Incubating) Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a>
</li>
<li>
- <a href="../../open-network-insight-3-most-asked-questions/">Apache Spot: Three Most-Asked Questions</a>
+ <a href="../../apache-spot-3-most-asked-questions/">Apache Spot (Incubating): 3 Most-Asked Questions</a>
</li>
</ul>
</div>
@@ -242,6 +228,9 @@
<h4 class="widgettitle">Archives</h4>
<ul>
<li>
+ <a href='../../2016/09/'>September 2016</a>
+ </li>
+ <li>
<a href='../../2016/08/'>August 2016</a>
</li>
<li>
diff --git a/category/data-science/feed/index.html b/category/data-science/feed/index.html
new file mode 100644
index 0000000..e2c33f6
--- /dev/null
+++ b/category/data-science/feed/index.html
@@ -0,0 +1,62 @@
+<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
+ xmlns:content="http://purl.org/rss/1.0/modules/content/"
+ xmlns:wfw="http://wellformedweb.org/CommentAPI/"
+ xmlns:dc="http://purl.org/dc/elements/1.1/"
+ xmlns:atom="http://www.w3.org/2005/Atom"
+ xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
+ xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
+ >
+
+<channel>
+ <title>Data Science – Apache Spot</title>
+ <atom:link href="http://spot.incubator.apache.org/category/data-science/feed/" rel="self" type="application/rss+xml" />
+ <link>http://spot.incubator.apache.org</link>
+ <description></description>
+ <lastBuildDate>Fri, 07 Oct 2016 00:20:13 +0000</lastBuildDate>
+ <language>en-US</language>
+ <sy:updatePeriod>hourly</sy:updatePeriod>
+ <sy:updateFrequency>1</sy:updateFrequency>
+
+ <item>
+ <title>Jupyter Notebooks for Data Analysis</title>
+ <link>http://spot.incubator.apache.org/jupyter-notebooks-for-data-analysis/</link>
+ <comments>http://spot.incubator.apache.org/jupyter-notebooks-for-data-analysis/#respond</comments>
+ <pubDate>Thu, 22 Sep 2016 21:28:12 +0000</pubDate>
+ <dc:creator><![CDATA[oni-admin]]></dc:creator>
+ <category><![CDATA[Data Science]]></category>
+ <category><![CDATA[Ipython Notebooks]]></category>
+ <category><![CDATA[Threat Analysis Tools]]></category>
+
+ <guid isPermaLink="false">http://spot.incubator.apache.org/?p=136</guid>
+ <description><![CDATA[Why Does Apache Spot Include iPython notebooks? The project team wants Apache Spot to be a versatile tool that can be used by anyone. This means that data scientists and developers need to be able to query and handle the source data to find all the information they need for their decision making. The iPython... <a class="excerpt-read-more" href="http://spot.incubator.apache.org/jupyter-notebooks-for-data-analysis/" title="Read Jupyter Notebooks for Data Analysis" [...]
+ <content:encoded><![CDATA[<p><strong>Why Does Apache Spot Include iPython notebooks? </strong></p>
+<p>The project team wants Apache Spot to be a versatile tool that can be used by anyone. This means that data scientists and developers need to be able to query and handle the source data to find all the information they need for their decision making. The iPython Notebook is an appropriate platform for easy data exploration. One of its biggest advantages is that it provides parallel and distributed computing to enable code execution and debugging in an interactive environment – thus the [...]
+<p>The iPython notebook is a web based interactive computational environment that provides access to the Python shell. While iPython notebooks were originally designed to work with the Python language, they support a number of other programming languages, including Ruby, Scala, Julia, R, Go, C, C++, Java and Perl. There are also multiple additional packages that can be used to get the most out of this highly-customizable tool.</p>
+<p>Starting on version 4.0, most notebook functionalities are now part of the Project Jupyter, while iPython remains as the kernel to work with Python code in the notebooks.</p>
+<img src="http://spot.incubator.apache.org/wp-content/uploads/2016/10/iPython-1.png" alt="ipython" width="361" height="107" class="aligncenter size-full wp-image-140" srcset="http://spot.incubator.apache.org/wp-content/uploads/2016/10/iPython-1.png 361w, http://spot.incubator.apache.org/wp-content/uploads/2016/10/iPython-1-300x89.png 300w" sizes="(max-width: 361px) 100vw, 361px" />
+<p><strong>IPython with Apache Spot for Network Threat Detection</strong></p>
+<p><em>NOTE: This is not intended to be a step-by-step tutorial on how to code a threat analysis in Apache Spot, but more like an introduction on how to approach the suspicions of a security breach.</em></p>
+<p>Although machine learning (ML) will do most of the work detecting anomalies in the traffic, Apache Spot also includes two notebook templates that can get you started on this. The <em>Threat_Investigation_master.ipynb</em> is designed to query the raw data table to find all connections in a day that are related to any threat you select – even connections that were not necessarily flagged as suspicious by ML on a first run. This gives us the chance to get a new data subset and here is w [...]
+<p>If you suspect of a specific type of attack in your network, you can get the whole story by answering the Five ‘W’s.</p>
+<p><strong><em>What? </em></strong></p>
+<p>Maybe there’s been an increase in the logs collected by the system, which indicates abnormal amounts of communication in your network. Or, the amount of POST requests in your network have risen overnight. This is the mystery that needs to be solved by researching through the anomalies previously detected by ML.</p>
+<p><strong><em>Who?</em></strong></p>
+<p>Assuming you have a network context, you can identify the name of the infected machine inside the network, as well as the name of the IP or DNS on the other side of the connection (if it is a known host). If you don’t have a network context or are using DHCP, this can be a little tricky to detect using only Netflow logs. But, that’s where DNS and Proxy logs, come in handy. Including a network context file with Apache Spot is really simple and can go a long way when identifying a threat.</p>
+<p><strong><em>When?</em></strong></p>
+<p>To have a broader visibility on the attack, you can customize the queries on the Threat investigation notebook to review the data through a wider time lapse – instead of just checking through the current day. With this, you could find an increase of a certain type of requests to one (or many) URIs and predict its future behavior.</p>
+<p><strong><em>Where?</em></strong></p>
+<p>When working only with DNS, having a destination URL might not say much about where your information is going to, but Apache Spot allows you to connect with a geolocation database to identify the location of the suspected attackers IP. Taking advantage of this option, you can visually locate the other end of the connection on a map. You might find that it’s pointing to a country banned by your company, indicating a leak.</p>
+<p><strong><em>Why?</em></strong></p>
+<p>This answer to “why” will depend highly on the result of the analysis. For instance, an excessive amount of POST requests from one machine inside the network to an unidentified URI can indicate a data mining attack. Tracing back to patient zero, you can find that this could have originated with a phishing email, malicious software installed by an employee or a one-time visitor’s infected machine that connected to your network.</p>
+<p><strong>How to Get Answers to the Five Ws Questions</strong></p>
+<p>All of the previous questions can be answered by looking at the raw data collected. Although performing elaborated queries directly to your database can seem tempting, this type of analysis with Hive, or even Impala, can be very time consuming. A better approach would be to use Pandas to read and transform your dataset into a relational structured dataframe. This lets you work with as if it were an offline structured relational database.</p>
+<p>Once you have your desired results and data subsets, you can use MatplotLib to easily graph your findings. (We cover this subject in more depth in another post.) Another advantage of the notebook is that you can download it as HTML or a PDF file to store locally and use it in a presentation – or just keep it for future reference.</p>
+<p><strong>Wrap Up</strong></p>
+<p>This post was meant to be just a brief introduction of how you can use iPython notebooks in Apache Spot to perform further data analysis and include it our executive report (in addition to the already included Story board). Although this is not the only way you can do this, it is a very interactive and fun way to do it. You’ll also see that the overall processing time is very short – thanks to the iPython notebook task parallelism ability.</p>
+<p>We want to hear from YOU! Have you used iPython notebooks before? How do you feel about having this tool in Apache Spot? If you’re interested in further data analysis through interactive charts, a new post is coming soon on D3 and jQuery data visualization. Also, check back soon to read more on this and other Cybersecurity subjects.</p>
+]]></content:encoded>
+ <wfw:commentRss>http://spot.incubator.apache.org/jupyter-notebooks-for-data-analysis/feed/</wfw:commentRss>
+ <slash:comments>0</slash:comments>
+ </item>
+ </channel>
+</rss>
diff --git a/2016/03/index.html b/category/data-science/index.html
similarity index 68%
copy from 2016/03/index.html
copy to category/data-science/index.html
index 4971dc0..d416f07 100644
--- a/2016/03/index.html
+++ b/category/data-science/index.html
@@ -12,7 +12,7 @@
<meta http-equiv="X-UA-Compatible" content="IE=edge">
- <title>Apache Spot 2016 March</title>
+ <title>Apache Spot Data Science</title>
<meta name="HandheldFriendly" content="True">
<meta name="MobileOptimized" content="320">
@@ -27,18 +27,15 @@
<meta name="msapplication-TileImage" content="../../wp-content/themes/oni/library/images/win8-tile-icon.png">
<meta name="theme-color" content="#121212">
- <link rel="pingback" href="../../xmlrpc.php">
+ <link rel="pingback" href="http://spot.incubator.apache.org/xmlrpc.php">
- <link rel='dns-prefetch' href='//fonts.googleapis.com' />
- <link rel='dns-prefetch' href='//s.w.org' />
<link rel="alternate" type="application/rss+xml" title="Apache Spot » Feed" href="../../feed/" />
<link rel="alternate" type="application/rss+xml" title="Apache Spot » Comments Feed" href="../../comments/feed/" />
+ <link rel="alternate" type="application/rss+xml" title="Apache Spot » Data Science Category Feed" href="../../category/data-science/feed/" />
<script type="text/javascript">
window._wpemojiSettings = {
- "baseUrl" : "https:\/\/s.w.org\/images\/core\/emoji\/2\/72x72\/",
+ "baseUrl" : "https:\/\/s.w.org\/images\/core\/emoji\/72x72\/",
"ext" : ".png",
- "svgUrl" : "https:\/\/s.w.org\/images\/core\/emoji\/2\/svg\/",
- "svgExt" : ".svg",
"source" : {
"concatemoji" : "http:\/\/spot.incubator.apache.org\/wp-includes\/js\/wp-emoji-release.min.js"
}
@@ -47,30 +44,24 @@
var c,
d,
e,
- f,
- g,
- h = b.createElement("canvas"),
- i = h.getContext && h.getContext("2d"),
- j = String.fromCharCode;
- if (!i || !i.fillText)
+ f = b.createElement("canvas"),
+ g = f.getContext && f.getContext("2d"),
+ h = String.fromCharCode;
+ if (!g || !g.fillText)
return !1;
- switch(i.textBaseline="top",i.font="600 32px Arial",a) {
+ switch(g.textBaseline="top",g.font="600 32px Arial",a) {
case"flag":
- return i.fillText(j(55356, 56806, 55356, 56826), 0, 0), !(h.toDataURL().length < 3e3) && (i.clearRect(0, 0, h.width, h.height), i.fillText(j(55356, 57331, 65039, 8205, 55356, 57096), 0, 0),
- c = h.toDataURL(), i.clearRect(0, 0, h.width, h.height), i.fillText(j(55356, 57331, 55356, 57096), 0, 0),
- d = h.toDataURL(), c !== d);
+ return g.fillText(h(55356, 56806, 55356, 56826), 0, 0), f.toDataURL().length > 3e3;
case"diversity":
- return i.fillText(j(55356, 57221), 0, 0),
- e = i.getImageData(16, 16, 1, 1).data,
- f = e[0] + "," + e[1] + "," + e[2] + "," + e[3], i.fillText(j(55356, 57221, 55356, 57343), 0, 0),
- e = i.getImageData(16, 16, 1, 1).data,
- g = e[0] + "," + e[1] + "," + e[2] + "," + e[3], f !== g;
+ return g.fillText(h(55356, 57221), 0, 0),
+ c = g.getImageData(16, 16, 1, 1).data,
+ d = c[0] + "," + c[1] + "," + c[2] + "," + c[3], g.fillText(h(55356, 57221, 55356, 57343), 0, 0),
+ c = g.getImageData(16, 16, 1, 1).data,
+ e = c[0] + "," + c[1] + "," + c[2] + "," + c[3], d !== e;
case"simple":
- return i.fillText(j(55357, 56835), 0, 0), 0 !== i.getImageData(16,16,1,1).data[0];
+ return g.fillText(h(55357, 56835), 0, 0), 0 !== g.getImageData(16,16,1,1).data[0];
case"unicode8":
- return i.fillText(j(55356, 57135), 0, 0), 0 !== i.getImageData(16,16,1,1).data[0];
- case"unicode9":
- return i.fillText(j(55358, 56631), 0, 0), 0 !== i.getImageData(16,16,1,1).data[0]
+ return g.fillText(h(55356, 57135), 0, 0), 0 !== g.getImageData(16,16,1,1).data[0]
}
return !1
}
@@ -84,7 +75,7 @@
g,
h,
i;
- for ( i = Array("simple", "flag", "unicode8", "diversity", "unicode9"), c.supports = {
+ for ( i = Array("simple", "flag", "unicode8", "diversity"), c.supports = {
everything : !0,
everythingExceptFlag : !0
},
@@ -116,7 +107,7 @@
<link rel='stylesheet' id='googleFonts-css' href='http://fonts.googleapis.com/css?family=Lato%3A400%2C700%2C400italic%2C700italic' type='text/css' media='all' />
<link rel='stylesheet' id='bones-stylesheet-css' href='../../wp-content/themes/oni/library/css/style.css' type='text/css' media='all' />
<!--[if lt IE 9]>
- <link rel='stylesheet' id='bones-ie-only-css' href='../../wp-content/themes/oni/library/css/ie.css' type='text/css' media='all' />
+ <link rel='stylesheet' id='bones-ie-only-css' href='http://spot.incubator.apache.org/wp-content/themes/oni/library/css/ie.css' type='text/css' media='all' />
<![endif]-->
<link rel='stylesheet' id='mm-css-css' href='../../wp-content/themes/oni/library/css/meanmenu.css' type='text/css' media='all' />
<script type='text/javascript' src='../../wp-content/themes/oni/library/js/libs/modernizr.custom.min.js'></script>
@@ -125,27 +116,9 @@
<script type='text/javascript' src='../../wp-content/themes/oni/library/js/jquery.meanmenu.js'></script>
<link rel='https://api.w.org/' href='../../wp-json/' />
- <script>
- (function(i, s, o, g, r, a, m) {
- i['GoogleAnalyticsObject'] = r;
- i[r] = i[r] ||
- function() {
- (i[r].q = i[r].q || []).push(arguments)
- }, i[r].l = 1 * new Date();
- a = s.createElement(o),
- m = s.getElementsByTagName(o)[0];
- a.async = 1;
- a.src = g;
- m.parentNode.insertBefore(a, m)
- })(window, document, 'script', '//www.google-analytics.com/analytics.js', 'ga');
-
- ga('create', 'UA-75955621-1', 'auto');
- ga('send', 'pageview');
-
- </script>
</head>
- <body class="archive date" itemscope itemtype="http://schema.org/WebPage">
+ <body class="archive category category-data-science category-11" itemscope itemtype="http://schema.org/WebPage">
<div id="container">
@@ -159,16 +132,16 @@
<nav role="navigation" itemscope itemtype="http://schema.org/SiteNavigationElement">
<ul id="menu-main-menu" class="nav top-nav cf">
- <li id="menu-item-129" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-129">
+ <li id="menu-item-4" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-4">
<a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight">Get Started</a>
</li>
<li id="menu-item-5" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-5">
<a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight#if-you-want-all-of-the-oni-code-at-once-just-clone-it">Download</a>
</li>
- <li id="menu-item-130" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-130">
+ <li id="menu-item-6" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-6">
<a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight#contributing-to-oni">Contribute</a>
</li>
- <li id="menu-item-106" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-106">
+ <li id="menu-item-7" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-7">
<a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight/wiki">Wiki</a>
</li>
<li id="menu-item-13" class="menu-item menu-item-type-post_type menu-item-object-page current_page_parent menu-item-13">
@@ -189,16 +162,16 @@
<main id="main" class="m-all t-2of3 d-5of7 cf" role="main" itemscope itemprop="mainContentOfPage" itemtype="http://schema.org/Blog">
- <h1 class="page-title">Month: March 2016</h1>
+ <h1 class="page-title">Category: Data Science</h1>
- <article id="post-62" class="cf post-62 post type-post status-publish format-standard hentry category-security-analytics tag-github tag-open-network-insight tag-open-source" role="article">
+ <article id="post-136" class="cf post-136 post type-post status-publish format-standard hentry category-data-science category-ipython-notebooks category-threat-analysis-tools" role="article">
<header class="entry-header article-header">
- <h3 class="h2 entry-title"><a href="../../open-network-insight-3-most-asked-questions/" rel="bookmark" title="Apache Spot: Three Most-Asked Questions">Apache Spot: Three Most-Asked Questions</a></h3>
+ <h3 class="h2 entry-title"><a href="../../jupyter-notebooks-for-data-analysis/" rel="bookmark" title="Jupyter Notebooks for Data Analysis">Jupyter Notebooks for Data Analysis</a></h3>
<p class="byline entry-meta vcard">
- <time class="updated entry-time" datetime="2016-03-29" itemprop="datePublished">
- March 29, 2016
+ <time class="updated entry-time" datetime="2016-09-22" itemprop="datePublished">
+ September 22, 2016
</time>
</span>
</p>
@@ -208,7 +181,7 @@
<section class="entry-content cf">
<p>
- While this is not the first blog post about Apache Spot, it is the first one by a creator of the solution. As a security data scientist in Intel’s Data Center Group, I joined a small team to start thinking about solving really hard problems in cloud analytics. The team grew, and out of that… <a class="excerpt-read-more" href="../../open-network-insight-3-most-asked-questions/" title="Read Apache Spot: Three Most-Asked Questions">Read more & [...]
+ Why Does Apache Spot Include iPython notebooks? The project team wants Apache Spot to be a versatile tool that can be used by anyone. This means that data scientists and developers need to be able to query and handle the source data to find all the information they need for their decision making. The iPython… <a class="excerpt-read-more" href="../../jupyter-notebooks-for-data-analysis/" title="Read Jupyter Notebooks for Data Analysis">Read more & [...]
</p>
</section>
@@ -227,13 +200,16 @@
<h4 class="widgettitle">Recent Posts</h4>
<ul>
<li>
- <a href="../../open-network-insight-oni-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/">Apache Spot and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a>
+ <a href="../../jupyter-notebooks-for-data-analysis/">Jupyter Notebooks for Data Analysis</a>
+ </li>
+ <li>
+ <a href="../../apache-spot-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/">Apache Spot (Incubating) and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a>
</li>
<li>
- <a href="../../how-open-network-insight-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/">How Apache Spot Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a>
+ <a href="../../how-apache-spot-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/">How Apache Spot (Incubating) Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a>
</li>
<li>
- <a href="../../open-network-insight-3-most-asked-questions/">Apache Spot: Three Most-Asked Questions</a>
+ <a href="../../apache-spot-3-most-asked-questions/">Apache Spot (Incubating): 3 Most-Asked Questions</a>
</li>
</ul>
</div>
@@ -241,6 +217,9 @@
<h4 class="widgettitle">Archives</h4>
<ul>
<li>
+ <a href='../../2016/09/'>September 2016</a>
+ </li>
+ <li>
<a href='../../2016/08/'>August 2016</a>
</li>
<li>
diff --git a/category/ipython-notebooks/feed/index.html b/category/ipython-notebooks/feed/index.html
new file mode 100644
index 0000000..1935eed
--- /dev/null
+++ b/category/ipython-notebooks/feed/index.html
@@ -0,0 +1,62 @@
+<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
+ xmlns:content="http://purl.org/rss/1.0/modules/content/"
+ xmlns:wfw="http://wellformedweb.org/CommentAPI/"
+ xmlns:dc="http://purl.org/dc/elements/1.1/"
+ xmlns:atom="http://www.w3.org/2005/Atom"
+ xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
+ xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
+ >
+
+<channel>
+ <title>Ipython Notebooks – Apache Spot</title>
+ <atom:link href="http://spot.incubator.apache.org/category/ipython-notebooks/feed/" rel="self" type="application/rss+xml" />
+ <link>http://spot.incubator.apache.org</link>
+ <description></description>
+ <lastBuildDate>Fri, 07 Oct 2016 00:20:13 +0000</lastBuildDate>
+ <language>en-US</language>
+ <sy:updatePeriod>hourly</sy:updatePeriod>
+ <sy:updateFrequency>1</sy:updateFrequency>
+
+ <item>
+ <title>Jupyter Notebooks for Data Analysis</title>
+ <link>http://spot.incubator.apache.org/jupyter-notebooks-for-data-analysis/</link>
+ <comments>http://spot.incubator.apache.org/jupyter-notebooks-for-data-analysis/#respond</comments>
+ <pubDate>Thu, 22 Sep 2016 21:28:12 +0000</pubDate>
+ <dc:creator><![CDATA[oni-admin]]></dc:creator>
+ <category><![CDATA[Data Science]]></category>
+ <category><![CDATA[Ipython Notebooks]]></category>
+ <category><![CDATA[Threat Analysis Tools]]></category>
+
+ <guid isPermaLink="false">http://spot.incubator.apache.org/?p=136</guid>
+ <description><![CDATA[Why Does Apache Spot Include iPython notebooks? The project team wants Apache Spot to be a versatile tool that can be used by anyone. This means that data scientists and developers need to be able to query and handle the source data to find all the information they need for their decision making. The iPython... <a class="excerpt-read-more" href="http://spot.incubator.apache.org/jupyter-notebooks-for-data-analysis/" title="Read Jupyter Notebooks for Data Analysis" [...]
+ <content:encoded><![CDATA[<p><strong>Why Does Apache Spot Include iPython notebooks? </strong></p>
+<p>The project team wants Apache Spot to be a versatile tool that can be used by anyone. This means that data scientists and developers need to be able to query and handle the source data to find all the information they need for their decision making. The iPython Notebook is an appropriate platform for easy data exploration. One of its biggest advantages is that it provides parallel and distributed computing to enable code execution and debugging in an interactive environment – thus the [...]
+<p>The iPython notebook is a web based interactive computational environment that provides access to the Python shell. While iPython notebooks were originally designed to work with the Python language, they support a number of other programming languages, including Ruby, Scala, Julia, R, Go, C, C++, Java and Perl. There are also multiple additional packages that can be used to get the most out of this highly-customizable tool.</p>
+<p>Starting on version 4.0, most notebook functionalities are now part of the Project Jupyter, while iPython remains as the kernel to work with Python code in the notebooks.</p>
+<img src="http://spot.incubator.apache.org/wp-content/uploads/2016/10/iPython-1.png" alt="ipython" width="361" height="107" class="aligncenter size-full wp-image-140" srcset="http://spot.incubator.apache.org/wp-content/uploads/2016/10/iPython-1.png 361w, http://spot.incubator.apache.org/wp-content/uploads/2016/10/iPython-1-300x89.png 300w" sizes="(max-width: 361px) 100vw, 361px" />
+<p><strong>IPython with Apache Spot for Network Threat Detection</strong></p>
+<p><em>NOTE: This is not intended to be a step-by-step tutorial on how to code a threat analysis in Apache Spot, but more like an introduction on how to approach the suspicions of a security breach.</em></p>
+<p>Although machine learning (ML) will do most of the work detecting anomalies in the traffic, Apache Spot also includes two notebook templates that can get you started on this. The <em>Threat_Investigation_master.ipynb</em> is designed to query the raw data table to find all connections in a day that are related to any threat you select – even connections that were not necessarily flagged as suspicious by ML on a first run. This gives us the chance to get a new data subset and here is w [...]
+<p>If you suspect of a specific type of attack in your network, you can get the whole story by answering the Five ‘W’s.</p>
+<p><strong><em>What? </em></strong></p>
+<p>Maybe there’s been an increase in the logs collected by the system, which indicates abnormal amounts of communication in your network. Or, the amount of POST requests in your network have risen overnight. This is the mystery that needs to be solved by researching through the anomalies previously detected by ML.</p>
+<p><strong><em>Who?</em></strong></p>
+<p>Assuming you have a network context, you can identify the name of the infected machine inside the network, as well as the name of the IP or DNS on the other side of the connection (if it is a known host). If you don’t have a network context or are using DHCP, this can be a little tricky to detect using only Netflow logs. But, that’s where DNS and Proxy logs, come in handy. Including a network context file with Apache Spot is really simple and can go a long way when identifying a threat.</p>
+<p><strong><em>When?</em></strong></p>
+<p>To have a broader visibility on the attack, you can customize the queries on the Threat investigation notebook to review the data through a wider time lapse – instead of just checking through the current day. With this, you could find an increase of a certain type of requests to one (or many) URIs and predict its future behavior.</p>
+<p><strong><em>Where?</em></strong></p>
+<p>When working only with DNS, having a destination URL might not say much about where your information is going to, but Apache Spot allows you to connect with a geolocation database to identify the location of the suspected attackers IP. Taking advantage of this option, you can visually locate the other end of the connection on a map. You might find that it’s pointing to a country banned by your company, indicating a leak.</p>
+<p><strong><em>Why?</em></strong></p>
+<p>This answer to “why” will depend highly on the result of the analysis. For instance, an excessive amount of POST requests from one machine inside the network to an unidentified URI can indicate a data mining attack. Tracing back to patient zero, you can find that this could have originated with a phishing email, malicious software installed by an employee or a one-time visitor’s infected machine that connected to your network.</p>
+<p><strong>How to Get Answers to the Five Ws Questions</strong></p>
+<p>All of the previous questions can be answered by looking at the raw data collected. Although performing elaborated queries directly to your database can seem tempting, this type of analysis with Hive, or even Impala, can be very time consuming. A better approach would be to use Pandas to read and transform your dataset into a relational structured dataframe. This lets you work with as if it were an offline structured relational database.</p>
+<p>Once you have your desired results and data subsets, you can use MatplotLib to easily graph your findings. (We cover this subject in more depth in another post.) Another advantage of the notebook is that you can download it as HTML or a PDF file to store locally and use it in a presentation – or just keep it for future reference.</p>
+<p><strong>Wrap Up</strong></p>
+<p>This post was meant to be just a brief introduction of how you can use iPython notebooks in Apache Spot to perform further data analysis and include it our executive report (in addition to the already included Story board). Although this is not the only way you can do this, it is a very interactive and fun way to do it. You’ll also see that the overall processing time is very short – thanks to the iPython notebook task parallelism ability.</p>
+<p>We want to hear from YOU! Have you used iPython notebooks before? How do you feel about having this tool in Apache Spot? If you’re interested in further data analysis through interactive charts, a new post is coming soon on D3 and jQuery data visualization. Also, check back soon to read more on this and other Cybersecurity subjects.</p>
+]]></content:encoded>
+ <wfw:commentRss>http://spot.incubator.apache.org/jupyter-notebooks-for-data-analysis/feed/</wfw:commentRss>
+ <slash:comments>0</slash:comments>
+ </item>
+ </channel>
+</rss>
diff --git a/2016/03/index.html b/category/ipython-notebooks/index.html
similarity index 69%
copy from 2016/03/index.html
copy to category/ipython-notebooks/index.html
index 4971dc0..2432f5e 100644
--- a/2016/03/index.html
+++ b/category/ipython-notebooks/index.html
@@ -12,7 +12,7 @@
<meta http-equiv="X-UA-Compatible" content="IE=edge">
- <title>Apache Spot 2016 March</title>
+ <title>Apache Spot Ipython Notebooks</title>
<meta name="HandheldFriendly" content="True">
<meta name="MobileOptimized" content="320">
@@ -27,18 +27,15 @@
<meta name="msapplication-TileImage" content="../../wp-content/themes/oni/library/images/win8-tile-icon.png">
<meta name="theme-color" content="#121212">
- <link rel="pingback" href="../../xmlrpc.php">
+ <link rel="pingback" href="http://spot.incubator.apache.org/xmlrpc.php">
- <link rel='dns-prefetch' href='//fonts.googleapis.com' />
- <link rel='dns-prefetch' href='//s.w.org' />
<link rel="alternate" type="application/rss+xml" title="Apache Spot » Feed" href="../../feed/" />
<link rel="alternate" type="application/rss+xml" title="Apache Spot » Comments Feed" href="../../comments/feed/" />
+ <link rel="alternate" type="application/rss+xml" title="Apache Spot » Ipython Notebooks Category Feed" href="../../category/ipython-notebooks/feed/" />
<script type="text/javascript">
window._wpemojiSettings = {
- "baseUrl" : "https:\/\/s.w.org\/images\/core\/emoji\/2\/72x72\/",
+ "baseUrl" : "https:\/\/s.w.org\/images\/core\/emoji\/72x72\/",
"ext" : ".png",
- "svgUrl" : "https:\/\/s.w.org\/images\/core\/emoji\/2\/svg\/",
- "svgExt" : ".svg",
"source" : {
"concatemoji" : "http:\/\/spot.incubator.apache.org\/wp-includes\/js\/wp-emoji-release.min.js"
}
@@ -47,30 +44,24 @@
var c,
d,
e,
- f,
- g,
- h = b.createElement("canvas"),
- i = h.getContext && h.getContext("2d"),
- j = String.fromCharCode;
- if (!i || !i.fillText)
+ f = b.createElement("canvas"),
+ g = f.getContext && f.getContext("2d"),
+ h = String.fromCharCode;
+ if (!g || !g.fillText)
return !1;
- switch(i.textBaseline="top",i.font="600 32px Arial",a) {
+ switch(g.textBaseline="top",g.font="600 32px Arial",a) {
case"flag":
- return i.fillText(j(55356, 56806, 55356, 56826), 0, 0), !(h.toDataURL().length < 3e3) && (i.clearRect(0, 0, h.width, h.height), i.fillText(j(55356, 57331, 65039, 8205, 55356, 57096), 0, 0),
- c = h.toDataURL(), i.clearRect(0, 0, h.width, h.height), i.fillText(j(55356, 57331, 55356, 57096), 0, 0),
- d = h.toDataURL(), c !== d);
+ return g.fillText(h(55356, 56806, 55356, 56826), 0, 0), f.toDataURL().length > 3e3;
case"diversity":
- return i.fillText(j(55356, 57221), 0, 0),
- e = i.getImageData(16, 16, 1, 1).data,
- f = e[0] + "," + e[1] + "," + e[2] + "," + e[3], i.fillText(j(55356, 57221, 55356, 57343), 0, 0),
- e = i.getImageData(16, 16, 1, 1).data,
- g = e[0] + "," + e[1] + "," + e[2] + "," + e[3], f !== g;
+ return g.fillText(h(55356, 57221), 0, 0),
+ c = g.getImageData(16, 16, 1, 1).data,
+ d = c[0] + "," + c[1] + "," + c[2] + "," + c[3], g.fillText(h(55356, 57221, 55356, 57343), 0, 0),
+ c = g.getImageData(16, 16, 1, 1).data,
+ e = c[0] + "," + c[1] + "," + c[2] + "," + c[3], d !== e;
case"simple":
- return i.fillText(j(55357, 56835), 0, 0), 0 !== i.getImageData(16,16,1,1).data[0];
+ return g.fillText(h(55357, 56835), 0, 0), 0 !== g.getImageData(16,16,1,1).data[0];
case"unicode8":
- return i.fillText(j(55356, 57135), 0, 0), 0 !== i.getImageData(16,16,1,1).data[0];
- case"unicode9":
- return i.fillText(j(55358, 56631), 0, 0), 0 !== i.getImageData(16,16,1,1).data[0]
+ return g.fillText(h(55356, 57135), 0, 0), 0 !== g.getImageData(16,16,1,1).data[0]
}
return !1
}
@@ -84,7 +75,7 @@
g,
h,
i;
- for ( i = Array("simple", "flag", "unicode8", "diversity", "unicode9"), c.supports = {
+ for ( i = Array("simple", "flag", "unicode8", "diversity"), c.supports = {
everything : !0,
everythingExceptFlag : !0
},
@@ -125,27 +116,9 @@
<script type='text/javascript' src='../../wp-content/themes/oni/library/js/jquery.meanmenu.js'></script>
<link rel='https://api.w.org/' href='../../wp-json/' />
- <script>
- (function(i, s, o, g, r, a, m) {
- i['GoogleAnalyticsObject'] = r;
- i[r] = i[r] ||
- function() {
- (i[r].q = i[r].q || []).push(arguments)
- }, i[r].l = 1 * new Date();
- a = s.createElement(o),
- m = s.getElementsByTagName(o)[0];
- a.async = 1;
- a.src = g;
- m.parentNode.insertBefore(a, m)
- })(window, document, 'script', '//www.google-analytics.com/analytics.js', 'ga');
-
- ga('create', 'UA-75955621-1', 'auto');
- ga('send', 'pageview');
-
- </script>
</head>
- <body class="archive date" itemscope itemtype="http://schema.org/WebPage">
+ <body class="archive category category-ipython-notebooks category-10" itemscope itemtype="http://schema.org/WebPage">
<div id="container">
@@ -159,16 +132,16 @@
<nav role="navigation" itemscope itemtype="http://schema.org/SiteNavigationElement">
<ul id="menu-main-menu" class="nav top-nav cf">
- <li id="menu-item-129" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-129">
+ <li id="menu-item-4" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-4">
<a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight">Get Started</a>
</li>
<li id="menu-item-5" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-5">
<a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight#if-you-want-all-of-the-oni-code-at-once-just-clone-it">Download</a>
</li>
- <li id="menu-item-130" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-130">
+ <li id="menu-item-6" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-6">
<a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight#contributing-to-oni">Contribute</a>
</li>
- <li id="menu-item-106" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-106">
+ <li id="menu-item-7" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-7">
<a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight/wiki">Wiki</a>
</li>
<li id="menu-item-13" class="menu-item menu-item-type-post_type menu-item-object-page current_page_parent menu-item-13">
@@ -189,16 +162,16 @@
<main id="main" class="m-all t-2of3 d-5of7 cf" role="main" itemscope itemprop="mainContentOfPage" itemtype="http://schema.org/Blog">
- <h1 class="page-title">Month: March 2016</h1>
+ <h1 class="page-title">Category: Ipython Notebooks</h1>
- <article id="post-62" class="cf post-62 post type-post status-publish format-standard hentry category-security-analytics tag-github tag-open-network-insight tag-open-source" role="article">
+ <article id="post-136" class="cf post-136 post type-post status-publish format-standard hentry category-data-science category-ipython-notebooks category-threat-analysis-tools" role="article">
<header class="entry-header article-header">
- <h3 class="h2 entry-title"><a href="../../open-network-insight-3-most-asked-questions/" rel="bookmark" title="Apache Spot: Three Most-Asked Questions">Apache Spot: Three Most-Asked Questions</a></h3>
+ <h3 class="h2 entry-title"><a href="../../jupyter-notebooks-for-data-analysis/" rel="bookmark" title="Jupyter Notebooks for Data Analysis">Jupyter Notebooks for Data Analysis</a></h3>
<p class="byline entry-meta vcard">
- <time class="updated entry-time" datetime="2016-03-29" itemprop="datePublished">
- March 29, 2016
+ <time class="updated entry-time" datetime="2016-09-22" itemprop="datePublished">
+ September 22, 2016
</time>
</span>
</p>
@@ -208,7 +181,7 @@
<section class="entry-content cf">
<p>
- While this is not the first blog post about Apache Spot, it is the first one by a creator of the solution. As a security data scientist in Intel’s Data Center Group, I joined a small team to start thinking about solving really hard problems in cloud analytics. The team grew, and out of that… <a class="excerpt-read-more" href="../../open-network-insight-3-most-asked-questions/" title="Read Apache Spot: Three Most-Asked Questions">Read more & [...]
+ Why Does Apache Spot Include iPython notebooks? The project team wants Apache Spot to be a versatile tool that can be used by anyone. This means that data scientists and developers need to be able to query and handle the source data to find all the information they need for their decision making. The iPython… <a class="excerpt-read-more" href="../../jupyter-notebooks-for-data-analysis/" title="Read Jupyter Notebooks for Data Analysis">Read more & [...]
</p>
</section>
@@ -227,13 +200,16 @@
<h4 class="widgettitle">Recent Posts</h4>
<ul>
<li>
- <a href="../../open-network-insight-oni-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/">Apache Spot and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a>
+ <a href="../../jupyter-notebooks-for-data-analysis/">Jupyter Notebooks for Data Analysis</a>
+ </li>
+ <li>
+ <a href="../../apache-spot-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/">Apache Spot (Incubating) and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a>
</li>
<li>
- <a href="../../how-open-network-insight-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/">How Apache Spot Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a>
+ <a href="../../how-apache-spot-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/">How Apache Spot (Incubating) Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a>
</li>
<li>
- <a href="../../open-network-insight-3-most-asked-questions/">Apache Spot: Three Most-Asked Questions</a>
+ <a href="../../apache-spot-3-most-asked-questions/">Apache Spot (Incubating): 3 Most-Asked Questions</a>
</li>
</ul>
</div>
@@ -241,6 +217,9 @@
<h4 class="widgettitle">Archives</h4>
<ul>
<li>
+ <a href='../../2016/09/'>September 2016</a>
+ </li>
+ <li>
<a href='../../2016/08/'>August 2016</a>
</li>
<li>
diff --git a/category/security-analytics/feed/index.html b/category/security-analytics/feed/index.html
index 690f185..c3706f5 100644
--- a/category/security-analytics/feed/index.html
+++ b/category/security-analytics/feed/index.html
@@ -19,8 +19,8 @@
<item>
<title>Apache Spot: Three Most-Asked Questions</title>
- <link>http://spot.incubator.apache.org/open-network-insight-3-most-asked-questions/</link>
- <comments>http://spot.incubator.apache.org/open-network-insight-3-most-asked-questions/#respond</comments>
+ <link>http://spot.incubator.apache.org/apache-spot-3-most-asked-questions/</link>
+ <comments>http://spot.incubator.apache.org/apache-spot-3-most-asked-questions/#respond</comments>
<pubDate>Tue, 29 Mar 2016 05:48:19 +0000</pubDate>
<dc:creator><![CDATA[oni-admin]]></dc:creator>
<category><![CDATA[Security Analytics]]></category>
@@ -29,7 +29,7 @@
<category><![CDATA[open source]]></category>
<guid isPermaLink="false">http://spot.incubator.apache.org/?p=62</guid>
- <description><![CDATA[While this is not the first blog post about Apache Spot, it is the first one by a creator of the solution. As a security data scientist in Intel’s Data Center Group, I joined a small team to start thinking about solving really hard problems in cloud analytics. The team grew, and out of that... <a class="excerpt-read-more" href="http://spot.incubator.apache.org/open-network-insight-3-most-asked-questions/" title="Read Apache Spot: Three Most-Asked Questions" [...]
+ <description><![CDATA[While this is not the first blog post about Apache Spot, it is the first one by a creator of the solution. As a security data scientist in Intel’s Data Center Group, I joined a small team to start thinking about solving really hard problems in cloud analytics. The team grew, and out of that... <a class="excerpt-read-more" href="http://spot.incubator.apache.org/apache-spot-3-most-asked-questions/" title="Read Apache Spot: Three Most-Asked Questions">Read mor [...]
<content:encoded><![CDATA[<p>While this is not the first blog post about Apache Spot, it is the first one by a creator of the solution. As a security data scientist in Intel’s Data Center Group, I joined a small team to start thinking about solving really hard problems in cloud analytics. The team grew, and out of that effort, came Apache Spot. Since we started talking about the project, these are the three questions I am asked the most.</p>
<p><strong>What Is Apache Spot?</strong><br />
Apache Spot is an open source, flow and packet analytics solution built on Hadoop. It combines big data processing, at-scale machine learning, and unique security analytics to put potential threats in front of defenders. While I am a data scientist today, I was a security investigator just a few years ago. I wanted to develop a solution that would put new tools and technology in play for defenders, but without requiring them to walk away from security and get a math degree.</p>
@@ -40,11 +40,11 @@ Apache Spot is an open source, flow and packet analytics solution built on Hadoo
<p>At Intel, I have worked in IT, for a security product company (McAfee), and in the Data Center Group. Intel IT was an early pioneer of the concept of proactive investigations to protect intellectual property. McAfee (now Intel Security Group) has a broad customer base in the realms of network, endpoint, and content security, to name only a few. And the Intel Data Center group has strategic partnerships with Cloudera and Accenture, as well as some pretty cool analytics efforts of their [...]
<p><strong>Why Open Source?</strong></p>
<p>I learned from my earlier efforts in security analytics, that to invite collaboration from academia, the public sector, and the private sector, open source software is an excellent choice. We are now seeking to build a community of developers, data scientists, and security enthusiasts to grow Apache Spot into something we can all be proud of. We have also chosen an Apache software license, so that it can enrich commercial software offerings as well.</p>
-<p>The greatest thing for me since we announced at RSA is to hear OTHER people talk about Apache Spot (formerly Open Network Insight or ONI), here are some of my favorites, from <a href="http://vision.cloudera.com/open-network-insight-changing-infosec-data-science-forever/" target="_blank">a Data Scientist @ eBay </a>, <a href="https://newsroom.accenture.com/news/accenture-introduces-the-accenture-cyber-intelligence-platform-to-help-organizations-continuously-predict-detect-and-combat-cy [...]
+<p>The greatest thing for me since we announced at RSA is to hear OTHER people talk about Apache Spot (formerly Open Network Insight or ONI), here are some of my favorites, from <a href="http://vision.cloudera.com/apache-spot-changing-infosec-data-science-forever/" target="_blank">a Data Scientist @ eBay </a>, <a href="https://newsroom.accenture.com/news/accenture-introduces-the-accenture-cyber-intelligence-platform-to-help-organizations-continuously-predict-detect-and-combat-cyber-attac [...]
<p>Fork us on Github!</p>
<p>Grant Babb</p>
]]></content:encoded>
- <wfw:commentRss>http://spot.incubator.apache.org/open-network-insight-3-most-asked-questions/feed/</wfw:commentRss>
+ <wfw:commentRss>http://spot.incubator.apache.org/apache-spot-3-most-asked-questions/feed/</wfw:commentRss>
<slash:comments>0</slash:comments>
</item>
</channel>
diff --git a/category/security-analytics/index.html b/category/security-analytics/index.html
index ea98511..9c716d8 100644
--- a/category/security-analytics/index.html
+++ b/category/security-analytics/index.html
@@ -12,7 +12,7 @@
<meta http-equiv="X-UA-Compatible" content="IE=edge">
- <title>Apache Spot Security Analytics</title>
+ <title>Security Analytics</title>
<meta name="HandheldFriendly" content="True">
<meta name="MobileOptimized" content="320">
@@ -126,24 +126,6 @@
<script type='text/javascript' src='../../wp-content/themes/oni/library/js/jquery.meanmenu.js'></script>
<link rel='https://api.w.org/' href='../../wp-json/' />
- <script>
- (function(i, s, o, g, r, a, m) {
- i['GoogleAnalyticsObject'] = r;
- i[r] = i[r] ||
- function() {
- (i[r].q = i[r].q || []).push(arguments)
- }, i[r].l = 1 * new Date();
- a = s.createElement(o),
- m = s.getElementsByTagName(o)[0];
- a.async = 1;
- a.src = g;
- m.parentNode.insertBefore(a, m)
- })(window, document, 'script', '//www.google-analytics.com/analytics.js', 'ga');
-
- ga('create', 'UA-75955621-1', 'auto');
- ga('send', 'pageview');
-
- </script>
</head>
<body class="archive category category-security-analytics category-4" itemscope itemtype="http://schema.org/WebPage">
@@ -161,16 +143,16 @@
<nav role="navigation" itemscope itemtype="http://schema.org/SiteNavigationElement">
<ul id="menu-main-menu" class="nav top-nav cf">
<li id="menu-item-129" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-129">
- <a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight">Get Started</a>
+ <a target="_blank" href="https://github.com/Open-Network-Insight/apache-spot">Get Started</a>
</li>
<li id="menu-item-5" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-5">
- <a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight#if-you-want-all-of-the-oni-code-at-once-just-clone-it">Download</a>
+ <a target="_blank" href="https://github.com/Open-Network-Insight/apache-spot#if-you-want-all-of-the-oni-code-at-once-just-clone-it">Download</a>
</li>
<li id="menu-item-130" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-130">
- <a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight#contributing-to-oni">Contribute</a>
+ <a target="_blank" href="https://github.com/Open-Network-Insight/apache-spot#contributing-to-oni">Contribute</a>
</li>
<li id="menu-item-106" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-106">
- <a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight/wiki">Wiki</a>
+ <a target="_blank" href="https://github.com/Open-Network-Insight/apache-spot/wiki">Wiki</a>
</li>
<li id="menu-item-13" class="menu-item menu-item-type-post_type menu-item-object-page current_page_parent menu-item-13">
<a href="../../blog/">Blog</a>
@@ -192,11 +174,11 @@
<h1 class="page-title">Category: Security Analytics</h1>
- <article id="post-62" class="cf post-62 post type-post status-publish format-standard hentry category-security-analytics tag-github tag-open-network-insight tag-open-source" role="article">
+ <article id="post-62" class="cf post-62 post type-post status-publish format-standard hentry category-security-analytics tag-github tag-apache-spot tag-open-source" role="article">
<header class="entry-header article-header">
- <h3 class="h2 entry-title"><a href="../../open-network-insight-3-most-asked-questions/" rel="bookmark" title="Apache Spot: Three Most-Asked Questions">Apache Spot: Three Most-Asked Questions</a></h3>
+ <h3 class="h2 entry-title"><a href="../../apache-spot-3-most-asked-questions/" rel="bookmark" title="Apache Spot: Three Most-Asked Questions">Apache Spot (Incubating): Three Most-Asked Questions</a></h3>
<p class="byline entry-meta vcard">
<time class="updated entry-time" datetime="2016-03-29" itemprop="datePublished">
March 29, 2016
@@ -209,7 +191,7 @@
<section class="entry-content cf">
<p>
- While this is not the first blog post about Apache Spot, it is the first one by a creator of the solution. As a security data scientist in Intel’s Data Center Group, I joined a small team to start thinking about solving really hard problems in cloud analytics. The team grew, and out of that… <a class="excerpt-read-more" href="../../open-network-insight-3-most-asked-questions/" title="Read Apache Spot: Three Most-Asked Questions">Read more & [...]
+ While this is not the first blog post about Apache Spot, it is the first one by a creator of the solution. As a security data scientist in Intel’s Data Center Group, I joined a small team to start thinking about solving really hard problems in cloud analytics. The team grew, and out of that… <a class="excerpt-read-more" href="../../apache-spot-3-most-asked-questions/" title="Read Apache Spot: Three Most-Asked Questions">Read more »</a>
</p>
</section>
@@ -228,13 +210,16 @@
<h4 class="widgettitle">Recent Posts</h4>
<ul>
<li>
- <a href="../../open-network-insight-oni-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/">Apache Spot and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a>
+ <a href="../../jupyter-notebooks-for-data-analysis/">Jupyter Notebooks for Data Analysis</a>
+ </li>
+ <li>
+ <a href="../../apache-spot-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/">Apache Spot (Incubating) and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a>
</li>
<li>
- <a href="../../how-open-network-insight-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/">How Apache Spot Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a>
+ <a href="../../how-apache-spot-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/">How Apache Spot (Incubating) Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a>
</li>
<li>
- <a href="../../open-network-insight-3-most-asked-questions/">Apache Spot: Three Most-Asked Questions</a>
+ <a href="../../apache-spot-3-most-asked-questions/">Apache Spot (Incubating): 3 Most-Asked Questions</a>
</li>
</ul>
</div>
@@ -242,6 +227,9 @@
<h4 class="widgettitle">Archives</h4>
<ul>
<li>
+ <a href='../../2016/09/'>September 2016</a>
+ </li>
+ <li>
<a href='../../2016/08/'>August 2016</a>
</li>
<li>
diff --git a/category/threat-analysis-tools/feed/index.html b/category/threat-analysis-tools/feed/index.html
new file mode 100644
index 0000000..0e0f486
--- /dev/null
+++ b/category/threat-analysis-tools/feed/index.html
@@ -0,0 +1,62 @@
+<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
+ xmlns:content="http://purl.org/rss/1.0/modules/content/"
+ xmlns:wfw="http://wellformedweb.org/CommentAPI/"
+ xmlns:dc="http://purl.org/dc/elements/1.1/"
+ xmlns:atom="http://www.w3.org/2005/Atom"
+ xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
+ xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
+ >
+
+<channel>
+ <title>Threat Analysis Tools – Apache Spot</title>
+ <atom:link href="http://spot.incubator.apache.org/category/threat-analysis-tools/feed/" rel="self" type="application/rss+xml" />
+ <link>http://spot.incubator.apache.org</link>
+ <description></description>
+ <lastBuildDate>Fri, 07 Oct 2016 00:20:13 +0000</lastBuildDate>
+ <language>en-US</language>
+ <sy:updatePeriod>hourly</sy:updatePeriod>
+ <sy:updateFrequency>1</sy:updateFrequency>
+
+ <item>
+ <title>Jupyter Notebooks for Data Analysis</title>
+ <link>http://spot.incubator.apache.org/jupyter-notebooks-for-data-analysis/</link>
+ <comments>http://spot.incubator.apache.org/jupyter-notebooks-for-data-analysis/#respond</comments>
+ <pubDate>Thu, 22 Sep 2016 21:28:12 +0000</pubDate>
+ <dc:creator><![CDATA[oni-admin]]></dc:creator>
+ <category><![CDATA[Data Science]]></category>
+ <category><![CDATA[Ipython Notebooks]]></category>
+ <category><![CDATA[Threat Analysis Tools]]></category>
+
+ <guid isPermaLink="false">http://spot.incubator.apache.org/?p=136</guid>
+ <description><![CDATA[Why Does Apache Spot Include iPython notebooks? The project team wants Apache Spot to be a versatile tool that can be used by anyone. This means that data scientists and developers need to be able to query and handle the source data to find all the information they need for their decision making. The iPython... <a class="excerpt-read-more" href="http://spot.incubator.apache.org/jupyter-notebooks-for-data-analysis/" title="Read Jupyter Notebooks for Data Analysis" [...]
+ <content:encoded><![CDATA[<p><strong>Why Does Apache Spot Include iPython notebooks? </strong></p>
+<p>The project team wants Apache Spot to be a versatile tool that can be used by anyone. This means that data scientists and developers need to be able to query and handle the source data to find all the information they need for their decision making. The iPython Notebook is an appropriate platform for easy data exploration. One of its biggest advantages is that it provides parallel and distributed computing to enable code execution and debugging in an interactive environment – thus the [...]
+<p>The iPython notebook is a web based interactive computational environment that provides access to the Python shell. While iPython notebooks were originally designed to work with the Python language, they support a number of other programming languages, including Ruby, Scala, Julia, R, Go, C, C++, Java and Perl. There are also multiple additional packages that can be used to get the most out of this highly-customizable tool.</p>
+<p>Starting on version 4.0, most notebook functionalities are now part of the Project Jupyter, while iPython remains as the kernel to work with Python code in the notebooks.</p>
+<img src="http://spot.incubator.apache.org/wp-content/uploads/2016/10/iPython-1.png" alt="ipython" width="361" height="107" class="aligncenter size-full wp-image-140" srcset="http://spot.incubator.apache.org/wp-content/uploads/2016/10/iPython-1.png 361w, http://spot.incubator.apache.org/wp-content/uploads/2016/10/iPython-1-300x89.png 300w" sizes="(max-width: 361px) 100vw, 361px" />
+<p><strong>IPython with Apache Spot for Network Threat Detection</strong></p>
+<p><em>NOTE: This is not intended to be a step-by-step tutorial on how to code a threat analysis in Apache Spot, but more like an introduction on how to approach the suspicions of a security breach.</em></p>
+<p>Although machine learning (ML) will do most of the work detecting anomalies in the traffic, Apache Spot also includes two notebook templates that can get you started on this. The <em>Threat_Investigation_master.ipynb</em> is designed to query the raw data table to find all connections in a day that are related to any threat you select – even connections that were not necessarily flagged as suspicious by ML on a first run. This gives us the chance to get a new data subset and here is w [...]
+<p>If you suspect of a specific type of attack in your network, you can get the whole story by answering the Five ‘W’s.</p>
+<p><strong><em>What? </em></strong></p>
+<p>Maybe there’s been an increase in the logs collected by the system, which indicates abnormal amounts of communication in your network. Or, the amount of POST requests in your network have risen overnight. This is the mystery that needs to be solved by researching through the anomalies previously detected by ML.</p>
+<p><strong><em>Who?</em></strong></p>
+<p>Assuming you have a network context, you can identify the name of the infected machine inside the network, as well as the name of the IP or DNS on the other side of the connection (if it is a known host). If you don’t have a network context or are using DHCP, this can be a little tricky to detect using only Netflow logs. But, that’s where DNS and Proxy logs, come in handy. Including a network context file with Apache Spot is really simple and can go a long way when identifying a threat.</p>
+<p><strong><em>When?</em></strong></p>
+<p>To have a broader visibility on the attack, you can customize the queries on the Threat investigation notebook to review the data through a wider time lapse – instead of just checking through the current day. With this, you could find an increase of a certain type of requests to one (or many) URIs and predict its future behavior.</p>
+<p><strong><em>Where?</em></strong></p>
+<p>When working only with DNS, having a destination URL might not say much about where your information is going to, but Apache Spot allows you to connect with a geolocation database to identify the location of the suspected attackers IP. Taking advantage of this option, you can visually locate the other end of the connection on a map. You might find that it’s pointing to a country banned by your company, indicating a leak.</p>
+<p><strong><em>Why?</em></strong></p>
+<p>This answer to “why” will depend highly on the result of the analysis. For instance, an excessive amount of POST requests from one machine inside the network to an unidentified URI can indicate a data mining attack. Tracing back to patient zero, you can find that this could have originated with a phishing email, malicious software installed by an employee or a one-time visitor’s infected machine that connected to your network.</p>
+<p><strong>How to Get Answers to the Five Ws Questions</strong></p>
+<p>All of the previous questions can be answered by looking at the raw data collected. Although performing elaborated queries directly to your database can seem tempting, this type of analysis with Hive, or even Impala, can be very time consuming. A better approach would be to use Pandas to read and transform your dataset into a relational structured dataframe. This lets you work with as if it were an offline structured relational database.</p>
+<p>Once you have your desired results and data subsets, you can use MatplotLib to easily graph your findings. (We cover this subject in more depth in another post.) Another advantage of the notebook is that you can download it as HTML or a PDF file to store locally and use it in a presentation – or just keep it for future reference.</p>
+<p><strong>Wrap Up</strong></p>
+<p>This post was meant to be just a brief introduction of how you can use iPython notebooks in Apache Spot to perform further data analysis and include it our executive report (in addition to the already included Story board). Although this is not the only way you can do this, it is a very interactive and fun way to do it. You’ll also see that the overall processing time is very short – thanks to the iPython notebook task parallelism ability.</p>
+<p>We want to hear from YOU! Have you used iPython notebooks before? How do you feel about having this tool in Apache Spot? If you’re interested in further data analysis through interactive charts, a new post is coming soon on D3 and jQuery data visualization. Also, check back soon to read more on this and other Cybersecurity subjects.</p>
+]]></content:encoded>
+ <wfw:commentRss>http://spot.incubator.apache.org/jupyter-notebooks-for-data-analysis/feed/</wfw:commentRss>
+ <slash:comments>0</slash:comments>
+ </item>
+ </channel>
+</rss>
diff --git a/2016/03/index.html b/category/threat-analysis-tools/index.html
similarity index 68%
copy from 2016/03/index.html
copy to category/threat-analysis-tools/index.html
index 4971dc0..0d13dd8 100644
--- a/2016/03/index.html
+++ b/category/threat-analysis-tools/index.html
@@ -12,7 +12,7 @@
<meta http-equiv="X-UA-Compatible" content="IE=edge">
- <title>Apache Spot 2016 March</title>
+ <title>Apache Spot Threat Analysis Tools</title>
<meta name="HandheldFriendly" content="True">
<meta name="MobileOptimized" content="320">
@@ -27,18 +27,15 @@
<meta name="msapplication-TileImage" content="../../wp-content/themes/oni/library/images/win8-tile-icon.png">
<meta name="theme-color" content="#121212">
- <link rel="pingback" href="../../xmlrpc.php">
+ <link rel="pingback" href="http://spot.incubator.apache.org/xmlrpc.php">
- <link rel='dns-prefetch' href='//fonts.googleapis.com' />
- <link rel='dns-prefetch' href='//s.w.org' />
<link rel="alternate" type="application/rss+xml" title="Apache Spot » Feed" href="../../feed/" />
<link rel="alternate" type="application/rss+xml" title="Apache Spot » Comments Feed" href="../../comments/feed/" />
+ <link rel="alternate" type="application/rss+xml" title="Apache Spot » Threat Analysis Tools Category Feed" href="../../category/threat-analysis-tools/feed/" />
<script type="text/javascript">
window._wpemojiSettings = {
- "baseUrl" : "https:\/\/s.w.org\/images\/core\/emoji\/2\/72x72\/",
+ "baseUrl" : "https:\/\/s.w.org\/images\/core\/emoji\/72x72\/",
"ext" : ".png",
- "svgUrl" : "https:\/\/s.w.org\/images\/core\/emoji\/2\/svg\/",
- "svgExt" : ".svg",
"source" : {
"concatemoji" : "http:\/\/spot.incubator.apache.org\/wp-includes\/js\/wp-emoji-release.min.js"
}
@@ -47,30 +44,24 @@
var c,
d,
e,
- f,
- g,
- h = b.createElement("canvas"),
- i = h.getContext && h.getContext("2d"),
- j = String.fromCharCode;
- if (!i || !i.fillText)
+ f = b.createElement("canvas"),
+ g = f.getContext && f.getContext("2d"),
+ h = String.fromCharCode;
+ if (!g || !g.fillText)
return !1;
- switch(i.textBaseline="top",i.font="600 32px Arial",a) {
+ switch(g.textBaseline="top",g.font="600 32px Arial",a) {
case"flag":
- return i.fillText(j(55356, 56806, 55356, 56826), 0, 0), !(h.toDataURL().length < 3e3) && (i.clearRect(0, 0, h.width, h.height), i.fillText(j(55356, 57331, 65039, 8205, 55356, 57096), 0, 0),
- c = h.toDataURL(), i.clearRect(0, 0, h.width, h.height), i.fillText(j(55356, 57331, 55356, 57096), 0, 0),
- d = h.toDataURL(), c !== d);
+ return g.fillText(h(55356, 56806, 55356, 56826), 0, 0), f.toDataURL().length > 3e3;
case"diversity":
- return i.fillText(j(55356, 57221), 0, 0),
- e = i.getImageData(16, 16, 1, 1).data,
- f = e[0] + "," + e[1] + "," + e[2] + "," + e[3], i.fillText(j(55356, 57221, 55356, 57343), 0, 0),
- e = i.getImageData(16, 16, 1, 1).data,
- g = e[0] + "," + e[1] + "," + e[2] + "," + e[3], f !== g;
+ return g.fillText(h(55356, 57221), 0, 0),
+ c = g.getImageData(16, 16, 1, 1).data,
+ d = c[0] + "," + c[1] + "," + c[2] + "," + c[3], g.fillText(h(55356, 57221, 55356, 57343), 0, 0),
+ c = g.getImageData(16, 16, 1, 1).data,
+ e = c[0] + "," + c[1] + "," + c[2] + "," + c[3], d !== e;
case"simple":
- return i.fillText(j(55357, 56835), 0, 0), 0 !== i.getImageData(16,16,1,1).data[0];
+ return g.fillText(h(55357, 56835), 0, 0), 0 !== g.getImageData(16,16,1,1).data[0];
case"unicode8":
- return i.fillText(j(55356, 57135), 0, 0), 0 !== i.getImageData(16,16,1,1).data[0];
- case"unicode9":
- return i.fillText(j(55358, 56631), 0, 0), 0 !== i.getImageData(16,16,1,1).data[0]
+ return g.fillText(h(55356, 57135), 0, 0), 0 !== g.getImageData(16,16,1,1).data[0]
}
return !1
}
@@ -84,7 +75,7 @@
g,
h,
i;
- for ( i = Array("simple", "flag", "unicode8", "diversity", "unicode9"), c.supports = {
+ for ( i = Array("simple", "flag", "unicode8", "diversity"), c.supports = {
everything : !0,
everythingExceptFlag : !0
},
@@ -116,7 +107,7 @@
<link rel='stylesheet' id='googleFonts-css' href='http://fonts.googleapis.com/css?family=Lato%3A400%2C700%2C400italic%2C700italic' type='text/css' media='all' />
<link rel='stylesheet' id='bones-stylesheet-css' href='../../wp-content/themes/oni/library/css/style.css' type='text/css' media='all' />
<!--[if lt IE 9]>
- <link rel='stylesheet' id='bones-ie-only-css' href='../../wp-content/themes/oni/library/css/ie.css' type='text/css' media='all' />
+ <link rel='stylesheet' id='bones-ie-only-css' href='http://spot.incubator.apache.org/wp-content/themes/oni/library/css/ie.css' type='text/css' media='all' />
<![endif]-->
<link rel='stylesheet' id='mm-css-css' href='../../wp-content/themes/oni/library/css/meanmenu.css' type='text/css' media='all' />
<script type='text/javascript' src='../../wp-content/themes/oni/library/js/libs/modernizr.custom.min.js'></script>
@@ -125,27 +116,9 @@
<script type='text/javascript' src='../../wp-content/themes/oni/library/js/jquery.meanmenu.js'></script>
<link rel='https://api.w.org/' href='../../wp-json/' />
- <script>
- (function(i, s, o, g, r, a, m) {
- i['GoogleAnalyticsObject'] = r;
- i[r] = i[r] ||
- function() {
- (i[r].q = i[r].q || []).push(arguments)
- }, i[r].l = 1 * new Date();
- a = s.createElement(o),
- m = s.getElementsByTagName(o)[0];
- a.async = 1;
- a.src = g;
- m.parentNode.insertBefore(a, m)
- })(window, document, 'script', '//www.google-analytics.com/analytics.js', 'ga');
-
- ga('create', 'UA-75955621-1', 'auto');
- ga('send', 'pageview');
-
- </script>
</head>
- <body class="archive date" itemscope itemtype="http://schema.org/WebPage">
+ <body class="archive category category-threat-analysis-tools category-9" itemscope itemtype="http://schema.org/WebPage">
<div id="container">
@@ -159,16 +132,16 @@
<nav role="navigation" itemscope itemtype="http://schema.org/SiteNavigationElement">
<ul id="menu-main-menu" class="nav top-nav cf">
- <li id="menu-item-129" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-129">
+ <li id="menu-item-4" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-4">
<a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight">Get Started</a>
</li>
<li id="menu-item-5" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-5">
<a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight#if-you-want-all-of-the-oni-code-at-once-just-clone-it">Download</a>
</li>
- <li id="menu-item-130" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-130">
+ <li id="menu-item-6" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-6">
<a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight#contributing-to-oni">Contribute</a>
</li>
- <li id="menu-item-106" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-106">
+ <li id="menu-item-7" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-7">
<a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight/wiki">Wiki</a>
</li>
<li id="menu-item-13" class="menu-item menu-item-type-post_type menu-item-object-page current_page_parent menu-item-13">
@@ -189,16 +162,16 @@
<main id="main" class="m-all t-2of3 d-5of7 cf" role="main" itemscope itemprop="mainContentOfPage" itemtype="http://schema.org/Blog">
- <h1 class="page-title">Month: March 2016</h1>
+ <h1 class="page-title">Category: Threat Analysis Tools</h1>
- <article id="post-62" class="cf post-62 post type-post status-publish format-standard hentry category-security-analytics tag-github tag-open-network-insight tag-open-source" role="article">
+ <article id="post-136" class="cf post-136 post type-post status-publish format-standard hentry category-data-science category-ipython-notebooks category-threat-analysis-tools" role="article">
<header class="entry-header article-header">
- <h3 class="h2 entry-title"><a href="../../open-network-insight-3-most-asked-questions/" rel="bookmark" title="Apache Spot: Three Most-Asked Questions">Apache Spot: Three Most-Asked Questions</a></h3>
+ <h3 class="h2 entry-title"><a href="../../jupyter-notebooks-for-data-analysis/" rel="bookmark" title="Jupyter Notebooks for Data Analysis">Jupyter Notebooks for Data Analysis</a></h3>
<p class="byline entry-meta vcard">
- <time class="updated entry-time" datetime="2016-03-29" itemprop="datePublished">
- March 29, 2016
+ <time class="updated entry-time" datetime="2016-09-22" itemprop="datePublished">
+ September 22, 2016
</time>
</span>
</p>
@@ -208,7 +181,7 @@
<section class="entry-content cf">
<p>
- While this is not the first blog post about Apache Spot, it is the first one by a creator of the solution. As a security data scientist in Intel’s Data Center Group, I joined a small team to start thinking about solving really hard problems in cloud analytics. The team grew, and out of that… <a class="excerpt-read-more" href="../../open-network-insight-3-most-asked-questions/" title="Read Apache Spot: Three Most-Asked Questions">Read more & [...]
+ Why Does Apache Spot Include iPython notebooks? The project team wants Apache Spot to be a versatile tool that can be used by anyone. This means that data scientists and developers need to be able to query and handle the source data to find all the information they need for their decision making. The iPython… <a class="excerpt-read-more" href="../../jupyter-notebooks-for-data-analysis/" title="Read Jupyter Notebooks for Data Analysis">Read more & [...]
</p>
</section>
@@ -227,13 +200,16 @@
<h4 class="widgettitle">Recent Posts</h4>
<ul>
<li>
- <a href="../../open-network-insight-oni-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/">Apache Spot and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a>
+ <a href="../../jupyter-notebooks-for-data-analysis/">Jupyter Notebooks for Data Analysis</a>
+ </li>
+ <li>
+ <a href="../../apache-spot-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/">Apache Spot (Incubating) and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a>
</li>
<li>
- <a href="../../how-open-network-insight-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/">How Apache Spot Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a>
+ <a href="../../how-apache-spot-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/">How Apache Spot (Incubating) Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a>
</li>
<li>
- <a href="../../open-network-insight-3-most-asked-questions/">Apache Spot: Three Most-Asked Questions</a>
+ <a href="../../apache-spot-3-most-asked-questions/">Apache Spot (Incubating): 3 Most-Asked Questions</a>
</li>
</ul>
</div>
@@ -241,6 +217,9 @@
<h4 class="widgettitle">Archives</h4>
<ul>
<li>
+ <a href='../../2016/09/'>September 2016</a>
+ </li>
+ <li>
<a href='../../2016/08/'>August 2016</a>
</li>
<li>
diff --git a/category/uncategorized/feed/index.html b/category/uncategorized/feed/index.html
index 123b112..4d92163 100644
--- a/category/uncategorized/feed/index.html
+++ b/category/uncategorized/feed/index.html
@@ -9,7 +9,7 @@
<channel>
<title>Uncategorized – Apache Spot</title>
- <atom:link href="http://spot.incubator.apache.org//category/uncategorized/feed/" rel="self" type="application/rss+xml" />
+ <atom:link href="http://spot.incubator.apache.org/category/uncategorized/feed/" rel="self" type="application/rss+xml" />
<link>http://spot.incubator.apache.org/</link>
<description></description>
<lastBuildDate>Tue, 27 Sep 2016 18:37:15 +0000</lastBuildDate>
@@ -17,19 +17,59 @@
<sy:updatePeriod>hourly</sy:updatePeriod>
<sy:updateFrequency>1</sy:updateFrequency>
+ <item>
+ <title>Jupyter Notebooks for Data Analysis</title>
+ <link>http://spot.incubator.apache.org/jupyter-notebooks-for-data-analysis/</link>
+ <comments>http://spot.incubator.apache.org/jupyter-notebooks-for-data-analysis/#respond</comments>
+ <pubDate>Thu, 22 Sep 2016 21:28:12 +0000</pubDate>
+ <dc:creator><![CDATA[oni-admin]]></dc:creator>
+ <category><![CDATA[Uncategorized]]></category>
+
+ <guid isPermaLink="false">http://spot.incubator.apache.org/?p=136</guid>
+ <description><![CDATA[Why Does Apache Spot Include iPython notebooks? The project team wants Apache Spot needs to be a versatile tool that can be used by anyone. This means that data scientists and developers need to be able to query and handle the source data to find all the information they need for their decision making. The... <a class="excerpt-read-more" href="http://spot.incubator.apache.org/jupyter-notebooks-for-data-analysis/" title="Read Jupyter Notebooks for Data Analy [...]
+ <content:encoded><![CDATA[<p><strong>Why Does Apache Spot Include iPython notebooks? </strong></p>
+<p>The project team wants Apache Spot needs to be a versatile tool that can be used by anyone. This means that data scientists and developers need to be able to query and handle the source data to find all the information they need for their decision making. The iPython Notebook is an appropriate platform for easy data exploration. One of its biggest advantages is that it provides parallel and distributed computing to enable code execution and debugging in an interactive environment – th [...]
+<p>The iPython notebook is a web based interactive computational environment that provides access to the Python shell. While iPython notebooks were originally designed to work with the Python language, they support a number of other programming languages, including Ruby, Scala, Julia, R, Go, C, C++, Java and Perl. There are also multiple additional packages that can be used to get the most out of this highly-customizable tool.</p>
+<p>Starting on version 4.0, most notebook functionalities are now part of the Project Jupyter, while iPython remains as the kernel to work with Python code in the notebooks.</p>
+<img src="http://spot.incubator.apache.org/wp-content/uploads/2016/10/iPython-1.png" alt="ipython" width="361" height="107" class="aligncenter size-full wp-image-140" srcset="http://spot.incubator.apache.org/wp-content/uploads/2016/10/iPython-1.png 361w, http://spot.incubator.apache.org/wp-content/uploads/2016/10/iPython-1-300x89.png 300w" sizes="(max-width: 361px) 100vw, 361px" />
+<p><strong>IPython with Apache Spot for Network Threat Detection</strong></p>
+<p><em>NOTE: This is not intended to be a step-by-step tutorial on how to code a threat analysis in Apache Spot, but more like an introduction on how to approach the suspicions of a security breach.</em></p>
+<p>Although machine learning (ML) will do most of the work detecting anomalies in the traffic, Apache Spot also includes two notebook templates that can get you started on this. The <em>Threat_Investigation_master.ipynb</em> is designed to query the raw data table to find all connections in a day that are related to any threat you select – even connections that were not necessarily flagged as suspicious by ML on a first run. This gives us the chance to get a new data subset and here is w [...]
+<p>If you suspect of a specific type of attack in your network, you can get the whole story by answering the Five ‘W’s.</p>
+<p><strong><em>What? </em></strong></p>
+<p>Maybe there’s been an increase in the logs collected by the system, which indicates abnormal amounts of communication in your network. Or, the amount of POST requests in your network have risen overnight. This is the mystery that needs to be solved by researching through the anomalies previously detected by ML.</p>
+<p><strong><em>Who?</em></strong></p>
+<p>Assuming you have a network context, you can identify the name of the infected machine inside the network, as well as the name of the IP or DNS on the other side of the connection (if it is a known host). If you don’t have a network context or are using DHCP, this can be a little tricky to detect using only Netflow logs. But, that’s where DNS and Proxy logs, come in handy. Including a network context file with Apache Spot is really simple and can go a long way when identifying a threat.</p>
+<p><strong><em>When?</em></strong></p>
+<p>To have a broader visibility on the attack, you can customize the queries on the Threat investigation notebook to review the data through a wider time lapse – instead of just checking through the current day. With this, you could find an increase of a certain type of requests to one (or many) URIs and predict its future behavior.</p>
+<p><strong><em>Where?</em></strong></p>
+<p>When working only with DNS, having a destination URL might not say much about where your information is going to, but Apache Spot allows you to connect with a geolocation database to identify the location of the suspected attackers IP. Taking advantage of this option, you can visually locate the other end of the connection on a map. You might find that it’s pointing to a country banned by your company, indicating a leak.</p>
+<p><strong><em>Why?</em></strong></p>
+<p>This answer to “why” will depend highly on the result of the analysis. For instance, an excessive amount of POST requests from one machine inside the network to an unidentified URI can indicate a data mining attack. Tracing back to patient zero, you can find that this could have originated with a phishing email, malicious software installed by an employee or a one-time visitor’s infected machine that connected to your network.</p>
+<p><strong>How to Get Answers to the Five Ws Questions</strong></p>
+<p>All of the previous questions can be answered by looking at the raw data collected. Although performing elaborated queries directly to your database can seem tempting, this type of analysis with Hive, or even Impala, can be very time consuming. A better approach would be to use Pandas to read and transform your dataset into a relational structured dataframe. This lets you work with as if it were an offline structured relational database.</p>
+<p>Once you have your desired results and data subsets, you can use MatplotLib to easily graph your findings. (We cover this subject in more depth in another post.) Another advantage of the notebook is that you can download it as HTML or a PDF file to store locally and use it in a presentation – or just keep it for future reference.</p>
+<p><strong>Wrap Up</strong></p>
+<p>This post was meant to be just a brief introduction of how you can use iPython notebooks in Apache Spot to perform further data analysis and include it our executive report (in addition to the already included Story board). Although this is not the only way you can do this, it is a very interactive and fun way to do it. You’ll also see that the overall processing time is very short – thanks to the iPython notebook task parallelism ability.</p>
+<p>We want to hear from YOU! Have you used iPython notebooks before? How do you feel about having this tool in Apache Spot? If you’re interested in further data analysis through interactive charts, a new post is coming soon on D3 and jQuery data visualization. Also, check back soon to read more on this and other Cybersecurity subjects.</p>
+]]></content:encoded>
+ <wfw:commentRss>http://spot.incubator.apache.org/jupyter-notebooks-for-data-analysis/feed/</wfw:commentRss>
+ <slash:comments>0</slash:comments>
+ </item>
+
<item>
- <title>How Apache Spot Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</title>
- <link>http://spot.incubator.apache.org//how-open-network-insight-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/</link>
- <comments>http://spot.incubator.apache.org//how-open-network-insight-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/#respond</comments>
+ <title>How Apache (Incubating) Spot Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</title>
+ <link>http://spot.incubator.apache.org/how-apache-spot-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/</link>
+ <comments>http://spot.incubator.apache.org/how-apache-spot-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/#respond</comments>
<pubDate>Mon, 08 Aug 2016 07:59:41 +0000</pubDate>
<dc:creator><![CDATA[oni-admin]]></dc:creator>
<category><![CDATA[Uncategorized]]></category>
- <guid isPermaLink="false">http://spot.incubator.apache.org//?p=113</guid>
- <description><![CDATA[About four years ago, the era of the Big Data analytics began. Paired with advanced analytics, massive volumes of data can be culled to not only inform critical decisions, but also to simulate sophisticated “what if” scenarios that allow companies to gain competitive advantages by generating and predicting different scenarios. For example, a financial services... <a class="excerpt-read-more" href="http://spot.incubator.apache.org//how-open-network-insight-helps-c [...]
+ <guid isPermaLink="false">http://spot.incubator.apache.org/?p=113</guid>
+ <description><![CDATA[About four years ago, the era of the Big Data analytics began. Paired with advanced analytics, massive volumes of data can be culled to not only inform critical decisions, but also to simulate sophisticated “what if” scenarios that allow companies to gain competitive advantages by generating and predicting different scenarios. For example, a financial services... <a class="excerpt-read-more" href="http://spot.incubator.apache.org/how-apache-spot-helps-create-well [...]
<content:encoded><![CDATA[<p>About four years ago, the era of the Big Data analytics began. Paired with advanced analytics, massive volumes of data can be culled to not only inform critical decisions, but also to simulate sophisticated “what if” scenarios that allow companies to gain competitive advantages by generating and predicting different scenarios. For example, a financial services company can more accurately determine what other products to offer a customer, and in what order [...]
<p>Apache Spot can help to solve most of these issues. Following is a description of the Apache Spot, which is designed to facilitate Big Data analytics scenarios like the financial services company’s question about the right product to offer customers.</p>
-<a href="http://spot.incubator.apache.org//wp-content/uploads/2016/09/ONI_Architecture-Diagram_1300_v4.png"><img src="http://spot.incubator.apache.org//wp-content/uploads/2016/09/ONI_Architecture-Diagram_1300_v4.png" alt="oni_architecture-diagram_1300_v4" width="1300" height="675" class="alignnone size-full wp-image-114" srcset="http://spot.incubator.apache.org//wp-content/uploads/2016/09/ONI_Architecture-Diagram_1300_v4.png 1300w, http://spot.incubator.apache.org//wp-content/uploads/201 [...]
+<a href="http://spot.incubator.apache.org/wp-content/uploads/2016/09/ONI_Architecture-Diagram_1300_v4.png"><img src="http://spot.incubator.apache.org/wp-content/uploads/2016/09/ONI_Architecture-Diagram_1300_v4.png" alt="oni_architecture-diagram_1300_v4" width="1300" height="675" class="alignnone size-full wp-image-114" srcset="http://spot.incubator.apache.org/wp-content/uploads/2016/09/ONI_Architecture-Diagram_1300_v4.png 1300w, http://spot.incubator.apache.org/wp-content/uploads/2016/09 [...]
<h3><strong>Apache Spot Core Components</strong></h3>
<p>The Apache Spot Core is composed of three main components — data integration (collectors), data store (HDFS here, but can also be a non-SQL database) and machine learning.</p>
<p>In this diagram, the top left shows Apache Spot Data Sources, which include the collection of the information that will be used to create a data lake. The process is simple. Define a pull or push from the source of information then capture this information on Apache Spot’s “collectors.” The collectors are processes that interpret the information that is sent, then write it to the HDFS system in the Apache Spot cluster. The HDFS stores the data lake and ensures that resources can grow [...]
@@ -40,7 +80,7 @@
<h3><strong>Customizable Open Source</strong></h3>
<p>Because Apache Spot is an open-source project, most of the components depicted here can be modified by the end user.</p>
]]></content:encoded>
- <wfw:commentRss>http://spot.incubator.apache.org//how-open-network-insight-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/feed/</wfw:commentRss>
+ <wfw:commentRss>http://spot.incubator.apache.org/how-apache-spot-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/feed/</wfw:commentRss>
<slash:comments>0</slash:comments>
</item>
</channel>
diff --git a/category/uncategorized/index.html b/category/uncategorized/index.html
index ef9156d..96f1ad4 100644
--- a/category/uncategorized/index.html
+++ b/category/uncategorized/index.html
@@ -127,24 +127,6 @@
<script type='text/javascript' src='../../wp-content/themes/oni/library/js/jquery.meanmenu.js'></script>
<link rel='https://api.w.org/' href='../../wp-json/' />
- <script>
- (function(i, s, o, g, r, a, m) {
- i['GoogleAnalyticsObject'] = r;
- i[r] = i[r] ||
- function() {
- (i[r].q = i[r].q || []).push(arguments)
- }, i[r].l = 1 * new Date();
- a = s.createElement(o),
- m = s.getElementsByTagName(o)[0];
- a.async = 1;
- a.src = g;
- m.parentNode.insertBefore(a, m)
- })(window, document, 'script', '//www.google-analytics.com/analytics.js', 'ga');
-
- ga('create', 'UA-75955621-1', 'auto');
- ga('send', 'pageview');
-
- </script>
</head>
<body class="archive category category-uncategorized category-1" itemscope itemtype="http://schema.org/WebPage">
@@ -162,16 +144,16 @@
<nav role="navigation" itemscope itemtype="http://schema.org/SiteNavigationElement">
<ul id="menu-main-menu" class="nav top-nav cf">
<li id="menu-item-129" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-129">
- <a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight">Get Started</a>
+ <a target="_blank" href="https://github.com/Open-Network-Insight/apache-spot">Get Started</a>
</li>
<li id="menu-item-5" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-5">
- <a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight#if-you-want-all-of-the-oni-code-at-once-just-clone-it">Download</a>
+ <a target="_blank" href="https://github.com/Open-Network-Insight/apache-spot#if-you-want-all-of-the-oni-code-at-once-just-clone-it">Download</a>
</li>
<li id="menu-item-130" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-130">
- <a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight#contributing-to-oni">Contribute</a>
+ <a target="_blank" href="https://github.com/Open-Network-Insight/apache-spot#contributing-to-oni">Contribute</a>
</li>
<li id="menu-item-106" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-106">
- <a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight/wiki">Wiki</a>
+ <a target="_blank" href="https://github.com/Open-Network-Insight/apache-spot/wiki">Wiki</a>
</li>
<li id="menu-item-13" class="menu-item menu-item-type-post_type menu-item-object-page current_page_parent menu-item-13">
<a href="../../blog/">Blog</a>
@@ -192,12 +174,13 @@
<main id="main" class="m-all t-2of3 d-5of7 cf" role="main" itemscope itemprop="mainContentOfPage" itemtype="http://schema.org/Blog">
<h1 class="page-title">Category: Uncategorized</h1>
+
<article id="post-113" class="cf post-113 post type-post status-publish format-standard hentry category-uncategorized" role="article">
<header class="entry-header article-header">
- <h3 class="h2 entry-title"><a href="../../how-open-network-insight-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/" rel="bookmark" title="How Apache Spot Helps Create Well-Stocked Data Lakes and Catch Powerful Insights">How Apache Spot Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a></h3>
+ <h3 class="h2 entry-title"><a href="../../how-apache-spot-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/" rel="bookmark" title="How Apache Spot Helps Create Well-Stocked Data Lakes and Catch Powerful Insights">How Apache Spot (Incubating) Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a></h3>
<p class="byline entry-meta vcard">
<time class="updated entry-time" datetime="2016-08-08" itemprop="datePublished">
August 8, 2016
@@ -210,7 +193,7 @@
<section class="entry-content cf">
<p>
- About four years ago, the era of the Big Data analytics began. Paired with advanced analytics, massive volumes of data can be culled to not only inform critical decisions, but also to simulate sophisticated “what if” scenarios that allow companies to gain competitive advantages by generating and predicting different scenarios. For example, a financial services… <a class="excerpt-read-more" href="../../how-open-network-insight-helps-create-well-st [...]
+ About four years ago, the era of the Big Data analytics began. Paired with advanced analytics, massive volumes of data can be culled to not only inform critical decisions, but also to simulate sophisticated “what if” scenarios that allow companies to gain competitive advantages by generating and predicting different scenarios. For example, a financial services… <a class="excerpt-read-more" href="../../how-apache-spot-helps-create-well-stocked-dat [...]
</p>
</section>
@@ -222,20 +205,23 @@
</article>
</main>
-
+
<div id="sidebar1" class="sidebar m-all t-1of3 d-2of7 last-col cf" role="complementary">
<div id="recent-posts-2" class="widget widget_recent_entries">
<h4 class="widgettitle">Recent Posts</h4>
<ul>
<li>
- <a href="../../open-network-insight-oni-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/">Apache Spot and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a>
+ <a href="../../jupyter-notebooks-for-data-analysis/">Jupyter Notebooks for Data Analysis</a>
</li>
<li>
- <a href="../../how-open-network-insight-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/">How Apache Spot Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a>
+ <a href="../../apache-spot-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/">Apache Spot (Incubating) and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a>
</li>
<li>
- <a href="../../open-network-insight-3-most-asked-questions/">Apache Spot: Three Most-Asked Questions</a>
+ <a href="../../how-apache-spot-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/">How Apache Spot (Incubating) Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a>
+ </li>
+ <li>
+ <a href="../../apache-spot-3-most-asked-questions/">Apache Spot (Incubating): 3 Most-Asked Questions</a>
</li>
</ul>
</div>
@@ -243,6 +229,9 @@
<h4 class="widgettitle">Archives</h4>
<ul>
<li>
+ <a href='../../2016/09/'>September 2016</a>
+ </li>
+ <li>
<a href='../../2016/08/'>August 2016</a>
</li>
<li>
diff --git a/feed/index.html b/feed/index.html
index 881aa0e..29f0268 100644
--- a/feed/index.html
+++ b/feed/index.html
@@ -9,7 +9,7 @@
<channel>
<title>Apache Spot</title>
- <atom:link href="http://spot.incubator.apache.org//feed/" rel="self" type="application/rss+xml" />
+ <atom:link href="http://spot.incubator.apache.org/feed/" rel="self" type="application/rss+xml" />
<link>http://spot.incubator.apache.org/</link>
<description></description>
<lastBuildDate>Tue, 27 Sep 2016 18:37:15 +0000</lastBuildDate>
@@ -17,16 +17,56 @@
<sy:updatePeriod>hourly</sy:updatePeriod>
<sy:updateFrequency>1</sy:updateFrequency>
+ <item>
+ <title>Jupyter Notebooks for Data Analysis</title>
+ <link>http://spot.incubator.apache.org/jupyter-notebooks-for-data-analysis/</link>
+ <comments>http://spot.incubator.apache.org/jupyter-notebooks-for-data-analysis/#respond</comments>
+ <pubDate>Thu, 22 Sep 2016 21:28:12 +0000</pubDate>
+ <dc:creator><![CDATA[oni-admin]]></dc:creator>
+ <category><![CDATA[Uncategorized]]></category>
+
+ <guid isPermaLink="false">http://spot.incubator.apache.org/?p=136</guid>
+ <description><![CDATA[Why Does Apache Spot Include iPython notebooks? The project team wants Apache Spot needs to be a versatile tool that can be used by anyone. This means that data scientists and developers need to be able to query and handle the source data to find all the information they need for their decision making. The... <a class="excerpt-read-more" href="http://spot.incubator.apache.org/jupyter-notebooks-for-data-analysis/" title="Read Jupyter Notebooks for Data Analy [...]
+ <content:encoded><![CDATA[<p><strong>Why Does Apache Spot Include iPython notebooks? </strong></p>
+<p>The project team wants Apache Spot needs to be a versatile tool that can be used by anyone. This means that data scientists and developers need to be able to query and handle the source data to find all the information they need for their decision making. The iPython Notebook is an appropriate platform for easy data exploration. One of its biggest advantages is that it provides parallel and distributed computing to enable code execution and debugging in an interactive environment – th [...]
+<p>The iPython notebook is a web based interactive computational environment that provides access to the Python shell. While iPython notebooks were originally designed to work with the Python language, they support a number of other programming languages, including Ruby, Scala, Julia, R, Go, C, C++, Java and Perl. There are also multiple additional packages that can be used to get the most out of this highly-customizable tool.</p>
+<p>Starting on version 4.0, most notebook functionalities are now part of the Project Jupyter, while iPython remains as the kernel to work with Python code in the notebooks.</p>
+<img src="http://spot.incubator.apache.org/wp-content/uploads/2016/10/iPython-1.png" alt="ipython" width="361" height="107" class="aligncenter size-full wp-image-140" srcset="http://spot.incubator.apache.org/wp-content/uploads/2016/10/iPython-1.png 361w, http://spot.incubator.apache.org/wp-content/uploads/2016/10/iPython-1-300x89.png 300w" sizes="(max-width: 361px) 100vw, 361px" />
+<p><strong>IPython with Apache Spot for Network Threat Detection</strong></p>
+<p><em>NOTE: This is not intended to be a step-by-step tutorial on how to code a threat analysis in Apache Spot, but more like an introduction on how to approach the suspicions of a security breach.</em></p>
+<p>Although machine learning (ML) will do most of the work detecting anomalies in the traffic, Apache Spot also includes two notebook templates that can get you started on this. The <em>Threat_Investigation_master.ipynb</em> is designed to query the raw data table to find all connections in a day that are related to any threat you select – even connections that were not necessarily flagged as suspicious by ML on a first run. This gives us the chance to get a new data subset and here is w [...]
+<p>If you suspect of a specific type of attack in your network, you can get the whole story by answering the Five ‘W’s.</p>
+<p><strong><em>What? </em></strong></p>
+<p>Maybe there’s been an increase in the logs collected by the system, which indicates abnormal amounts of communication in your network. Or, the amount of POST requests in your network have risen overnight. This is the mystery that needs to be solved by researching through the anomalies previously detected by ML.</p>
+<p><strong><em>Who?</em></strong></p>
+<p>Assuming you have a network context, you can identify the name of the infected machine inside the network, as well as the name of the IP or DNS on the other side of the connection (if it is a known host). If you don’t have a network context or are using DHCP, this can be a little tricky to detect using only Netflow logs. But, that’s where DNS and Proxy logs, come in handy. Including a network context file with Apache Spot is really simple and can go a long way when identifying a threat.</p>
+<p><strong><em>When?</em></strong></p>
+<p>To have a broader visibility on the attack, you can customize the queries on the Threat investigation notebook to review the data through a wider time lapse – instead of just checking through the current day. With this, you could find an increase of a certain type of requests to one (or many) URIs and predict its future behavior.</p>
+<p><strong><em>Where?</em></strong></p>
+<p>When working only with DNS, having a destination URL might not say much about where your information is going to, but Apache Spot allows you to connect with a geolocation database to identify the location of the suspected attackers IP. Taking advantage of this option, you can visually locate the other end of the connection on a map. You might find that it’s pointing to a country banned by your company, indicating a leak.</p>
+<p><strong><em>Why?</em></strong></p>
+<p>This answer to “why” will depend highly on the result of the analysis. For instance, an excessive amount of POST requests from one machine inside the network to an unidentified URI can indicate a data mining attack. Tracing back to patient zero, you can find that this could have originated with a phishing email, malicious software installed by an employee or a one-time visitor’s infected machine that connected to your network.</p>
+<p><strong>How to Get Answers to the Five Ws Questions</strong></p>
+<p>All of the previous questions can be answered by looking at the raw data collected. Although performing elaborated queries directly to your database can seem tempting, this type of analysis with Hive, or even Impala, can be very time consuming. A better approach would be to use Pandas to read and transform your dataset into a relational structured dataframe. This lets you work with as if it were an offline structured relational database.</p>
+<p>Once you have your desired results and data subsets, you can use MatplotLib to easily graph your findings. (We cover this subject in more depth in another post.) Another advantage of the notebook is that you can download it as HTML or a PDF file to store locally and use it in a presentation – or just keep it for future reference.</p>
+<p><strong>Wrap Up</strong></p>
+<p>This post was meant to be just a brief introduction of how you can use iPython notebooks in Apache Spot to perform further data analysis and include it our executive report (in addition to the already included Story board). Although this is not the only way you can do this, it is a very interactive and fun way to do it. You’ll also see that the overall processing time is very short – thanks to the iPython notebook task parallelism ability.</p>
+<p>We want to hear from YOU! Have you used iPython notebooks before? How do you feel about having this tool in Apache Spot? If you’re interested in further data analysis through interactive charts, a new post is coming soon on D3 and jQuery data visualization. Also, check back soon to read more on this and other Cybersecurity subjects.</p>
+]]></content:encoded>
+ <wfw:commentRss>http://spot.incubator.apache.org/jupyter-notebooks-for-data-analysis/feed/</wfw:commentRss>
+ <slash:comments>0</slash:comments>
+ </item>
+
<item>
- <title>Apache Spot and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</title>
- <link>http://spot.incubator.apache.org//open-network-insight-oni-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/</link>
- <comments>http://spot.incubator.apache.org//open-network-insight-oni-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/#respond</comments>
+ <title>Apache Spot (Incubating) and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</title>
+ <link>http://spot.incubator.apache.org/apache-spot-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/</link>
+ <comments>http://spot.incubator.apache.org/apache-spot-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/#respond</comments>
<pubDate>Mon, 08 Aug 2016 08:00:32 +0000</pubDate>
<dc:creator><![CDATA[oni-admin]]></dc:creator>
<category><![CDATA[Cybersecurity]]></category>
- <guid isPermaLink="false">http://spot.incubator.apache.org//?p=117</guid>
- <description><![CDATA[The “first” documented cybersecurity case was the worm replication, which was initiated by Robert T. Morris on November 2, 1988. Wow! Here we are in 2016, 28 years later, with viruses and worms giving way to Trojan horses and polymorphic code. Nowadays, we are also fighting against DDoS, phishing, spear phishing attacks, command and controls... <a class="excerpt-read-more" href="http://spot.incubator.apache.org//open-network-insight-oni-and-cybersecurity-using-ne [...]
+ <guid isPermaLink="false">http://spot.incubator.apache.org/?p=117</guid>
+ <description><![CDATA[The “first” documented cybersecurity case was the worm replication, which was initiated by Robert T. Morris on November 2, 1988. Wow! Here we are in 2016, 28 years later, with viruses and worms giving way to Trojan horses and polymorphic code. Nowadays, we are also fighting against DDoS, phishing, spear phishing attacks, command and controls... <a class="excerpt-read-more" href="http://spot.incubator.apache.org/apache-spot-and-cybersecurity-using-netflows-to-dete [...]
<content:encoded><![CDATA[<p>The “first” documented cybersecurity case was the worm replication, which was initiated by Robert T. Morris on November 2, 1988. Wow! Here we are in 2016, 28 years later, with viruses and worms giving way to Trojan horses and polymorphic code. Nowadays, we are also fighting against DDoS, phishing, spear phishing attacks, command and controls along with APTs such as Aurora, Zeus, Red October and Stuxnet. What happened with our security controls on each att [...]
<p>Despite heroic efforts, internal and external security controls, no matter if they are preventive, detective or corrective, can be bypassed by different situations or misconfigurations. Capabilities to detect bugs or vulnerabilities in code, protocol, etc. are still limited. When we consider how to close these gaps, we have two options:</p>
<ol>
@@ -41,25 +81,25 @@
<p>Apache Spot offers a solution. It was designed to gather, store and analyze Big Data. In fact, Apache Spot is an ideal solution for this cybersecurity challenge. Apache Spot can integrate many different data sources in a data lake then add operational context to the data by linking configuration, inventory, service databases and other data stores. This helps you to prioritize the actions to take under different attack, malware, APT and hacking scenarios. With Apache Spot, attacks that [...]
<p>Not only can Apache Spot collect, store and analyze billions of NetFlow packets, but it can also be adapted to meet the unique requirements of your organization. How? Apache Spot is an open-source project.</p>
<p><strong>But Wait, There’s More</strong></p>
-<p>Check out “<a href="http://spot.incubator.apache.org//how-open-network-insight-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/"><u>How Apache Spot Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</u></a>” to learn more about the underlying Apache Spot architecture.</p>
+<p>Check out “<a href="http://spot.incubator.apache.org/how-apache-spot-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/"><u>How Apache Spot Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</u></a>” to learn more about the underlying Apache Spot architecture.</p>
<p>This is the first of a series of blogs that we will be writing about cybersecurity, so check back to read more.</p>
]]></content:encoded>
- <wfw:commentRss>http://spot.incubator.apache.org//open-network-insight-oni-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/feed/</wfw:commentRss>
+ <wfw:commentRss>http://spot.incubator.apache.org/apache-spot-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/feed/</wfw:commentRss>
<slash:comments>0</slash:comments>
</item>
<item>
<title>How Apache Spot Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</title>
- <link>http://spot.incubator.apache.org//how-open-network-insight-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/</link>
- <comments>http://spot.incubator.apache.org//how-open-network-insight-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/#respond</comments>
+ <link>http://spot.incubator.apache.org/how-apache-spot-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/</link>
+ <comments>http://spot.incubator.apache.org/how-apache-spot-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/#respond</comments>
<pubDate>Mon, 08 Aug 2016 07:59:41 +0000</pubDate>
<dc:creator><![CDATA[oni-admin]]></dc:creator>
<category><![CDATA[Uncategorized]]></category>
- <guid isPermaLink="false">http://spot.incubator.apache.org//?p=113</guid>
- <description><![CDATA[About four years ago, the era of the Big Data analytics began. Paired with advanced analytics, massive volumes of data can be culled to not only inform critical decisions, but also to simulate sophisticated “what if” scenarios that allow companies to gain competitive advantages by generating and predicting different scenarios. For example, a financial services... <a class="excerpt-read-more" href="http://spot.incubator.apache.org//how-open-network-insight-helps-c [...]
+ <guid isPermaLink="false">http://spot.incubator.apache.org/?p=113</guid>
+ <description><![CDATA[About four years ago, the era of the Big Data analytics began. Paired with advanced analytics, massive volumes of data can be culled to not only inform critical decisions, but also to simulate sophisticated “what if” scenarios that allow companies to gain competitive advantages by generating and predicting different scenarios. For example, a financial services... <a class="excerpt-read-more" href="http://spot.incubator.apache.org/how-apache-spot-helps-create-well [...]
<content:encoded><![CDATA[<p>About four years ago, the era of the Big Data analytics began. Paired with advanced analytics, massive volumes of data can be culled to not only inform critical decisions, but also to simulate sophisticated “what if” scenarios that allow companies to gain competitive advantages by generating and predicting different scenarios. For example, a financial services company can more accurately determine what other products to offer a customer, and in what order [...]
<p>Apache Spot can help to solve most of these issues. Following is a description of the Apache Spot, which is designed to facilitate Big Data analytics scenarios like the financial services company’s question about the right product to offer customers.</p>
-<a href="http://spot.incubator.apache.org//wp-content/uploads/2016/09/ONI_Architecture-Diagram_1300_v4.png"><img src="http://spot.incubator.apache.org//wp-content/uploads/2016/09/ONI_Architecture-Diagram_1300_v4.png" alt="oni_architecture-diagram_1300_v4" width="1300" height="675" class="alignnone size-full wp-image-114" srcset="http://spot.incubator.apache.org//wp-content/uploads/2016/09/ONI_Architecture-Diagram_1300_v4.png 1300w, http://spot.incubator.apache.org//wp-content/uploads/201 [...]
+<a href="http://spot.incubator.apache.org/wp-content/uploads/2016/09/ONI_Architecture-Diagram_1300_v4.png"><img src="http://spot.incubator.apache.org/wp-content/uploads/2016/09/ONI_Architecture-Diagram_1300_v4.png" alt="oni_architecture-diagram_1300_v4" width="1300" height="675" class="alignnone size-full wp-image-114" srcset="http://spot.incubator.apache.org/wp-content/uploads/2016/09/ONI_Architecture-Diagram_1300_v4.png 1300w, http://spot.incubator.apache.org/wp-content/uploads/2016/09 [...]
<h3><strong>Apache Spot Core Components</strong></h3>
<p>The Apache Spot Core is composed of three main components — data integration (collectors), data store (HDFS here, but can also be a non-SQL database) and machine learning.</p>
<p>In this diagram, the top left shows Apache Spot Data Sources, which include the collection of the information that will be used to create a data lake. The process is simple. Define a pull or push from the source of information then capture this information on Apache Spot’s “collectors.” The collectors are processes that interpret the information that is sent, then write it to the HDFS system in the Apache Spot cluster. The HDFS stores the data lake and ensures that resources can grow [...]
@@ -70,13 +110,13 @@
<h3><strong>Customizable Open Source</strong></h3>
<p>Because Apache Spot is an open-source project, most of the components depicted here can be modified by the end user.</p>
]]></content:encoded>
- <wfw:commentRss>http://spot.incubator.apache.org//how-open-network-insight-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/feed/</wfw:commentRss>
+ <wfw:commentRss>http://spot.incubator.apache.org/how-apache-spot-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/feed/</wfw:commentRss>
<slash:comments>0</slash:comments>
</item>
<item>
- <title>Apache Spot: Three Most-Asked Questions</title>
- <link>http://spot.incubator.apache.org//open-network-insight-3-most-asked-questions/</link>
- <comments>http://spot.incubator.apache.org//open-network-insight-3-most-asked-questions/#respond</comments>
+ <title>Apache Spot (Incubating): Three Most-Asked Questions</title>
+ <link>http://spot.incubator.apache.org/apache-spot-3-most-asked-questions/</link>
+ <comments>http://spot.incubator.apache.org/apache-spot-3-most-asked-questions/#respond</comments>
<pubDate>Tue, 29 Mar 2016 05:48:19 +0000</pubDate>
<dc:creator><![CDATA[oni-admin]]></dc:creator>
<category><![CDATA[Security Analytics]]></category>
@@ -84,23 +124,23 @@
<category><![CDATA[open network insight]]></category>
<category><![CDATA[open source]]></category>
- <guid isPermaLink="false">http://spot.incubator.apache.org//?p=62</guid>
- <description><![CDATA[While this is not the first blog post about Apache Spot, it is the first one by a creator of the solution. As a security data scientist in Intel’s Data Center Group, I joined a small team to start thinking about solving really hard problems in cloud analytics. The team grew, and out of that... <a class="excerpt-read-more" href="http://spot.incubator.apache.org//open-network-insight-3-most-asked-questions/" title="Read Apache Spot: Three Most-Asked Questions [...]
+ <guid isPermaLink="false">http://spot.incubator.apache.org/?p=62</guid>
+ <description><![CDATA[While this is not the first blog post about Apache Spot, it is the first one by a creator of the solution. As a security data scientist in Intel’s Data Center Group, I joined a small team to start thinking about solving really hard problems in cloud analytics. The team grew, and out of that... <a class="excerpt-read-more" href="http://spot.incubator.apache.org/apache-spot-3-most-asked-questions/" title="Read Apache Spot: Three Most-Asked Questions">Read mor [...]
<content:encoded><![CDATA[<p>While this is not the first blog post about Apache Spot, it is the first one by a creator of the solution. As a security data scientist in Intel’s Data Center Group, I joined a small team to start thinking about solving really hard problems in cloud analytics. The team grew, and out of that effort, came Apache Spot. Since we started talking about the project, these are the three questions I am asked the most.</p>
<p><strong>What Is Apache Spot?</strong><br />
Apache Spot is an open source, flow and packet analytics solution built on Hadoop. It combines big data processing, at-scale machine learning, and unique security analytics to put potential threats in front of defenders. While I am a data scientist today, I was a security investigator just a few years ago. I wanted to develop a solution that would put new tools and technology in play for defenders, but without requiring them to walk away from security and get a math degree.</p>
<p>We wanted to start with the hard problems, so we looked at the emerging need to analyze data that was produced at a scale outside what a lot of security solutions could handle. The data is being created today, and lack of visibility into that data gives attackers a profound advantage. Also, in this new era of security, many defenders (public and private sector) have to answer to their citizens and customers when these threats occur. In other words, an event that says “this attac [...]
-<a href="http://spot.incubator.apache.org//wp-content/uploads/2016/03/FloCon2015.png" rel="attachment wp-att-66"><img class="aligncenter size-full wp-image-66" src="http://spot.incubator.apache.org//wp-content/uploads/2016/03/FloCon2015.png" alt="open source packet and flow analytics" width="960" height="720" srcset="http://spot.incubator.apache.org//wp-content/uploads/2016/03/FloCon2015.png 960w, http://spot.incubator.apache.org//wp-content/uploads/2016/03/FloCon2015-300x225.png 300w, h [...]
+<a href="http://spot.incubator.apache.org/wp-content/uploads/2016/03/FloCon2015.png" rel="attachment wp-att-66"><img class="aligncenter size-full wp-image-66" src="http://spot.incubator.apache.org/wp-content/uploads/2016/03/FloCon2015.png" alt="open source packet and flow analytics" width="960" height="720" srcset="http://spot.incubator.apache.org/wp-content/uploads/2016/03/FloCon2015.png 960w, http://spot.incubator.apache.org/wp-content/uploads/2016/03/FloCon2015-300x225.png 300w, http: [...]
<p>The gist is that while processing is a challenge at higher scales, the amount of insight gained is higher when analyzing flows and packets from key protocols (like DNS). And that’s how we got here.</p>
<p><strong>Why Intel?</strong></p>
<p>At Intel, I have worked in IT, for a security product company (McAfee), and in the Data Center Group. Intel IT was an early pioneer of the concept of proactive investigations to protect intellectual property. McAfee (now Intel Security Group) has a broad customer base in the realms of network, endpoint, and content security, to name only a few. And the Intel Data Center group has strategic partnerships with Cloudera and Accenture, as well as some pretty cool analytics efforts of their [...]
<p><strong>Why Open Source?</strong></p>
<p>I learned from my earlier efforts in security analytics, that to invite collaboration from academia, the public sector, and the private sector, open source software is an excellent choice. We are now seeking to build a community of developers, data scientists, and security enthusiasts to grow Apache Spot into something we can all be proud of. We have also chosen an Apache software license, so that it can enrich commercial software offerings as well.</p>
-<p>The greatest thing for me since we announced at RSA is to hear OTHER people talk about Apache Spot (formerly Open Network Insight or ONI), here are some of my favorites, from <a href="http://vision.cloudera.com/open-network-insight-changing-infosec-data-science-forever/" target="_blank">a Data Scientist @ eBay </a>, <a href="https://newsroom.accenture.com/news/accenture-introduces-the-accenture-cyber-intelligence-platform-to-help-organizations-continuously-predict-detect-and-combat-cy [...]
+<p>The greatest thing for me since we announced at RSA is to hear OTHER people talk about Apache Spot (formerly Open Network Insight or ONI), here are some of my favorites, from <a href="http://vision.cloudera.com/apache-spot-changing-infosec-data-science-forever/" target="_blank">a Data Scientist @ eBay </a>, <a href="https://newsroom.accenture.com/news/accenture-introduces-the-accenture-cyber-intelligence-platform-to-help-organizations-continuously-predict-detect-and-combat-cyber-attac [...]
<p>Fork us on Github!</p>
<p>Grant Babb</p>
]]></content:encoded>
- <wfw:commentRss>http://spot.incubator.apache.org//open-network-insight-3-most-asked-questions/feed/</wfw:commentRss>
+ <wfw:commentRss>http://spot.incubator.apache.org/apache-spot-3-most-asked-questions/feed/</wfw:commentRss>
<slash:comments>0</slash:comments>
</item>
</channel>
diff --git a/how-apache-spot-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/feed/index.html b/how-apache-spot-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/feed/index.html
new file mode 100644
index 0000000..62b7e33
--- /dev/null
+++ b/how-apache-spot-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/feed/index.html
@@ -0,0 +1,18 @@
+<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
+ xmlns:content="http://purl.org/rss/1.0/modules/content/"
+ xmlns:dc="http://purl.org/dc/elements/1.1/"
+ xmlns:atom="http://www.w3.org/2005/Atom"
+ xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
+
+ >
+<channel>
+ <title>Comments on: How Apache Spot (Incubating) Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</title>
+ <atom:link href="http://spot.incubator.apache.org/how-apache-spot-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/feed/" rel="self" type="application/rss+xml" />
+ <link>http://spot.incubator.apache.org/how-apache-spot-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/</link>
+ <description></description>
+ <lastBuildDate></lastBuildDate>
+ <sy:updatePeriod>hourly</sy:updatePeriod>
+ <sy:updateFrequency>1</sy:updateFrequency>
+
+</channel>
+</rss>
diff --git a/blog/index.html b/how-apache-spot-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/index.html
similarity index 60%
copy from blog/index.html
copy to how-apache-spot-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/index.html
index 81a376e..4249e9e 100644
--- a/blog/index.html
+++ b/how-apache-spot-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/index.html
@@ -12,7 +12,7 @@
<meta http-equiv="X-UA-Compatible" content="IE=edge">
- <title>Apache Spot Blog</title>
+ <title>How Apache Spot (Incubating) Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</title>
<meta name="HandheldFriendly" content="True">
<meta name="MobileOptimized" content="320">
@@ -33,6 +33,7 @@
<link rel='dns-prefetch' href='//s.w.org' />
<link rel="alternate" type="application/rss+xml" title="Apache Spot » Feed" href="../feed/" />
<link rel="alternate" type="application/rss+xml" title="Apache Spot » Comments Feed" href="../comments/feed/" />
+ <link rel="alternate" type="application/rss+xml" title="Apache Spot » How Apache Spot Helps Create Well-Stocked Data Lakes and Catch Powerful Insights Comments Feed" href="../how-open-network-insight-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/feed/" />
<script type="text/javascript">
window._wpemojiSettings = {
"baseUrl" : "https:\/\/s.w.org\/images\/core\/emoji\/2\/72x72\/",
@@ -124,28 +125,14 @@
<script type='text/javascript' src='../wp-includes/js/jquery/jquery-migrate.min.js'></script>
<script type='text/javascript' src='../wp-content/themes/oni/library/js/jquery.meanmenu.js'></script>
<link rel='https://api.w.org/' href='../wp-json/' />
+ <link rel="canonical" href="../how-open-network-insight-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/" />
+ <link rel='shortlink' href='../?p=113' />
+ <link rel="alternate" type="application/json+oembed" href="../wp-json/oembed/1.0/embed?url=http%3A%2F%2Fnolamarketing.com%2Fclient%2Foni%2Fhow-open-network-insight-helps-create-well-stocked-data-lakes-and-catch-powerful-insights%2F" />
+ <link rel="alternate" type="text/xml+oembed" href="../wp-json/oembed/1.0/embed?url=http%3A%2F%2Fnolamarketing.com%2Fclient%2Foni%2Fhow-open-network-insight-helps-create-well-stocked-data-lakes-and-catch-powerful-insights%2F&format=xml" />
- <script>
- (function(i, s, o, g, r, a, m) {
- i['GoogleAnalyticsObject'] = r;
- i[r] = i[r] ||
- function() {
- (i[r].q = i[r].q || []).push(arguments)
- }, i[r].l = 1 * new Date();
- a = s.createElement(o),
- m = s.getElementsByTagName(o)[0];
- a.async = 1;
- a.src = g;
- m.parentNode.insertBefore(a, m)
- })(window, document, 'script', '//www.google-analytics.com/analytics.js', 'ga');
-
- ga('create', 'UA-75955621-1', 'auto');
- ga('send', 'pageview');
-
- </script>
</head>
- <body class="blog" itemscope itemtype="http://schema.org/WebPage">
+ <body class="single single-post postid-113 single-format-standard" itemscope itemtype="http://schema.org/WebPage">
<div id="container">
@@ -171,7 +158,7 @@
<li id="menu-item-106" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-106">
<a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight/wiki">Wiki</a>
</li>
- <li id="menu-item-13" class="menu-item menu-item-type-post_type menu-item-object-page current-menu-item page_item page-item-10 current_page_item current_page_parent menu-item-13">
+ <li id="menu-item-13" class="menu-item menu-item-type-post_type menu-item-object-page current_page_parent menu-item-13">
<a href="../blog/">Blog</a>
</li>
</ul>
@@ -186,39 +173,17 @@
<div id="content">
<div id="inner-content" class="wrap cf">
- <main id="main" class="m-all t-2of3 d-5of7 cf" role="main" itemscope itemprop="mainContentOfPage" itemtype="http://schema.org/Blog">
- <article id="post-117" class="cf post-117 post type-post status-publish format-standard hentry category-cybersecurity" role="article">
-
- <header class="article-header">
-
- <h1 class="h2 entry-title"><a href="../open-network-insight-oni-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/" rel="bookmark" title="Apache Spot and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure">Apache Spot and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a></h1>
- <p class="byline entry-meta vcard">
- <time class="updated entry-time" datetime="2016-08-08" itemprop="datePublished">
- August 8, 2016
- </time>
- </span>
- </p>
-
- </header>
-
- <section class="entry-content cf">
- <div class="m-all t-1of4 d-2of7"></div>
- <div class="m-all t-3of4 d-5of7">
- <p>
- The “first” documented cybersecurity case was the worm replication, which was initiated by Robert T. Morris on November 2, 1988. Wow! Here we are in 2016, 28 years later, with viruses and worms giving way to Trojan horses and polymorphic code. Nowadays, we are also fighting against DDoS, phishing, spear phishing attacks, command and controls… <a class="excerpt-read-more" href="../open-network-insight-oni-and-cybersecurity-using-netflows-to-de [...]
- </p>
- </div>
- </section>
+ <main id="main" class="m-all t-2of3 d-5of7 cf" role="main" itemscope itemprop="mainContentOfPage" itemtype="http://schema.org/Blog">
- </article>
+ <article id="post-113" class="cf post-113 post type-post status-publish format-standard hentry category-uncategorized" role="article" itemscope itemprop="blogPost" itemtype="http://schema.org/BlogPosting">
- <article id="post-113" class="cf post-113 post type-post status-publish format-standard hentry category-uncategorized" role="article">
+ <header class="article-header entry-header">
- <header class="article-header">
+ <h1 class="entry-title single-title" itemprop="headline" rel="bookmark">How Apache Spot (Incubating) Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</h1>
- <h1 class="h2 entry-title"><a href="../how-open-network-insight-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/" rel="bookmark" title="How Apache Spot Helps Create Well-Stocked Data Lakes and Catch Powerful Insights">How Apache Spot Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a></h1>
<p class="byline entry-meta vcard">
+
<time class="updated entry-time" datetime="2016-08-08" itemprop="datePublished">
August 8, 2016
</time>
@@ -226,40 +191,39 @@
</p>
</header>
-
- <section class="entry-content cf">
- <div class="m-all t-1of4 d-2of7"></div>
- <div class="m-all t-3of4 d-5of7">
- <p>
- About four years ago, the era of the Big Data analytics began. Paired with advanced analytics, massive volumes of data can be culled to not only inform critical decisions, but also to simulate sophisticated “what if” scenarios that allow companies to gain competitive advantages by generating and predicting different scenarios. For example, a financial services… <a class="excerpt-read-more" href="../how-open-network-insight-helps-create-well-s [...]
- </p>
- </div>
- </section>
-
- </article>
-
- <article id="post-62" class="cf post-62 post type-post status-publish format-standard hentry category-security-analytics tag-github tag-open-network-insight tag-open-source" role="article">
-
- <header class="article-header">
-
- <h1 class="h2 entry-title"><a href="../open-network-insight-3-most-asked-questions/" rel="bookmark" title="Apache Spot: Three Most-Asked Questions">Apache Spot: Three Most-Asked Questions</a></h1>
- <p class="byline entry-meta vcard">
- <time class="updated entry-time" datetime="2016-03-29" itemprop="datePublished">
- March 29, 2016
- </time>
- </span>
+ <section class="entry-content cf" itemprop="articleBody">
+ <p>
+ About four years ago, the era of the Big Data analytics began. Paired with advanced analytics, massive volumes of data can be culled to not only inform critical decisions, but also to simulate sophisticated “what if” scenarios that allow companies to gain competitive advantages by generating and predicting different scenarios. For example, a financial services company can more accurately determine what other products to offer a customer, and in what or [...]
+ </p>
+ <p>
+ Apache Spot can help to solve most of these issues. Following is a description of the Apache Spot, which is designed to facilitate Big Data analytics scenarios like the financial services company’s question about the right product to offer customers.
+ </p>
+ <a href="../wp-content/uploads/2016/09/ONI_Architecture-Diagram_1300_v4.png"><img src="../wp-content/uploads/2016/09/ONI_Architecture-Diagram_1300_v4.png" alt="oni_architecture-diagram_1300_v4" width="1300" height="675" class="alignnone size-full wp-image-114" srcset="../wp-content/uploads/2016/09/ONI_Architecture-Diagram_1300_v4.png 1300w, ../wp-content/uploads/2016/09/ONI_Architecture-Diagram_1300_v4-300x156.png 300w, ../wp-content/uploads/2016/09/ONI_Ar [...]
+ <h3><strong>Apache Spot Core Components</strong></h3>
+ <p>
+ The Apache Spot Core is composed of three main components — data integration (collectors), data store (HDFS here, but can also be a non-SQL database) and machine learning.
+ </p>
+ <p>
+ In this diagram, the top left shows Apache Spot Data Sources, which include the collection of the information that will be used to create a data lake. The process is simple. Define a pull or push from the source of information then capture this information on Apache Spot’s “collectors.” The collectors are processes that interpret the information that is sent, then write it to the HDFS system in the Apache Spot cluster. The HDFS stores the data lake and [...]
</p>
+ <h3><strong>Operational Analytics</strong></h3>
+ <p>
+ As part of operational analytics, Apache Spot executes different batch processes that add information to machine learning results to provide meaning and context. Using the financial services product example, basic customer data could be augmented with information about other customers in the same region along with information about which products those customers recommended or complained about. Basically, the data scientists can “play” with the data us [...]
+ </p>
+ <h3><strong>Visualizing Results</strong></h3>
+ <p>
+ The Apache Spot GUI displays the results that the machine learning algorithms generate. Results are represented such that it is easy to identify both the most common things as well as find the most suspicious or uncommon information that is part of the data lake.
+ </p>
+ <h3><strong>Customizable Open Source</strong></h3>
+ <p>
+ Because Apache Spot is an open-source project, most of the components depicted here can be modified by the end user.
+ </p>
+ </section>
+ <footer class="article-footer">
- </header>
+ filed under: <a href="../category/uncategorized/" rel="category tag">Uncategorized</a>
- <section class="entry-content cf">
- <div class="m-all t-1of4 d-2of7"></div>
- <div class="m-all t-3of4 d-5of7">
- <p>
- While this is not the first blog post about Apache Spot, it is the first one by a creator of the solution. As a security data scientist in Intel’s Data Center Group, I joined a small team to start thinking about solving really hard problems in cloud analytics. The team grew, and out of that… <a class="excerpt-read-more" href="../open-network-insight-3-most-asked-questions/" title="Read Apache Spot: Three Most-Asked Questions">Read more [...]
- </p>
- </div>
- </section>
+ </footer>
</article>
@@ -271,13 +235,16 @@
<h4 class="widgettitle">Recent Posts</h4>
<ul>
<li>
- <a href="../open-network-insight-oni-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/">Apache Spot and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a>
+ <a href="../jupyter-notebooks-for-data-analysis/">Jupyter Notebooks for Data Analysis</a>
+ </li>
+ <li>
+ <a href="../apache-spot-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/">Apache Spot (Incubating) and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a>
</li>
<li>
- <a href="../how-open-network-insight-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/">How Apache Spot Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a>
+ <a href="../how-apache-spot-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/">How Apache Spot (Incubating) Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a>
</li>
<li>
- <a href="../open-network-insight-3-most-asked-questions/">Apache Spot: Three Most-Asked Questions</a>
+ <a href="../apache-spot-3-most-asked-questions/">Apache Spot (Incubating): 3 Most-Asked Questions</a>
</li>
</ul>
</div>
@@ -285,6 +252,9 @@
<h4 class="widgettitle">Archives</h4>
<ul>
<li>
+ <a href='../2016/09/'>September 2016</a>
+ </li>
+ <li>
<a href='../2016/08/'>August 2016</a>
</li>
<li>
@@ -314,6 +284,7 @@
</div>
+ <script type='text/javascript' src='../wp-includes/js/comment-reply.min.js'></script>
<script type='text/javascript' src='../wp-content/themes/oni/library/js/scripts.js'></script>
<script type='text/javascript' src='../wp-includes/js/wp-embed.min.js'></script>
diff --git a/index.html b/index.html
index 31a6a1e..360738c 100644
--- a/index.html
+++ b/index.html
@@ -42,7 +42,8 @@
"source" : {
"concatemoji" : "http:\/\/spot.incubator.apache.org\/wp-includes\/js\/wp-emoji-release.min.js"
}
- }; ! function(a, b, c) {
+ };
+ ! function(a, b, c) {
function d(a) {
var c,
d,
@@ -129,24 +130,6 @@
<link rel="alternate" type="application/json+oembed" href="wp-json/oembed/1.0/embed?url=http%3A%2F%2Fnolamarketing.com%2Fclient%2Foni%2F" />
<link rel="alternate" type="text/xml+oembed" href="wp-json/oembed/1.0/embed?url=http%3A%2F%2Fnolamarketing.com%2Fclient%2Foni%2F&format=xml" />
- <script>
- (function(i, s, o, g, r, a, m) {
- i['GoogleAnalyticsObject'] = r;
- i[r] = i[r] ||
- function() {
- (i[r].q = i[r].q || []).push(arguments)
- }, i[r].l = 1 * new Date();
- a = s.createElement(o),
- m = s.getElementsByTagName(o)[0];
- a.async = 1;
- a.src = g;
- m.parentNode.insertBefore(a, m)
- })(window, document, 'script', '//www.google-analytics.com/analytics.js', 'ga');
-
- ga('create', 'UA-75955621-1', 'auto');
- ga('send', 'pageview');
-
- </script>
</head>
<body class="home page page-id-8 page-template-default" itemscope itemtype="http://schema.org/WebPage">
@@ -354,45 +337,6 @@
<p class="btn-margin center">
Spot accelerates the development of cybersecurity applications by providing a cybersecurity analytics framework. This means more solutions can be created faster. This is because Spot allows organizations to focus developing the analytics and visualizations for applications that discover cybercrime rather than spending time building systems to ingest, integrate, store, and process myriad volumes or varieties of security data.
</p>
- <!--
- <div class="quotes">
-
- <div class="table">
- <div class="table-row">
- <div class="table-cell quote-man">
- <img src="wp-content/themes/oni/library/images/quote-person.png" alt="" />
- <h3>Senior Analyst</h3>
- </div>
- <div class="table-cell">
- <p>...that allows me to see and customize the data and scripts to my enviroment. I want control over how the solution works.”</p> </div>
- </div>
- </div>
-
- <div class="table">
- <div class="table-row">
- <div class="table-cell quote-man">
- <img src="wp-content/themes/oni/library/images/quote-person.png" alt="" />
- <h3>Junior Analyst</h3>
- </div>
- <div class="table-cell">
- <p>...that automatically alerts me to actionable suspicious events and ways to optimize my network in a timely fashion. Help me investigate these events and tell the story across stakeholders in my organization.”
- </p> </div>
- </div>
- </div>
-
- <div class="table">
- <div class="table-row">
- <div class="table-cell quote-man">
- <img src="wp-content/themes/oni/library/images/quote-person.png" alt="" />
- <h3>Decision Maker</h3>
- </div>
- <div class="table-cell">
- <p>...tells me the story of what happened in a way I can understand so I can make decisions as a result.”
- </p> </div>
- </div>
- </div>
-
- </div>-->
</div>
</div>
@@ -447,43 +391,6 @@
</div>
- <!--
- <div class="m-all t-1of2 d-4of7 left">
- <img src="wp-content/uploads/2016/03/use-case-graphic.png" alt="" />
- </div>
- <div class="m-all t-1of2 d-3of7 right">
- <div class="table">
- <div class="table-row">
- <div class="table-cell number">
- <img src="wp-content/themes/oni/library/images/one.png" alt="" />
- </div>
- <div class="table-cell">
- <p>Deliver suspicious connections in <strong>ranked</strong> order with multiple data points such as time, traffic flow, and more.</p> </div>
- </div>
- </div>
-
- <div class="table">
- <div class="table-row">
- <div class="table-cell number">
- <img src="wp-content/themes/oni/library/images/two.png" alt="" />
- </div>
- <div class="table-cell">
- <p>Human input helps the system evolve. <strong>Quickly eliminate false positives from the lineup.</strong></p> </div>
- </div>
- </div>
-
- <div class="table">
- <div class="table-row">
- <div class="table-cell number">
- <img src="wp-content/themes/oni/library/images/three.png" alt="" />
- </div>
- <div class="table-cell">
- <p>And make every item on the list <strong>worth your time</strong> to investigate.</p> </div>
- </div>
- </div>
-
- </div> -->
-
</div>
<div class="arrow">
@@ -498,7 +405,7 @@
<div id="more-info">
<div class="wrap cf">
<p class="social-icons">
- <a href="mailto:info@open-network-insight.org"><span class="icon-envelope"></span></a><a href="https://twitter.com/onihadoop" target="_blank"><span class="icon-twitter"></span></a>
+ <a href="mailto:info@open-network-insight.org"><span class="icon-envelope"></span></a><a href="https://twitter.com/ApacheSpot" target="_blank"><span class="icon-twitter"></span><a href="http://open-network-insight.org:3000/" target="_blank"><img src="wp-content/themes/oni/library/images/add-to-slack.png" alt="" class="add-to-slack" /></a></a>
</p>
<p>
diff --git a/jupyter-notebooks-for-data-analysis/feed/index.html b/jupyter-notebooks-for-data-analysis/feed/index.html
new file mode 100644
index 0000000..0ce0406
--- /dev/null
+++ b/jupyter-notebooks-for-data-analysis/feed/index.html
@@ -0,0 +1,18 @@
+<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
+ xmlns:content="http://purl.org/rss/1.0/modules/content/"
+ xmlns:dc="http://purl.org/dc/elements/1.1/"
+ xmlns:atom="http://www.w3.org/2005/Atom"
+ xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
+
+ >
+<channel>
+ <title>Comments on: Jupyter Notebooks for Data Analysis</title>
+ <atom:link href="http://spot.incubator.apache.org/jupyter-notebooks-for-data-analysis/feed/" rel="self" type="application/rss+xml" />
+ <link>http://spot.incubator.apache.org/jupyter-notebooks-for-data-analysis/</link>
+ <description></description>
+ <lastBuildDate></lastBuildDate>
+ <sy:updatePeriod>hourly</sy:updatePeriod>
+ <sy:updateFrequency>1</sy:updateFrequency>
+
+</channel>
+</rss>
diff --git a/jupyter-notebooks-for-data-analysis/index.html b/jupyter-notebooks-for-data-analysis/index.html
new file mode 100644
index 0000000..cbd0e04
--- /dev/null
+++ b/jupyter-notebooks-for-data-analysis/index.html
@@ -0,0 +1,333 @@
+<!doctype html>
+
+<!--[if lt IE 7]><html lang="en-US" class="no-js lt-ie9 lt-ie8 lt-ie7"><![endif]-->
+<!--[if (IE 7)&!(IEMobile)]><html lang="en-US" class="no-js lt-ie9 lt-ie8"><![endif]-->
+<!--[if (IE 8)&!(IEMobile)]><html lang="en-US" class="no-js lt-ie9"><![endif]-->
+<!--[if gt IE 8]><!-->
+<html lang="en-US" class="no-js">
+ <!--<![endif]-->
+
+ <head>
+ <meta charset="utf-8">
+
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+
+ <title>Apache Spot Jupyter Notebooks for Data Analysis</title>
+
+ <meta name="HandheldFriendly" content="True">
+ <meta name="MobileOptimized" content="320">
+ <meta name="viewport" content="width=device-width, initial-scale=1"/>
+
+ <link rel="apple-touch-icon" href="../wp-content/themes/oni/library/images/apple-touch-icon.png">
+ <link rel="icon" href="../wp-content/themes/oni/favicon.png">
+ <!--[if IE]>
+ <link rel="shortcut icon" href="http://spot.incubator.apache.org../wp-content/themes/oni/favicon.ico">
+ <![endif]-->
+ <meta name="msapplication-TileColor" content="#f01d4f">
+ <meta name="msapplication-TileImage" content="../wp-content/themes/oni/library/images/win8-tile-icon.png">
+ <meta name="theme-color" content="#121212">
+
+ <link rel="pingback" href="http://spot.incubator.apache.org/xmlrpc.php">
+
+ <link rel="alternate" type="application/rss+xml" title="Apache Spot » Feed" href="http://spot.incubator.apache.org/feed/" />
+ <link rel="alternate" type="application/rss+xml" title="Apache Spot » Comments Feed" href="http://spot.incubator.apache.org/comments/feed/" />
+ <link rel="alternate" type="application/rss+xml" title="Apache Spot » Jupyter Notebooks for Data Analysis Comments Feed" href="http://spot.incubator.apache.org/jupyter-notebooks-for-data-analysis/feed/" />
+ <script type="text/javascript">
+ window._wpemojiSettings = {
+ "baseUrl" : "https:\/\/s.w.org\/images\/core\/emoji\/72x72\/",
+ "ext" : ".png",
+ "source" : {
+ "concatemoji" : "http:\/\/spot.incubator.apache.org\/wp-includes\/js\/wp-emoji-release.min.js"
+ }
+ }; ! function(a, b, c) {
+ function d(a) {
+ var c,
+ d,
+ e,
+ f = b.createElement("canvas"),
+ g = f.getContext && f.getContext("2d"),
+ h = String.fromCharCode;
+ if (!g || !g.fillText)
+ return !1;
+ switch(g.textBaseline="top",g.font="600 32px Arial",a) {
+ case"flag":
+ return g.fillText(h(55356, 56806, 55356, 56826), 0, 0), f.toDataURL().length > 3e3;
+ case"diversity":
+ return g.fillText(h(55356, 57221), 0, 0),
+ c = g.getImageData(16, 16, 1, 1).data,
+ d = c[0] + "," + c[1] + "," + c[2] + "," + c[3], g.fillText(h(55356, 57221, 55356, 57343), 0, 0),
+ c = g.getImageData(16, 16, 1, 1).data,
+ e = c[0] + "," + c[1] + "," + c[2] + "," + c[3], d !== e;
+ case"simple":
+ return g.fillText(h(55357, 56835), 0, 0), 0 !== g.getImageData(16,16,1,1).data[0];
+ case"unicode8":
+ return g.fillText(h(55356, 57135), 0, 0), 0 !== g.getImageData(16,16,1,1).data[0]
+ }
+ return !1
+ }
+
+ function e(a) {
+ var c = b.createElement("script");
+ c.src = a, c.type = "text/javascript", b.getElementsByTagName("head")[0].appendChild(c)
+ }
+
+ var f,
+ g,
+ h,
+ i;
+ for ( i = Array("simple", "flag", "unicode8", "diversity"), c.supports = {
+ everything : !0,
+ everythingExceptFlag : !0
+ },
+ h = 0; h < i.length; h++)
+ c.supports[i[h]] = d(i[h]), c.supports.everything = c.supports.everything && c.supports[i[h]], "flag" !== i[h] && (c.supports.everythingExceptFlag = c.supports.everythingExceptFlag && c.supports[i[h]]);
+ c.supports.everythingExceptFlag = c.supports.everythingExceptFlag && !c.supports.flag, c.DOMReady = !1, c.readyCallback = function() {
+ c.DOMReady = !0
+ }, c.supports.everything || ( g = function() {
+ c.readyCallback()
+ }, b.addEventListener ? (b.addEventListener("DOMContentLoaded", g, !1), a.addEventListener("load", g, !1)) : (a.attachEvent("onload", g), b.attachEvent("onreadystatechange", function() {
+ "complete" === b.readyState && c.readyCallback()
+ })),
+ f = c.source || {}, f.concatemoji ? e(f.concatemoji) : f.wpemoji && f.twemoji && (e(f.twemoji), e(f.wpemoji)))
+ }(window, document, window._wpemojiSettings);
+ </script>
+ <style type="text/css">
+ img.wp-smiley, img.emoji {
+ display: inline !important;
+ border: none !important;
+ box-shadow: none !important;
+ height: 1em !important;
+ width: 1em !important;
+ margin: 0 .07em !important;
+ vertical-align: -0.1em !important;
+ background: none !important;
+ padding: 0 !important;
+ }
+ </style>
+ <link rel='stylesheet' id='googleFonts-css' href='http://fonts.googleapis.com/css?family=Lato%3A400%2C700%2C400italic%2C700italic' type='text/css' media='all' />
+ <link rel='stylesheet' id='bones-stylesheet-css' href='../wp-content/themes/oni/library/css/style.css' type='text/css' media='all' />
+ <!--[if lt IE 9]>
+ <link rel='stylesheet' id='bones-ie-only-css' href='http://spot.incubator.apache.org../wp-content/themes/oni/library/css/ie.css' type='text/css' media='all' />
+ <![endif]-->
+ <link rel='stylesheet' id='mm-css-css' href='../wp-content/themes/oni/library/css/meanmenu.css' type='text/css' media='all' />
+ <script type='text/javascript' src='../wp-content/themes/oni/library/js/libs/modernizr.custom.min.js'></script>
+ <script type='text/javascript' src='../wp-includes/js/jquery/jquery.js'></script>
+ <script type='text/javascript' src='../wp-includes/js/jquery/jquery-migrate.min.js'></script>
+ <script type='text/javascript' src='../wp-content/themes/oni/library/js/jquery.meanmenu.js'></script>
+ <link rel='https://api.w.org/' href='../wp-json/' />
+ <link rel="canonical" href="http://spot.incubator.apache.org/jupyter-notebooks-for-data-analysis/" />
+ <!--<link rel='shortlink' href='http://spot.incubator.apache.org/?p=136' />-->
+ <link rel="alternate" type="application/json+oembed" href="http://spot.incubator.apache.org/wp-json/oembed/1.0/embed?url=http%3A%2F%2Fopen-network-insight.org%2Fjupyter-notebooks-for-data-analysis%2F" />
+ <link rel="alternate" type="text/xml+oembed" href="http://spot.incubator.apache.org/wp-json/oembed/1.0/embed?url=http%3A%2F%2Fopen-network-insight.org%2Fjupyter-notebooks-for-data-analysis%2F&format=xml" />
+
+
+ </head>
+
+ <body class="single single-post postid-136 single-format-standard" itemscope itemtype="http://schema.org/WebPage">
+
+ <div id="container">
+
+ <header class="header" role="banner" itemscope itemtype="http://schema.org/WPHeader">
+
+ <div id="inner-header" class="wrap cf">
+
+ <p id="logo" class="h1" itemscope itemtype="http://schema.org/Organization">
+ <a href="http://spot.incubator.apache.org" rel="nofollow"><img src="../wp-content/themes/oni/library/images/logo.png" alt="Open Network Insight" /></a>
+ </p>
+
+ <nav role="navigation" itemscope itemtype="http://schema.org/SiteNavigationElement">
+ <ul id="menu-main-menu" class="nav top-nav cf">
+ <li id="menu-item-4" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-4">
+ <a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight">Get Started</a>
+ </li>
+ <li id="menu-item-5" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-5">
+ <a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight#if-you-want-all-of-the-oni-code-at-once-just-clone-it">Download</a>
+ </li>
+ <li id="menu-item-6" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-6">
+ <a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight#contributing-to-oni">Contribute</a>
+ </li>
+ <li id="menu-item-7" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-7">
+ <a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight/wiki">Wiki</a>
+ </li>
+ <li id="menu-item-13" class="menu-item menu-item-type-post_type menu-item-object-page current_page_parent menu-item-13">
+ <a href="../blog/">Blog</a>
+ </li>
+ </ul>
+ </nav>
+
+ </div>
+
+ </header>
+
+ <div id="mobile-nav"></div>
+
+ <div id="content">
+
+ <div id="inner-content" class="wrap cf">
+
+ <main id="main" class="m-all t-2of3 d-5of7 cf" role="main" itemscope itemprop="mainContentOfPage" itemtype="http://schema.org/Blog">
+
+ <article id="post-136" class="cf post-136 post type-post status-publish format-standard hentry category-uncategorized" role="article" itemscope itemprop="blogPost" itemtype="http://schema.org/BlogPosting">
+
+ <header class="article-header entry-header">
+
+ <h1 class="entry-title single-title" itemprop="headline" rel="bookmark">Jupyter Notebooks for Data Analysis</h1>
+
+ <p class="byline entry-meta vcard">
+
+ <time class="updated entry-time" datetime="2016-09-22" itemprop="datePublished">
+ September 22, 2016
+ </time>
+ </span>
+ </p>
+
+ </header>
+ <section class="entry-content cf" itemprop="articleBody">
+ <p>
+ <strong>Why Does Apache Spot Include iPython notebooks? </strong>
+ </p>
+ <p>
+ The project team wants Apache Spot to be a versatile tool that can be used by anyone. This means that data scientists and developers need to be able to query and handle the source data to find all the information they need for their decision making. The iPython Notebook is an appropriate platform for easy data exploration. One of its biggest advantages is that it provides parallel and distributed computing to enable code execution and debugging in an i [...]
+ </p>
+ <p>
+ The iPython notebook is a web based interactive computational environment that provides access to the Python shell. While iPython notebooks were originally designed to work with the Python language, they support a number of other programming languages, including Ruby, Scala, Julia, R, Go, C, C++, Java and Perl. There are also multiple additional packages that can be used to get the most out of this highly-customizable tool.
+ </p>
+ <p>
+ Starting on version 4.0, most notebook functionalities are now part of the Project Jupyter, while iPython remains as the kernel to work with Python code in the notebooks.
+ </p>
+ <img src="../wp-content/uploads/2016/10/iPython-1.png" alt="ipython" width="361" height="107" class="aligncenter size-full wp-image-140" srcset="../wp-content/uploads/2016/10/iPython-1.png 361w, ../wp-content/uploads/2016/10/iPython-1-300x89.png 300w" sizes="(max-width: 361px) 100vw, 361px" />
+ <p>
+ <strong>IPython with Apache Spot for Network Threat Detection</strong>
+ </p>
+ <p>
+ <em>NOTE: This is not intended to be a step-by-step tutorial on how to code a threat analysis in Apache Spot, but more like an introduction on how to approach the suspicions of a security breach.</em>
+ </p>
+ <p>
+ Although machine learning (ML) will do most of the work detecting anomalies in the traffic, Apache Spot also includes two notebook templates that can get you started on this. The <em>Threat_Investigation_master.ipynb</em> is designed to query the raw data table to find all connections in a day that are related to any threat you select – even connections that were not necessarily flagged as suspicious by ML on a first run. This gives us the chance to ge [...]
+ </p>
+ <p>
+ If you suspect of a specific type of attack in your network, you can get the whole story by answering the Five ‘W’s.
+ </p>
+ <p>
+ <strong><em>What? </em></strong>
+ </p>
+ <p>
+ Maybe there’s been an increase in the logs collected by the system, which indicates abnormal amounts of communication in your network. Or, the amount of POST requests in your network have risen overnight. This is the mystery that needs to be solved by researching through the anomalies previously detected by ML.
+ </p>
+ <p>
+ <strong><em>Who?</em></strong>
+ </p>
+ <p>
+ Assuming you have a network context, you can identify the name of the infected machine inside the network, as well as the name of the IP or DNS on the other side of the connection (if it is a known host). If you don’t have a network context or are using DHCP, this can be a little tricky to detect using only Netflow logs. But, that’s where DNS and Proxy logs, come in handy. Including a network context file with Apache Spot is really simple and can go a [...]
+ </p>
+ <p>
+ <strong><em>When?</em></strong>
+ </p>
+ <p>
+ To have a broader visibility on the attack, you can customize the queries on the Threat investigation notebook to review the data through a wider time lapse – instead of just checking through the current day. With this, you could find an increase of a certain type of requests to one (or many) URIs and predict its future behavior.
+ </p>
+ <p>
+ <strong><em>Where?</em></strong>
+ </p>
+ <p>
+ When working only with DNS, having a destination URL might not say much about where your information is going to, but Apache Spot allows you to connect with a geolocation database to identify the location of the suspected attackers IP. Taking advantage of this option, you can visually locate the other end of the connection on a map. You might find that it’s pointing to a country banned by your company, indicating a leak.
+ </p>
+ <p>
+ <strong><em>Why?</em></strong>
+ </p>
+ <p>
+ This answer to “why” will depend highly on the result of the analysis. For instance, an excessive amount of POST requests from one machine inside the network to an unidentified URI can indicate a data mining attack. Tracing back to patient zero, you can find that this could have originated with a phishing email, malicious software installed by an employee or a one-time visitor’s infected machine that connected to your network.
+ </p>
+ <p>
+ <strong>How to Get Answers to the Five Ws Questions</strong>
+ </p>
+ <p>
+ All of the previous questions can be answered by looking at the raw data collected. Although performing elaborated queries directly to your database can seem tempting, this type of analysis with Hive, or even Impala, can be very time consuming. A better approach would be to use Pandas to read and transform your dataset into a relational structured dataframe. This lets you work with as if it were an offline structured relational database.
+ </p>
+ <p>
+ Once you have your desired results and data subsets, you can use MatplotLib to easily graph your findings. (We cover this subject in more depth in another post.) Another advantage of the notebook is that you can download it as HTML or a PDF file to store locally and use it in a presentation – or just keep it for future reference.
+ </p>
+ <p>
+ <strong>Wrap Up</strong>
+ </p>
+ <p>
+ This post was meant to be just a brief introduction of how you can use iPython notebooks in Apache Spot to perform further data analysis and include it our executive report (in addition to the already included Story board). Although this is not the only way you can do this, it is a very interactive and fun way to do it. You’ll also see that the overall processing time is very short – thanks to the iPython notebook task parallelism ability.
+ </p>
+ <p>
+ We want to hear from YOU! Have you used iPython notebooks before? How do you feel about having this tool in Apache Spot? If you’re interested in further data analysis through interactive charts, a new post is coming soon on D3 and jQuery data visualization. Also, check back soon to read more on this and other Cybersecurity subjects.
+ </p>
+ </section>
+
+ <footer class="article-footer">
+
+ filed under: <a href="../category/data-science/" rel="category tag">Data Science</a>, <a href="../category/ipython-notebooks/" rel="category tag">Ipython Notebooks</a>, <a href="../category/threat-analysis-tools/" rel="category tag">Threat Analysis Tools</a>
+
+ </footer>
+
+ </article>
+
+ </main>
+
+ <div id="sidebar1" class="sidebar m-all t-1of3 d-2of7 last-col cf" role="complementary">
+
+ <div id="recent-posts-2" class="widget widget_recent_entries">
+ <h4 class="widgettitle">Recent Posts</h4>
+ <ul>
+ <li>
+ <a href="../jupyter-notebooks-for-data-analysis/">Jupyter Notebooks for Data Analysis</a>
+ </li>
+ <li>
+ <a href="../apache-spot-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/">Apache Spot (Incubating) and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a>
+ </li>
+ <li>
+ <a href="../how-apache-spot-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/">How Apache Spot (Incubating) Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a>
+ </li>
+ <li>
+ <a href="../apache-spot-3-most-asked-questions/">Apache Spot (Incubating): 3 Most-Asked Questions</a>
+ </li>
+ </ul>
+ </div>
+ <div id="archives-2" class="widget widget_archive">
+ <h4 class="widgettitle">Archives</h4>
+ <ul>
+ <li>
+ <a href='../2016/09/'>September 2016</a>
+ </li>
+ <li>
+ <a href='../2016/08/'>August 2016</a>
+ </li>
+ <li>
+ <a href='../2016/03/'>March 2016</a>
+ </li>
+ </ul>
+ </div>
+
+ </div>
+
+ </div>
+
+ </div>
+
+ <footer class="footer" role="contentinfo" itemscope itemtype="http://schema.org/WPFooter">
+
+ <div id="inner-footer" class="wrap cf">
+
+ <nav role="navigation"></nav>
+ <p class="source-org copyright" style="text-align:center;">
+ © 2016 Apache Spot.
+ </p>
+
+ </div>
+
+ </footer>
+
+ </div>
+
+ <script type='text/javascript' src='/wp-includes/js/comment-reply.min.js'></script>
+ <script type='text/javascript' src='../wp-content/themes/oni/library/js/scripts.js'></script>
+ <script type='text/javascript' src='/wp-includes/js/wp-embed.min.js'></script>
+
+ </body>
+
+</html>
+<!-- end of site. what a ride! -->
diff --git a/tag/github/feed/index.html b/tag/github/feed/index.html
index f385614..71bcefa 100644
--- a/tag/github/feed/index.html
+++ b/tag/github/feed/index.html
@@ -18,9 +18,9 @@
<sy:updateFrequency>1</sy:updateFrequency>
<item>
- <title>Apache Spot: Three Most-Asked Questions</title>
- <link>http://spot.incubator.apache.org/open-network-insight-3-most-asked-questions/</link>
- <comments>http://spot.incubator.apache.org/open-network-insight-3-most-asked-questions/#respond</comments>
+ <title>Apache Spot (Incubating): Three Most-Asked Questions</title>
+ <link>http://spot.incubator.apache.org/apache-spot-3-most-asked-questions/</link>
+ <comments>http://spot.incubator.apache.org/apache-spot-3-most-asked-questions/#respond</comments>
<pubDate>Tue, 29 Mar 2016 05:48:19 +0000</pubDate>
<dc:creator><![CDATA[oni-admin]]></dc:creator>
<category><![CDATA[Security Analytics]]></category>
@@ -29,7 +29,7 @@
<category><![CDATA[open source]]></category>
<guid isPermaLink="false">http://spot.incubator.apache.org/?p=62</guid>
- <description><![CDATA[While this is not the first blog post about Apache Spot, it is the first one by a creator of the solution. As a security data scientist in Intel’s Data Center Group, I joined a small team to start thinking about solving really hard problems in cloud analytics. The team grew, and out of that... <a class="excerpt-read-more" href="http://spot.incubator.apache.org/open-network-insight-3-most-asked-questions/" title="Read Apache Spot: Three Most-Asked Questions" [...]
+ <description><![CDATA[While this is not the first blog post about Apache Spot, it is the first one by a creator of the solution. As a security data scientist in Intel’s Data Center Group, I joined a small team to start thinking about solving really hard problems in cloud analytics. The team grew, and out of that... <a class="excerpt-read-more" href="http://spot.incubator.apache.org/apache-spot-3-most-asked-questions/" title="Read Apache Spot: Three Most-Asked Questions">Read mor [...]
<content:encoded><![CDATA[<p>While this is not the first blog post about Apache Spot, it is the first one by a creator of the solution. As a security data scientist in Intel’s Data Center Group, I joined a small team to start thinking about solving really hard problems in cloud analytics. The team grew, and out of that effort, came Apache Spot. Since we started talking about the project, these are the three questions I am asked the most.</p>
<p><strong>What Is Apache Spot?</strong><br />
Apache Spot is an open source, flow and packet analytics solution built on Hadoop. It combines big data processing, at-scale machine learning, and unique security analytics to put potential threats in front of defenders. While I am a data scientist today, I was a security investigator just a few years ago. I wanted to develop a solution that would put new tools and technology in play for defenders, but without requiring them to walk away from security and get a math degree.</p>
@@ -40,11 +40,11 @@ Apache Spot is an open source, flow and packet analytics solution built on Hadoo
<p>At Intel, I have worked in IT, for a security product company (McAfee), and in the Data Center Group. Intel IT was an early pioneer of the concept of proactive investigations to protect intellectual property. McAfee (now Intel Security Group) has a broad customer base in the realms of network, endpoint, and content security, to name only a few. And the Intel Data Center group has strategic partnerships with Cloudera and Accenture, as well as some pretty cool analytics efforts of their [...]
<p><strong>Why Open Source?</strong></p>
<p>I learned from my earlier efforts in security analytics, that to invite collaboration from academia, the public sector, and the private sector, open source software is an excellent choice. We are now seeking to build a community of developers, data scientists, and security enthusiasts to grow Apache Spot into something we can all be proud of. We have also chosen an Apache software license, so that it can enrich commercial software offerings as well.</p>
-<p>The greatest thing for me since we announced at RSA is to hear OTHER people talk about Apache Spot (formerly Open Network Insight or ONI), here are some of my favorites, from <a href="http://vision.cloudera.com/open-network-insight-changing-infosec-data-science-forever/" target="_blank">a Data Scientist @ eBay </a>, <a href="https://newsroom.accenture.com/news/accenture-introduces-the-accenture-cyber-intelligence-platform-to-help-organizations-continuously-predict-detect-and-combat-cy [...]
+<p>The greatest thing for me since we announced at RSA is to hear OTHER people talk about Apache Spot (formerly Open Network Insight or ONI), here are some of my favorites, from <a href="http://vision.cloudera.com/apache-spot-changing-infosec-data-science-forever/" target="_blank">a Data Scientist @ eBay </a>, <a href="https://newsroom.accenture.com/news/accenture-introduces-the-accenture-cyber-intelligence-platform-to-help-organizations-continuously-predict-detect-and-combat-cyber-attac [...]
<p>Fork us on Github!</p>
<p>Grant Babb</p>
]]></content:encoded>
- <wfw:commentRss>http://spot.incubator.apache.org/open-network-insight-3-most-asked-questions/feed/</wfw:commentRss>
+ <wfw:commentRss>http://spot.incubator.apache.org/apache-spot-3-most-asked-questions/feed/</wfw:commentRss>
<slash:comments>0</slash:comments>
</item>
</channel>
diff --git a/tag/github/index.html b/tag/github/index.html
index a084a84..1d36a96 100644
--- a/tag/github/index.html
+++ b/tag/github/index.html
@@ -12,7 +12,7 @@
<meta http-equiv="X-UA-Compatible" content="IE=edge">
- <title>Apache Spot github</title>
+ <title>Github</title>
<meta name="HandheldFriendly" content="True">
<meta name="MobileOptimized" content="320">
@@ -126,24 +126,6 @@
<script type='text/javascript' src='../../wp-content/themes/oni/library/js/jquery.meanmenu.js'></script>
<link rel='https://api.w.org/' href='../../wp-json/' />
- <script>
- (function(i, s, o, g, r, a, m) {
- i['GoogleAnalyticsObject'] = r;
- i[r] = i[r] ||
- function() {
- (i[r].q = i[r].q || []).push(arguments)
- }, i[r].l = 1 * new Date();
- a = s.createElement(o),
- m = s.getElementsByTagName(o)[0];
- a.async = 1;
- a.src = g;
- m.parentNode.insertBefore(a, m)
- })(window, document, 'script', '//www.google-analytics.com/analytics.js', 'ga');
-
- ga('create', 'UA-75955621-1', 'auto');
- ga('send', 'pageview');
-
- </script>
</head>
<body class="archive tag tag-github tag-6" itemscope itemtype="http://schema.org/WebPage">
@@ -161,16 +143,16 @@
<nav role="navigation" itemscope itemtype="http://schema.org/SiteNavigationElement">
<ul id="menu-main-menu" class="nav top-nav cf">
<li id="menu-item-129" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-129">
- <a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight">Get Started</a>
+ <a target="_blank" href="https://github.com/Open-Network-Insight/apache-spot">Get Started</a>
</li>
<li id="menu-item-5" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-5">
- <a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight#if-you-want-all-of-the-oni-code-at-once-just-clone-it">Download</a>
+ <a target="_blank" href="https://github.com/Open-Network-Insight/apache-spot#if-you-want-all-of-the-oni-code-at-once-just-clone-it">Download</a>
</li>
<li id="menu-item-130" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-130">
- <a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight#contributing-to-oni">Contribute</a>
+ <a target="_blank" href="https://github.com/Open-Network-Insight/apache-spot#contributing-to-oni">Contribute</a>
</li>
<li id="menu-item-106" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-106">
- <a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight/wiki">Wiki</a>
+ <a target="_blank" href="https://github.com/Open-Network-Insight/apache-spot/wiki">Wiki</a>
</li>
<li id="menu-item-13" class="menu-item menu-item-type-post_type menu-item-object-page current_page_parent menu-item-13">
<a href="../../blog/">Blog</a>
@@ -192,11 +174,11 @@
<h1 class="page-title">Tag: github</h1>
- <article id="post-62" class="cf post-62 post type-post status-publish format-standard hentry category-security-analytics tag-github tag-open-network-insight tag-open-source" role="article">
+ <article id="post-62" class="cf post-62 post type-post status-publish format-standard hentry category-security-analytics tag-github tag-apache-spot tag-open-source" role="article">
<header class="entry-header article-header">
- <h3 class="h2 entry-title"><a href="../../open-network-insight-3-most-asked-questions/" rel="bookmark" title="Apache Spot: Three Most-Asked Questions">Apache Spot: Three Most-Asked Questions</a></h3>
+ <h3 class="h2 entry-title"><a href="../../apache-spot-3-most-asked-questions/" rel="bookmark" title="Apache Spot: Three Most-Asked Questions">Apache Spot (Incubating): Three Most-Asked Questions</a></h3>
<p class="byline entry-meta vcard">
<time class="updated entry-time" datetime="2016-03-29" itemprop="datePublished">
March 29, 2016
@@ -209,7 +191,7 @@
<section class="entry-content cf">
<p>
- While this is not the first blog post about Apache Spot, it is the first one by a creator of the solution. As a security data scientist in Intel’s Data Center Group, I joined a small team to start thinking about solving really hard problems in cloud analytics. The team grew, and out of that… <a class="excerpt-read-more" href="../../open-network-insight-3-most-asked-questions/" title="Read Apache Spot: Three Most-Asked Questions">Read more & [...]
+ While this is not the first blog post about Apache Spot, it is the first one by a creator of the solution. As a security data scientist in Intel’s Data Center Group, I joined a small team to start thinking about solving really hard problems in cloud analytics. The team grew, and out of that… <a class="excerpt-read-more" href="../../apache-spot-3-most-asked-questions/" title="Read Apache Spot: Three Most-Asked Questions">Read more »</a>
</p>
</section>
@@ -228,13 +210,16 @@
<h4 class="widgettitle">Recent Posts</h4>
<ul>
<li>
- <a href="../../open-network-insight-oni-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/">Apache Spot and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a>
+ <a href="../../jupyter-notebooks-for-data-analysis/">Jupyter Notebooks for Data Analysis</a>
+ </li>
+ <li>
+ <a href="../../apache-spot-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/">Apache Spot (Incubating) and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a>
</li>
<li>
- <a href="../../how-open-network-insight-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/">How Apache Spot Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a>
+ <a href="../../how-apache-spot-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/">How Apache Spot (Incubating) Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a>
</li>
<li>
- <a href="../../open-network-insight-3-most-asked-questions/">Apache Spot: Three Most-Asked Questions</a>
+ <a href="../../apache-spot-3-most-asked-questions/">Apache Spot (Incubating): 3 Most-Asked Questions</a>
</li>
</ul>
</div>
@@ -242,6 +227,9 @@
<h4 class="widgettitle">Archives</h4>
<ul>
<li>
+ <a href='../../2016/09/'>September 2016</a>
+ </li>
+ <li>
<a href='../../2016/08/'>August 2016</a>
</li>
<li>
diff --git a/tag/open-network-insight/feed/index.html b/tag/open-network-insight/feed/index.html
index e9db2f3..b30ebf2 100644
--- a/tag/open-network-insight/feed/index.html
+++ b/tag/open-network-insight/feed/index.html
@@ -8,8 +8,8 @@
>
<channel>
- <title>open network insight – Apache Spot</title>
- <atom:link href="http://spot.incubator.apache.org/tag/open-network-insight/feed/" rel="self" type="application/rss+xml" />
+ <title>Open Network Insight – Apache Spot</title>
+ <atom:link href="http://spot.incubator.apache.org/tag/apache-spot/feed/" rel="self" type="application/rss+xml" />
<link>http://spot.incubator.apache.org</link>
<description></description>
<lastBuildDate>Tue, 27 Sep 2016 18:37:15 +0000</lastBuildDate>
@@ -18,9 +18,9 @@
<sy:updateFrequency>1</sy:updateFrequency>
<item>
- <title>Apache Spot: Three Most-Asked Questions</title>
- <link>http://spot.incubator.apache.org/open-network-insight-3-most-asked-questions/</link>
- <comments>http://spot.incubator.apache.org/open-network-insight-3-most-asked-questions/#respond</comments>
+ <title>Apache Spot (Incubating): Three Most-Asked Questions</title>
+ <link>http://spot.incubator.apache.org/apache-spot-3-most-asked-questions/</link>
+ <comments>http://spot.incubator.apache.org/apache-spot-3-most-asked-questions/#respond</comments>
<pubDate>Tue, 29 Mar 2016 05:48:19 +0000</pubDate>
<dc:creator><![CDATA[oni-admin]]></dc:creator>
<category><![CDATA[Security Analytics]]></category>
@@ -29,7 +29,7 @@
<category><![CDATA[open source]]></category>
<guid isPermaLink="false">http://spot.incubator.apache.org/?p=62</guid>
- <description><![CDATA[While this is not the first blog post about Apache Spot, it is the first one by a creator of the solution. As a security data scientist in Intel’s Data Center Group, I joined a small team to start thinking about solving really hard problems in cloud analytics. The team grew, and out of that... <a class="excerpt-read-more" href="http://spot.incubator.apache.org/open-network-insight-3-most-asked-questions/" title="Read Apache Spot: Three Most-Asked Questions" [...]
+ <description><![CDATA[While this is not the first blog post about Apache Spot, it is the first one by a creator of the solution. As a security data scientist in Intel’s Data Center Group, I joined a small team to start thinking about solving really hard problems in cloud analytics. The team grew, and out of that... <a class="excerpt-read-more" href="http://spot.incubator.apache.org/apache-spot-3-most-asked-questions/" title="Read Apache Spot: Three Most-Asked Questions">Read mor [...]
<content:encoded><![CDATA[<p>While this is not the first blog post about Apache Spot, it is the first one by a creator of the solution. As a security data scientist in Intel’s Data Center Group, I joined a small team to start thinking about solving really hard problems in cloud analytics. The team grew, and out of that effort, came Apache Spot. Since we started talking about the project, these are the three questions I am asked the most.</p>
<p><strong>What Is Apache Spot?</strong><br />
Apache Spot is an open source, flow and packet analytics solution built on Hadoop. It combines big data processing, at-scale machine learning, and unique security analytics to put potential threats in front of defenders. While I am a data scientist today, I was a security investigator just a few years ago. I wanted to develop a solution that would put new tools and technology in play for defenders, but without requiring them to walk away from security and get a math degree.</p>
@@ -40,11 +40,11 @@ Apache Spot is an open source, flow and packet analytics solution built on Hadoo
<p>At Intel, I have worked in IT, for a security product company (McAfee), and in the Data Center Group. Intel IT was an early pioneer of the concept of proactive investigations to protect intellectual property. McAfee (now Intel Security Group) has a broad customer base in the realms of network, endpoint, and content security, to name only a few. And the Intel Data Center group has strategic partnerships with Cloudera and Accenture, as well as some pretty cool analytics efforts of their [...]
<p><strong>Why Open Source?</strong></p>
<p>I learned from my earlier efforts in security analytics, that to invite collaboration from academia, the public sector, and the private sector, open source software is an excellent choice. We are now seeking to build a community of developers, data scientists, and security enthusiasts to grow Apache Spot into something we can all be proud of. We have also chosen an Apache software license, so that it can enrich commercial software offerings as well.</p>
-<p>The greatest thing for me since we announced at RSA is to hear OTHER people talk about Apache Spot (formerly Open Network Insight or ONI), here are some of my favorites, from <a href="http://vision.cloudera.com/open-network-insight-changing-infosec-data-science-forever/" target="_blank">a Data Scientist @ eBay </a>, <a href="https://newsroom.accenture.com/news/accenture-introduces-the-accenture-cyber-intelligence-platform-to-help-organizations-continuously-predict-detect-and-combat-cy [...]
+<p>The greatest thing for me since we announced at RSA is to hear OTHER people talk about Apache Spot (formerly Open Network Insight or ONI), here are some of my favorites, from <a href="http://vision.cloudera.com/apache-spot-changing-infosec-data-science-forever/" target="_blank">a Data Scientist @ eBay </a>, <a href="https://newsroom.accenture.com/news/accenture-introduces-the-accenture-cyber-intelligence-platform-to-help-organizations-continuously-predict-detect-and-combat-cyber-attac [...]
<p>Fork us on Github!</p>
<p>Grant Babb</p>
]]></content:encoded>
- <wfw:commentRss>http://spot.incubator.apache.org/open-network-insight-3-most-asked-questions/feed/</wfw:commentRss>
+ <wfw:commentRss>http://spot.incubator.apache.org/apache-spot-3-most-asked-questions/feed/</wfw:commentRss>
<slash:comments>0</slash:comments>
</item>
</channel>
diff --git a/tag/open-network-insight/index.html b/tag/open-network-insight/index.html
index 0b20aed..64c6c02 100644
--- a/tag/open-network-insight/index.html
+++ b/tag/open-network-insight/index.html
@@ -12,7 +12,7 @@
<meta http-equiv="X-UA-Compatible" content="IE=edge">
- <title>Apache Spot open network insight</title>
+ <title>Open Network Insight</title>
<meta name="HandheldFriendly" content="True">
<meta name="MobileOptimized" content="320">
@@ -33,7 +33,7 @@
<link rel='dns-prefetch' href='//s.w.org' />
<link rel="alternate" type="application/rss+xml" title="Apache Spot » Feed" href="../../feed/" />
<link rel="alternate" type="application/rss+xml" title="Apache Spot » Comments Feed" href="../../comments/feed/" />
- <link rel="alternate" type="application/rss+xml" title="Apache Spot » open network insight Tag Feed" href="../../tag/open-network-insight/feed/" />
+ <link rel="alternate" type="application/rss+xml" title="Apache Spot » open network insight Tag Feed" href="../../tag/apache-spot/feed/" />
<script type="text/javascript">
window._wpemojiSettings = {
"baseUrl" : "https:\/\/s.w.org\/images\/core\/emoji\/2\/72x72\/",
@@ -126,27 +126,9 @@
<script type='text/javascript' src='../../wp-content/themes/oni/library/js/jquery.meanmenu.js'></script>
<link rel='https://api.w.org/' href='../../wp-json/' />
- <script>
- (function(i, s, o, g, r, a, m) {
- i['GoogleAnalyticsObject'] = r;
- i[r] = i[r] ||
- function() {
- (i[r].q = i[r].q || []).push(arguments)
- }, i[r].l = 1 * new Date();
- a = s.createElement(o),
- m = s.getElementsByTagName(o)[0];
- a.async = 1;
- a.src = g;
- m.parentNode.insertBefore(a, m)
- })(window, document, 'script', '//www.google-analytics.com/analytics.js', 'ga');
-
- ga('create', 'UA-75955621-1', 'auto');
- ga('send', 'pageview');
-
- </script>
</head>
- <body class="archive tag tag-open-network-insight tag-7" itemscope itemtype="http://schema.org/WebPage">
+ <body class="archive tag tag-apache-spot tag-7" itemscope itemtype="http://schema.org/WebPage">
<div id="container">
@@ -161,16 +143,16 @@
<nav role="navigation" itemscope itemtype="http://schema.org/SiteNavigationElement">
<ul id="menu-main-menu" class="nav top-nav cf">
<li id="menu-item-129" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-129">
- <a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight">Get Started</a>
+ <a target="_blank" href="https://github.com/Open-Network-Insight/apache-spot">Get Started</a>
</li>
<li id="menu-item-5" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-5">
- <a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight#if-you-want-all-of-the-oni-code-at-once-just-clone-it">Download</a>
+ <a target="_blank" href="https://github.com/Open-Network-Insight/apache-spot#if-you-want-all-of-the-oni-code-at-once-just-clone-it">Download</a>
</li>
<li id="menu-item-130" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-130">
- <a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight#contributing-to-oni">Contribute</a>
+ <a target="_blank" href="https://github.com/Open-Network-Insight/apache-spot#contributing-to-oni">Contribute</a>
</li>
<li id="menu-item-106" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-106">
- <a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight/wiki">Wiki</a>
+ <a target="_blank" href="https://github.com/Open-Network-Insight/apache-spot/wiki">Wiki</a>
</li>
<li id="menu-item-13" class="menu-item menu-item-type-post_type menu-item-object-page current_page_parent menu-item-13">
<a href="../../blog/">Blog</a>
@@ -192,11 +174,11 @@
<h1 class="page-title">Tag: open network insight</h1>
- <article id="post-62" class="cf post-62 post type-post status-publish format-standard hentry category-security-analytics tag-github tag-open-network-insight tag-open-source" role="article">
+ <article id="post-62" class="cf post-62 post type-post status-publish format-standard hentry category-security-analytics tag-github tag-apache-spot tag-open-source" role="article">
<header class="entry-header article-header">
- <h3 class="h2 entry-title"><a href="../../open-network-insight-3-most-asked-questions/" rel="bookmark" title="Apache Spot: Three Most-Asked Questions">Apache Spot: Three Most-Asked Questions</a></h3>
+ <h3 class="h2 entry-title"><a href="../../apache-spot-3-most-asked-questions/" rel="bookmark" title="Apache Spot: Three Most-Asked Questions">Apache Spot (Incubating): Three Most-Asked Questions</a></h3>
<p class="byline entry-meta vcard">
<time class="updated entry-time" datetime="2016-03-29" itemprop="datePublished">
March 29, 2016
@@ -209,7 +191,7 @@
<section class="entry-content cf">
<p>
- While this is not the first blog post about Apache Spot, it is the first one by a creator of the solution. As a security data scientist in Intel’s Data Center Group, I joined a small team to start thinking about solving really hard problems in cloud analytics. The team grew, and out of that… <a class="excerpt-read-more" href="../../open-network-insight-3-most-asked-questions/" title="Read Apache Spot: Three Most-Asked Questions">Read more & [...]
+ While this is not the first blog post about Apache Spot, it is the first one by a creator of the solution. As a security data scientist in Intel’s Data Center Group, I joined a small team to start thinking about solving really hard problems in cloud analytics. The team grew, and out of that… <a class="excerpt-read-more" href="../../apache-spot-3-most-asked-questions/" title="Read Apache Spot: Three Most-Asked Questions">Read more »</a>
</p>
</section>
@@ -228,13 +210,16 @@
<h4 class="widgettitle">Recent Posts</h4>
<ul>
<li>
- <a href="../../open-network-insight-oni-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/">Apache Spot and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a>
+ <a href="../../jupyter-notebooks-for-data-analysis/">Jupyter Notebooks for Data Analysis</a>
+ </li>
+ <li>
+ <a href="../../apache-spot-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/">Apache Spot (Incubating) and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a>
</li>
<li>
- <a href="../../how-open-network-insight-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/">How Apache Spot Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a>
+ <a href="../../how-apache-spot-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/">How Apache Spot (Incubating) Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a>
</li>
<li>
- <a href="../../open-network-insight-3-most-asked-questions/">Apache Spot: Three Most-Asked Questions</a>
+ <a href="../../apache-spot-3-most-asked-questions/">Apache Spot (Incubating): 3 Most-Asked Questions</a>
</li>
</ul>
</div>
@@ -242,6 +227,9 @@
<h4 class="widgettitle">Archives</h4>
<ul>
<li>
+ <a href='../../2016/09/'>September 2016</a>
+ </li>
+ <li>
<a href='../../2016/08/'>August 2016</a>
</li>
<li>
diff --git a/tag/open-source/feed/index.html b/tag/open-source/feed/index.html
index a971588..42cd1c0 100644
--- a/tag/open-source/feed/index.html
+++ b/tag/open-source/feed/index.html
@@ -18,7 +18,7 @@
<sy:updateFrequency>1</sy:updateFrequency>
<item>
- <title>Apache Spot: Three Most-Asked Questions</title>
+ <title>Apache Spot (Incubating): Three Most-Asked Questions</title>
<link>http://spot.incubator.apache.org/open-network-insight-3-most-asked-questions/</link>
<comments>http://spot.incubator.apache.org/open-network-insight-3-most-asked-questions/#respond</comments>
<pubDate>Tue, 29 Mar 2016 05:48:19 +0000</pubDate>
diff --git a/tag/open-source/index.html b/tag/open-source/index.html
index aae3a92..8cb018f 100644
--- a/tag/open-source/index.html
+++ b/tag/open-source/index.html
@@ -126,24 +126,6 @@
<script type='text/javascript' src='../../wp-content/themes/oni/library/js/jquery.meanmenu.js'></script>
<link rel='https://api.w.org/' href='../../wp-json/' />
- <script>
- (function(i, s, o, g, r, a, m) {
- i['GoogleAnalyticsObject'] = r;
- i[r] = i[r] ||
- function() {
- (i[r].q = i[r].q || []).push(arguments)
- }, i[r].l = 1 * new Date();
- a = s.createElement(o),
- m = s.getElementsByTagName(o)[0];
- a.async = 1;
- a.src = g;
- m.parentNode.insertBefore(a, m)
- })(window, document, 'script', '//www.google-analytics.com/analytics.js', 'ga');
-
- ga('create', 'UA-75955621-1', 'auto');
- ga('send', 'pageview');
-
- </script>
</head>
<body class="archive tag tag-open-source tag-5" itemscope itemtype="http://schema.org/WebPage">
@@ -161,16 +143,16 @@
<nav role="navigation" itemscope itemtype="http://schema.org/SiteNavigationElement">
<ul id="menu-main-menu" class="nav top-nav cf">
<li id="menu-item-129" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-129">
- <a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight">Get Started</a>
+ <a target="_blank" href="https://github.com/Open-Network-Insight/apache-spot">Get Started</a>
</li>
<li id="menu-item-5" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-5">
- <a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight#if-you-want-all-of-the-oni-code-at-once-just-clone-it">Download</a>
+ <a target="_blank" href="https://github.com/Open-Network-Insight/apache-spot#if-you-want-all-of-the-oni-code-at-once-just-clone-it">Download</a>
</li>
<li id="menu-item-130" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-130">
- <a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight#contributing-to-oni">Contribute</a>
+ <a target="_blank" href="https://github.com/Open-Network-Insight/apache-spot#contributing-to-oni">Contribute</a>
</li>
<li id="menu-item-106" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-106">
- <a target="_blank" href="https://github.com/Open-Network-Insight/open-network-insight/wiki">Wiki</a>
+ <a target="_blank" href="https://github.com/Open-Network-Insight/apache-spot/wiki">Wiki</a>
</li>
<li id="menu-item-13" class="menu-item menu-item-type-post_type menu-item-object-page current_page_parent menu-item-13">
<a href="../../blog/">Blog</a>
@@ -192,11 +174,11 @@
<h1 class="page-title">Tag: open source</h1>
- <article id="post-62" class="cf post-62 post type-post status-publish format-standard hentry category-security-analytics tag-github tag-open-network-insight tag-open-source" role="article">
+ <article id="post-62" class="cf post-62 post type-post status-publish format-standard hentry category-security-analytics tag-github tag-apache-spot tag-open-source" role="article">
<header class="entry-header article-header">
- <h3 class="h2 entry-title"><a href="../../open-network-insight-3-most-asked-questions/" rel="bookmark" title="Apache Spot: Three Most-Asked Questions">Apache Spot: Three Most-Asked Questions</a></h3>
+ <h3 class="h2 entry-title"><a href="../../apache-spot-3-most-asked-questions/" rel="bookmark" title="Apache Spot: Three Most-Asked Questions">Apache Spot (Incubating): Three Most-Asked Questions</a></h3>
<p class="byline entry-meta vcard">
<time class="updated entry-time" datetime="2016-03-29" itemprop="datePublished">
March 29, 2016
@@ -209,7 +191,7 @@
<section class="entry-content cf">
<p>
- While this is not the first blog post about Apache Spot, it is the first one by a creator of the solution. As a security data scientist in Intel’s Data Center Group, I joined a small team to start thinking about solving really hard problems in cloud analytics. The team grew, and out of that… <a class="excerpt-read-more" href="../../open-network-insight-3-most-asked-questions/" title="Read Apache Spot: Three Most-Asked Questions">Read more & [...]
+ While this is not the first blog post about Apache Spot, it is the first one by a creator of the solution. As a security data scientist in Intel’s Data Center Group, I joined a small team to start thinking about solving really hard problems in cloud analytics. The team grew, and out of that… <a class="excerpt-read-more" href="../../apache-spot-3-most-asked-questions/" title="Read Apache Spot: Three Most-Asked Questions">Read more »</a>
</p>
</section>
@@ -228,13 +210,16 @@
<h4 class="widgettitle">Recent Posts</h4>
<ul>
<li>
- <a href="../../open-network-insight-oni-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/">Apache Spot and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a>
+ <a href="../../jupyter-notebooks-for-data-analysis/">Jupyter Notebooks for Data Analysis</a>
+ </li>
+ <li>
+ <a href="../../apache-spot-and-cybersecurity-using-netflows-to-detect-threats-to-critical-infrastructure/">Apache Spot (Incubating) and Cybersecurity — Using NetFlows to Detect Threats to Critical Infrastructure</a>
</li>
<li>
- <a href="../../how-open-network-insight-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/">How Apache Spot Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a>
+ <a href="../../how-apache-spot-helps-create-well-stocked-data-lakes-and-catch-powerful-insights/">How Apache Spot (Incubating) Helps Create Well-Stocked Data Lakes and Catch Powerful Insights</a>
</li>
<li>
- <a href="../../open-network-insight-3-most-asked-questions/">Apache Spot: Three Most-Asked Questions</a>
+ <a href="../../apache-spot-3-most-asked-questions/">Apache Spot (Incubating): 3 Most-Asked Questions</a>
</li>
</ul>
</div>
@@ -242,6 +227,9 @@
<h4 class="widgettitle">Archives</h4>
<ul>
<li>
+ <a href='../../2016/09/'>September 2016</a>
+ </li>
+ <li>
<a href='../../2016/08/'>August 2016</a>
</li>
<li>
diff --git a/wp-content/themes/oni/favicon.ico b/wp-content/themes/oni/favicon.ico
new file mode 100644
index 0000000..b04db3c
Binary files /dev/null and b/wp-content/themes/oni/favicon.ico differ
diff --git a/wp-content/themes/oni/favicon.png b/wp-content/themes/oni/favicon.png
index e68d33b..1e360dd 100644
Binary files a/wp-content/themes/oni/favicon.png and b/wp-content/themes/oni/favicon.png differ
diff --git a/wp-content/themes/oni/library/css/style.css b/wp-content/themes/oni/library/css/style.css
index 92851fd..b8dd11e 100644
--- a/wp-content/themes/oni/library/css/style.css
+++ b/wp-content/themes/oni/library/css/style.css
@@ -739,6 +739,8 @@ POSTS & CONTENT STYLES
.social-icons a {font-size:2em;}
.social-icons a:hover {color:#fff;}
.social-icons a:first-child {margin-right:15px;}
+.add-to-slack {margin:0 0 0 15px;}
+.add-to-slack {position:relative;top:5px;}
#content { margin-top: 2em; }
diff --git a/wp-content/themes/oni/library/images/add-to-slack.png b/wp-content/themes/oni/library/images/add-to-slack.png
new file mode 100644
index 0000000..93649ed
Binary files /dev/null and b/wp-content/themes/oni/library/images/add-to-slack.png differ
diff --git a/wp-content/themes/oni/library/images/apple-touch-icon.png b/wp-content/themes/oni/library/images/apple-touch-icon.png
index 6c9b524..b561378 100644
Binary files a/wp-content/themes/oni/library/images/apple-touch-icon.png and b/wp-content/themes/oni/library/images/apple-touch-icon.png differ
diff --git a/wp-content/themes/oni/library/images/win8-tile-icon.png b/wp-content/themes/oni/library/images/win8-tile-icon.png
new file mode 100644
index 0000000..2392bed
Binary files /dev/null and b/wp-content/themes/oni/library/images/win8-tile-icon.png differ
diff --git a/wp-content/uploads/2016/03/advantages-1.png b/wp-content/uploads/2016/03/advantages-1.png
index 2487375..e266dfa 100644
Binary files a/wp-content/uploads/2016/03/advantages-1.png and b/wp-content/uploads/2016/03/advantages-1.png differ
diff --git a/wp-content/uploads/2016/03/how-it-works.png b/wp-content/uploads/2016/03/how-it-works.png
index 14c5ced..798ac8f 100644
Binary files a/wp-content/uploads/2016/03/how-it-works.png and b/wp-content/uploads/2016/03/how-it-works.png differ
diff --git a/wp-content/uploads/2016/09/ONI_Architecture-Diagram_1300_v4.png b/wp-content/uploads/2016/09/ONI_Architecture-Diagram_1300_v4.png
index edb900a..4034c7d 100644
Binary files a/wp-content/uploads/2016/09/ONI_Architecture-Diagram_1300_v4.png and b/wp-content/uploads/2016/09/ONI_Architecture-Diagram_1300_v4.png differ
diff --git a/wp-content/uploads/2016/09/ONI_Architecture-Diagram_1300_v4.png b/wp-content/uploads/2016/09/ONI_Architecture-Diagram_1300_v4_old.png
similarity index 100%
copy from wp-content/uploads/2016/09/ONI_Architecture-Diagram_1300_v4.png
copy to wp-content/uploads/2016/09/ONI_Architecture-Diagram_1300_v4_old.png
diff --git a/wp-content/uploads/2016/10/iPython-1-300x89.png b/wp-content/uploads/2016/10/iPython-1-300x89.png
new file mode 100644
index 0000000..a17760c
Binary files /dev/null and b/wp-content/uploads/2016/10/iPython-1-300x89.png differ
diff --git a/wp-content/uploads/2016/10/iPython-1.png b/wp-content/uploads/2016/10/iPython-1.png
new file mode 100644
index 0000000..80b8ea7
Binary files /dev/null and b/wp-content/uploads/2016/10/iPython-1.png differ