You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spot.apache.org by le...@apache.org on 2017/06/23 22:40:55 UTC

[5/5] incubator-spot git commit: Added advanced mode notebooks for proxy and dns

Added advanced mode notebooks for proxy and dns


Project: http://git-wip-us.apache.org/repos/asf/incubator-spot/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spot/commit/74a61880
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spot/tree/74a61880
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spot/diff/74a61880

Branch: refs/heads/apache/SPOT-35_graphql_api
Commit: 74a61880f31d5610832c2189bac85d7336049fa8
Parents: 2efa3f5
Author: LedaLima <le...@apache.org>
Authored: Thu Jun 22 13:02:27 2017 -0500
Committer: LedaLima <ga...@intel.com>
Committed: Fri Jun 23 17:39:07 2017 -0500

----------------------------------------------------------------------
 .../ipynb_templates/Advanced_Mode_master.ipynb  | 256 +++++++++++++++++++
 .../ipynb_templates/Advanced_Mode_master.ipynb  | 256 +++++++++++++++++++
 2 files changed, 512 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-spot/blob/74a61880/spot-oa/oa/dns/ipynb_templates/Advanced_Mode_master.ipynb
----------------------------------------------------------------------
diff --git a/spot-oa/oa/dns/ipynb_templates/Advanced_Mode_master.ipynb b/spot-oa/oa/dns/ipynb_templates/Advanced_Mode_master.ipynb
new file mode 100644
index 0000000..423152a
--- /dev/null
+++ b/spot-oa/oa/dns/ipynb_templates/Advanced_Mode_master.ipynb
@@ -0,0 +1,256 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Apache Spot's Ipython Advanced Mode\n",
+    "## DNS\n",
+    "\n",
+    "This guide provides examples about how to request data, show data with some cool libraries like pandas and more.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**Import Libraries**\n",
+    "\n",
+    "The next cell will import the necessary libraries to execute the functions. Do not remove"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "import datetime\n",
+    "import pandas as pd\n",
+    "import numpy as np\n",
+    "import linecache, bisect\n",
+    "import os\n",
+    "\n",
+    "spath = os.getcwd()\n",
+    "path = spath.split(\"/\")\n",
+    "date = path[len(path)-1]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**Request Data**\n",
+    "\n",
+    "In order to request data we are using Graphql (a query language for APIs, more info at: http://graphql.org/).\n",
+    "\n",
+    "We provide the function to make a data request, all you need is a query and variables\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "def makeGraphqlRequest(query, variables):\n",
+    "    return GraphQLClient.request(query, variables)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now that we have a function, we can run a query like this:\n",
+    "\n",
+    "*Note: There's no need to manually set the date for the query, by default the code will read the date from the current path"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "suspicious_query = \"\"\"query($date:SpotDateType) {\n",
+    "                            dns {\n",
+    "                              suspicious(date:$date)\n",
+    "                                  { clientIp\n",
+    "                                    clientIpSev\n",
+    "                                    dnsQuery\n",
+    "                                    dnsQueryClass\n",
+    "                                    dnsQueryClassLabel\n",
+    "                                    dnsQueryRcode\n",
+    "                                    dnsQueryRcodeLabel\n",
+    "                                    dnsQueryRep\n",
+    "                                    dnsQuerySev\n",
+    "                                    dnsQueryType\n",
+    "                                    dnsQueryTypeLabel\n",
+    "                                    frameLength\n",
+    "                                    frameTime\n",
+    "                                    networkContext\n",
+    "                                    score\n",
+    "                                    tld\n",
+    "                                    unixTimestamp                                 \n",
+    "                                  }\n",
+    "                            }\n",
+    "                    }\"\"\"\n",
+    "\n",
+    "##If you want to use a different date for your query, switch the \n",
+    "##commented/uncommented following lines\n",
+    "\n",
+    "variables={\n",
+    "    'date': datetime.datetime.strptime(date, '%Y%m%d').strftime('%Y-%m-%d')\n",
+    "#     'date': \"2016-10-08\"\n",
+    "    }\n",
+    " \n",
+    "suspicious_request = makeGraphqlRequest(suspicious_query,variables)\n",
+    "\n",
+    "##The variable suspicious_request will contain the resulting data from the query.\n",
+    "results = suspicious_request['data']['dns']['suspicious']\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "##Pandas Dataframes\n",
+    "\n",
+    "The following cell loads the results into a pandas dataframe\n",
+    "\n",
+    "For more information on how to use pandas, you can learn more here: https://pandas.pydata.org/pandas-docs/stable/10min.html"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "df = pd.read_json(json.dumps(results))\n",
+    "##Printing only the selected column list from the dataframe\n",
+    "##Unless specified otherwise, \n",
+    "print df[['clientIp', 'unixTimestamp','tld', 'dnsQuery','dnsQueryRcode','dnsQueryRcodeLabel']]\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "##Additional operations \n",
+    "\n",
+    "Additional operations can be performed on the dataframe like sorting the data, filtering it and grouping it\n",
+    "\n",
+    "**Filtering the data**"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "##Filter results where the destination port = 3389\n",
+    "##The resulting data will be stored in df2 \n",
+    "\n",
+    "df2 = df[df['tld'].isin(['sjc04-login.dotomi.com'])]\n",
+    "print df2[['clientIp', 'unixTimestamp','tld', 'dnsQuery','dnsQueryRcode','dnsQueryRcodeLabel']]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**Ordering the data**"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false,
+    "scrolled": true
+   },
+   "outputs": [],
+   "source": [
+    "srtd = df.sort_values(by=\"tld\")\n",
+    "print srtd[['clientIp', 'unixTimestamp','tld', 'dnsQuery','dnsQueryRcode','dnsQueryRcodeLabel']]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**Grouping the data**"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "## This command will group the results by pairs of source-destination IP\n",
+    "## summarizing all other columns \n",
+    "grpd = df.groupby(['clientIp','tld']).count()\n",
+    "## This will print the resulting dataframe displaying the input and output bytes columnns\n",
+    "print grpd[[\"dnsQuery\"]]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "##Sandbox\n",
+    "\n",
+    "At this point you can perform your own analysis using the previously provided functions as a guide.\n",
+    "\n",
+    "Happy threat hunting!"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "#Your code here"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 2",
+   "language": "python",
+   "name": "python2"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 2
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython2",
+   "version": "2.7.5"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}

http://git-wip-us.apache.org/repos/asf/incubator-spot/blob/74a61880/spot-oa/oa/proxy/ipynb_templates/Advanced_Mode_master.ipynb
----------------------------------------------------------------------
diff --git a/spot-oa/oa/proxy/ipynb_templates/Advanced_Mode_master.ipynb b/spot-oa/oa/proxy/ipynb_templates/Advanced_Mode_master.ipynb
new file mode 100644
index 0000000..423152a
--- /dev/null
+++ b/spot-oa/oa/proxy/ipynb_templates/Advanced_Mode_master.ipynb
@@ -0,0 +1,256 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Apache Spot's Ipython Advanced Mode\n",
+    "## DNS\n",
+    "\n",
+    "This guide provides examples about how to request data, show data with some cool libraries like pandas and more.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**Import Libraries**\n",
+    "\n",
+    "The next cell will import the necessary libraries to execute the functions. Do not remove"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "import datetime\n",
+    "import pandas as pd\n",
+    "import numpy as np\n",
+    "import linecache, bisect\n",
+    "import os\n",
+    "\n",
+    "spath = os.getcwd()\n",
+    "path = spath.split(\"/\")\n",
+    "date = path[len(path)-1]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**Request Data**\n",
+    "\n",
+    "In order to request data we are using Graphql (a query language for APIs, more info at: http://graphql.org/).\n",
+    "\n",
+    "We provide the function to make a data request, all you need is a query and variables\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "def makeGraphqlRequest(query, variables):\n",
+    "    return GraphQLClient.request(query, variables)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now that we have a function, we can run a query like this:\n",
+    "\n",
+    "*Note: There's no need to manually set the date for the query, by default the code will read the date from the current path"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "suspicious_query = \"\"\"query($date:SpotDateType) {\n",
+    "                            dns {\n",
+    "                              suspicious(date:$date)\n",
+    "                                  { clientIp\n",
+    "                                    clientIpSev\n",
+    "                                    dnsQuery\n",
+    "                                    dnsQueryClass\n",
+    "                                    dnsQueryClassLabel\n",
+    "                                    dnsQueryRcode\n",
+    "                                    dnsQueryRcodeLabel\n",
+    "                                    dnsQueryRep\n",
+    "                                    dnsQuerySev\n",
+    "                                    dnsQueryType\n",
+    "                                    dnsQueryTypeLabel\n",
+    "                                    frameLength\n",
+    "                                    frameTime\n",
+    "                                    networkContext\n",
+    "                                    score\n",
+    "                                    tld\n",
+    "                                    unixTimestamp                                 \n",
+    "                                  }\n",
+    "                            }\n",
+    "                    }\"\"\"\n",
+    "\n",
+    "##If you want to use a different date for your query, switch the \n",
+    "##commented/uncommented following lines\n",
+    "\n",
+    "variables={\n",
+    "    'date': datetime.datetime.strptime(date, '%Y%m%d').strftime('%Y-%m-%d')\n",
+    "#     'date': \"2016-10-08\"\n",
+    "    }\n",
+    " \n",
+    "suspicious_request = makeGraphqlRequest(suspicious_query,variables)\n",
+    "\n",
+    "##The variable suspicious_request will contain the resulting data from the query.\n",
+    "results = suspicious_request['data']['dns']['suspicious']\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "##Pandas Dataframes\n",
+    "\n",
+    "The following cell loads the results into a pandas dataframe\n",
+    "\n",
+    "For more information on how to use pandas, you can learn more here: https://pandas.pydata.org/pandas-docs/stable/10min.html"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "df = pd.read_json(json.dumps(results))\n",
+    "##Printing only the selected column list from the dataframe\n",
+    "##Unless specified otherwise, \n",
+    "print df[['clientIp', 'unixTimestamp','tld', 'dnsQuery','dnsQueryRcode','dnsQueryRcodeLabel']]\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "##Additional operations \n",
+    "\n",
+    "Additional operations can be performed on the dataframe like sorting the data, filtering it and grouping it\n",
+    "\n",
+    "**Filtering the data**"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "##Filter results where the destination port = 3389\n",
+    "##The resulting data will be stored in df2 \n",
+    "\n",
+    "df2 = df[df['tld'].isin(['sjc04-login.dotomi.com'])]\n",
+    "print df2[['clientIp', 'unixTimestamp','tld', 'dnsQuery','dnsQueryRcode','dnsQueryRcodeLabel']]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**Ordering the data**"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false,
+    "scrolled": true
+   },
+   "outputs": [],
+   "source": [
+    "srtd = df.sort_values(by=\"tld\")\n",
+    "print srtd[['clientIp', 'unixTimestamp','tld', 'dnsQuery','dnsQueryRcode','dnsQueryRcodeLabel']]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**Grouping the data**"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "## This command will group the results by pairs of source-destination IP\n",
+    "## summarizing all other columns \n",
+    "grpd = df.groupby(['clientIp','tld']).count()\n",
+    "## This will print the resulting dataframe displaying the input and output bytes columnns\n",
+    "print grpd[[\"dnsQuery\"]]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "##Sandbox\n",
+    "\n",
+    "At this point you can perform your own analysis using the previously provided functions as a guide.\n",
+    "\n",
+    "Happy threat hunting!"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "#Your code here"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 2",
+   "language": "python",
+   "name": "python2"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 2
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython2",
+   "version": "2.7.5"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}