You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spot.apache.org by le...@apache.org on 2017/06/23 22:40:55 UTC
[5/5] incubator-spot git commit: Added advanced mode notebooks for
proxy and dns
Added advanced mode notebooks for proxy and dns
Project: http://git-wip-us.apache.org/repos/asf/incubator-spot/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spot/commit/74a61880
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spot/tree/74a61880
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spot/diff/74a61880
Branch: refs/heads/apache/SPOT-35_graphql_api
Commit: 74a61880f31d5610832c2189bac85d7336049fa8
Parents: 2efa3f5
Author: LedaLima <le...@apache.org>
Authored: Thu Jun 22 13:02:27 2017 -0500
Committer: LedaLima <ga...@intel.com>
Committed: Fri Jun 23 17:39:07 2017 -0500
----------------------------------------------------------------------
.../ipynb_templates/Advanced_Mode_master.ipynb | 256 +++++++++++++++++++
.../ipynb_templates/Advanced_Mode_master.ipynb | 256 +++++++++++++++++++
2 files changed, 512 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spot/blob/74a61880/spot-oa/oa/dns/ipynb_templates/Advanced_Mode_master.ipynb
----------------------------------------------------------------------
diff --git a/spot-oa/oa/dns/ipynb_templates/Advanced_Mode_master.ipynb b/spot-oa/oa/dns/ipynb_templates/Advanced_Mode_master.ipynb
new file mode 100644
index 0000000..423152a
--- /dev/null
+++ b/spot-oa/oa/dns/ipynb_templates/Advanced_Mode_master.ipynb
@@ -0,0 +1,256 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Apache Spot's Ipython Advanced Mode\n",
+ "## DNS\n",
+ "\n",
+ "This guide provides examples about how to request data, show data with some cool libraries like pandas and more.\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "**Import Libraries**\n",
+ "\n",
+ "The next cell will import the necessary libraries to execute the functions. Do not remove"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "import datetime\n",
+ "import pandas as pd\n",
+ "import numpy as np\n",
+ "import linecache, bisect\n",
+ "import os\n",
+ "\n",
+ "spath = os.getcwd()\n",
+ "path = spath.split(\"/\")\n",
+ "date = path[len(path)-1]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "**Request Data**\n",
+ "\n",
+ "In order to request data we are using Graphql (a query language for APIs, more info at: http://graphql.org/).\n",
+ "\n",
+ "We provide the function to make a data request, all you need is a query and variables\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "def makeGraphqlRequest(query, variables):\n",
+ " return GraphQLClient.request(query, variables)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now that we have a function, we can run a query like this:\n",
+ "\n",
+ "*Note: There's no need to manually set the date for the query, by default the code will read the date from the current path"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "suspicious_query = \"\"\"query($date:SpotDateType) {\n",
+ " dns {\n",
+ " suspicious(date:$date)\n",
+ " { clientIp\n",
+ " clientIpSev\n",
+ " dnsQuery\n",
+ " dnsQueryClass\n",
+ " dnsQueryClassLabel\n",
+ " dnsQueryRcode\n",
+ " dnsQueryRcodeLabel\n",
+ " dnsQueryRep\n",
+ " dnsQuerySev\n",
+ " dnsQueryType\n",
+ " dnsQueryTypeLabel\n",
+ " frameLength\n",
+ " frameTime\n",
+ " networkContext\n",
+ " score\n",
+ " tld\n",
+ " unixTimestamp \n",
+ " }\n",
+ " }\n",
+ " }\"\"\"\n",
+ "\n",
+ "##If you want to use a different date for your query, switch the \n",
+ "##commented/uncommented following lines\n",
+ "\n",
+ "variables={\n",
+ " 'date': datetime.datetime.strptime(date, '%Y%m%d').strftime('%Y-%m-%d')\n",
+ "# 'date': \"2016-10-08\"\n",
+ " }\n",
+ " \n",
+ "suspicious_request = makeGraphqlRequest(suspicious_query,variables)\n",
+ "\n",
+ "##The variable suspicious_request will contain the resulting data from the query.\n",
+ "results = suspicious_request['data']['dns']['suspicious']\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "##Pandas Dataframes\n",
+ "\n",
+ "The following cell loads the results into a pandas dataframe\n",
+ "\n",
+ "For more information on how to use pandas, you can learn more here: https://pandas.pydata.org/pandas-docs/stable/10min.html"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "df = pd.read_json(json.dumps(results))\n",
+ "##Printing only the selected column list from the dataframe\n",
+ "##Unless specified otherwise, \n",
+ "print df[['clientIp', 'unixTimestamp','tld', 'dnsQuery','dnsQueryRcode','dnsQueryRcodeLabel']]\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "##Additional operations \n",
+ "\n",
+ "Additional operations can be performed on the dataframe like sorting the data, filtering it and grouping it\n",
+ "\n",
+ "**Filtering the data**"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "##Filter results where the destination port = 3389\n",
+ "##The resulting data will be stored in df2 \n",
+ "\n",
+ "df2 = df[df['tld'].isin(['sjc04-login.dotomi.com'])]\n",
+ "print df2[['clientIp', 'unixTimestamp','tld', 'dnsQuery','dnsQueryRcode','dnsQueryRcodeLabel']]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "**Ordering the data**"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": false,
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "srtd = df.sort_values(by=\"tld\")\n",
+ "print srtd[['clientIp', 'unixTimestamp','tld', 'dnsQuery','dnsQueryRcode','dnsQueryRcodeLabel']]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "**Grouping the data**"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "## This command will group the results by pairs of source-destination IP\n",
+ "## summarizing all other columns \n",
+ "grpd = df.groupby(['clientIp','tld']).count()\n",
+ "## This will print the resulting dataframe displaying the input and output bytes columnns\n",
+ "print grpd[[\"dnsQuery\"]]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "##Sandbox\n",
+ "\n",
+ "At this point you can perform your own analysis using the previously provided functions as a guide.\n",
+ "\n",
+ "Happy threat hunting!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "#Your code here"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 2",
+ "language": "python",
+ "name": "python2"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 2
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython2",
+ "version": "2.7.5"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}
http://git-wip-us.apache.org/repos/asf/incubator-spot/blob/74a61880/spot-oa/oa/proxy/ipynb_templates/Advanced_Mode_master.ipynb
----------------------------------------------------------------------
diff --git a/spot-oa/oa/proxy/ipynb_templates/Advanced_Mode_master.ipynb b/spot-oa/oa/proxy/ipynb_templates/Advanced_Mode_master.ipynb
new file mode 100644
index 0000000..423152a
--- /dev/null
+++ b/spot-oa/oa/proxy/ipynb_templates/Advanced_Mode_master.ipynb
@@ -0,0 +1,256 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Apache Spot's Ipython Advanced Mode\n",
+ "## DNS\n",
+ "\n",
+ "This guide provides examples about how to request data, show data with some cool libraries like pandas and more.\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "**Import Libraries**\n",
+ "\n",
+ "The next cell will import the necessary libraries to execute the functions. Do not remove"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "import datetime\n",
+ "import pandas as pd\n",
+ "import numpy as np\n",
+ "import linecache, bisect\n",
+ "import os\n",
+ "\n",
+ "spath = os.getcwd()\n",
+ "path = spath.split(\"/\")\n",
+ "date = path[len(path)-1]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "**Request Data**\n",
+ "\n",
+ "In order to request data we are using Graphql (a query language for APIs, more info at: http://graphql.org/).\n",
+ "\n",
+ "We provide the function to make a data request, all you need is a query and variables\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "def makeGraphqlRequest(query, variables):\n",
+ " return GraphQLClient.request(query, variables)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now that we have a function, we can run a query like this:\n",
+ "\n",
+ "*Note: There's no need to manually set the date for the query, by default the code will read the date from the current path"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "suspicious_query = \"\"\"query($date:SpotDateType) {\n",
+ " dns {\n",
+ " suspicious(date:$date)\n",
+ " { clientIp\n",
+ " clientIpSev\n",
+ " dnsQuery\n",
+ " dnsQueryClass\n",
+ " dnsQueryClassLabel\n",
+ " dnsQueryRcode\n",
+ " dnsQueryRcodeLabel\n",
+ " dnsQueryRep\n",
+ " dnsQuerySev\n",
+ " dnsQueryType\n",
+ " dnsQueryTypeLabel\n",
+ " frameLength\n",
+ " frameTime\n",
+ " networkContext\n",
+ " score\n",
+ " tld\n",
+ " unixTimestamp \n",
+ " }\n",
+ " }\n",
+ " }\"\"\"\n",
+ "\n",
+ "##If you want to use a different date for your query, switch the \n",
+ "##commented/uncommented following lines\n",
+ "\n",
+ "variables={\n",
+ " 'date': datetime.datetime.strptime(date, '%Y%m%d').strftime('%Y-%m-%d')\n",
+ "# 'date': \"2016-10-08\"\n",
+ " }\n",
+ " \n",
+ "suspicious_request = makeGraphqlRequest(suspicious_query,variables)\n",
+ "\n",
+ "##The variable suspicious_request will contain the resulting data from the query.\n",
+ "results = suspicious_request['data']['dns']['suspicious']\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "##Pandas Dataframes\n",
+ "\n",
+ "The following cell loads the results into a pandas dataframe\n",
+ "\n",
+ "For more information on how to use pandas, you can learn more here: https://pandas.pydata.org/pandas-docs/stable/10min.html"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "df = pd.read_json(json.dumps(results))\n",
+ "##Printing only the selected column list from the dataframe\n",
+ "##Unless specified otherwise, \n",
+ "print df[['clientIp', 'unixTimestamp','tld', 'dnsQuery','dnsQueryRcode','dnsQueryRcodeLabel']]\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "##Additional operations \n",
+ "\n",
+ "Additional operations can be performed on the dataframe like sorting the data, filtering it and grouping it\n",
+ "\n",
+ "**Filtering the data**"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "##Filter results where the destination port = 3389\n",
+ "##The resulting data will be stored in df2 \n",
+ "\n",
+ "df2 = df[df['tld'].isin(['sjc04-login.dotomi.com'])]\n",
+ "print df2[['clientIp', 'unixTimestamp','tld', 'dnsQuery','dnsQueryRcode','dnsQueryRcodeLabel']]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "**Ordering the data**"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": false,
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "srtd = df.sort_values(by=\"tld\")\n",
+ "print srtd[['clientIp', 'unixTimestamp','tld', 'dnsQuery','dnsQueryRcode','dnsQueryRcodeLabel']]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "**Grouping the data**"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "## This command will group the results by pairs of source-destination IP\n",
+ "## summarizing all other columns \n",
+ "grpd = df.groupby(['clientIp','tld']).count()\n",
+ "## This will print the resulting dataframe displaying the input and output bytes columnns\n",
+ "print grpd[[\"dnsQuery\"]]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "##Sandbox\n",
+ "\n",
+ "At this point you can perform your own analysis using the previously provided functions as a guide.\n",
+ "\n",
+ "Happy threat hunting!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "#Your code here"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 2",
+ "language": "python",
+ "name": "python2"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 2
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython2",
+ "version": "2.7.5"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}