[4/5] incubator-spot git commit: Addind migration documentation

[ev...@apache.org]

Addind migration documentation


Project: http://git-wip-us.apache.org/repos/asf/incubator-spot/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spot/commit/10d07ec6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spot/tree/10d07ec6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spot/diff/10d07ec6

Branch: refs/heads/SPOT-35_graphql_api
Commit: 10d07ec69f56dea7a8543d24446e5d0f63e3cbb9
Parents: 7150915
Author: Moy Valdovinos <mo...@gmail.com>
Authored: Tue Jun 20 14:39:15 2017 -0500
Committer: Everardo Lopez Sandoval (Intel) <el...@elopezsa-mac02.zpn.intel.com>
Committed: Wed Jun 21 17:20:53 2017 -0500

----------------------------------------------------------------------
 spot-setup/migration/README.md | 68 +++++++++++++++++++++++++++++++++++++
 1 file changed, 68 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-spot/blob/10d07ec6/spot-setup/migration/README.md
----------------------------------------------------------------------
diff --git a/spot-setup/migration/README.md b/spot-setup/migration/README.md
new file mode 100644
index 0000000..d344322
--- /dev/null
+++ b/spot-setup/migration/README.md
@@ -0,0 +1,68 @@
+## Data Migration from Spot 0.9 to Spot 1.0
+
+This document is intended for any developer or sysadmin who wants to migrate their existing data from Spot 0.9 to Spot 1.0. In Spot 0.9, data was stored in CSV files in a given location. In Spot 1.0, data is stored in Impala tables. The purposes of these scripts are to migrate independently each use case (flow, proxy and dns) from those CSV files the new Impala tables.
+
+This migration process is optional and only for those users who want to keep usable their data generated by ML and OA (scores, edges, chords, dendros, ingest summaries, storyboards, threat investigations and timelines) in the previous Spot version. 
+
+### Requirements
+
+You must run first new Spot setup installation to have the new tables created in Impala. 
+
+### CSV to Impala Tables Mapping
+
+**Flow**
+
+CSV File | Impala Table
+---------|-------
+flow_scores.csv | flow_scores
+chord-*.tsv  | flow_chords 
+edge-*.tsv | flow_edge
+is_*.csv | flow_ingest_summary
+threats.csv | flow_storyboard
+flow_scores.csv (only scored values) | flow_threat_investigation
+sbdet-*.tsv | flow_timeline
+
+**DNS**
+
+CSV File | Impala Table
+---------|-------
+flow_scores.csv | dns_scores
+edge-*.csv | dns_edge
+dendro-*.csv | dns_dendro
+threat-dendro-*.csv | dns_threat_dendro
+is_*.csv | dns_ingest_summary
+threats.csv | dns_storyboard
+dns_scores.csv (only scored values) | dns_threat_investigation
+
+**Proxy**
+
+CSV File | Impala Table
+---------|-------
+edge-*.csv | proxy_edge
+is_*.csv | proxy_ingest_summary
+proxy_scores.csv | proxy_scores
+threats.csv | proxy_storyboard
+proxy_scores.csv (only scored values) | proxy_threat_investigation
+timeline-*.csv | proxy_timeline
+
+### Data Flow
+
+These scripts will read each of the CSV from the existing location and import data to Impala tables accordingly, creating first a staging database and tables to load the records in the CSV and then insert that data into the new Spot 1.0 tables. You must execute this migration process from the server where Spot 0.9 CSV files are located. You may run one or all script for each use case (flow, dns and proxy) according to your needs and your existing data.
+
+### Execution
+
+```shell
+./migrate_old_dns_data.sh OLD_DATA_PATH STAGING_DB_NAME STAGING_DB_HDFS_PATH NEW_SPOT_IMPALA_DB IMPALA_DAEMON
+```
+
+where variables mean:
+- **OLD_DATA_PATH** - Path to existing Spot 0.9 CSV files in the local filesystem 
+- **STAGING_DB_NAME** - Name of the staging database to be created to temporarily store these records
+- **STAGING_DB_HDFS_PATH** - HDFS path of the staging database to be created to temporarily store these records
+- **NEW_SPOT_IMPALA_DB** - Database name of the Spot 1.0 Impala tables. Use the same as in the spot.conf when the new Spot release was installed 
+- **IMPALA_DAEMON** - Choose an Impala daemon to be used to run scripts' queries.
+
+Example:
+```shell
+./migrate_old_dns_data.sh '/home/spot/spot-csv-data' 'spot_migration' '/user/spotuser/spot_migration/' 'migrated' 'node01'
+```