You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@metron.apache.org by ni...@apache.org on 2016/05/06 17:39:09 UTC
[1/2] incubator-metron git commit: METRON-129 Created Vagrant images
for Metron development (merrimanr via nickwallen) closes
apache/incubator-metron#105
Repository: incubator-metron
Updated Branches:
refs/heads/master 4c16de905 -> ec431b8dd
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/quick-dev-platform/run_enrichment_role.sh
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/quick-dev-platform/run_enrichment_role.sh b/metron-deployment/vagrant/quick-dev-platform/run_enrichment_role.sh
new file mode 100755
index 0000000..e26bc16
--- /dev/null
+++ b/metron-deployment/vagrant/quick-dev-platform/run_enrichment_role.sh
@@ -0,0 +1,20 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+./run_ansible_role.sh enrichment
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/singlenode-vagrant/.gitignore
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/singlenode-vagrant/.gitignore b/metron-deployment/vagrant/singlenode-vagrant/.gitignore
deleted file mode 100644
index 8000dd9..0000000
--- a/metron-deployment/vagrant/singlenode-vagrant/.gitignore
+++ /dev/null
@@ -1 +0,0 @@
-.vagrant
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/singlenode-vagrant/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/singlenode-vagrant/README.md b/metron-deployment/vagrant/singlenode-vagrant/README.md
deleted file mode 100644
index 8650570..0000000
--- a/metron-deployment/vagrant/singlenode-vagrant/README.md
+++ /dev/null
@@ -1,102 +0,0 @@
-Apache Metron on Virtualbox
-===========================
-
-This project fully automates the provisioning of Apache Metron on single, virtualized host running on Virtualbox. Metron is composed of many components and installing all of these on a single host, especially a virtualized one, will greatly stress your computer. To work sufficiently this will require at least 8 GB of RAM and a fair amount of patience.
-
-Getting Started
----------------
-
-### Prerequisites
-
-The computer used to deploy Apache Metron will need to have the following components installed.
-
- - [Ansible](https://github.com/ansible/ansible) 2.0.0.2
- - [Vagrant](https://www.vagrantup.com) 1.8.1
- - [Virtualbox](virtualbox.org) 5.0.16
- - Python 2.7.11
- - Maven 3.3.9
-
-Any platform that supports these tools is suitable, but the following instructions cover installation on Mac OS X only. The easiest means of installing these tools on a Mac is to use the excellent [Homebrew](http://brew.sh/) project.
-
-1. Install Homebrew by running the following command in a terminal. Refer to the [Homebrew](http://brew.sh/) home page for the latest installation instructions.
-
- ```
- /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
- ```
-
-2. With Homebrew installed, run the following command in a terminal to install all of the required tools.
-
- ```
- brew cask install vagrant virtualbox
- brew install brew-pip maven git
- pip install ansible==2.0.0.2
- ```
-
-3. Install Vagrant Hostmanager.
-
- ```
- vagrant plugin install vagrant-hostmanager
- ```
-
-### Metron
-
-Now that the hard part is done, start the Metron deployment process.
-
-1. Build Metron
-
- ```
- cd metron-platform
- mvn clean package -DskipTests
- ```
-
-2. Deploy Metron
-
- ```
- cd metron-deployment/vagrant/singlenode-vagrant
- vagrant up
- ```
-
- Should the process fail before completing the deployment, the following command will continue the deployment process without re-instantiating the host.
-
- ```
- vagrant provision
- ```
-
- In addition to re-running the entire provisioning play book, you may now re-run an individual Ansible tag or a collection of tags in the following ways.
-
- ```
- ./run_ansible_role.sh web
- ```
- or
- ```
- vagrant --ansible-tags="web" provision
- ```
- Will re-run the web role on the Vagrant image. This will re-install (if necessary) and start the UI.
-
- A collection of tags is specified as a comma separated list.
-
- ```
- ./run_ansbile_role.sh "sensors,enrichment"
-
- ```
-
- Tags are listed in the playbooks, some frequently used tags:
- + hdp-install - Install HDP
- + hdp-deploy - Deploy and Start HDP Services (will start all Hadoop Services)
- + sensors - Deploy and Start Sensors.
- + enrichment - Deploy and Start Enrichment Topology.
-
- Note: there is a convienence script, ```./run_enrichment_role.sh```, which runs the enrichment tag.
-
-### Explore Metron
-
-Navigate to the following resources to explore your newly minted Apache Metron environment.
-
- - [Metron](http://node1:8080)
- - [Ambari](http://node1:5000)
-
-Connecting to the host through SSH is as simple as running the following command.
-
- ```
- vagrant ssh
- ```
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/singlenode-vagrant/Vagrantfile
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/singlenode-vagrant/Vagrantfile b/metron-deployment/vagrant/singlenode-vagrant/Vagrantfile
deleted file mode 100644
index f9f4b84..0000000
--- a/metron-deployment/vagrant/singlenode-vagrant/Vagrantfile
+++ /dev/null
@@ -1,92 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-require 'getoptlong'
-
-ansibleTags=''
-ansibleSkipTags=''
-
-begin
- opts = GetoptLong.new(
- [ '--ansible-tags', GetoptLong::OPTIONAL_ARGUMENT ],
- [ '--ansible-skip-tags', GetoptLong::OPTIONAL_ARGUMENT ]
- )
-
- opts.quiet = TRUE
-
- opts.each do |opt, arg|
- case opt
- when '--ansible-tags'
- ansibleTags=arg
- when '--ansible-skip-tags'
- ansibleSkipTags=arg
- end
- end
-rescue Exception => ignored
-#Ignore to allow other opts to be passed to Vagrant
-end
-
-puts " Running with ansible-tags: " + ansibleTags.split(",").to_s if ansibleTags != ''
-puts " Running with ansible-skip-tags: " + ansibleSkipTags.split(",").to_s if ansibleSkipTags != ''
-
-hosts = [{
- hostname: "node1",
- ip: "192.168.66.121",
- memory: "8192",
- cpus: 4,
- promisc: 2 # enables promisc on the 'Nth' network interface
-}]
-
-Vagrant.configure(2) do |config|
-
- # all hosts built on centos 6
- config.vm.box = "bento/centos-6.7"
- config.ssh.insert_key = true
-
- # enable the hostmanager plugin
- config.hostmanager.enabled = true
- config.hostmanager.manage_host = true
-
- # host definition
- hosts.each_with_index do |host, index|
- config.vm.define host[:hostname] do |node|
-
- # host settings
- node.vm.hostname = host[:hostname]
- node.vm.network "private_network", ip: host[:ip]
-
- # vm settings
- node.vm.provider "virtualbox" do |vb|
- vb.memory = host[:memory]
- vb.cpus = host[:cpus]
-
- # enable promisc mode on the network interface
- if host.has_key?(:promisc)
- vb.customize ["modifyvm", :id, "--nicpromisc#{host[:promisc]}", "allow-all"]
- end
- end
- end
- end
-
- # provisioning
- config.vm.provision :ansible do |ansible|
- ansible.playbook = "../../playbooks/metron_full_install.yml"
- ansible.sudo = true
- ansible.tags = ansibleTags.split(",") if ansibleTags != ''
- ansible.skip_tags = ansibleSkipTags.split(",") if ansibleSkipTags != ''
- ansible.inventory_path = "../../inventory/singlenode-vagrant"
- end
-end
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/singlenode-vagrant/ansible.cfg
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/singlenode-vagrant/ansible.cfg b/metron-deployment/vagrant/singlenode-vagrant/ansible.cfg
deleted file mode 100644
index 74f3fdf..0000000
--- a/metron-deployment/vagrant/singlenode-vagrant/ansible.cfg
+++ /dev/null
@@ -1,27 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-[defaults]
-host_key_checking = false
-library = ../../extra_modules
-roles_path = ../../roles
-pipelining = True
-log_path = ./ansible.log
-
-
-# fix for "ssh throws 'unix domain socket too long' " problem
-[ssh_connection]
-control_path = %(directory)s/%%h-%%p-%%r
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/singlenode-vagrant/run_ansible_role.sh
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/singlenode-vagrant/run_ansible_role.sh b/metron-deployment/vagrant/singlenode-vagrant/run_ansible_role.sh
deleted file mode 100755
index 0e5d831..0000000
--- a/metron-deployment/vagrant/singlenode-vagrant/run_ansible_role.sh
+++ /dev/null
@@ -1,20 +0,0 @@
-#!/bin/bash
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-vagrant --ansible-tags="$@" provision
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/singlenode-vagrant/run_enrichment_role.sh
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/singlenode-vagrant/run_enrichment_role.sh b/metron-deployment/vagrant/singlenode-vagrant/run_enrichment_role.sh
deleted file mode 100755
index e26bc16..0000000
--- a/metron-deployment/vagrant/singlenode-vagrant/run_enrichment_role.sh
+++ /dev/null
@@ -1,20 +0,0 @@
-#!/bin/bash
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-./run_ansible_role.sh enrichment
[2/2] incubator-metron git commit: METRON-129 Created Vagrant images
for Metron development (merrimanr via nickwallen) closes
apache/incubator-metron#105
Posted by ni...@apache.org.
METRON-129 Created Vagrant images for Metron development (merrimanr via nickwallen) closes apache/incubator-metron#105
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/ec431b8d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/ec431b8d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/ec431b8d
Branch: refs/heads/master
Commit: ec431b8dd2a6bb5c275ef3825c1c7516ec4ffce9
Parents: 4c16de9
Author: merrimanr <me...@gmail.com>
Authored: Fri May 6 13:38:21 2016 -0400
Committer: Nick Allen <ni...@nickallen.org>
Committed: Fri May 6 13:38:21 2016 -0400
----------------------------------------------------------------------
metron-deployment/README.md | 2 +-
.../inventory/full-dev-platform/group_vars/all | 86 ++++++++++++++++
.../inventory/full-dev-platform/hosts | 52 ++++++++++
.../inventory/singlenode-vagrant/group_vars/all | 86 ----------------
.../inventory/singlenode-vagrant/hosts | 52 ----------
.../vagrant/codelab-platform/README | 46 +++++++++
.../vagrant/codelab-platform/Vagrantfile | 94 +++++++++++++++++
.../vagrant/codelab-platform/ansible.cfg | 27 +++++
.../vagrant/codelab-platform/launch_image.sh | 20 ++++
.../codelab-platform/run_ansible_role.sh | 20 ++++
.../codelab-platform/run_enrichment_role.sh | 20 ++++
.../vagrant/devimage-vagrant/README.md | 61 -----------
.../vagrant/devimage-vagrant/Vagrantfile | 94 -----------------
.../vagrant/devimage-vagrant/ansible.cfg | 27 -----
.../devimage-vagrant/launch_dev_image.sh | 21 ----
.../devimage-vagrant/run_ansible_role.sh | 20 ----
.../devimage-vagrant/run_enrichment_role.sh | 20 ----
.../vagrant/fastcapa-test-platform/.gitignore | 1 +
.../vagrant/fastcapa-test-platform/README.md | 73 +++++++++++++
.../vagrant/fastcapa-test-platform/Vagrantfile | 71 +++++++++++++
.../vagrant/fastcapa-test-platform/ansible.cfg | 22 ++++
.../vagrant/fastcapa-test-platform/playbook.yml | 51 ++++++++++
.../tasks/validate-messages-received.yml | 38 +++++++
.../tasks/validate-packets-sent.yml | 41 ++++++++
.../fastcapa-test-platform/vars/main.yml | 38 +++++++
.../vagrant/fastcapa-vagrant/.gitignore | 1 -
.../vagrant/fastcapa-vagrant/README.md | 73 -------------
.../vagrant/fastcapa-vagrant/Vagrantfile | 71 -------------
.../vagrant/fastcapa-vagrant/ansible.cfg | 22 ----
.../vagrant/fastcapa-vagrant/playbook.yml | 51 ----------
.../tasks/validate-messages-received.yml | 38 -------
.../tasks/validate-packets-sent.yml | 41 --------
.../vagrant/fastcapa-vagrant/vars/main.yml | 38 -------
.../vagrant/full-dev-platform/.gitignore | 1 +
.../vagrant/full-dev-platform/README.md | 102 +++++++++++++++++++
.../vagrant/full-dev-platform/Vagrantfile | 92 +++++++++++++++++
.../vagrant/full-dev-platform/ansible.cfg | 27 +++++
.../full-dev-platform/run_ansible_role.sh | 20 ++++
.../full-dev-platform/run_enrichment_role.sh | 20 ++++
.../vagrant/quick-dev-platform/README.md | 61 +++++++++++
.../vagrant/quick-dev-platform/Vagrantfile | 94 +++++++++++++++++
.../vagrant/quick-dev-platform/ansible.cfg | 27 +++++
.../quick-dev-platform/launch_dev_image.sh | 21 ++++
.../quick-dev-platform/run_ansible_role.sh | 20 ++++
.../quick-dev-platform/run_enrichment_role.sh | 20 ++++
.../vagrant/singlenode-vagrant/.gitignore | 1 -
.../vagrant/singlenode-vagrant/README.md | 102 -------------------
.../vagrant/singlenode-vagrant/Vagrantfile | 92 -----------------
.../vagrant/singlenode-vagrant/ansible.cfg | 27 -----
.../singlenode-vagrant/run_ansible_role.sh | 20 ----
.../singlenode-vagrant/run_enrichment_role.sh | 20 ----
51 files changed, 1206 insertions(+), 979 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/README.md b/metron-deployment/README.md
index c6236d9..edf8a37 100644
--- a/metron-deployment/README.md
+++ b/metron-deployment/README.md
@@ -88,7 +88,7 @@ run this:
- Hostmanager plugin for vagrant - Run `vagrant plugin install vagrant-hostmanager` on the machine where Vagrant is
installed
-Navigate to `incubator-metron/metron-deployment/vagrant/singlenode-vagrant` and run `vagrant up`. This also provides a good
+Navigate to `incubator-metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`. This also provides a good
example of how to run a full end-to-end Metron install.
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/inventory/full-dev-platform/group_vars/all
----------------------------------------------------------------------
diff --git a/metron-deployment/inventory/full-dev-platform/group_vars/all b/metron-deployment/inventory/full-dev-platform/group_vars/all
new file mode 100644
index 0000000..decd7f3
--- /dev/null
+++ b/metron-deployment/inventory/full-dev-platform/group_vars/all
@@ -0,0 +1,86 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+#Ambari variables
+ambari_host: "{{ groups.ambari_master[0] }}"
+hdp_host_group: "{{ groups.ambari_slave }}"
+ambari_port: 8080
+ambari_user: admin
+ambari_password: admin
+cluster_type: single_node_vm
+
+# hbase
+pcap_hbase_table: pcap
+tracker_hbase_table: access_tracker
+threatintel_hbase_table: threatintel
+enrichment_hbase_table: enrichment
+
+# metron variables
+metron_version: 0.1BETA
+metron_directory: /usr/metron/{{ metron_version }}
+java_home: /usr/jdk64/jdk1.8.0_40
+bro_version: "2.4.1"
+fixbuf_version: "1.7.1"
+yaf_version: "2.8.0"
+daq_version: "2.0.6-1"
+pycapa_repo: "https://github.com/OpenSOC/pycapa.git"
+pycapa_home: "/opt/pycapa"
+snort_version: "2.9.8.0-1"
+snort_alert_csv_path: "/var/log/snort/alert.csv"
+
+#data directories - only required to override defaults
+zookeeper_data_dir: "/data1/hadoop/zookeeper"
+namenode_checkpoint_dir: "/data1/hadoop/hdfs/namesecondary"
+namenode_name_dir: "/data1/hadoop/hdfs/namenode"
+datanode_data_dir: "/data1/hadoop/hdfs/data,/data2/hadoop/hdfs/data"
+journalnode_edits_dir: "/data1/hadoop/hdfs/journalnode"
+nodemanager_local_dirs: "/data1/hadoop/yarn/local"
+timeline_ldb_store_path: "/data1/hadoop/yarn/timeline"
+timeline_ldb_state_path: "/data1/hadoop/yarn/timeline"
+nodemanager_log_dirs: "/data1/hadoop/yarn/log"
+jhs_recovery_store_ldb_path: "/data1/hadoop/mapreduce/jhs"
+storm_local_dir: "/data1/hadoop/storm"
+kafka_log_dirs: "/data1/kafka-log"
+elasticsearch_data_dir: "/data1/elasticsearch,/data2/elasticsearch"
+
+ambari_server_mem: 512
+threat_intel_bulk_load: False
+
+# sensors
+sensor_test_mode: True
+install_pycapa: False
+install_bro: True
+install_snort: True
+install_yaf: True
+pcap_replay: True
+sniff_interface: eth1
+pcap_replay_interface: "{{ sniff_interface }}"
+storm_parser_topologies:
+ - "bro"
+ - "snort"
+ - "yaf"
+pcapservice_port: 8081
+
+#Search
+install_elasticsearch: True
+install_solr: False
+solr_collection_name: Metron
+solr_number_shards: 1
+solr_replication_factor: 1
+elasticsearch_transport_port: 9300
+elasticsearch_network_interface: eth1
+elasticsearch_web_port: 9200
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/inventory/full-dev-platform/hosts
----------------------------------------------------------------------
diff --git a/metron-deployment/inventory/full-dev-platform/hosts b/metron-deployment/inventory/full-dev-platform/hosts
new file mode 100644
index 0000000..93c53c2
--- /dev/null
+++ b/metron-deployment/inventory/full-dev-platform/hosts
@@ -0,0 +1,52 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+[ambari_master]
+node1
+
+[ambari_slave]
+node1
+
+[hadoop_client]
+node1
+
+[enrichment]
+node1
+
+[search]
+node1
+
+[web]
+node1
+
+[sensors]
+node1
+
+[pcap_server]
+node1
+
+[mysql]
+node1
+
+[metron:children]
+enrichment
+search
+web
+sensors
+mysql
+hadoop_client
+pcap_server
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/inventory/singlenode-vagrant/group_vars/all
----------------------------------------------------------------------
diff --git a/metron-deployment/inventory/singlenode-vagrant/group_vars/all b/metron-deployment/inventory/singlenode-vagrant/group_vars/all
deleted file mode 100644
index decd7f3..0000000
--- a/metron-deployment/inventory/singlenode-vagrant/group_vars/all
+++ /dev/null
@@ -1,86 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#Ambari variables
-ambari_host: "{{ groups.ambari_master[0] }}"
-hdp_host_group: "{{ groups.ambari_slave }}"
-ambari_port: 8080
-ambari_user: admin
-ambari_password: admin
-cluster_type: single_node_vm
-
-# hbase
-pcap_hbase_table: pcap
-tracker_hbase_table: access_tracker
-threatintel_hbase_table: threatintel
-enrichment_hbase_table: enrichment
-
-# metron variables
-metron_version: 0.1BETA
-metron_directory: /usr/metron/{{ metron_version }}
-java_home: /usr/jdk64/jdk1.8.0_40
-bro_version: "2.4.1"
-fixbuf_version: "1.7.1"
-yaf_version: "2.8.0"
-daq_version: "2.0.6-1"
-pycapa_repo: "https://github.com/OpenSOC/pycapa.git"
-pycapa_home: "/opt/pycapa"
-snort_version: "2.9.8.0-1"
-snort_alert_csv_path: "/var/log/snort/alert.csv"
-
-#data directories - only required to override defaults
-zookeeper_data_dir: "/data1/hadoop/zookeeper"
-namenode_checkpoint_dir: "/data1/hadoop/hdfs/namesecondary"
-namenode_name_dir: "/data1/hadoop/hdfs/namenode"
-datanode_data_dir: "/data1/hadoop/hdfs/data,/data2/hadoop/hdfs/data"
-journalnode_edits_dir: "/data1/hadoop/hdfs/journalnode"
-nodemanager_local_dirs: "/data1/hadoop/yarn/local"
-timeline_ldb_store_path: "/data1/hadoop/yarn/timeline"
-timeline_ldb_state_path: "/data1/hadoop/yarn/timeline"
-nodemanager_log_dirs: "/data1/hadoop/yarn/log"
-jhs_recovery_store_ldb_path: "/data1/hadoop/mapreduce/jhs"
-storm_local_dir: "/data1/hadoop/storm"
-kafka_log_dirs: "/data1/kafka-log"
-elasticsearch_data_dir: "/data1/elasticsearch,/data2/elasticsearch"
-
-ambari_server_mem: 512
-threat_intel_bulk_load: False
-
-# sensors
-sensor_test_mode: True
-install_pycapa: False
-install_bro: True
-install_snort: True
-install_yaf: True
-pcap_replay: True
-sniff_interface: eth1
-pcap_replay_interface: "{{ sniff_interface }}"
-storm_parser_topologies:
- - "bro"
- - "snort"
- - "yaf"
-pcapservice_port: 8081
-
-#Search
-install_elasticsearch: True
-install_solr: False
-solr_collection_name: Metron
-solr_number_shards: 1
-solr_replication_factor: 1
-elasticsearch_transport_port: 9300
-elasticsearch_network_interface: eth1
-elasticsearch_web_port: 9200
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/inventory/singlenode-vagrant/hosts
----------------------------------------------------------------------
diff --git a/metron-deployment/inventory/singlenode-vagrant/hosts b/metron-deployment/inventory/singlenode-vagrant/hosts
deleted file mode 100644
index 93c53c2..0000000
--- a/metron-deployment/inventory/singlenode-vagrant/hosts
+++ /dev/null
@@ -1,52 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-[ambari_master]
-node1
-
-[ambari_slave]
-node1
-
-[hadoop_client]
-node1
-
-[enrichment]
-node1
-
-[search]
-node1
-
-[web]
-node1
-
-[sensors]
-node1
-
-[pcap_server]
-node1
-
-[mysql]
-node1
-
-[metron:children]
-enrichment
-search
-web
-sensors
-mysql
-hadoop_client
-pcap_server
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/codelab-platform/README
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/codelab-platform/README b/metron-deployment/vagrant/codelab-platform/README
new file mode 100644
index 0000000..afc6ab7
--- /dev/null
+++ b/metron-deployment/vagrant/codelab-platform/README
@@ -0,0 +1,46 @@
+Developer Image for Apache Metron on Virtualbox
+===============================================
+
+This image is a fully functional Metron installation that has been pre-loaded with Ambari, HDP and Metron.
+
+Getting Started
+---------------
+
+### Prerequisites
+
+As with the Singlenode Full Image, the computer used to deploy Apache Metron will need to have the following components installed.
+
+ - [Ansible](https://github.com/ansible/ansible) 2.0.0.2
+ - [Vagrant](https://www.vagrantup.com) 1.8.1
+ - [Virtualbox](virtualbox.org) 5.0.16
+ - Python 2.7.11
+ - Maven 3.3.9
+
+
+### Launch the Metron Development Image
+
+Start the image with the following commands:
+
+ ```
+ cd metron-deployment/vagrant/codelab-platform
+ ./launch_dev_image.sh
+ ```
+
+### Work with Metron
+
+As you build out new capabilities for Metron, you will need to re-deploy the Storm topologies. To do so, first HALT the running Storm topologies and then run:
+
+```
+./run_enrichment_role.sh
+```
+
+Remember Navigate to the following resources to explore your newly minted Apache Metron environment.
+
+ - [Metron](http://node1:8080)
+ - [Ambari](http://node1:5000)
+
+Connecting to the host through SSH is as simple as running the following command.
+
+ ```
+ vagrant ssh
+ ```
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/codelab-platform/Vagrantfile
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/codelab-platform/Vagrantfile b/metron-deployment/vagrant/codelab-platform/Vagrantfile
new file mode 100644
index 0000000..b56f649
--- /dev/null
+++ b/metron-deployment/vagrant/codelab-platform/Vagrantfile
@@ -0,0 +1,94 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+require 'getoptlong'
+
+ansibleTags=''
+ansibleSkipTags=''
+
+begin
+ opts = GetoptLong.new(
+ [ '--ansible-tags', GetoptLong::OPTIONAL_ARGUMENT ],
+ [ '--ansible-skip-tags', GetoptLong::OPTIONAL_ARGUMENT ]
+ )
+
+ opts.quiet = TRUE
+
+ opts.each do |opt, arg|
+ case opt
+ when '--ansible-tags'
+ ansibleTags=arg
+ when '--ansible-skip-tags'
+ ansibleSkipTags=arg
+ end
+ end
+rescue Exception => ignored
+#Ignore to allow other opts to be passed to Vagrant
+end
+
+puts " Running with ansible-tags: " + ansibleTags.split(",").to_s if ansibleTags != ''
+puts " Running with ansible-skip-tags: " + ansibleSkipTags.split(",").to_s if ansibleSkipTags != ''
+
+hosts = [{
+ hostname: "node1",
+ ip: "192.168.66.121",
+ memory: "8192",
+ cpus: 4,
+ promisc: 2 # enables promisc on the 'Nth' network interface
+}]
+
+Vagrant.configure(2) do |config|
+
+ # all hosts built on centos 6
+ config.vm.box = "metron/hackathon_05042016"
+ config.ssh.insert_key = true
+
+ # enable the hostmanager plugin
+ config.hostmanager.enabled = true
+ config.hostmanager.manage_host = true
+
+ # host definition
+ hosts.each_with_index do |host, index|
+ config.vm.define host[:hostname] do |node|
+
+ # host settings
+ node.vm.hostname = host[:hostname]
+ node.vm.network "private_network", ip: host[:ip]
+
+ # vm settings
+ node.vm.provider "virtualbox" do |vb|
+ vb.memory = host[:memory]
+ vb.cpus = host[:cpus]
+
+ # enable promisc mode on the network interface
+ if host.has_key?(:promisc)
+ vb.customize ["modifyvm", :id, "--nicpromisc#{host[:promisc]}", "allow-all"]
+ end
+ end
+ end
+ end
+
+ if ansibleSkipTags != '' or ansibleTags != ''
+ config.vm.provision :ansible do |ansible|
+ ansible.playbook = "../../playbooks/metron_full_install.yml"
+ ansible.sudo = true
+ ansible.tags = ansibleTags.split(",") if ansibleTags != ''
+ ansible.skip_tags = ansibleSkipTags.split(",") if ansibleSkipTags != ''
+ ansible.inventory_path = "../../inventory/full-dev-platform"
+ end
+ end
+
+end
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/codelab-platform/ansible.cfg
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/codelab-platform/ansible.cfg b/metron-deployment/vagrant/codelab-platform/ansible.cfg
new file mode 100644
index 0000000..74f3fdf
--- /dev/null
+++ b/metron-deployment/vagrant/codelab-platform/ansible.cfg
@@ -0,0 +1,27 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+[defaults]
+host_key_checking = false
+library = ../../extra_modules
+roles_path = ../../roles
+pipelining = True
+log_path = ./ansible.log
+
+
+# fix for "ssh throws 'unix domain socket too long' " problem
+[ssh_connection]
+control_path = %(directory)s/%%h-%%p-%%r
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/codelab-platform/launch_image.sh
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/codelab-platform/launch_image.sh b/metron-deployment/vagrant/codelab-platform/launch_image.sh
new file mode 100755
index 0000000..45ebe4b
--- /dev/null
+++ b/metron-deployment/vagrant/codelab-platform/launch_image.sh
@@ -0,0 +1,20 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+vagrant --ansible-tags="hdp-deploy,sensors" up
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/codelab-platform/run_ansible_role.sh
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/codelab-platform/run_ansible_role.sh b/metron-deployment/vagrant/codelab-platform/run_ansible_role.sh
new file mode 100755
index 0000000..0e5d831
--- /dev/null
+++ b/metron-deployment/vagrant/codelab-platform/run_ansible_role.sh
@@ -0,0 +1,20 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+vagrant --ansible-tags="$@" provision
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/codelab-platform/run_enrichment_role.sh
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/codelab-platform/run_enrichment_role.sh b/metron-deployment/vagrant/codelab-platform/run_enrichment_role.sh
new file mode 100755
index 0000000..e26bc16
--- /dev/null
+++ b/metron-deployment/vagrant/codelab-platform/run_enrichment_role.sh
@@ -0,0 +1,20 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+./run_ansible_role.sh enrichment
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/devimage-vagrant/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/devimage-vagrant/README.md b/metron-deployment/vagrant/devimage-vagrant/README.md
deleted file mode 100644
index af66682..0000000
--- a/metron-deployment/vagrant/devimage-vagrant/README.md
+++ /dev/null
@@ -1,61 +0,0 @@
-Developer Image for Apache Metron on Virtualbox
-===============================================
-
-This image is designed for more rapid deployment of Metron for use by Metron developers. It uses a base image that has been pre-loaded with Ambari and HDP.
-
-Getting Started
----------------
-
-### Prerequisites
-
-As with the Singlenode Full Image, the computer used to deploy Apache Metron will need to have the following components installed.
-
- - [Ansible](https://github.com/ansible/ansible) 2.0.0.2
- - [Vagrant](https://www.vagrantup.com) 1.8.1
- - [Virtualbox](virtualbox.org) 5.0.16
- - Python 2.7.11
- - Maven 3.3.9
-
-
-### Launch the Metron Development Image
-
-Now that the hard part is done, start the Metron deployment process.
-
-1. Build Metron
-
- ```
- cd metron-platform
- mvn clean package -DskipTests
- ```
-
-2. Deploy Metron
-
- ```
- cd metron-deployment/vagrant/devimage-vagrant
- ./launch_dev_image.sh
- ```
-
- Should the process fail before completing the deployment, the following command will continue the deployment process without re-instantiating the host.
-
- ```
- vagrant --ansible-tags="hdp-deploy,metron" provision
- ```
-
-### Work with Metron
-
-As you build out new capabilities for Metron, you will need to re-deploy the Storm topologies. To do so, first HALT the running Storm topologies and then run:
-
-```
-./run_enrichment_role.sh
-```
-
-Remember Navigate to the following resources to explore your newly minted Apache Metron environment.
-
- - [Metron](http://node1:8080)
- - [Ambari](http://node1:5000)
-
-Connecting to the host through SSH is as simple as running the following command.
-
- ```
- vagrant ssh
- ```
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/devimage-vagrant/Vagrantfile
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/devimage-vagrant/Vagrantfile b/metron-deployment/vagrant/devimage-vagrant/Vagrantfile
deleted file mode 100644
index 29e91b0..0000000
--- a/metron-deployment/vagrant/devimage-vagrant/Vagrantfile
+++ /dev/null
@@ -1,94 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-require 'getoptlong'
-
-ansibleTags=''
-ansibleSkipTags=''
-
-begin
- opts = GetoptLong.new(
- [ '--ansible-tags', GetoptLong::OPTIONAL_ARGUMENT ],
- [ '--ansible-skip-tags', GetoptLong::OPTIONAL_ARGUMENT ]
- )
-
- opts.quiet = TRUE
-
- opts.each do |opt, arg|
- case opt
- when '--ansible-tags'
- ansibleTags=arg
- when '--ansible-skip-tags'
- ansibleSkipTags=arg
- end
- end
-rescue Exception => ignored
-#Ignore to allow other opts to be passed to Vagrant
-end
-
-puts " Running with ansible-tags: " + ansibleTags.split(",").to_s if ansibleTags != ''
-puts " Running with ansible-skip-tags: " + ansibleSkipTags.split(",").to_s if ansibleSkipTags != ''
-
-hosts = [{
- hostname: "node1",
- ip: "192.168.66.121",
- memory: "8192",
- cpus: 4,
- promisc: 2 # enables promisc on the 'Nth' network interface
-}]
-
-Vagrant.configure(2) do |config|
-
- # all hosts built on centos 6
- config.vm.box = "metron/hdp-base"
- config.ssh.insert_key = true
-
- # enable the hostmanager plugin
- config.hostmanager.enabled = true
- config.hostmanager.manage_host = true
-
- # host definition
- hosts.each_with_index do |host, index|
- config.vm.define host[:hostname] do |node|
-
- # host settings
- node.vm.hostname = host[:hostname]
- node.vm.network "private_network", ip: host[:ip]
-
- # vm settings
- node.vm.provider "virtualbox" do |vb|
- vb.memory = host[:memory]
- vb.cpus = host[:cpus]
-
- # enable promisc mode on the network interface
- if host.has_key?(:promisc)
- vb.customize ["modifyvm", :id, "--nicpromisc#{host[:promisc]}", "allow-all"]
- end
- end
- end
- end
-
- if ansibleSkipTags != '' or ansibleTags != ''
- config.vm.provision :ansible do |ansible|
- ansible.playbook = "../../playbooks/metron_full_install.yml"
- ansible.sudo = true
- ansible.tags = ansibleTags.split(",") if ansibleTags != ''
- ansible.skip_tags = ansibleSkipTags.split(",") if ansibleSkipTags != ''
- ansible.inventory_path = "../../inventory/singlenode-vagrant"
- end
- end
-
-end
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/devimage-vagrant/ansible.cfg
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/devimage-vagrant/ansible.cfg b/metron-deployment/vagrant/devimage-vagrant/ansible.cfg
deleted file mode 100644
index 74f3fdf..0000000
--- a/metron-deployment/vagrant/devimage-vagrant/ansible.cfg
+++ /dev/null
@@ -1,27 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-[defaults]
-host_key_checking = false
-library = ../../extra_modules
-roles_path = ../../roles
-pipelining = True
-log_path = ./ansible.log
-
-
-# fix for "ssh throws 'unix domain socket too long' " problem
-[ssh_connection]
-control_path = %(directory)s/%%h-%%p-%%r
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/devimage-vagrant/launch_dev_image.sh
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/devimage-vagrant/launch_dev_image.sh b/metron-deployment/vagrant/devimage-vagrant/launch_dev_image.sh
deleted file mode 100755
index dc91679..0000000
--- a/metron-deployment/vagrant/devimage-vagrant/launch_dev_image.sh
+++ /dev/null
@@ -1,21 +0,0 @@
-#!/bin/bash
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-vagrant --ansible-tags="hdp-deploy,metron" up
-
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/devimage-vagrant/run_ansible_role.sh
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/devimage-vagrant/run_ansible_role.sh b/metron-deployment/vagrant/devimage-vagrant/run_ansible_role.sh
deleted file mode 100755
index 0e5d831..0000000
--- a/metron-deployment/vagrant/devimage-vagrant/run_ansible_role.sh
+++ /dev/null
@@ -1,20 +0,0 @@
-#!/bin/bash
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-vagrant --ansible-tags="$@" provision
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/devimage-vagrant/run_enrichment_role.sh
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/devimage-vagrant/run_enrichment_role.sh b/metron-deployment/vagrant/devimage-vagrant/run_enrichment_role.sh
deleted file mode 100755
index e26bc16..0000000
--- a/metron-deployment/vagrant/devimage-vagrant/run_enrichment_role.sh
+++ /dev/null
@@ -1,20 +0,0 @@
-#!/bin/bash
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-./run_ansible_role.sh enrichment
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/fastcapa-test-platform/.gitignore
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/fastcapa-test-platform/.gitignore b/metron-deployment/vagrant/fastcapa-test-platform/.gitignore
new file mode 100644
index 0000000..a8b42eb
--- /dev/null
+++ b/metron-deployment/vagrant/fastcapa-test-platform/.gitignore
@@ -0,0 +1 @@
+*.retry
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/fastcapa-test-platform/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/fastcapa-test-platform/README.md b/metron-deployment/vagrant/fastcapa-test-platform/README.md
new file mode 100644
index 0000000..9bb0a39
--- /dev/null
+++ b/metron-deployment/vagrant/fastcapa-test-platform/README.md
@@ -0,0 +1,73 @@
+Fastcapa Test Environment
+=========================
+
+Provides a test environment for the development and testing of Fastcapa. The environment is automatically validated after it is created to ensure that Fastcapa is behaving correctly.
+
+Two virtualized nodes are launched with Vagrant that can communicate with one another over a private network.
+- The `source` node uses Metron's `pcap_replay` functionality to transmit raw network packet data over a private network.
+- The `sink` node is running `fastcapa` and is capturing these network packets.
+- Fastcapa then transforms and bundles the packets into a message.
+- The message is sent to a Kafka broker running on the `source` node.
+
+Getting Started
+---------------
+
+Simply run `vagrant up` to launch the environment. Automated tests are executed after provisioning completes to ensure that Fastcapa and the rest of the environment is functioning properly.
+
+```
+$ vagrant up
+==> source: Running provisioner: ansible...
+ source: Running ansible-playbook...
+...
+TASK [debug] *******************************************************************
+ok: [source] => {
+ "msg": "Successfully received packets sent from pcap-replay!"
+}
+...
+TASK [debug] *******************************************************************
+ok: [source] => {
+ "msg": "Successfully received a Kafka message from fastcapa!"
+}
+```
+
+Going Deeper
+------------
+
+This section will outline in more detail the environment and how to interact with it.
+
+### `source`
+
+To validate that the `source` node is functioning properly, run the following commands.
+
+First, ensure that the `pcap-replay` service is running.
+
+```
+vagrant ssh source
+sudo service pcap-replay status
+```
+
+Use `tcpdump` to ensure that the raw packet data is being sent over the private network. Enter 'CTRL-C' to kill the `tcpdump` process once you are able to see that packets are being sent.
+
+```
+sudo yum -y install tcpdump
+sudo tcpdump -i enp0s8
+```
+
+### `sink`
+
+Next validate that the `sink` is functioning properly. Run the following commands starting from the host operating system.
+
+First, ensure that the `fastcapa` service is running.
+
+```
+vagrant ssh sink
+service fastcapa status
+```
+
+Ensure that the raw network packet data is being received by Kafka.
+
+```
+/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic pcap
+```
+
+Enter 'CTRL-C' to kill the `kafka-console-consumer` process once you are able to see that packets are being sent. These packets will appear to be gibberish in the console. This is the raw binary network packet data after all.
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/fastcapa-test-platform/Vagrantfile
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/fastcapa-test-platform/Vagrantfile b/metron-deployment/vagrant/fastcapa-test-platform/Vagrantfile
new file mode 100644
index 0000000..7f58417
--- /dev/null
+++ b/metron-deployment/vagrant/fastcapa-test-platform/Vagrantfile
@@ -0,0 +1,71 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+Vagrant.configure("2") do |config|
+
+ # enable hostmanager
+ config.hostmanager.enabled = true
+ config.hostmanager.manage_host = true
+
+ #
+ # source
+ #
+ config.vm.define "source" do |node|
+
+ # host settings
+ node.vm.hostname = "source"
+ node.vm.box = "bento/centos-6.7"
+ node.ssh.insert_key = "true"
+ node.vm.network :private_network, ip: "192.168.33.10", netmask: "255.255.255.0"
+
+ # provider
+ node.vm.provider "virtualbox" do |vb|
+ vb.memory = 1024
+ vb.cpus = 1
+ end
+ end
+
+ #
+ # sink
+ #
+ config.vm.define "sink" do |node|
+
+ # host settings
+ node.vm.hostname = "sink"
+ node.vm.box = "bento/centos-7.1"
+ node.ssh.insert_key = "true"
+ node.vm.network :private_network, ip: "192.168.33.11", netmask: "255.255.255.0"
+
+ # provider
+ node.vm.provider "virtualbox" do |vb|
+ vb.memory = 4096
+ vb.cpus = 3
+
+ # network adapter settings; [Am79C970A|Am79C973|82540EM|82543GC|82545EM|virtio]
+ vb.customize ["modifyvm", :id, "--nicpromisc2", "allow-all"]
+ vb.customize ["modifyvm", :id, "--nictype2","82545EM"]
+ end
+
+ # provision host
+ node.vm.provision :ansible do |ansible|
+ ansible.limit = "all"
+ ansible.playbook = "playbook.yml"
+ end
+ end
+
+
+end
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/fastcapa-test-platform/ansible.cfg
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/fastcapa-test-platform/ansible.cfg b/metron-deployment/vagrant/fastcapa-test-platform/ansible.cfg
new file mode 100644
index 0000000..9c650c2
--- /dev/null
+++ b/metron-deployment/vagrant/fastcapa-test-platform/ansible.cfg
@@ -0,0 +1,22 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+[defaults]
+host_key_checking = false
+library = ../../extra_modules
+roles_path = ../../roles
+pipelining = True
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/fastcapa-test-platform/playbook.yml
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/fastcapa-test-platform/playbook.yml b/metron-deployment/vagrant/fastcapa-test-platform/playbook.yml
new file mode 100644
index 0000000..61509c4
--- /dev/null
+++ b/metron-deployment/vagrant/fastcapa-test-platform/playbook.yml
@@ -0,0 +1,51 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+---
+#
+# the 'source' produces network traffic
+#
+- hosts: source
+ become: yes
+ vars_files:
+ - vars/main.yml
+ roles:
+ - role: kafka-broker
+ - role: pcap_replay
+ - { role: pycapa, install_pycapa_service: False }
+ - { role: sensor-test-mode, pcap_replay: True, install_yaf: False, install_snort: False }
+
+#
+# the 'sink' consumes network traffic
+#
+- hosts: sink
+ become: yes
+ vars_files:
+ - vars/main.yml
+ roles:
+ - role: librdkafka
+ - role: fastcapa
+
+#
+# validate the environment - needs to run on `source` node
+#
+- hosts: source
+ become: yes
+ vars_files:
+ - vars/main.yml
+ tasks:
+ - include: tasks/validate-packets-sent.yml
+ - include: tasks/validate-messages-received.yml
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/fastcapa-test-platform/tasks/validate-messages-received.yml
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/fastcapa-test-platform/tasks/validate-messages-received.yml b/metron-deployment/vagrant/fastcapa-test-platform/tasks/validate-messages-received.yml
new file mode 100644
index 0000000..5109a0e
--- /dev/null
+++ b/metron-deployment/vagrant/fastcapa-test-platform/tasks/validate-messages-received.yml
@@ -0,0 +1,38 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+---
+- set_fact:
+ kafka_consumer_cmd: "{{ kafka_broker_home}}/bin/kafka-console-consumer.sh --zookeeper {{ zookeeper_url }} --topic {{ fastcapa_topic }} --max-messages 10 --from-beginning"
+
+- name: Attempting to receive message(s) sent by fastcapa
+ shell: "{{ kafka_consumer_cmd }}"
+ async: 30
+ register: kafka_consumer
+
+- name: Waiting to receive message(s) sent by fastcapa
+ async_status: jid={{ kafka_consumer.ansible_job_id }}
+ register: job
+ until: job.finished
+ retries: 30
+
+- fail:
+ msg: "No messages received from fastcapa within timeout: {{ kafka_consumer_cmd }}"
+ when: job.finished != 1
+
+- debug: var=job
+
+- debug: msg="Successfully received a Kafka message from fastcapa!"
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/fastcapa-test-platform/tasks/validate-packets-sent.yml
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/fastcapa-test-platform/tasks/validate-packets-sent.yml b/metron-deployment/vagrant/fastcapa-test-platform/tasks/validate-packets-sent.yml
new file mode 100644
index 0000000..25347dd
--- /dev/null
+++ b/metron-deployment/vagrant/fastcapa-test-platform/tasks/validate-packets-sent.yml
@@ -0,0 +1,41 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+---
+- name: Install tcpdump
+ yum: name=tcpdump
+
+- set_fact:
+ sniff_cmd: "tcpdump -i {{ pcap_replay_interface }} -c 10"
+
+- name: "Attempting to sniff packet(s)"
+ shell: "{{ sniff_cmd }}"
+ async: 30
+ register: sniffer
+
+- name: Waiting to receive packet(s)
+ async_status: jid={{ sniffer.ansible_job_id }}
+ register: job
+ until: job.finished
+ retries: 30
+
+- fail:
+ msg: "No packets received from pcap-replay within timeout: {{ sniff_cmd }}"
+ when: job.finished != 1
+
+- debug: var=job
+
+- debug: msg="Successfully received packets sent from pcap-replay!"
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/fastcapa-test-platform/vars/main.yml
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/fastcapa-test-platform/vars/main.yml b/metron-deployment/vagrant/fastcapa-test-platform/vars/main.yml
new file mode 100644
index 0000000..8e4584e
--- /dev/null
+++ b/metron-deployment/vagrant/fastcapa-test-platform/vars/main.yml
@@ -0,0 +1,38 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+---
+dpdk_device: ["00:08.0"]
+dpdk_target: "x86_64-native-linuxapp-gcc"
+num_huge_pages: 512
+fastcapa_portmask: "0x01"
+fastcapa_topic: pcap
+kafka_broker_url: source:9092
+zookeeper_url: source:2181
+pcap_replay_interface: eth1
+kafka_broker_home: /usr/hdp/current/kafka-broker/
+
+# dummy variables for pycapa's dependence on ambari_gather_facts
+cluster_name: dummy
+namenode_host: dummy
+core_site_tag: dummy
+hdfs_url: dummy
+kafka_broker_hosts: dummy
+kafka_broker_tag: dummy
+kafka_broker_port: dummy
+zookeeper_hosts: dummy
+zookeeper_tag: dummy
+zookeeper_port: dummy
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/fastcapa-vagrant/.gitignore
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/fastcapa-vagrant/.gitignore b/metron-deployment/vagrant/fastcapa-vagrant/.gitignore
deleted file mode 100644
index a8b42eb..0000000
--- a/metron-deployment/vagrant/fastcapa-vagrant/.gitignore
+++ /dev/null
@@ -1 +0,0 @@
-*.retry
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/fastcapa-vagrant/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/fastcapa-vagrant/README.md b/metron-deployment/vagrant/fastcapa-vagrant/README.md
deleted file mode 100644
index 9bb0a39..0000000
--- a/metron-deployment/vagrant/fastcapa-vagrant/README.md
+++ /dev/null
@@ -1,73 +0,0 @@
-Fastcapa Test Environment
-=========================
-
-Provides a test environment for the development and testing of Fastcapa. The environment is automatically validated after it is created to ensure that Fastcapa is behaving correctly.
-
-Two virtualized nodes are launched with Vagrant that can communicate with one another over a private network.
-- The `source` node uses Metron's `pcap_replay` functionality to transmit raw network packet data over a private network.
-- The `sink` node is running `fastcapa` and is capturing these network packets.
-- Fastcapa then transforms and bundles the packets into a message.
-- The message is sent to a Kafka broker running on the `source` node.
-
-Getting Started
----------------
-
-Simply run `vagrant up` to launch the environment. Automated tests are executed after provisioning completes to ensure that Fastcapa and the rest of the environment is functioning properly.
-
-```
-$ vagrant up
-==> source: Running provisioner: ansible...
- source: Running ansible-playbook...
-...
-TASK [debug] *******************************************************************
-ok: [source] => {
- "msg": "Successfully received packets sent from pcap-replay!"
-}
-...
-TASK [debug] *******************************************************************
-ok: [source] => {
- "msg": "Successfully received a Kafka message from fastcapa!"
-}
-```
-
-Going Deeper
-------------
-
-This section will outline in more detail the environment and how to interact with it.
-
-### `source`
-
-To validate that the `source` node is functioning properly, run the following commands.
-
-First, ensure that the `pcap-replay` service is running.
-
-```
-vagrant ssh source
-sudo service pcap-replay status
-```
-
-Use `tcpdump` to ensure that the raw packet data is being sent over the private network. Enter 'CTRL-C' to kill the `tcpdump` process once you are able to see that packets are being sent.
-
-```
-sudo yum -y install tcpdump
-sudo tcpdump -i enp0s8
-```
-
-### `sink`
-
-Next validate that the `sink` is functioning properly. Run the following commands starting from the host operating system.
-
-First, ensure that the `fastcapa` service is running.
-
-```
-vagrant ssh sink
-service fastcapa status
-```
-
-Ensure that the raw network packet data is being received by Kafka.
-
-```
-/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic pcap
-```
-
-Enter 'CTRL-C' to kill the `kafka-console-consumer` process once you are able to see that packets are being sent. These packets will appear to be gibberish in the console. This is the raw binary network packet data after all.
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/fastcapa-vagrant/Vagrantfile
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/fastcapa-vagrant/Vagrantfile b/metron-deployment/vagrant/fastcapa-vagrant/Vagrantfile
deleted file mode 100644
index 7f58417..0000000
--- a/metron-deployment/vagrant/fastcapa-vagrant/Vagrantfile
+++ /dev/null
@@ -1,71 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-Vagrant.configure("2") do |config|
-
- # enable hostmanager
- config.hostmanager.enabled = true
- config.hostmanager.manage_host = true
-
- #
- # source
- #
- config.vm.define "source" do |node|
-
- # host settings
- node.vm.hostname = "source"
- node.vm.box = "bento/centos-6.7"
- node.ssh.insert_key = "true"
- node.vm.network :private_network, ip: "192.168.33.10", netmask: "255.255.255.0"
-
- # provider
- node.vm.provider "virtualbox" do |vb|
- vb.memory = 1024
- vb.cpus = 1
- end
- end
-
- #
- # sink
- #
- config.vm.define "sink" do |node|
-
- # host settings
- node.vm.hostname = "sink"
- node.vm.box = "bento/centos-7.1"
- node.ssh.insert_key = "true"
- node.vm.network :private_network, ip: "192.168.33.11", netmask: "255.255.255.0"
-
- # provider
- node.vm.provider "virtualbox" do |vb|
- vb.memory = 4096
- vb.cpus = 3
-
- # network adapter settings; [Am79C970A|Am79C973|82540EM|82543GC|82545EM|virtio]
- vb.customize ["modifyvm", :id, "--nicpromisc2", "allow-all"]
- vb.customize ["modifyvm", :id, "--nictype2","82545EM"]
- end
-
- # provision host
- node.vm.provision :ansible do |ansible|
- ansible.limit = "all"
- ansible.playbook = "playbook.yml"
- end
- end
-
-
-end
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/fastcapa-vagrant/ansible.cfg
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/fastcapa-vagrant/ansible.cfg b/metron-deployment/vagrant/fastcapa-vagrant/ansible.cfg
deleted file mode 100644
index 9c650c2..0000000
--- a/metron-deployment/vagrant/fastcapa-vagrant/ansible.cfg
+++ /dev/null
@@ -1,22 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-[defaults]
-host_key_checking = false
-library = ../../extra_modules
-roles_path = ../../roles
-pipelining = True
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/fastcapa-vagrant/playbook.yml
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/fastcapa-vagrant/playbook.yml b/metron-deployment/vagrant/fastcapa-vagrant/playbook.yml
deleted file mode 100644
index 61509c4..0000000
--- a/metron-deployment/vagrant/fastcapa-vagrant/playbook.yml
+++ /dev/null
@@ -1,51 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
-#
-# the 'source' produces network traffic
-#
-- hosts: source
- become: yes
- vars_files:
- - vars/main.yml
- roles:
- - role: kafka-broker
- - role: pcap_replay
- - { role: pycapa, install_pycapa_service: False }
- - { role: sensor-test-mode, pcap_replay: True, install_yaf: False, install_snort: False }
-
-#
-# the 'sink' consumes network traffic
-#
-- hosts: sink
- become: yes
- vars_files:
- - vars/main.yml
- roles:
- - role: librdkafka
- - role: fastcapa
-
-#
-# validate the environment - needs to run on `source` node
-#
-- hosts: source
- become: yes
- vars_files:
- - vars/main.yml
- tasks:
- - include: tasks/validate-packets-sent.yml
- - include: tasks/validate-messages-received.yml
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/fastcapa-vagrant/tasks/validate-messages-received.yml
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/fastcapa-vagrant/tasks/validate-messages-received.yml b/metron-deployment/vagrant/fastcapa-vagrant/tasks/validate-messages-received.yml
deleted file mode 100644
index 5109a0e..0000000
--- a/metron-deployment/vagrant/fastcapa-vagrant/tasks/validate-messages-received.yml
+++ /dev/null
@@ -1,38 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
-- set_fact:
- kafka_consumer_cmd: "{{ kafka_broker_home}}/bin/kafka-console-consumer.sh --zookeeper {{ zookeeper_url }} --topic {{ fastcapa_topic }} --max-messages 10 --from-beginning"
-
-- name: Attempting to receive message(s) sent by fastcapa
- shell: "{{ kafka_consumer_cmd }}"
- async: 30
- register: kafka_consumer
-
-- name: Waiting to receive message(s) sent by fastcapa
- async_status: jid={{ kafka_consumer.ansible_job_id }}
- register: job
- until: job.finished
- retries: 30
-
-- fail:
- msg: "No messages received from fastcapa within timeout: {{ kafka_consumer_cmd }}"
- when: job.finished != 1
-
-- debug: var=job
-
-- debug: msg="Successfully received a Kafka message from fastcapa!"
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/fastcapa-vagrant/tasks/validate-packets-sent.yml
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/fastcapa-vagrant/tasks/validate-packets-sent.yml b/metron-deployment/vagrant/fastcapa-vagrant/tasks/validate-packets-sent.yml
deleted file mode 100644
index 25347dd..0000000
--- a/metron-deployment/vagrant/fastcapa-vagrant/tasks/validate-packets-sent.yml
+++ /dev/null
@@ -1,41 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
-- name: Install tcpdump
- yum: name=tcpdump
-
-- set_fact:
- sniff_cmd: "tcpdump -i {{ pcap_replay_interface }} -c 10"
-
-- name: "Attempting to sniff packet(s)"
- shell: "{{ sniff_cmd }}"
- async: 30
- register: sniffer
-
-- name: Waiting to receive packet(s)
- async_status: jid={{ sniffer.ansible_job_id }}
- register: job
- until: job.finished
- retries: 30
-
-- fail:
- msg: "No packets received from pcap-replay within timeout: {{ sniff_cmd }}"
- when: job.finished != 1
-
-- debug: var=job
-
-- debug: msg="Successfully received packets sent from pcap-replay!"
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/fastcapa-vagrant/vars/main.yml
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/fastcapa-vagrant/vars/main.yml b/metron-deployment/vagrant/fastcapa-vagrant/vars/main.yml
deleted file mode 100644
index 8e4584e..0000000
--- a/metron-deployment/vagrant/fastcapa-vagrant/vars/main.yml
+++ /dev/null
@@ -1,38 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
-dpdk_device: ["00:08.0"]
-dpdk_target: "x86_64-native-linuxapp-gcc"
-num_huge_pages: 512
-fastcapa_portmask: "0x01"
-fastcapa_topic: pcap
-kafka_broker_url: source:9092
-zookeeper_url: source:2181
-pcap_replay_interface: eth1
-kafka_broker_home: /usr/hdp/current/kafka-broker/
-
-# dummy variables for pycapa's dependence on ambari_gather_facts
-cluster_name: dummy
-namenode_host: dummy
-core_site_tag: dummy
-hdfs_url: dummy
-kafka_broker_hosts: dummy
-kafka_broker_tag: dummy
-kafka_broker_port: dummy
-zookeeper_hosts: dummy
-zookeeper_tag: dummy
-zookeeper_port: dummy
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/full-dev-platform/.gitignore
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/full-dev-platform/.gitignore b/metron-deployment/vagrant/full-dev-platform/.gitignore
new file mode 100644
index 0000000..8000dd9
--- /dev/null
+++ b/metron-deployment/vagrant/full-dev-platform/.gitignore
@@ -0,0 +1 @@
+.vagrant
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/full-dev-platform/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/full-dev-platform/README.md b/metron-deployment/vagrant/full-dev-platform/README.md
new file mode 100644
index 0000000..4242be2
--- /dev/null
+++ b/metron-deployment/vagrant/full-dev-platform/README.md
@@ -0,0 +1,102 @@
+Apache Metron on Virtualbox
+===========================
+
+This project fully automates the provisioning of Apache Metron on single, virtualized host running on Virtualbox. Metron is composed of many components and installing all of these on a single host, especially a virtualized one, will greatly stress your computer. To work sufficiently this will require at least 8 GB of RAM and a fair amount of patience.
+
+Getting Started
+---------------
+
+### Prerequisites
+
+The computer used to deploy Apache Metron will need to have the following components installed.
+
+ - [Ansible](https://github.com/ansible/ansible) 2.0.0.2
+ - [Vagrant](https://www.vagrantup.com) 1.8.1
+ - [Virtualbox](virtualbox.org) 5.0.16
+ - Python 2.7.11
+ - Maven 3.3.9
+
+Any platform that supports these tools is suitable, but the following instructions cover installation on Mac OS X only. The easiest means of installing these tools on a Mac is to use the excellent [Homebrew](http://brew.sh/) project.
+
+1. Install Homebrew by running the following command in a terminal. Refer to the [Homebrew](http://brew.sh/) home page for the latest installation instructions.
+
+ ```
+ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
+ ```
+
+2. With Homebrew installed, run the following command in a terminal to install all of the required tools.
+
+ ```
+ brew cask install vagrant virtualbox
+ brew install brew-pip maven git
+ pip install ansible==2.0.0.2
+ ```
+
+3. Install Vagrant Hostmanager.
+
+ ```
+ vagrant plugin install vagrant-hostmanager
+ ```
+
+### Metron
+
+Now that the hard part is done, start the Metron deployment process.
+
+1. Build Metron
+
+ ```
+ cd metron-platform
+ mvn clean package -DskipTests
+ ```
+
+2. Deploy Metron
+
+ ```
+ cd metron-deployment/vagrant/full-dev-platform
+ vagrant up
+ ```
+
+ Should the process fail before completing the deployment, the following command will continue the deployment process without re-instantiating the host.
+
+ ```
+ vagrant provision
+ ```
+
+ In addition to re-running the entire provisioning play book, you may now re-run an individual Ansible tag or a collection of tags in the following ways.
+
+ ```
+ ./run_ansible_role.sh web
+ ```
+ or
+ ```
+ vagrant --ansible-tags="web" provision
+ ```
+ Will re-run the web role on the Vagrant image. This will re-install (if necessary) and start the UI.
+
+ A collection of tags is specified as a comma separated list.
+
+ ```
+ ./run_ansbile_role.sh "sensors,enrichment"
+
+ ```
+
+ Tags are listed in the playbooks, some frequently used tags:
+ + hdp-install - Install HDP
+ + hdp-deploy - Deploy and Start HDP Services (will start all Hadoop Services)
+ + sensors - Deploy and Start Sensors.
+ + enrichment - Deploy and Start Enrichment Topology.
+
+ Note: there is a convienence script, ```./run_enrichment_role.sh```, which runs the enrichment tag.
+
+### Explore Metron
+
+Navigate to the following resources to explore your newly minted Apache Metron environment.
+
+ - [Metron](http://node1:8080)
+ - [Ambari](http://node1:5000)
+
+Connecting to the host through SSH is as simple as running the following command.
+
+ ```
+ vagrant ssh
+ ```
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/full-dev-platform/Vagrantfile
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/full-dev-platform/Vagrantfile b/metron-deployment/vagrant/full-dev-platform/Vagrantfile
new file mode 100644
index 0000000..1061dcc
--- /dev/null
+++ b/metron-deployment/vagrant/full-dev-platform/Vagrantfile
@@ -0,0 +1,92 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+require 'getoptlong'
+
+ansibleTags=''
+ansibleSkipTags=''
+
+begin
+ opts = GetoptLong.new(
+ [ '--ansible-tags', GetoptLong::OPTIONAL_ARGUMENT ],
+ [ '--ansible-skip-tags', GetoptLong::OPTIONAL_ARGUMENT ]
+ )
+
+ opts.quiet = TRUE
+
+ opts.each do |opt, arg|
+ case opt
+ when '--ansible-tags'
+ ansibleTags=arg
+ when '--ansible-skip-tags'
+ ansibleSkipTags=arg
+ end
+ end
+rescue Exception => ignored
+#Ignore to allow other opts to be passed to Vagrant
+end
+
+puts " Running with ansible-tags: " + ansibleTags.split(",").to_s if ansibleTags != ''
+puts " Running with ansible-skip-tags: " + ansibleSkipTags.split(",").to_s if ansibleSkipTags != ''
+
+hosts = [{
+ hostname: "node1",
+ ip: "192.168.66.121",
+ memory: "8192",
+ cpus: 4,
+ promisc: 2 # enables promisc on the 'Nth' network interface
+}]
+
+Vagrant.configure(2) do |config|
+
+ # all hosts built on centos 6
+ config.vm.box = "bento/centos-6.7"
+ config.ssh.insert_key = true
+
+ # enable the hostmanager plugin
+ config.hostmanager.enabled = true
+ config.hostmanager.manage_host = true
+
+ # host definition
+ hosts.each_with_index do |host, index|
+ config.vm.define host[:hostname] do |node|
+
+ # host settings
+ node.vm.hostname = host[:hostname]
+ node.vm.network "private_network", ip: host[:ip]
+
+ # vm settings
+ node.vm.provider "virtualbox" do |vb|
+ vb.memory = host[:memory]
+ vb.cpus = host[:cpus]
+
+ # enable promisc mode on the network interface
+ if host.has_key?(:promisc)
+ vb.customize ["modifyvm", :id, "--nicpromisc#{host[:promisc]}", "allow-all"]
+ end
+ end
+ end
+ end
+
+ # provisioning
+ config.vm.provision :ansible do |ansible|
+ ansible.playbook = "../../playbooks/metron_full_install.yml"
+ ansible.sudo = true
+ ansible.tags = ansibleTags.split(",") if ansibleTags != ''
+ ansible.skip_tags = ansibleSkipTags.split(",") if ansibleSkipTags != ''
+ ansible.inventory_path = "../../inventory/full-dev-platform"
+ end
+end
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/full-dev-platform/ansible.cfg
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/full-dev-platform/ansible.cfg b/metron-deployment/vagrant/full-dev-platform/ansible.cfg
new file mode 100644
index 0000000..74f3fdf
--- /dev/null
+++ b/metron-deployment/vagrant/full-dev-platform/ansible.cfg
@@ -0,0 +1,27 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+[defaults]
+host_key_checking = false
+library = ../../extra_modules
+roles_path = ../../roles
+pipelining = True
+log_path = ./ansible.log
+
+
+# fix for "ssh throws 'unix domain socket too long' " problem
+[ssh_connection]
+control_path = %(directory)s/%%h-%%p-%%r
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/full-dev-platform/run_ansible_role.sh
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/full-dev-platform/run_ansible_role.sh b/metron-deployment/vagrant/full-dev-platform/run_ansible_role.sh
new file mode 100755
index 0000000..0e5d831
--- /dev/null
+++ b/metron-deployment/vagrant/full-dev-platform/run_ansible_role.sh
@@ -0,0 +1,20 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+vagrant --ansible-tags="$@" provision
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/full-dev-platform/run_enrichment_role.sh
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/full-dev-platform/run_enrichment_role.sh b/metron-deployment/vagrant/full-dev-platform/run_enrichment_role.sh
new file mode 100755
index 0000000..e26bc16
--- /dev/null
+++ b/metron-deployment/vagrant/full-dev-platform/run_enrichment_role.sh
@@ -0,0 +1,20 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+./run_ansible_role.sh enrichment
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/quick-dev-platform/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/quick-dev-platform/README.md b/metron-deployment/vagrant/quick-dev-platform/README.md
new file mode 100644
index 0000000..6d9a2d0
--- /dev/null
+++ b/metron-deployment/vagrant/quick-dev-platform/README.md
@@ -0,0 +1,61 @@
+Developer Image for Apache Metron on Virtualbox
+===============================================
+
+This image is designed for more rapid deployment of Metron for use by Metron developers. It uses a base image that has been pre-loaded with Ambari and HDP.
+
+Getting Started
+---------------
+
+### Prerequisites
+
+As with the Singlenode Full Image, the computer used to deploy Apache Metron will need to have the following components installed.
+
+ - [Ansible](https://github.com/ansible/ansible) 2.0.0.2
+ - [Vagrant](https://www.vagrantup.com) 1.8.1
+ - [Virtualbox](virtualbox.org) 5.0.16
+ - Python 2.7.11
+ - Maven 3.3.9
+
+
+### Launch the Metron Development Image
+
+Now that the hard part is done, start the Metron deployment process.
+
+1. Build Metron
+
+ ```
+ cd metron-platform
+ mvn clean package -DskipTests
+ ```
+
+2. Deploy Metron
+
+ ```
+ cd metron-deployment/vagrant/quick-dev-platform
+ ./launch_dev_image.sh
+ ```
+
+ Should the process fail before completing the deployment, the following command will continue the deployment process without re-instantiating the host.
+
+ ```
+ vagrant --ansible-tags="hdp-deploy,metron" provision
+ ```
+
+### Work with Metron
+
+As you build out new capabilities for Metron, you will need to re-deploy the Storm topologies. To do so, first HALT the running Storm topologies and then run:
+
+```
+./run_enrichment_role.sh
+```
+
+Remember Navigate to the following resources to explore your newly minted Apache Metron environment.
+
+ - [Metron](http://node1:8080)
+ - [Ambari](http://node1:5000)
+
+Connecting to the host through SSH is as simple as running the following command.
+
+ ```
+ vagrant ssh
+ ```
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/quick-dev-platform/Vagrantfile
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/quick-dev-platform/Vagrantfile b/metron-deployment/vagrant/quick-dev-platform/Vagrantfile
new file mode 100644
index 0000000..1509252
--- /dev/null
+++ b/metron-deployment/vagrant/quick-dev-platform/Vagrantfile
@@ -0,0 +1,94 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+require 'getoptlong'
+
+ansibleTags=''
+ansibleSkipTags=''
+
+begin
+ opts = GetoptLong.new(
+ [ '--ansible-tags', GetoptLong::OPTIONAL_ARGUMENT ],
+ [ '--ansible-skip-tags', GetoptLong::OPTIONAL_ARGUMENT ]
+ )
+
+ opts.quiet = TRUE
+
+ opts.each do |opt, arg|
+ case opt
+ when '--ansible-tags'
+ ansibleTags=arg
+ when '--ansible-skip-tags'
+ ansibleSkipTags=arg
+ end
+ end
+rescue Exception => ignored
+#Ignore to allow other opts to be passed to Vagrant
+end
+
+puts " Running with ansible-tags: " + ansibleTags.split(",").to_s if ansibleTags != ''
+puts " Running with ansible-skip-tags: " + ansibleSkipTags.split(",").to_s if ansibleSkipTags != ''
+
+hosts = [{
+ hostname: "node1",
+ ip: "192.168.66.121",
+ memory: "8192",
+ cpus: 4,
+ promisc: 2 # enables promisc on the 'Nth' network interface
+}]
+
+Vagrant.configure(2) do |config|
+
+ # all hosts built on centos 6
+ config.vm.box = "metron/hdp-base"
+ config.ssh.insert_key = true
+
+ # enable the hostmanager plugin
+ config.hostmanager.enabled = true
+ config.hostmanager.manage_host = true
+
+ # host definition
+ hosts.each_with_index do |host, index|
+ config.vm.define host[:hostname] do |node|
+
+ # host settings
+ node.vm.hostname = host[:hostname]
+ node.vm.network "private_network", ip: host[:ip]
+
+ # vm settings
+ node.vm.provider "virtualbox" do |vb|
+ vb.memory = host[:memory]
+ vb.cpus = host[:cpus]
+
+ # enable promisc mode on the network interface
+ if host.has_key?(:promisc)
+ vb.customize ["modifyvm", :id, "--nicpromisc#{host[:promisc]}", "allow-all"]
+ end
+ end
+ end
+ end
+
+ if ansibleSkipTags != '' or ansibleTags != ''
+ config.vm.provision :ansible do |ansible|
+ ansible.playbook = "../../playbooks/metron_full_install.yml"
+ ansible.sudo = true
+ ansible.tags = ansibleTags.split(",") if ansibleTags != ''
+ ansible.skip_tags = ansibleSkipTags.split(",") if ansibleSkipTags != ''
+ ansible.inventory_path = "../../inventory/full-dev-platform"
+ end
+ end
+
+end
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/quick-dev-platform/ansible.cfg
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/quick-dev-platform/ansible.cfg b/metron-deployment/vagrant/quick-dev-platform/ansible.cfg
new file mode 100644
index 0000000..74f3fdf
--- /dev/null
+++ b/metron-deployment/vagrant/quick-dev-platform/ansible.cfg
@@ -0,0 +1,27 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+[defaults]
+host_key_checking = false
+library = ../../extra_modules
+roles_path = ../../roles
+pipelining = True
+log_path = ./ansible.log
+
+
+# fix for "ssh throws 'unix domain socket too long' " problem
+[ssh_connection]
+control_path = %(directory)s/%%h-%%p-%%r
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/quick-dev-platform/launch_dev_image.sh
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/quick-dev-platform/launch_dev_image.sh b/metron-deployment/vagrant/quick-dev-platform/launch_dev_image.sh
new file mode 100755
index 0000000..dc91679
--- /dev/null
+++ b/metron-deployment/vagrant/quick-dev-platform/launch_dev_image.sh
@@ -0,0 +1,21 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+vagrant --ansible-tags="hdp-deploy,metron" up
+
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/ec431b8d/metron-deployment/vagrant/quick-dev-platform/run_ansible_role.sh
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/quick-dev-platform/run_ansible_role.sh b/metron-deployment/vagrant/quick-dev-platform/run_ansible_role.sh
new file mode 100755
index 0000000..0e5d831
--- /dev/null
+++ b/metron-deployment/vagrant/quick-dev-platform/run_ansible_role.sh
@@ -0,0 +1,20 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+vagrant --ansible-tags="$@" provision