You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@metron.apache.org by ni...@apache.org on 2017/11/30 15:30:55 UTC
metron git commit: METRON-1239 Drop extra dev environments
(nickwallen) closes apache/metron#852
Repository: metron
Updated Branches:
refs/heads/master 5a1bb6784 -> f08440481
METRON-1239 Drop extra dev environments (nickwallen) closes apache/metron#852
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/f0844048
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/f0844048
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/f0844048
Branch: refs/heads/master
Commit: f08440481bb95e135e59deadc6f6fd35775c66a2
Parents: 5a1bb67
Author: nickwallen <ni...@nickallen.org>
Authored: Thu Nov 30 09:58:39 2017 -0500
Committer: nickallen <ni...@apache.org>
Committed: Thu Nov 30 09:58:39 2017 -0500
----------------------------------------------------------------------
metron-deployment/Kerberos-manual-setup.md | 20 +-
metron-deployment/README.md | 9 +-
metron-deployment/amazon-ec2/run.sh | 2 +-
.../inventory/quick-dev-platform/group_vars/all | 89 -------
.../inventory/quick-dev-platform/hosts | 47 ----
.../packaging/docker/ansible-docker/README.md | 5 +-
.../packaging/packer-build/README.md | 18 +-
.../packer-build/quick-dev-centos-6.7.json | 132 -----------
metron-deployment/playbooks/metron_install.yml | 11 +-
metron-deployment/roles/quick_dev/meta/main.yml | 19 --
.../roles/quick_dev/tasks/main.yml | 75 ------
metron-deployment/roles/sensor-stubs/README.md | 10 +-
metron-deployment/vagrant/README.md | 2 -
.../vagrant/codelab-platform/README.md | 65 -----
.../vagrant/codelab-platform/Vagrantfile | 94 --------
.../vagrant/codelab-platform/ansible.cfg | 27 ---
.../vagrant/codelab-platform/run.sh | 20 --
.../codelab-platform/run_ansible_role.sh | 20 --
.../codelab-platform/run_enrichment_role.sh | 20 --
.../vagrant/full-dev-platform/Vagrantfile | 2 +-
.../vagrant/quick-dev-platform/README.md | 77 ------
.../vagrant/quick-dev-platform/Vagrantfile | 94 --------
.../vagrant/quick-dev-platform/ansible.cfg | 27 ---
.../quick-dev-platform/run_ansible_role.sh | 20 --
.../quick-dev-platform/run_enrichment_role.sh | 20 --
metron-interface/metron-alerts/README.md | 4 +-
metron-interface/metron-config/README.md | 2 +-
metron-interface/metron-rest/README.md | 28 +--
.../apache/metron/rest/utils/ReadMeUtils.java | 130 ----------
.../metron-rest/src/test/resources/README.vm | 236 -------------------
30 files changed, 47 insertions(+), 1278 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/f0844048/metron-deployment/Kerberos-manual-setup.md
----------------------------------------------------------------------
diff --git a/metron-deployment/Kerberos-manual-setup.md b/metron-deployment/Kerberos-manual-setup.md
index f42b7ed..5d44b2a 100644
--- a/metron-deployment/Kerberos-manual-setup.md
+++ b/metron-deployment/Kerberos-manual-setup.md
@@ -1,7 +1,7 @@
Kerberos Setup
==============
-This document provides instructions for kerberizing Metron's Vagrant-based development environments; "Quick Dev" and "Full Dev". These instructions do not cover the Ambari MPack or sensors. General Kerberization notes can be found in the metron-deployment [README.md](../README.md).
+This document provides instructions for kerberizing Metron's Vagrant-based development environments. These instructions do not cover the Ambari MPack or sensors. General Kerberization notes can be found in the metron-deployment [README.md](../README.md).
* [Setup](#setup)
* [Setup a KDC](#setup-a-kdc)
@@ -17,7 +17,7 @@ This document provides instructions for kerberizing Metron's Vagrant-based devel
Setup
-----
-1. Deploy a Vagrant development environment; either [Full Dev](vagrant/full-dev-platform/README.md) or [Quick Dev](vagrant/quick-dev-platform/README.md).
+1. Deploy the [development environment.](vagrant/full-dev-platform/README.md).
1. Export the following environment variables. These need to be set for the remainder of the instructions. Replace `node1` with the appropriate hosts, if you are running Metron anywhere other than Vagrant.
@@ -73,7 +73,7 @@ Setup a KDC
cp -f /etc/krb5.conf /var/lib/ambari-server/resources/scripts
```
-1. Ensure that the KDC can issue renewable tickets. This may be necessary on a real cluster, but should not be on [Full Dev](vagrant/full-dev-platform/README.md).
+1. Ensure that the KDC can issue renewable tickets. This may be necessary on a real cluster, but should not be on [Full Dev](vagrant/full-dev-platform/README.md).
Edit `/var/kerberos/krb5kdc/kdc.conf` and ensure the following is added to the `realm` section
@@ -93,9 +93,9 @@ Setup a KDC
/etc/rc.d/init.d/krb5kdc start
chkconfig krb5kdc on
```
-
+
1. Start the Kerberos Admin service and ensure that it starts on boot.
-
+
```
/etc/rc.d/init.d/kadmin start
chkconfig kadmin on
@@ -106,9 +106,9 @@ Setup a KDC
```
kadmin.local -q "addprinc admin/admin"
```
-
+
1. Setup the `metron` principal. You will `kinit` as the `metron` principal when running topologies. You will be prompted for a password; do not forget it.
-
+
```
kadmin.local -q "addprinc metron"
```
@@ -120,7 +120,7 @@ Verify KDC
1. Ticket renewal is disallowed by default in many Linux distributions. If the KDC cannot issue renewable tickets, an error will be thrown when starting Metron's Storm topologies:
```
- Exception in thread "main" java.lang.RuntimeException:
+ Exception in thread "main" java.lang.RuntimeException:
java.lang.RuntimeException: The TGT found is not renewable
```
@@ -132,8 +132,8 @@ Verify KDC
* If the 'R' flags are present, you may skip to next section.
* If the 'R' flags are absent, you will need to follow the below steps:
-
-1. If the KDC is already setup, then editing `max_life` and `max_renewable_life` in `/var/kerberos/krb5kdc/kdc.conf`, then restarting `kadmin` and `krb5kdc` services will not change the policies for existing users.
+
+1. If the KDC is already setup, then editing `max_life` and `max_renewable_life` in `/var/kerberos/krb5kdc/kdc.conf`, then restarting `kadmin` and `krb5kdc` services will not change the policies for existing users.
You need to set the renew lifetime for existing users and the `krbtgt` realm. Modify the appropriate principals to allow renewable tickets using the following commands. Adjust the parameters to match your desired KDC parameters:
http://git-wip-us.apache.org/repos/asf/metron/blob/f0844048/metron-deployment/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/README.md b/metron-deployment/README.md
index 133f523..73e7e87 100644
--- a/metron-deployment/README.md
+++ b/metron-deployment/README.md
@@ -32,9 +32,7 @@ The Ambari playbook will install a Hadoop cluster including the Metron Services
Currently, the playbooks supports building a local development cluster running on one node or deploying to a 10 node cluster on AWS EC2.
## Vagrant
-There are current two Vagrant modes, full-dev and quick-dev. Full-dev installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure.
-Quick-dev re-installs the core Metron Services (Parsing, Enrichment, and Indexing)on a pre-built instance.
-Use quick-dev for testing out changes to core Metron services.
+There is a development environment based on Vagrant that is referred to as "Full Dev". This installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure.
### Prerequsities
- Install [Vagrant](https://www.vagrantup.com/) (5.0.16+)
@@ -44,9 +42,6 @@ installed
### Full-Dev
Navigate to `metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`.
-### Quick-Dev
-Navigate to `metron/metron-deployment/vagrant/quick-dev-platform` and run `vagrant up`.
-
## Ambari Management Pack
An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update.
@@ -148,7 +143,7 @@ rpm -i <package>
The dashboards installed by the Kibana custom action are managed by the dashboard.p file. This file is created by exporting existing dashboards from a running Kibana instance.
-To create a new version of the file, make any necessary changes to Kibana (e.g. on quick-dev), and export with the appropriate script.
+To create a new version of the file, make any necessary changes to Kibana (e.g. on full-dev), and export with the appropriate script.
```
python packaging/ambari/metron-mpack/src/main/resources/common-services/KIBANA/4.5.1/package/scripts/dashboard/dashboardindex.py \
http://git-wip-us.apache.org/repos/asf/metron/blob/f0844048/metron-deployment/amazon-ec2/run.sh
----------------------------------------------------------------------
diff --git a/metron-deployment/amazon-ec2/run.sh b/metron-deployment/amazon-ec2/run.sh
index bdaabf0..96c07af 100755
--- a/metron-deployment/amazon-ec2/run.sh
+++ b/metron-deployment/amazon-ec2/run.sh
@@ -62,6 +62,6 @@ $DEPLOYDIR/../scripts/platform-info.sh >> $LOGFILE
cd $DEPLOYDIR
export EC2_INI_PATH=conf/ec2.ini
ansible-playbook -i ec2.py playbook.yml \
- --skip-tags="quick_dev,sensor-stubs" \
+ --skip-tags="sensor-stubs" \
--extra-vars="env=$ENV" \
$EXTRA_ARGS
http://git-wip-us.apache.org/repos/asf/metron/blob/f0844048/metron-deployment/inventory/quick-dev-platform/group_vars/all
----------------------------------------------------------------------
diff --git a/metron-deployment/inventory/quick-dev-platform/group_vars/all b/metron-deployment/inventory/quick-dev-platform/group_vars/all
deleted file mode 100644
index d2d8590..0000000
--- a/metron-deployment/inventory/quick-dev-platform/group_vars/all
+++ /dev/null
@@ -1,89 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-# which services should be started?
-services_to_start:
- - pcap-service
- - snort
- - snort-logs
- - bro
- - pcap-replay
-
-# ambari
-ambari_host: "{{ groups.ambari_master[0] }}"
-hdp_host_group: "{{ groups.ambari_slave }}"
-ambari_port: 8080
-ambari_user: admin
-ambari_password: admin
-cluster_type: single_node_vm
-ambari_server_mem: 512
-
-# hbase
-pcap_hbase_table: pcap
-tracker_hbase_table: access_tracker
-threatintel_hbase_table: threatintel
-enrichment_hbase_table: enrichment
-
-# metron
-metron_version: 0.4.2
-metron_directory: /usr/metron/{{ metron_version }}
-bro_version: "2.5.2"
-fixbuf_version: "1.7.1"
-yaf_version: "2.8.0"
-daq_version: "2.0.6-1"
-pycapa_repo: "https://github.com/OpenSOC/pycapa.git"
-pycapa_home: "/opt/pycapa"
-snort_version: "2.9.8.0-1"
-snort_alert_csv_path: "/var/log/snort/alert.csv"
-threat_intel_bulk_load: False
-
-# data directories - only required to override defaults
-zookeeper_data_dir: "/data1/hadoop/zookeeper"
-namenode_checkpoint_dir: "/data1/hadoop/hdfs/namesecondary"
-namenode_name_dir: "/data1/hadoop/hdfs/namenode"
-datanode_data_dir: "/data1/hadoop/hdfs/data,/data2/hadoop/hdfs/data"
-journalnode_edits_dir: "/data1/hadoop/hdfs/journalnode"
-nodemanager_local_dirs: "/data1/hadoop/yarn/local"
-timeline_ldb_store_path: "/data1/hadoop/yarn/timeline"
-timeline_ldb_state_path: "/data1/hadoop/yarn/timeline"
-nodemanager_log_dirs: "/data1/hadoop/yarn/log"
-jhs_recovery_store_ldb_path: "/data1/hadoop/mapreduce/jhs"
-storm_local_dir: "/data1/hadoop/storm"
-kafka_log_dirs: "/data1/kafka-log"
-elasticsearch_data_dir: "/data1/elasticsearch,/data2/elasticsearch"
-
-# sensors
-sensor_test_mode: True
-install_pycapa: False
-install_bro: True
-install_snort: True
-install_yaf: False
-install_pcap_replay: True
-sniff_interface: eth1
-pcap_replay_interface: "{{ sniff_interface }}"
-pcapservice_port: 8081
-
-# search
-install_elasticsearch: True
-install_solr: False
-solr_collection_name: Metron
-solr_number_shards: 1
-solr_replication_factor: 1
-elasticsearch_transport_port: 9300
-## The elasticsearch_network_interface must be in the form specified for Elasticsearch, with leading and trailing underscores.
-elasticsearch_network_interface: _eth1_
-elasticsearch_web_port: 9200
http://git-wip-us.apache.org/repos/asf/metron/blob/f0844048/metron-deployment/inventory/quick-dev-platform/hosts
----------------------------------------------------------------------
diff --git a/metron-deployment/inventory/quick-dev-platform/hosts b/metron-deployment/inventory/quick-dev-platform/hosts
deleted file mode 100644
index 9bd9ea1..0000000
--- a/metron-deployment/inventory/quick-dev-platform/hosts
+++ /dev/null
@@ -1,47 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-[ambari_master]
-node1
-
-[ambari_slave]
-node1
-
-[metron]
-node1
-
-[search]
-node1
-
-[sensors]
-node1
-
-[pcap_server]
-node1
-
-[web]
-node1
-
-[zeppelin]
-node1
-
-[monit:children]
-sensors
-pcap_server
-
-[local]
-127.0.0.1
http://git-wip-us.apache.org/repos/asf/metron/blob/f0844048/metron-deployment/packaging/docker/ansible-docker/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/docker/ansible-docker/README.md b/metron-deployment/packaging/docker/ansible-docker/README.md
index 0c41d10..1d0d22b 100644
--- a/metron-deployment/packaging/docker/ansible-docker/README.md
+++ b/metron-deployment/packaging/docker/ansible-docker/README.md
@@ -25,11 +25,11 @@ After running the container:
2. run build commands, for example:
- build metron without tests : `mvn clean package -DskipTests`
- build metron and build the rpms as well : `mvn clean install && cd metron-deployment && mvn package -P build-rpms`
-
+
If you wish to use this build with a vagrant instance, then after building with rpms as above, modify
your usual vagrant up command to skip the build role, as so:
-`vagrant --ansible-skip-tags="build,quick_dev" up`
+`vagrant --ansible-skip-tags="build" up`
## Using the Container for deployment
@@ -37,4 +37,3 @@ your usual vagrant up command to skip the build role, as so:
> Note these instructions are outdated
Full instructions are found on the wiki at https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65144361
-
http://git-wip-us.apache.org/repos/asf/metron/blob/f0844048/metron-deployment/packaging/packer-build/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/packer-build/README.md b/metron-deployment/packaging/packer-build/README.md
index d0c5484..78217a0 100644
--- a/metron-deployment/packaging/packer-build/README.md
+++ b/metron-deployment/packaging/packer-build/README.md
@@ -6,7 +6,6 @@ Based on the fantastic [Bento](https://github.com/chef/bento) project developed
Images Provided
---------------------
- base-centos-6.7: Centos 6.7 + HDP. Used in the full-dev-platform Vagrant image
-- quick-dev-centos-6.7: Centos 6.7 + HDP + Metron. Used for the quick-dev-platform Vagrant image.
Prerequisites
---------------------
@@ -14,27 +13,23 @@ Prerequisites
- [Virtualbox](https://www.virtualbox.org/) 5.0.16+ (Tested with 5.0.20)
Build Both Images
-----------------------
+----------------------
Navigate to \<your-project-directory\>/metron-deployment/packer-build
Execute bin/bento build
-
+
Packer will build both images and export .box files to the ./builds directory.
-
+
Build Single Images
-----------------------
+----------------------
Navigate to *your-project-directory*/metron-deployment/packer-build
* Base Centos (full-dev)
```
bin/bento build base-centos-6.7.json
```
- * Quick Dev
- ```
-bin/bento build quick-dev-centos-6.7.json
-```
Using Your New Box File
-----------------------
-Modify the relevant Vagrantfile (full-dev-platform or quick-dev-platform) replacing the lines:
+----------------------
+Modify the relevant Vagrantfile (full-dev-platform) replacing the lines:
```
<pre><code>config.vm.box = "<i>box_name</i>"
config.ssh.insert_key = true</code></pre>
@@ -48,4 +43,3 @@ config.ssh.insert_key = true</code></pre>
Launch the image as usual.
Node: Vagrant will cache boxes, you can force Vagrant to reload your box by running <code>vagrant box remove <i>test_box_name</i></code> before launching your new image.
-
http://git-wip-us.apache.org/repos/asf/metron/blob/f0844048/metron-deployment/packaging/packer-build/quick-dev-centos-6.7.json
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/packer-build/quick-dev-centos-6.7.json b/metron-deployment/packaging/packer-build/quick-dev-centos-6.7.json
deleted file mode 100644
index 96680cb..0000000
--- a/metron-deployment/packaging/packer-build/quick-dev-centos-6.7.json
+++ /dev/null
@@ -1,132 +0,0 @@
-{
- "builders": [
- {
- "type": "virtualbox-iso",
- "boot_command": [
- "<tab> text ks=http://{{ .HTTPIP }}:{{ .HTTPPort }}/{{user `ks_path`}} ksdevice=eth0<enter><wait>"
- ],
- "boot_wait": "10s",
- "disk_size": 40960,
- "guest_additions_path": "VBoxGuestAdditions_{{.Version}}.iso",
- "guest_os_type": "RedHat_64",
- "hard_drive_interface": "sata",
- "headless": "{{ user `headless` }}",
- "http_directory": "http",
- "iso_checksum": "{{user `iso_checksum`}}",
- "iso_checksum_type": "{{user `iso_checksum_type`}}",
- "iso_url": "{{user `mirror`}}/{{user `mirror_directory`}}/{{user `iso_name`}}",
- "output_directory": "packer-{{user `template`}}-virtualbox",
- "shutdown_command": "echo 'vagrant' | sudo -S /sbin/halt -h -p",
- "ssh_password": "vagrant",
- "ssh_port": 22,
- "ssh_username": "vagrant",
- "ssh_wait_timeout": "10000s",
- "vboxmanage": [
- [
- "modifyvm",
- "{{.Name}}",
- "--memory",
- "{{ user `memory` }}"
- ],
- [
- "modifyvm",
- "{{.Name}}",
- "--nic2",
- "hostonly"
- ],
- [
- "modifyvm",
- "{{.Name}}",
- "--hostonlyadapter2",
- "vboxnet0"
- ],
- [
- "modifyvm",
- "{{.Name}}",
- "--cpus",
- "{{ user `cpus` }}"
- ]
- ],
- "virtualbox_version_file": ".vbox_version",
- "vm_name": "{{ user `template` }}"
- }
- ],
- "post-processors": [
- {
- "output": "builds/{{user `box_basename`}}.{{.Provider}}.box",
- "type": "vagrant",
- "compression_level": 9
- }
- ],
- "provisioners": [
- {
- "destination": "/tmp/bento-metadata.json",
- "source": "{{user `metadata`}}",
- "type": "file"
- },
- {
- "type": "shell",
- "scripts": [
- "scripts/common/metadata.sh",
- "scripts/common/sshd.sh",
- "scripts/centos/networking.sh",
- "scripts/common/vagrant.sh",
- "scripts/centos/update_all.sh",
- "scripts/common/reboot.sh"
- ],
- "environment_vars": [
- "HOME_DIR=/home/vagrant",
- "http_proxy={{user `http_proxy`}}",
- "https_proxy={{user `https_proxy`}}",
- "no_proxy={{user `no_proxy`}}"
- ],
- "execute_command": "echo 'vagrant' | {{.Vars}} sudo -S -E sh -eux '{{.Path}}'"
- },
- {
- "type" : "ansible",
- "pause_before" : "2m",
- "playbook_file" : "./ansible/playbook.yml",
- "sftp_command": "/usr/libexec/openssh/sftp-server",
- "user" : "vagrant",
- "host_alias" : "node1",
- "groups" : ["packer","ambari_master","ambari_slave","metron","search","web","sensors","pcap_server","zeppelin","monit"],
- "extra_arguments": [ "--extra-vars","metron_rpm_glob=../../packaging/docker/rpm-docker/target/RPMS/noarch/*.rpm metron_build_dir=../../..","--skip-tags","report,sensors,quick_dev" ]
- },
- {
- "type": "shell",
- "scripts": [
- "scripts/common/vmtools.sh",
- "scripts/centos/cleanup.sh",
- "scripts/common/minimize.sh"
- ],
- "environment_vars": [
- "HOME_DIR=/home/vagrant",
- "http_proxy={{user `http_proxy`}}",
- "https_proxy={{user `https_proxy`}}",
- "no_proxy={{user `no_proxy`}}"
- ],
- "execute_command": "echo 'vagrant' | {{.Vars}} sudo -S -E sh -eux '{{.Path}}'"
- }
- ],
- "variables": {
- "box_basename": "quick-dev-centos-6.7",
- "build_timestamp": "{{isotime \"20060102150405\"}}",
- "cpus": "1",
- "git_revision": "__unknown_git_revision__",
- "headless": "",
- "http_proxy": "{{env `http_proxy`}}",
- "https_proxy": "{{env `https_proxy`}}",
- "iso_checksum": "c0c1a05d3d74fb093c6232003da4b22b0680f59d3b2fa2cb7da736bc40b3f2c5",
- "iso_checksum_type": "sha256",
- "iso_name": "CentOS-6.7-x86_64-bin-DVD1.iso",
- "ks_path": "centos-6.7/ks.cfg",
- "memory": "8192",
- "metadata": "floppy/dummy_metadata.json",
- "mirror": "http://archive.kernel.org/centos-vault",
- "mirror_directory": "6.7/isos/x86_64",
- "name": "quick-dev-centos-6.7",
- "no_proxy": "{{env `no_proxy`}}",
- "template": "quick-dev-centos-6.7",
- "version": "2.1.TIMESTAMP"
- }
-}
http://git-wip-us.apache.org/repos/asf/metron/blob/f0844048/metron-deployment/playbooks/metron_install.yml
----------------------------------------------------------------------
diff --git a/metron-deployment/playbooks/metron_install.yml b/metron-deployment/playbooks/metron_install.yml
index 333207a..998394d 100644
--- a/metron-deployment/playbooks/metron_install.yml
+++ b/metron-deployment/playbooks/metron_install.yml
@@ -18,13 +18,10 @@
- hosts: metron
become: true
roles:
- - role: ambari_slave
- - { role: metron-builder, tags: ['build'] }
- - role: metron-rpms
- - role: quick_dev
- tags:
- - quick_dev
- -
+ - { role: ambari_slave }
+ - { role: metron-builder, tags: ['build'] }
+ - { role: metron-rpms }
+
- hosts: ec2
become: true
tasks:
http://git-wip-us.apache.org/repos/asf/metron/blob/f0844048/metron-deployment/roles/quick_dev/meta/main.yml
----------------------------------------------------------------------
diff --git a/metron-deployment/roles/quick_dev/meta/main.yml b/metron-deployment/roles/quick_dev/meta/main.yml
deleted file mode 100644
index c3d807b..0000000
--- a/metron-deployment/roles/quick_dev/meta/main.yml
+++ /dev/null
@@ -1,19 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
-dependencies:
- - ambari_gather_facts
http://git-wip-us.apache.org/repos/asf/metron/blob/f0844048/metron-deployment/roles/quick_dev/tasks/main.yml
----------------------------------------------------------------------
diff --git a/metron-deployment/roles/quick_dev/tasks/main.yml b/metron-deployment/roles/quick_dev/tasks/main.yml
deleted file mode 100644
index 341e02e..0000000
--- a/metron-deployment/roles/quick_dev/tasks/main.yml
+++ /dev/null
@@ -1,75 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
-- name: Delete the Metron Components from Ambari
- ambari_service_state:
- host: "{{ groups.ambari_master[0] }}"
- port: "{{ ambari_port }}"
- username: "{{ ambari_user }}"
- password: "{{ ambari_password }}"
- cluster_name: "{{ cluster_name }}"
- state: deleted
- component_name: "{{ item }}"
- component_host: "{{ inventory_hostname }}"
- with_items:
- - METRON_ENRICHMENT_MASTER
- - METRON_INDEXING
- - METRON_PARSERS
-
-- name: Remove the Metron packages
- package:
- name: "{{ item }}"
- state: absent
- with_items:
- - metron-common
- - metron-data-management
- - metron-parsers
- - metron-enrichment
- - metron-indexing
- - metron-elasticsearch
-
-- name: Create local repo with new packages
- shell: "createrepo /localrepo"
- args:
- creates: /localrepo/repodata
-
-- name: Re-install the Metron Packages via Ambari
- ambari_service_state:
- host: "{{ groups.ambari_master[0] }}"
- port: "{{ ambari_port }}"
- username: "{{ ambari_user }}"
- password: "{{ ambari_password }}"
- cluster_name: "{{ cluster_name }}"
- state: stopped
- component_name: "{{ item }}"
- component_host: "{{ inventory_hostname }}"
- wait_for_complete: True
- with_items:
- - METRON_ENRICHMENT_MASTER
- - METRON_INDEXING
- - METRON_PARSERS
-
-- name: Start the ambari cluster
- ambari_cluster_state:
- host: "{{ groups.ambari_master[0] }}"
- port: "{{ ambari_port }}"
- username: "{{ ambari_user }}"
- password: "{{ ambari_password }}"
- cluster_name: "{{ cluster_name }}"
- cluster_state: started
- wait_for_complete: True
-
http://git-wip-us.apache.org/repos/asf/metron/blob/f0844048/metron-deployment/roles/sensor-stubs/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/roles/sensor-stubs/README.md b/metron-deployment/roles/sensor-stubs/README.md
index be9b7af..5acd36e 100644
--- a/metron-deployment/roles/sensor-stubs/README.md
+++ b/metron-deployment/roles/sensor-stubs/README.md
@@ -2,7 +2,7 @@
# Sensor Stubs
A service has been created to simulate the behavior of a sensor by sending canned telemetry data to a Kafka topic. These "Sensor Stubs" consume fewer resources than the actual sensor that they replace.
-
+
### (Q) How do the sensor stubs work?
The stubs are installed with a set of canned data for each sensor type; Bro, Snort and YAF. A subset of this canned data is randomly selected and sent to the Kafka topic in batches. The timestamp of each message is updated to match current system time.
@@ -15,7 +15,7 @@ The number of telemetry messages sent in each batch, along with the time delay b
Using the default playbooks, this role can be installed by using the Ansible tag `sensor-stubs`. This service is installed on the same hosts where the sensors would be; defined by the `sensors` host group.
-The defaults for the "Quick Dev" and "Full Dev" environments have been changed so that the Sensor Stubs are installed by default, rather than the sensors themselves. The Amazon EC2 environment continues to install the original sensors by default.
+The defaults for the "Full Dev" environment have been changed so that the Sensor Stubs are installed by default, rather than the sensors themselves. The Amazon EC2 environment continues to install the original sensors by default.
### (Q) How do I use the sensor stubs?
@@ -71,15 +71,13 @@ Stopping sensor-stubs...
### (Q) How do I install the original sensors?
-The default behavior can be changed by skipping the `sensor-stubs` flag and including the `sensors` flag. For example, to deploy "Quick Dev" with the original sensors run the following command.
+The default behavior can be changed by skipping the `sensor-stubs` flag and including the `sensors` flag. For example, to deploy "Full Dev" with the original sensors run the following command.
```
-cd metron-deployment/vagrant/quick-dev-platform
+cd metron-deployment/vagrant/full-dev-platform
vagrant --ansible-skip-tags="sensor-stubs,solr" up
```
### (Q) Where does the mock data come from?
The data produced by the sensor stubs was generated by running the sensors against the example pcap file that is distributed with Metron. This ensures that the data produced by the sensor stubs is similar to the data produced when using the actual sensors.
-
-
http://git-wip-us.apache.org/repos/asf/metron/blob/f0844048/metron-deployment/vagrant/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/README.md b/metron-deployment/vagrant/README.md
index b629a1f..9e48302 100644
--- a/metron-deployment/vagrant/README.md
+++ b/metron-deployment/vagrant/README.md
@@ -1,6 +1,4 @@
# Vagrant Deployment
-- Codelab Platform
- Fast CAPA Test Platform
- Full Dev Platform
-- Quick Dev Platform
http://git-wip-us.apache.org/repos/asf/metron/blob/f0844048/metron-deployment/vagrant/codelab-platform/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/codelab-platform/README.md b/metron-deployment/vagrant/codelab-platform/README.md
deleted file mode 100644
index 893f97d..0000000
--- a/metron-deployment/vagrant/codelab-platform/README.md
+++ /dev/null
@@ -1,65 +0,0 @@
-Developer Image for Apache Metron on Virtualbox
-===============================================
-
-This image is a fully functional Metron installation that has been pre-loaded with Ambari, HDP and Metron.
-
-Getting Started
----------------
-
-### Prerequisites
-
-As with the Singlenode Full Image, the computer used to deploy Apache Metron will need to have the following components installed.
-
- - [Ansible](https://github.com/ansible/ansible) (2.0.0.2 or 2.2.2.0)
- - [Vagrant](https://www.vagrantup.com) 1.8.1
- - [Virtualbox](https://virtualbox.org) 5.0.16
- - Python 2.7.11
- - Maven 3.3.9
-
-### Ensure vagrant hostmanager is installed
-
-To check and make sure you have the plugin installed execute the following:
-
- ```
- vagrant plugin list
- ```
-
-If you have it installed you should see the following listed in the output:
-
- ```
- vagrant-hostmanager (1.8.1)
- ```
-If it is not installed, you can install it with the following command:
-
- ```
- vagrant plugin install vagrant-hostmanager
- ```
-
-### Launch the Metron Development Image
-
-Start the image with the following commands:
-
- ```
- cd metron-deployment/vagrant/codelab-platform
- ./run.sh
- ```
-
-### Work with Metron
-
-As you build out new capabilities for Metron, you will need to re-deploy the Storm topologies. To do so, first HALT the running Storm topologies and then run:
-
-```
-./run_enrichment_role.sh
-```
-
-Remember Navigate to the following resources to explore your newly minted Apache Metron environment.
-
- - [Metron](http://node1:5000)
- - [Ambari](http://node1:8080)
- - [Services](http://node1:2812)
-
-Connecting to the host through SSH is as simple as running the following command.
-
- ```
- vagrant ssh
- ```
http://git-wip-us.apache.org/repos/asf/metron/blob/f0844048/metron-deployment/vagrant/codelab-platform/Vagrantfile
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/codelab-platform/Vagrantfile b/metron-deployment/vagrant/codelab-platform/Vagrantfile
deleted file mode 100644
index 2fa1b93..0000000
--- a/metron-deployment/vagrant/codelab-platform/Vagrantfile
+++ /dev/null
@@ -1,94 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-require 'getoptlong'
-
-ansibleTags=''
-ansibleSkipTags=''
-
-begin
- opts = GetoptLong.new(
- [ '--ansible-tags', GetoptLong::OPTIONAL_ARGUMENT ],
- [ '--ansible-skip-tags', GetoptLong::OPTIONAL_ARGUMENT ]
- )
-
- opts.quiet = TRUE
-
- opts.each do |opt, arg|
- case opt
- when '--ansible-tags'
- ansibleTags=arg
- when '--ansible-skip-tags'
- ansibleSkipTags=arg
- end
- end
-rescue Exception => ignored
-#Ignore to allow other opts to be passed to Vagrant
-end
-
-puts " Running with ansible-tags: " + ansibleTags.split(",").to_s if ansibleTags != ''
-puts " Running with ansible-skip-tags: " + ansibleSkipTags.split(",").to_s if ansibleSkipTags != ''
-
-hosts = [{
- hostname: "node1",
- ip: "192.168.66.121",
- memory: "8192",
- cpus: 4,
- promisc: 2 # enables promisc on the 'Nth' network interface
-}]
-
-Vagrant.configure(2) do |config|
-
- # all hosts built on centos 6
- config.vm.box = "metron/codelab"
- config.ssh.insert_key = true
-
- # enable the hostmanager plugin
- config.hostmanager.enabled = true
- config.hostmanager.manage_host = true
-
- # host definition
- hosts.each_with_index do |host, index|
- config.vm.define host[:hostname] do |node|
-
- # host settings
- node.vm.hostname = host[:hostname]
- node.vm.network "private_network", ip: host[:ip]
-
- # vm settings
- node.vm.provider "virtualbox" do |vb|
- vb.memory = host[:memory]
- vb.cpus = host[:cpus]
-
- # enable promisc mode on the network interface
- if host.has_key?(:promisc)
- vb.customize ["modifyvm", :id, "--nicpromisc#{host[:promisc]}", "allow-all"]
- end
- end
- end
- end
-
- if ansibleSkipTags != '' or ansibleTags != ''
- config.vm.provision :ansible do |ansible|
- ansible.playbook = "../../playbooks/metron_full_install.yml"
- ansible.sudo = true
- ansible.tags = ansibleTags.split(",") if ansibleTags != ''
- ansible.skip_tags = ansibleSkipTags.split(",") if ansibleSkipTags != ''
- ansible.inventory_path = "../../inventory/full-dev-platform"
- end
- end
-
-end
http://git-wip-us.apache.org/repos/asf/metron/blob/f0844048/metron-deployment/vagrant/codelab-platform/ansible.cfg
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/codelab-platform/ansible.cfg b/metron-deployment/vagrant/codelab-platform/ansible.cfg
deleted file mode 100644
index 74f3fdf..0000000
--- a/metron-deployment/vagrant/codelab-platform/ansible.cfg
+++ /dev/null
@@ -1,27 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-[defaults]
-host_key_checking = false
-library = ../../extra_modules
-roles_path = ../../roles
-pipelining = True
-log_path = ./ansible.log
-
-
-# fix for "ssh throws 'unix domain socket too long' " problem
-[ssh_connection]
-control_path = %(directory)s/%%h-%%p-%%r
http://git-wip-us.apache.org/repos/asf/metron/blob/f0844048/metron-deployment/vagrant/codelab-platform/run.sh
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/codelab-platform/run.sh b/metron-deployment/vagrant/codelab-platform/run.sh
deleted file mode 100755
index 498098c..0000000
--- a/metron-deployment/vagrant/codelab-platform/run.sh
+++ /dev/null
@@ -1,20 +0,0 @@
-#!/bin/bash
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-vagrant --ansible-tags="hdp-deploy,start" --ansible-skip-tags="solr" up
http://git-wip-us.apache.org/repos/asf/metron/blob/f0844048/metron-deployment/vagrant/codelab-platform/run_ansible_role.sh
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/codelab-platform/run_ansible_role.sh b/metron-deployment/vagrant/codelab-platform/run_ansible_role.sh
deleted file mode 100755
index 0e5d831..0000000
--- a/metron-deployment/vagrant/codelab-platform/run_ansible_role.sh
+++ /dev/null
@@ -1,20 +0,0 @@
-#!/bin/bash
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-vagrant --ansible-tags="$@" provision
http://git-wip-us.apache.org/repos/asf/metron/blob/f0844048/metron-deployment/vagrant/codelab-platform/run_enrichment_role.sh
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/codelab-platform/run_enrichment_role.sh b/metron-deployment/vagrant/codelab-platform/run_enrichment_role.sh
deleted file mode 100755
index e26bc16..0000000
--- a/metron-deployment/vagrant/codelab-platform/run_enrichment_role.sh
+++ /dev/null
@@ -1,20 +0,0 @@
-#!/bin/bash
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-./run_ansible_role.sh enrichment
http://git-wip-us.apache.org/repos/asf/metron/blob/f0844048/metron-deployment/vagrant/full-dev-platform/Vagrantfile
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/full-dev-platform/Vagrantfile b/metron-deployment/vagrant/full-dev-platform/Vagrantfile
index 1c96ded..15df777 100644
--- a/metron-deployment/vagrant/full-dev-platform/Vagrantfile
+++ b/metron-deployment/vagrant/full-dev-platform/Vagrantfile
@@ -17,7 +17,7 @@
require 'getoptlong'
ansibleTags=''
-ansibleSkipTags='sensors,quick_dev'
+ansibleSkipTags='sensors'
begin
opts = GetoptLong.new(
http://git-wip-us.apache.org/repos/asf/metron/blob/f0844048/metron-deployment/vagrant/quick-dev-platform/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/quick-dev-platform/README.md b/metron-deployment/vagrant/quick-dev-platform/README.md
deleted file mode 100644
index ca1168b..0000000
--- a/metron-deployment/vagrant/quick-dev-platform/README.md
+++ /dev/null
@@ -1,77 +0,0 @@
-Quick Development Platform
-==========================
-
-This project fully automates the provisioning and deployment of Apache Metron and all necessary prerequisites on a single, virtualized host running on Virtualbox.
-
-This image is designed for quick deployment of a single node Metron cluster running on Virtualbox. This platform is ideal for use by Metron developers. It uses a base image that has been pre-loaded with Ambari and HDP.
-
-Metron is composed of many components and installing all of these on a single host, especially a virtualized one, will greatly stress the resources of the host. The host will require at least 8 GB of RAM and a fair amount of patience. It is highly recommended that you shut down all unnecessary services. To that end the vagrant file configuration defaults to disabling solr and yaf.
-
-Getting Started
----------------
-
-### Prerequisites
-
-As with the Full Development Platform (`metron-deployment/vagrant/full-dev-platform`), the computer used to deploy Apache Metron will need the following components installed.
-
- - [Ansible](https://github.com/ansible/ansible) (2.0.0.2 or 2.2.2.0)
- - [Docker](https://www.docker.com/community-edition)
- - [Vagrant](https://www.vagrantup.com) 1.8.1
- - [Vagrant Hostmanager Plugin](https://github.com/devopsgroup-io/vagrant-hostmanager) `vagrant plugin install vagrant-hostmanager`
- - [Virtualbox](https://virtualbox.org) 5.0.16
- - Python 2.7.11
- - Maven 3.3.9
-
-#### macOS
-
- Any platform that supports these tools is suitable, but the following instructions cover installation on macOS. The easiest means of installing these tools on a Mac is to use the excellent [Homebrew](http://brew.sh/) project.
-
- 1. Install Homebrew by following the instructions at [Homebrew](http://brew.sh/).
-
- 1. Run the following command in a terminal to install all of the required tools.
-
- ```
- brew cask install vagrant virtualbox java docker
- brew install maven git
- ```
-
- 1. Install Ansible by following the instructions [here](http://docs.ansible.com/ansible/intro_installation.html#latest-releases-via-pip).
-
-### Deploy Metron
-
-1. Ensure that the Docker service is running.
-
-1. Deploy Metron
-
- ```
- cd metron-deployment/vagrant/quick-dev-platform
- vagrant up
- ```
-
- Should the process fail before completing the deployment, the following command will continue the deployment process without re-instantiating the host.
-
- ```
- vagrant provision
- ```
-
-### Explore Metron
-
-Navigate to the following resources to explore your newly minted Apache Metron environment.
-
- - [Ambari](http://node1:8080)
- - [Metron](http://node1:5000)
- - [Services](http://node1:2812)
-
-### Working with Metron
-
-As you build out new capabilities for Metron, you will need to re-deploy the Storm topologies. To do so, first HALT the running Storm topologies and then run the following command.
-
-```
-./run_enrichment_role.sh
-```
-
-Connecting to the host through SSH is as simple as running the following command.
-
-```
-vagrant ssh
-```
http://git-wip-us.apache.org/repos/asf/metron/blob/f0844048/metron-deployment/vagrant/quick-dev-platform/Vagrantfile
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/quick-dev-platform/Vagrantfile b/metron-deployment/vagrant/quick-dev-platform/Vagrantfile
deleted file mode 100644
index 1c18df5..0000000
--- a/metron-deployment/vagrant/quick-dev-platform/Vagrantfile
+++ /dev/null
@@ -1,94 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-require 'getoptlong'
-
-ansibleTags='quick_dev,report'
-ansibleSkipTags=''
-
-begin
- opts = GetoptLong.new(
- [ '--ansible-tags', GetoptLong::OPTIONAL_ARGUMENT ],
- [ '--ansible-skip-tags', GetoptLong::OPTIONAL_ARGUMENT ]
- )
-
- opts.quiet = TRUE
-
- opts.each do |opt, arg|
- case opt
- when '--ansible-tags'
- ansibleTags=arg
- when '--ansible-skip-tags'
- ansibleSkipTags=arg
- end
- end
-rescue Exception => ignored
-#Ignore to allow other opts to be passed to Vagrant
-end
-
-puts " Running with ansible-tags: " + ansibleTags.split(",").to_s if ansibleTags != ''
-puts " Running with ansible-skip-tags: " + ansibleSkipTags.split(",").to_s if ansibleSkipTags != ''
-
-hosts = [{
- hostname: "node1",
- ip: "192.168.66.121",
- memory: "8192",
- cpus: 4,
- promisc: 2 # enables promisc on the 'Nth' network interface
-}]
-
-Vagrant.configure(2) do |config|
-
- # all hosts built on centos 6
- config.vm.box = "metron/quick_dev"
- config.ssh.insert_key = true
-
- # enable the hostmanager plugin
- config.hostmanager.enabled = true
- config.hostmanager.manage_host = true
-
- # host definition
- hosts.each_with_index do |host, index|
- config.vm.define host[:hostname] do |node|
-
- # host settings
- node.vm.hostname = host[:hostname]
- node.vm.network "private_network", ip: host[:ip]
-
- # vm settings
- node.vm.provider "virtualbox" do |vb|
- vb.memory = host[:memory]
- vb.cpus = host[:cpus]
-
- # enable promisc mode on the network interface
- if host.has_key?(:promisc)
- vb.customize ["modifyvm", :id, "--nicpromisc#{host[:promisc]}", "allow-all"]
- end
- end
- end
- end
-
- if ansibleSkipTags != '' or ansibleTags != ''
- config.vm.provision :ansible do |ansible|
- ansible.playbook = "../../playbooks/metron_full_install.yml"
- ansible.sudo = true
- ansible.tags = ansibleTags.split(",") if ansibleTags != ''
- ansible.skip_tags = ansibleSkipTags.split(",") if ansibleSkipTags != ''
- ansible.inventory_path = "../../inventory/quick-dev-platform"
- end
- end
-
-end
http://git-wip-us.apache.org/repos/asf/metron/blob/f0844048/metron-deployment/vagrant/quick-dev-platform/ansible.cfg
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/quick-dev-platform/ansible.cfg b/metron-deployment/vagrant/quick-dev-platform/ansible.cfg
deleted file mode 100644
index 74f3fdf..0000000
--- a/metron-deployment/vagrant/quick-dev-platform/ansible.cfg
+++ /dev/null
@@ -1,27 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-[defaults]
-host_key_checking = false
-library = ../../extra_modules
-roles_path = ../../roles
-pipelining = True
-log_path = ./ansible.log
-
-
-# fix for "ssh throws 'unix domain socket too long' " problem
-[ssh_connection]
-control_path = %(directory)s/%%h-%%p-%%r
http://git-wip-us.apache.org/repos/asf/metron/blob/f0844048/metron-deployment/vagrant/quick-dev-platform/run_ansible_role.sh
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/quick-dev-platform/run_ansible_role.sh b/metron-deployment/vagrant/quick-dev-platform/run_ansible_role.sh
deleted file mode 100755
index 0e5d831..0000000
--- a/metron-deployment/vagrant/quick-dev-platform/run_ansible_role.sh
+++ /dev/null
@@ -1,20 +0,0 @@
-#!/bin/bash
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-vagrant --ansible-tags="$@" provision
http://git-wip-us.apache.org/repos/asf/metron/blob/f0844048/metron-deployment/vagrant/quick-dev-platform/run_enrichment_role.sh
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/quick-dev-platform/run_enrichment_role.sh b/metron-deployment/vagrant/quick-dev-platform/run_enrichment_role.sh
deleted file mode 100755
index e26bc16..0000000
--- a/metron-deployment/vagrant/quick-dev-platform/run_enrichment_role.sh
+++ /dev/null
@@ -1,20 +0,0 @@
-#!/bin/bash
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-./run_ansible_role.sh enrichment
http://git-wip-us.apache.org/repos/asf/metron/blob/f0844048/metron-interface/metron-alerts/README.md
----------------------------------------------------------------------
diff --git a/metron-interface/metron-alerts/README.md b/metron-interface/metron-alerts/README.md
index 1e1acb9..8480a69 100644
--- a/metron-interface/metron-alerts/README.md
+++ b/metron-interface/metron-alerts/README.md
@@ -15,7 +15,7 @@ Alert GUIDs must be double-quoted when being searched on to ensure correctness o
### Search for Comments
Users cannot search for the contents of the comment's in the Alerts-UI
-### Meta alerts
+### Meta alerts
Grouping/faceting requests and other aggregations do not return meta alerts. This is because it's not clear what the intended results should be when there are multiple matching items.
Sorting has a similar caveat, in that if we are matching on multiple alerts, there is no well defined sort.
@@ -25,7 +25,7 @@ Alerts that are contained in a a meta alert are generally excluded from search r
## Prerequisites
* The Metron REST application should be up and running and Elasticsearch should have some alerts populated by Metron topologies
* The Management UI should be installed (which includes [Express](https://expressjs.com/))
-* The alerts can be populated using Quick Dev, Full Dev or any other setup
+* The alerts can be populated using Full Dev or any other setup
* UI is developed using angular4 and uses angular-cli
* node.JS >= 7.8.0
http://git-wip-us.apache.org/repos/asf/metron/blob/f0844048/metron-interface/metron-config/README.md
----------------------------------------------------------------------
diff --git a/metron-interface/metron-config/README.md b/metron-interface/metron-config/README.md
index f0a69e3..1b20731 100644
--- a/metron-interface/metron-config/README.md
+++ b/metron-interface/metron-config/README.md
@@ -5,7 +5,7 @@ This module provides a user interface for management functions in Metron.
## Prerequisites
* A network accessible Metron REST application
-* nodejs v6.9+ (nodejs can be installed on quick dev with `curl --silent --location https://rpm.nodesource.com/setup_6.x | bash - && yum install -y nodejs`)
+* nodejs v6.9+ (nodejs can be installed on Full Dev with `curl --silent --location https://rpm.nodesource.com/setup_6.x | bash - && yum install -y nodejs`)
## Installation
http://git-wip-us.apache.org/repos/asf/metron/blob/f0844048/metron-interface/metron-rest/README.md
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/README.md b/metron-interface/metron-rest/README.md
index e46f865..4a16808 100644
--- a/metron-interface/metron-rest/README.md
+++ b/metron-interface/metron-rest/README.md
@@ -276,13 +276,13 @@ Request and Response objects are JSON formatted. The JSON schemas are available
* alerts - The alerts to be escalated
* Returns:
* 200 - Alerts were escalated
-
+
### `GET /api/v1/alert/profile`
* Description: Retrieves the current user's alerts profile
* Returns:
* 200 - Alerts profile
* 404 - The current user does not have an alerts profile
-
+
### `GET /api/v1/alert/profile/all`
* Description: Retrieves all users' alerts profiles. Only users that are part of the "ROLE_ADMIN" role are allowed to get all alerts profiles.
* Returns:
@@ -429,21 +429,21 @@ Request and Response objects are JSON formatted. The JSON schemas are available
* request - Meta alert create request which includes a list of alert get requests and a list of custom groups used to annotate a meta alert.
* Returns:
* 200 - The GUID of the new meta alert
-
+
### `POST /api/v1/metaalert/add/alert`
* Description: Adds an alert to an existing meta alert. An alert will not be added if it is already contained in a meta alert.
* Input:
* request - Meta alert add request which includes a meta alert GUID and list of alert get requests
* Returns:
* 200 - Returns 'true' if the alert was added and 'false' if the meta alert did not change.
-
+
### `POST /api/v1/metaalert/remove/alert`
* Description: Removes an alert from an existing meta alert. If the alert to be removed is not in a meta alert, 'false' will be returned.
* Input:
* request - Meta alert remove request which includes a meta alert GUID and list of alert get requests
* Returns:
* 200 - Returns 'true' if the alert was removed and 'false' if the meta alert did not change.
-
+
### `POST /api/v1/metaalert/update/status/{guid}/{status}`
* Description: Updates the status of a meta alert to either 'ACTIVE' or 'INACTIVE'.
* Input:
@@ -458,7 +458,7 @@ Request and Response objects are JSON formatted. The JSON schemas are available
* searchRequest - Search request
* Returns:
* 200 - Search response
-
+
### `POST /api/v1/search/group`
* Description: Searches the indexing store and returns field groups. GUIDs must be quoted to ensure correct results. Groups are hierarchical and nested in the order the fields appear in the 'groups' request parameter. The default sorting within groups is by count descending. A groupOrder type of count will sort based on then number of documents in a group while a groupType of term will sort by the groupBy term.
* Input:
@@ -466,10 +466,10 @@ Request and Response objects are JSON formatted. The JSON schemas are available
* indices - list of indices to search
* query - lucene query
* scoreField - field used to compute a total score for each group
- * groups - List of groups (field name and sort order)
+ * groups - List of groups (field name and sort order)
* Returns:
* 200 - Group response
-
+
### `GET /api/v1/search/findOne`
* Description: Returns latest document for a guid and sensor
* Input:
@@ -486,7 +486,7 @@ Request and Response objects are JSON formatted. The JSON schemas are available
* Returns:
* 200 - Document representing the output
* 404 - Document with UUID and sensor type not found
-
+
### `GET /api/v1/search/column/metadata`
* Description: Get index column metadata for a list of sensor types with duplicates removed. Column names and types for each sensor are retrieved from the most recent index. Columns that exist in multiple indices with different types will default to type 'other'.
* Input:
@@ -745,7 +745,7 @@ Request and Response objects are JSON formatted. The JSON schemas are available
### `GET /api/v1/storm/supervisors`
* Description: Retrieves the status of all Storm Supervisors
* Returns:
- * 200 - Returns a list of the status of all Storm Supervisors
+ * 200 - Returns a list of the status of all Storm Supervisors
### `PATCH /api/v1/update/patch`
* Description: Update a document with a patch
@@ -804,7 +804,7 @@ Request and Response objects are JSON formatted. The JSON schemas are available
## Testing
-Profiles are includes for both the metron-docker and Quick Dev environments.
+Profiles are includes for both the metron-docker and Full Dev environments.
### metron-docker
@@ -816,9 +816,9 @@ mvn spring-boot:run -Drun.profiles=docker,dev
The metron-rest application will be available at http://localhost:8080/swagger-ui.html#/.
-### Quick Dev
+### Full Dev
-Start the [Quick Dev](../../metron-deployment/vagrant/quick-dev-platform) environment. Build the metron-rest module and start it with the Spring Boot Maven plugin:
+Start the [Full Dev](../../metron-deployment/vagrant/full-dev-platform) environment. Build the metron-rest module and start it with the Spring Boot Maven plugin:
```
mvn clean package
mvn spring-boot:run -Drun.profiles=vagrant,dev
@@ -826,7 +826,7 @@ mvn spring-boot:run -Drun.profiles=vagrant,dev
The metron-rest application will be available at http://localhost:8080/swagger-ui.html#/.
-To run the application locally on the Quick Dev host (node1), follow the [Installation](#installation) instructions above. Then set the METRON_SPRING_PROFILES_ACTIVE variable in `/etc/default/metron`:
+To run the application locally on the Full Dev host (node1), follow the [Installation](#installation) instructions above. Then set the METRON_SPRING_PROFILES_ACTIVE variable in `/etc/default/metron`:
```
METRON_SPRING_PROFILES_ACTIVE="vagrant,dev"
```
http://git-wip-us.apache.org/repos/asf/metron/blob/f0844048/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/utils/ReadMeUtils.java
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/utils/ReadMeUtils.java b/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/utils/ReadMeUtils.java
deleted file mode 100644
index d9522f2..0000000
--- a/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/utils/ReadMeUtils.java
+++ /dev/null
@@ -1,130 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.metron.rest.utils;
-
-import io.swagger.annotations.ApiOperation;
-import io.swagger.annotations.ApiParam;
-import io.swagger.annotations.ApiResponse;
-import io.swagger.annotations.ApiResponses;
-import org.apache.velocity.Template;
-import org.apache.velocity.VelocityContext;
-import org.apache.velocity.app.Velocity;
-import org.apache.velocity.runtime.RuntimeConstants;
-import org.apache.velocity.runtime.resource.loader.ClasspathResourceLoader;
-import org.reflections.Reflections;
-import org.springframework.web.bind.annotation.RequestMapping;
-import org.springframework.web.bind.annotation.RequestMethod;
-import org.springframework.web.bind.annotation.RestController;
-
-import java.io.FileWriter;
-import java.lang.reflect.Method;
-import java.lang.reflect.Parameter;
-import java.security.Principal;
-import java.util.ArrayList;
-import java.util.Collections;
-import java.util.List;
-import java.util.Set;
-
-public class ReadMeUtils {
-
- public static void main(String[] args) throws Exception {
- String path;
- if (args.length == 0) {
- System.out.println("README output path was not set. Defaulting to 'metron-interface/metron-rest/README.md'");
- path = "metron-interface/metron-rest/README.md";
- } else {
- path = args[0];
- }
-
- Reflections reflections = new Reflections("org.apache.metron.rest.controller");
- List<RestControllerInfo> endpoints = new ArrayList<>();
- Set<Class<?>> controllers = reflections.getTypesAnnotatedWith(RestController.class);
- for(Class<?> controller: controllers) {
- RequestMapping controllerRequestMapping = controller.getAnnotation(RequestMapping.class);
- String pathPrefix;
- if (controllerRequestMapping == null) {
- pathPrefix = "";
- } else {
- pathPrefix = controllerRequestMapping.value()[0];
- }
- for(Method method: controller.getDeclaredMethods()) {
- RestControllerInfo restControllerInfo = new RestControllerInfo();
- RequestMapping requestMapping = method.getAnnotation(RequestMapping.class);
- if (requestMapping == null) {
- throw new Exception(String.format("@RequestMapping annotation missing from method %s.%s. Every controller method must have a @RequestMapping annotation.", controller.getSimpleName(), method.getName()));
- }
- ApiOperation apiOperation = method.getAnnotation(ApiOperation.class);
- if (apiOperation == null) {
- throw new Exception(String.format("@ApiOperation annotation missing from method %s.%s. Every controller method must have an @ApiOperation annotation.", controller.getSimpleName(), method.getName()));
- }
- ApiResponse[] apiResponseArray;
- ApiResponses apiResponses = method.getAnnotation(ApiResponses.class);
- if (apiResponses == null) {
- ApiResponse apiResponse = method.getAnnotation(ApiResponse.class);
- if (apiResponse == null) {
- throw new Exception(String.format("@ApiResponses or @ApiResponse annotation missing from method %s.%s. Every controller method must have an @ApiResponses or @ApiResponse annotation.", controller.getSimpleName(), method.getName()));
- } else {
- apiResponseArray = new ApiResponse[]{ apiResponse };
- }
- } else {
- apiResponseArray = apiResponses.value();
- }
- String[] requestMappingValue = requestMapping.value();
- if (requestMappingValue.length == 0) {
- restControllerInfo.setPath(pathPrefix);
- } else {
- restControllerInfo.setPath(pathPrefix + requestMappingValue[0]);
- }
- restControllerInfo.setDescription(apiOperation.value());
- RequestMethod requestMethod;
- if (requestMapping.method().length == 0) {
- requestMethod = RequestMethod.GET;
- } else {
- requestMethod = requestMapping.method()[0];
- }
- restControllerInfo.setMethod(requestMethod);
- for (ApiResponse apiResponse: apiResponseArray) {
- restControllerInfo.addResponse(apiResponse.message(), apiResponse.code());
- }
- for(Parameter parameter: method.getParameters()) {
- if (!parameter.getType().equals(Principal.class)) {
- ApiParam apiParam = parameter.getAnnotation(ApiParam.class);
- if (apiParam == null) {
- throw new Exception(String.format("@ApiParam annotation missing from parameter %s.%s.%s. Every controller method parameter must have an @ApiParam annotation.", controller.getSimpleName(), method.getName(), parameter.getName()));
- }
- restControllerInfo.addParameterDescription(apiParam.name(), apiParam.value());
- }
- }
- endpoints.add(restControllerInfo);
- }
- }
- Collections.sort(endpoints, (o1, o2) -> o1.getPath().compareTo(o2.getPath()));
-
- Velocity.setProperty(RuntimeConstants.RESOURCE_LOADER, "classpath");
- Velocity.setProperty("classpath.resource.loader.class", ClasspathResourceLoader.class.getName());
- Velocity.init();
-
- VelocityContext context = new VelocityContext();
- context.put( "endpoints", endpoints );
-
- Template template = Velocity.getTemplate("README.vm");
- FileWriter fileWriter = new FileWriter(path);
- template.merge( context, fileWriter );
- fileWriter.close();
- }
-}
http://git-wip-us.apache.org/repos/asf/metron/blob/f0844048/metron-interface/metron-rest/src/test/resources/README.vm
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/test/resources/README.vm b/metron-interface/metron-rest/src/test/resources/README.vm
deleted file mode 100644
index e6e6472..0000000
--- a/metron-interface/metron-rest/src/test/resources/README.vm
+++ /dev/null
@@ -1,236 +0,0 @@
-#[[#]]# Metron REST
-
-This module provides a RESTful API for interacting with Metron.
-
-#[[##]]# Prerequisites
-
-* A running Metron cluster
-* Java 8 installed
-* Storm CLI and Metron topology scripts (start_parser_topology.sh, start_enrichment_topology.sh, start_elasticsearch_topology.sh) installed
-* A relational database
-
-#[[##]]# Installation
-
-#[[###]]# From Source
-
-1. Package the application with Maven:
-```
-mvn clean package
-```
-
-1. Untar the archive in the $METRON_HOME directory. The directory structure will look like:
-```
-config
- rest_application.yml
-bin
- metron-rest
-lib
- metron-rest-$METRON_VERSION.jar
-```
-
-1. Copy the `$METRON_HOME/bin/metron-rest` script to `/etc/init.d/metron-rest`
-
-#[[###]]# From Package Manager
-
-1. Deploy the RPM at `/metron/metron-deployment/packaging/docker/rpm-docker/target/RPMS/noarch/metron-rest-$METRON_VERSION-*.noarch.rpm`
-
-1. Install the RPM with:
- ```
- rpm -ih metron-rest-$METRON_VERSION-*.noarch.rpm
- ```
-
-#[[##]]# Configuration
-
-The REST application depends on several configuration parameters:
-
-| Environment Variable | Description | Required | Default |
-| ------------------------------------- | ----------------------------------------------------------------- | -------- | ---------------------- |
-| METRON_USER | Run the application as this user | Optional | metron |
-| METRON_LOG_DIR | Directory where the log file is written | Optional | /var/log/metron/ |
-| METRON_PID_DIR | Directory where the pid file is written | Optional | /var/run/metron/ |
-| METRON_REST_PORT | REST application port | Optional | 8082 |
-| METRON_JVMFLAGS | JVM flags added to the start command | Optional | |
-| METRON_SPRING_PROFILES_ACTIVE | Active Spring profiles (see [below](#spring-profiles)) | Optional | |
-| METRON_JDBC_DRIVER | JDBC driver class | Required | |
-| METRON_JDBC_URL | JDBC url | Required | |
-| METRON_JDBC_USERNAME | JDBC username | Required | |
-| METRON_JDBC_PLATFORM | JDBC platform (one of h2, mysql, postgres, oracle | Required | |
-| METRON_JDBC_CLIENT_PATH | Path to JDBC client jar | Optional | H2 is bundled |
-| METRON_TEMP_GROK_PATH | Temporary directory used to test grok statements | Optional | ./patterns/temp |
-| METRON_DEFAULT_GROK_PATH | Defaults HDFS directory used to store grok statements | Optional | /apps/metron/patterns |
-| METRON_SPRING_OPTIONS | Additional Spring input parameters | Optional | |
-| ZOOKEEPER | Zookeeper quorum (ex. node1:2181,node2:2181) | Required | |
-| BROKERLIST | Kafka Broker list (ex. node1:6667,node2:6667) | Required | |
-| HDFS_URL | HDFS url or `fs.defaultFS` Hadoop setting (ex. hdfs://node1:8020) | Required | |
-| SECURITY_ENABLED | Enables Kerberos support | Optional | false |
-| METRON_PRINCIPAL_NAME | Kerberos principal for the metron user | Optional | |
-| METRON_SERVICE_KEYTAB | Path to the Kerberos keytab for the metron user | Optional | |
-
-These are set in the `/etc/sysconfig/metron` file.
-
-#[[##]]# Database setup
-
-The REST application persists data in a relational database and requires a dedicated database user and database (see https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-sql.html for more detail).
-
-#[[###]]# Development
-
-The REST application comes with embedded database support for development purposes (https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-sql.html#boot-features-embedded-database-support).
-
-For example, edit these variables in `/etc/sysconfig/metron` before starting the application to configure H2:
-```
-METRON_JDBC_DRIVER="org.h2.Driver"
-METRON_JDBC_URL="jdbc:h2:file:~/metrondb"
-METRON_JDBC_USERNAME="root"
-METRON_JDBC_PASSWORD='root"
-METRON_JDBC_PLATFORM="h2"
-```
-
-#[[###]]# Production
-
-The REST application should be configured with a production-grade database outside of development.
-
-For example, the following configures the application for MySQL:
-
-1. Install MySQL if not already available (this example uses version 5.7, installation instructions can be found [here](https://dev.mysql.com/doc/refman/5.7/en/linux-installation-yum-repo.html))
-
-1. Create a metron user and REST database and permission the user for that database:
-```
-CREATE USER 'metron'@'node1' IDENTIFIED BY 'Myp@ssw0rd';
-CREATE DATABASE IF NOT EXISTS metronrest;
-GRANT ALL PRIVILEGES ON metronrest.* TO 'metron'@'node1';
-```
-
-1. Install the MySQL JDBC client onto the REST application host and configurate the METRON_JDBC_CLIENT_PATH variable:
-```
-cd $METRON_HOME/lib
-wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.41.tar.gz
-tar xf mysql-connector-java-5.1.41.tar.gz
-```
-
-1. Edit these variables in `/etc/sysconfig/metron` to configure the REST application for MySQL:
-```
-METRON_JDBC_DRIVER="com.mysql.jdbc.Driver"
-METRON_JDBC_URL="jdbc:mysql://mysql_host:3306/metronrest"
-METRON_JDBC_USERNAME="metron"
-METRON_JDBC_PLATFORM="mysql"
-METRON_JDBC_CLIENT_PATH=$METRON_HOME/lib/mysql-connector-java-5.1.41/mysql-connector-java-5.1.41-bin.jar
-```
-
-#[[##]]# Usage
-
-After configuration is complete, the REST application can be managed as a service:
-```
-service metron-rest start
-```
-
-If a production database is configured, the JDBC password should be passed in as the first argument on startup:
-```
-service metron-rest start Myp@ssw0rd
-```
-
-The REST application can be accessed with the Swagger UI at http://host:port/swagger-ui.html#/. The default port is 8082.
-
-#[[##]]# Security
-
-#[[###]]# Authentication
-
-The metron-rest module uses [Spring Security](http://projects.spring.io/spring-security/) for authentication and stores user credentials in the relational database configured above. The required tables are created automatically the first time the application is started so that should be done first. For example (continuing the MySQL example above), users can be added by connecting to MySQL and running:
-```
-use metronrest;
-insert into users (username, password, enabled) values ('your_username','your_password',1);
-insert into authorities (username, authority) values ('your_username', 'ROLE_USER');
-```
-
-#[[###]]# Kerberos
-
-Metron REST can be configured for a cluster with Kerberos enabled. A client JAAS file is required for Kafka and Zookeeper and a Kerberos keytab for the metron user principal is required for all other services. Configure these settings in the `/etc/sysconfig/metron` file:
-```
-SECURITY_ENABLED=true
-METRON_JVMFLAGS="-Djava.security.auth.login.config=$METRON_HOME/client_jaas.conf"
-METRON_PRINCIPAL_NAME="metron@EXAMPLE.COM"
-METRON_SERVICE_KEYTAB="/etc/security/keytabs/metron.keytab"
-```
-
-#[[##]]# Spring Profiles
-
-The REST application comes with a few [Spring Profiles](http://docs.spring.io/autorepo/docs/spring-boot/current/reference/html/boot-features-profiles.html) to aid in testing and development.
-
-| Profile | Description |
-| ------------------------ | --------------------------------------------- |
-| test | sets variables to in-memory services, only used for integration testing |
-| dev | adds a test user to the database with credentials `user/password` |
-| vagrant | sets configuration variables to match the Metron vagrant environment |
-| docker | sets configuration variables to match the Metron dcoker environment |
-
-Setting active profiles is done with the METRON_SPRING_PROFILES_ACTIVE variable. For example, set this variable in `/etc/sysconfig/metron` to configure the REST application for the Vagrant environment and add a test user:
-```
-METRON_SPRING_PROFILES_ACTIVE="vagrant,dev"
-```
-
-#[[##]]# API
-
-Request and Response objects are JSON formatted. The JSON schemas are available in the Swagger UI.
-
-| |
-| ---------- |
-#foreach( $restControllerInfo in $endpoints )
-| [ `$restControllerInfo.getMethod().toString() $restControllerInfo.getPath()`](#$restControllerInfo.getMethod().toString().toLowerCase()-$restControllerInfo.getPath().toLowerCase().replaceAll("[/\{\}]", ""))|
-#end
-
-#foreach( $restControllerInfo in $endpoints )
-#[[###]]# `$restControllerInfo.getMethod().toString() $restControllerInfo.getPath()`
- * Description: $restControllerInfo.getDescription()
-#if($restControllerInfo.getParameterDescriptions().size() > 0)
- * Input:
-#end
-#foreach( $parameterDescription in $restControllerInfo.getParameterDescriptions().entrySet())
- * $parameterDescription.getKey() - $parameterDescription.getValue()
-#end
- * Returns:
-#foreach( $response in $restControllerInfo.getResponses())
- * $response.getCode() - $response.getMessage()
-#end
-
-#end
-#[[##]]# Testing
-
-Profiles are includes for both the metron-docker and Quick Dev environments.
-
-#[[###]]# metron-docker
-
-Start the [metron-docker](../../metron-docker) environment. Build the metron-rest module and start it with the Spring Boot Maven plugin:
-```
-mvn clean package
-mvn spring-boot:run -Drun.profiles=docker,dev
-```
-The metron-rest application will be available at http://localhost:8080/swagger-ui.html#/.
-
-#[[###]]# Quick Dev
-
-Start the [Quick Dev](../../metron-deployment/vagrant/quick-dev-platform) environment. Build the metron-rest module and start it with the Spring Boot Maven plugin:
-```
-mvn clean package
-mvn spring-boot:run -Drun.profiles=vagrant,dev
-```
-The metron-rest application will be available at http://localhost:8080/swagger-ui.html#/.
-
-To run the application locally on the Quick Dev host (node1), follow the [Installation](#installation) instructions above. Then set the METRON_SPRING_PROFILES_ACTIVE variable in `/etc/sysconfig/metron`:
-```
-METRON_SPRING_PROFILES_ACTIVE="vagrant,dev"
-```
-and start the application:
-```
-service metron-rest start
-```
-
-In a cluster with Kerberos enabled, update the security settings in `/etc/sysconfig/metron`. Security is disabled by default in the `vagrant` Spring profile so that setting must be overriden with the METRON_SPRING_OPTIONS variable:
-```
-METRON_SPRING_PROFILES_ACTIVE="vagrant,dev"
-METRON_JVMFLAGS="-Djava.security.auth.login.config=$METRON_HOME/client_jaas.conf"
-METRON_SPRING_OPTIONS="--kerberos.enabled=true"
-```
-The metron-rest application will be available at http://node1:8082/swagger-ui.html#/.
-
-#[[##]]# License
-
-This project depends on the Java Transaction API. See https://java.net/projects/jta-spec/ for more details.