You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@metron.apache.org by rm...@apache.org on 2016/04/26 16:46:38 UTC
[50/51] [partial] incubator-metron git commit: METRON-113 Project
Reorganization (merrimanr) closes apache/incubator-metron#88
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/0117987e/deployment/amazon-ec2/README.md
----------------------------------------------------------------------
diff --git a/deployment/amazon-ec2/README.md b/deployment/amazon-ec2/README.md
deleted file mode 100644
index b4dcc6f..0000000
--- a/deployment/amazon-ec2/README.md
+++ /dev/null
@@ -1,211 +0,0 @@
-Apache Metron on Amazon EC2
-===========================
-
-This project fully automates the provisioning of Apache Metron on Amazon EC2 infrastructure. Starting with only your Amazon EC2 credentials, this project will create a fully-functioning, end-to-end, multi-node cluster running Apache Metron.
-
-Getting Started
----------------
-
-### Prerequisites
-
-The host that will drive the provisioning process will need to have [Ansible](https://github.com/ansible/ansible), Python and PIP installed. In most cases, a development laptop serves this purpose just fine. Also, install the Python library `boto` and its dependencies.
-
-```
-pip install boto six
-```
-
-Ensure that an SSH key has been generated and stored at `~/.ssh/id_rsa.pub`. In most cases this key will already exist and no further action will be needed.
-
-### Create User
-
-1. Use Amazon's [Identity and Access Management](https://console.aws.amazon.com/iam/) tool to create a user account by navigating to `Users > Create New User`.
-
-2. Grant the user permission by clicking on `Permissions > Attach Policy` and add the following policies.
-
- ```
- AmazonEC2FullAccess
- AmazonVPCFullAccess
- ```
-
-3. Create an access key for the user by clicking on `Security Credentials > Create Access Key`. Save the provided access key values in a safe place. These values cannot be retrieved from the web console at a later time.
-
-4. Use the access key by exporting its values to the shell's environment. This allows Ansible to authenticate with Amazon EC2. For example:
-
- ```
- export AWS_ACCESS_KEY_ID="AKIAI6NRFEO27E5FFELQ"
- export AWS_SECRET_ACCESS_KEY="vTDydWJQnAer7OWauUS150i+9Np7hfCXrrVVP6ed"
- ```
-
-### Deploy Metron
-
-1. Ensure that Metron's streaming topology uber-jar has been built.
-
- ```
- cd ../../metron-streaming
- mvn clean package -DskipTests
- ```
-
-2. Start the Metron playbook. A full Metron deployment can consume up to 60 minutes. Grab a coffee, relax and practice mindfulness meditation. If the playbook fails mid-stream for any reason, simply re-run it.
-
- ```
- export EC2_INI_PATH=conf/ec2.ini
- ansible-playbook -i ec2.py playbook.yml
- ```
-
-### Explore Metron
-
-1. After the deployment has completed successfully, a message like the following will be displayed. Navigate to the specified resources to explore your newly minted Apache Metron environment.
-
- ```
- TASK [debug] *******************************************************************
- ok: [localhost] => {
- "Success": [
- "Apache Metron deployed successfully",
- " Metron @ http://ec2-52-37-255-142.us-west-2.compute.amazonaws.com:5000",
- " Ambari @ http://ec2-52-37-225-202.us-west-2.compute.amazonaws.com:8080",
- " Sensors @ ec2-52-37-225-202.us-west-2.compute.amazonaws.com on tap0",
- "For additional information, see https://metron.incubator.apache.org/'"
- ]
- }
- ```
-
-2. Each of the provisioned hosts will be accessible from the internet. Connecting to one over SSH as the user `centos` will not require a password as it will authenticate with the pre-defined SSH key.
-
- ```
- ssh centos@ec2-52-91-215-174.compute-1.amazonaws.com
- ```
-
-Advanced Usage
---------------
-
-### Multiple Environments
-
-This process can support provisioning of multiple, isolated environments. Simply change the `env` settings in `conf/defaults.yml`. For example, you might provision separate development, test, and production environments.
-
-```
-env: metron-test
-```
-
-### Selective Provisioning
-
-To provision only subsets of the entire Metron deployment, Ansible tags can be specified. For example, to only deploy the sensors on an Amazon EC2 environment, run the following command.
-
-```
-ansible-playbook -i ec2.py playbook.yml --tags "ec2,sensors"
-```
-
-### Custom SSH Key
-
-
-By default, the playbook will attempt to register your public SSH key `~/.ssh/id_rsa.pub` with each provisioned host. This enables Ansible to communicate with each host using an SSH connection. If would prefer to use another key simply add the path to the public key file to the `key_file` property in `conf/defaults.yml`.
-
-For example, generate a new SSH key for Metron that will be stored at `~/.ssh/my-metron-key`.
-
-```
-$ ssh-keygen -q -f ~/.ssh/my-metron-key
-Enter passphrase (empty for no passphrase):
-Enter same passphrase again:
-```
-
-Add the path to the newly created SSH public key to `conf/defaults.yml`.
-
-```
-key_file: ~/.ssh/metron-private-key.pub
-```
-
-Common Errors
--------------
-
-### Error: 'No handler was ready to authenticate...Check your credentials'
-
-```
-TASK [Define keypair] **********************************************************
-failed: [localhost] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXbcb1AlWsEPP
- r9jEFrn0yun3PYNidJ/...david@hasselhoff.com) => {"failed": true, "item": "ssh-r
- sa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXbcb1AlWsEPPr9jEFr... david@hasselhoff.com",
- "msg": "No handler was ready to authenticate. 1 handlers were checked.
- ['HmacAuthV4Handler'] Check your credentials"}
-```
-
-#### Solution 1
-
-This occurs when Ansible does not have the correct AWS access keys. The following commands must return a valid access key that is defined within Amazon's [Identity and Access Management](https://console.aws.amazon.com/iam/) console.
-
-```
-$ echo $AWS_ACCESS_KEY_ID
-AKIAI6NRFEO27E5FFELQ
-
-$ echo $AWS_SECRET_ACCESS_KEY
-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
-```
-
-#### Solution 2
-
-This error can occur if you have exported the correct AWS access key, but you are using `sudo` to run the Ansible playbook. Do not use the `sudo` command when running the Ansible playbook.
-
-### Error: 'OptInRequired: ... you need to accept terms and subscribe'
-
-```
-TASK [metron-test: Instantiate 1 host(s) as sensors,ambari_master,metron,ec2] **
-fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg":
-"Instance creation failed => OptInRequired: In order to use this AWS Marketplace
-product you need to accept terms and subscribe. To do so please visit
-http://aws.amazon.com/marketplace/pp?sku=6x5jmcajty9edm3f211pqjfn2"}
-to retry, use: --limit @playbook.retry
-```
-
-#### Solution
-
-Apache Metron uses the [official CentOS 6 Amazon Machine Image](https://aws.amazon.com/marketplace/pp?sku=6x5jmcajty9edm3f211pqjfn2) when provisioning hosts. Amazon requires that you accept certain terms and conditions when using any Amazon Machine Image (AMI). Follow the link provided in the error message to accept the terms and conditions then re-run the playbook.
-
-### Error: 'PendingVerification: Your account is currently being verified'
-
-```
-TASK [metron-test: Instantiate 1 host(s) as sensors,ambari_master,metron,ec2] **
-fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg":
-"Instance creation failed => PendingVerification: Your account is currently
-being verified. Verification normally takes less than 2 hours. Until your
-account is verified, you may not be able to launch additional instances or
-create additional volumes. If you are still receiving this message after more
-than 2 hours, please let us know by writing to aws-verification@amazon.com. We
-appreciate your patience."}
-to retry, use: --limit @playbook.retry
-```
-
-#### Solution
-
-This will occur if you are attempting to deploy Apache Metron using a newly created Amazon Web Services account. Follow the advice of the message and wait until Amazon's verification process is complete. Amazon has some additional [advice for dealing with this error and more](http://docs.aws.amazon.com/AWSEC2/latest/APIReference/errors-overview.html).
-
-> Your account is pending verification. Until the verification process is complete, you may not be able to carry out requests with this account. If you have questions, contact [AWS Support](http://console.aws.amazon.com/support/home#/).
-
-### Error: 'Instance creation failed => InstanceLimitExceeded'
-
-```
-TASK [metron-test: Instantiate 3 host(s) as search,metron,ec2] *****************
-fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg":
-"Instance creation failed => InstanceLimitExceeded: You have requested more
-instances (11) than your current instance limit of 10 allows for the specified
-instance type. Please visit http://aws.amazon.com/contact-us/ec2-request to
-request an adjustment to this limit."}
-to retry, use: --limit @playbook.retry
-```
-
-#### Solution
-
-This will occur if Apache Metron attempts to deploy more host instances than allowed by your account. The total number of instances required for Apache Metron can be reduced by editing `deployment/amazon-ec/playbook.yml`. Perhaps a better alternative is to request of Amazon that this limit be increased. Amazon has some additional [advice for dealing with this error and more](http://docs.aws.amazon.com/AWSEC2/latest/APIReference/errors-overview.html).
-
-> You've reached the limit on the number of instances you can run concurrently. The limit depends on the instance type. For more information, see [How many instances can I run in Amazon EC2](http://aws.amazon.com/ec2/faqs/#How_many_instances_can_I_run_in_Amazon_EC2). If you need additional instances, complete the [Amazon EC2 Instance Request Form](https://console.aws.amazon.com/support/home#/case/create?issueType=service-limit-increase&limitType=service-code-ec2-instances).
-
-### Error: 'SSH encountered an unknown error during the connection'
-
-```
-TASK [setup] *******************************************************************
-fatal: [ec2-52-26-113-221.us-west-2.compute.amazonaws.com]: UNREACHABLE! => {
- "changed": false, "msg": "SSH encountered an unknown error during the
- connection. We recommend you re-run the command using -vvvv, which will enable
- SSH debugging output to help diagnose the issue", "unreachable": true}
-```
-
-#### Solution
-
-This most often indicates that Ansible cannot connect to the host with the SSH key that it has access to. This could occur if hosts are provisioned with one SSH key, but the playbook is executed subsequently with a different SSH key. The issue can be addressed by either altering the `key_file` variable to point to the key that was used to provision the hosts or by simply terminating all hosts and re-running the playbook.
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/0117987e/deployment/amazon-ec2/ansible.cfg
----------------------------------------------------------------------
diff --git a/deployment/amazon-ec2/ansible.cfg b/deployment/amazon-ec2/ansible.cfg
deleted file mode 100644
index c8f26c4..0000000
--- a/deployment/amazon-ec2/ansible.cfg
+++ /dev/null
@@ -1,28 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-[defaults]
-host_key_checking = False
-library = ../extra_modules
-roles_path = ../roles
-pipelining = True
-remote_user = centos
-forks = 20
-log_path = ./ansible.log
-
-# fix for "ssh throws 'unix domain socket too long' " problem
-[ssh_connection]
-control_path = %(directory)s/%%h-%%p-%%r
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/0117987e/deployment/amazon-ec2/conf/defaults.yml
----------------------------------------------------------------------
diff --git a/deployment/amazon-ec2/conf/defaults.yml b/deployment/amazon-ec2/conf/defaults.yml
deleted file mode 100644
index 76c4b98..0000000
--- a/deployment/amazon-ec2/conf/defaults.yml
+++ /dev/null
@@ -1,80 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
-# ec2
-env: metron-test
-region: us-west-2
-instance_type: m4.xlarge
-image: ami-05cf2265
-volume_type: standard
-key_name: metron-key
-xvda_vol_size: 50
-xvdb_vol_size: 100
-xvdc_vol_size: 100
-
-# ambari
-ambari_host: "{{ groups.ambari_master[0] }}"
-ambari_port: 8080
-ambari_user: admin
-ambari_password: admin
-cluster_type: small_cluster
-
-# hbase
-pcap_hbase_table: pcap
-tracker_hbase_table: access_tracker
-threatintel_hbase_table: threatintel
-enrichment_hbase_table: enrichment
-
-# kafka
-num_partitions: 3
-retention_in_gb: 25
-
-# metron variables
-metron_version: 0.1BETA
-java_home: /usr/jdk64/jdk1.8.0_40
-pcapservice_port: 8081
-
-# sensors
-sensor_test_mode: True
-sniff_interface: tap0
-snort_alert_csv_path: "/var/log/snort/alert.csv"
-pcap_replay: True
-pcap_replay_interface: tap0
-install_tap: True
-pcap_path: /opt/pcap-replay
-
-# data directories
-zookeeper_data_dir: "/data1/hadoop/zookeeper"
-namenode_checkpoint_dir: "/data1/hadoop/hdfs/namesecondary"
-namenode_name_dir: "/data1/hadoop/hdfs/namenode"
-datanode_data_dir: "/data1/hadoop/hdfs/data,/data2/hadoop/hdfs/data"
-journalnode_edits_dir: "/data1/hadoop/hdfs/journalnode"
-nodemanager_local_dirs: "/data1/hadoop/yarn/local"
-timeline_ldb_store_path: "/data1/hadoop/yarn/timeline"
-timeline_ldb_state_path: "/data1/hadoop/yarn/timeline"
-nodemanager_log_dirs: "/data1/hadoop/yarn/log"
-jhs_recovery_store_ldb_path: "/data1/hadoop/mapreduce/jhs"
-storm_local_dir: "/data1/hadoop/storm"
-kafka_log_dirs: "/data2/kafka-log"
-elasticsearch_data_dir: "/data1/elasticsearch,/data2/elasticsearch"
-
-#Search
-install_elasticsearch: True
-install_solr: False
-elasticsearch_transport_port: 9300
-elasticsearch_network_interface: eth0
-elasticsearch_web_port: 9200
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/0117987e/deployment/amazon-ec2/conf/ec2.ini
----------------------------------------------------------------------
diff --git a/deployment/amazon-ec2/conf/ec2.ini b/deployment/amazon-ec2/conf/ec2.ini
deleted file mode 100755
index 646ffaf..0000000
--- a/deployment/amazon-ec2/conf/ec2.ini
+++ /dev/null
@@ -1,105 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
-# Ansible EC2 external inventory script settings.
-#
-# Refer to https://github.com/ansible/ansible/blob/devel/contrib/inventory/ec2.ini
-# for additional information on available settings
-#
-
-[ec2]
-
-# AWS regions to make calls to. Set this to 'all' to make request to all regions
-# in AWS and merge the results together. Alternatively, set this to a comma
-# separated list of regions. E.g. 'us-east-1,us-west-1,us-west-2'
-regions = all
-
-# When generating inventory, Ansible needs to know how to address a server.
-# Each EC2 instance has a lot of variables associated with it. Here is the list:
-# http://docs.pythonboto.org/en/latest/ref/ec2.html#module-boto.ec2.instance
-# Below are 2 variables that are used as the address of a server:
-# - destination_variable
-# - vpc_destination_variable
-
-# This is the normal destination variable to use. If you are running Ansible
-# from outside EC2, then 'public_dns_name' makes the most sense. If you are
-# running Ansible from within EC2, then perhaps you want to use the internal
-# address, and should set this to 'private_dns_name'. The key of an EC2 tag
-# may optionally be used; however the boto instance variables hold precedence
-# in the event of a collision.
-destination_variable = public_dns_name
-
-# For server inside a VPC, using DNS names may not make sense. When an instance
-# has 'subnet_id' set, this variable is used. If the subnet is public, setting
-# this to 'ip_address' will return the public IP address. For instances in a
-# private subnet, this should be set to 'private_ip_address', and Ansible must
-# be run from within EC2. The key of an EC2 tag may optionally be used; however
-# the boto instance variables hold precedence in the event of a collision.
-# WARNING: - instances that are in the private vpc, _without_ public ip address
-# will not be listed in the inventory until You set:
-# vpc_destination_variable = private_ip_address
-#vpc_destination_variable = ip_address
-vpc_destination_variable = public_dns_name
-
-# To exclude RDS instances from the inventory, uncomment and set to False.
-#rds = False
-
-# To exclude ElastiCache instances from the inventory, uncomment and set to False.
-#elasticache = False
-
-# API calls to EC2 are slow. For this reason, we cache the results of an API
-# call. Set this to the path you want cache files to be written to. Two files
-# will be written to this directory:
-# - ansible-ec2.cache
-# - ansible-ec2.index
-cache_path = ~/.ansible/tmp
-
-# The number of seconds a cache file is considered valid. After this many
-# seconds, a new API call will be made, and the cache file will be updated.
-# To disable the cache, set this value to 0
-cache_max_age = 300
-
-# Organize groups into a nested/hierarchy instead of a flat namespace.
-nested_groups = False
-
-# Replace - tags when creating groups to avoid issues with ansible
-replace_dash_in_groups = True
-
-# If set to true, any tag of the form "a,b,c" is expanded into a list
-# and the results are used to create additional tag_* inventory groups.
-expand_csv_tags = False
-
-# The EC2 inventory output can become very large. To manage its size,
-# configure which groups should be created.
-group_by_instance_id = True
-group_by_region = True
-group_by_availability_zone = True
-group_by_ami_id = True
-group_by_instance_type = True
-group_by_key_pair = True
-group_by_vpc_id = True
-group_by_security_group = True
-group_by_tag_keys = True
-group_by_tag_none = True
-group_by_route53_names = True
-group_by_rds_engine = True
-group_by_rds_parameter_group = True
-group_by_elasticache_engine = True
-group_by_elasticache_cluster = True
-group_by_elasticache_parameter_group = True
-group_by_elasticache_replication_group = True
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/0117987e/deployment/amazon-ec2/playbook.yml
----------------------------------------------------------------------
diff --git a/deployment/amazon-ec2/playbook.yml b/deployment/amazon-ec2/playbook.yml
deleted file mode 100644
index 16d281d..0000000
--- a/deployment/amazon-ec2/playbook.yml
+++ /dev/null
@@ -1,80 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
-#
-# instantiate the hosts on amazon ec2
-#
-- hosts: localhost
- vars_files:
- - conf/defaults.yml
- tasks:
- - include: tasks/create-keypair.yml
- - include: tasks/create-vpc.yml
- - include: tasks/create-open-inbound-security-group.yml
- - include: tasks/create-open-outbound-security-group.yml
- - include: tasks/create-hosts.yml host_count=1 host_type=sensors,ambari_master,metron,ec2
- - include: tasks/create-hosts.yml host_count=4 host_type=ambari_slave,ec2
- - include: tasks/create-hosts.yml host_count=1 host_type=ambari_slave,hadoop_client,metron,ec2
- - include: tasks/create-hosts.yml host_count=1 host_type=ambari_slave,enrichment,metron,ec2
- - include: tasks/create-hosts.yml host_count=2 host_type=search,metron,ec2
- - include: tasks/create-hosts.yml host_count=1 host_type=web,mysql,metron,ec2
- tags:
- - ec2
-
-#
-# wait for all ec2 hosts to come up
-#
-- hosts: ec2
- become: True
- vars_files:
- - conf/defaults.yml
- gather_facts: False
- tasks:
- - include: tasks/check-hosts.yml
- tags:
- - ec2
- - wait
-
-#
-# mount additional data volumes on all ec2 hosts
-#
-- hosts: ec2
- become: True
- vars_files:
- - conf/defaults.yml
- tasks:
- - include: tasks/mount-volume.yml vol_src=/dev/xvdb vol_mnt=/data1
- - include: tasks/mount-volume.yml vol_src=/dev/xvdc vol_mnt=/data2
- - include: tasks/check-volume.yml vol_name=xvda vol_src=/dev/xvda vol_size={{ xvda_vol_size }}
- tags:
- - ec2
-
-#
-# build the metron cluster
-#
-- include: ../playbooks/metron_full_install.yml
-
-#
-# provisioning report
-#
-- hosts: localhost
- vars_files:
- - conf/defaults.yml
- tasks:
- - include: tasks/provisioning-report.yml
- tags:
- - ec2
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/0117987e/deployment/amazon-ec2/tasks/check-hosts.yml
----------------------------------------------------------------------
diff --git a/deployment/amazon-ec2/tasks/check-hosts.yml b/deployment/amazon-ec2/tasks/check-hosts.yml
deleted file mode 100644
index 1a4b2c7..0000000
--- a/deployment/amazon-ec2/tasks/check-hosts.yml
+++ /dev/null
@@ -1,20 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
-- name: Wait for connectivity to host(s)
- local_action: wait_for host={{ inventory_hostname }} state=started timeout=300 delay=10
- become: False
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/0117987e/deployment/amazon-ec2/tasks/check-volume.yml
----------------------------------------------------------------------
diff --git a/deployment/amazon-ec2/tasks/check-volume.yml b/deployment/amazon-ec2/tasks/check-volume.yml
deleted file mode 100644
index b7ac63d..0000000
--- a/deployment/amazon-ec2/tasks/check-volume.yml
+++ /dev/null
@@ -1,26 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
-- name: "Check size of volume {{ vol_src }}"
- shell: "lsblk | grep part | grep {{ vol_name }} | awk '{ print $4}' | sed 's/[^0-9]//g'"
- register: current_size
-
-- name: "Status of {{ vol_src }} volume"
- debug: msg="volume={{ vol_src }} current={{ current_size.stdout|int }} expected={{ vol_size|int }}"
-
-- include: expand-volume.yml vol_src={{ vol_src }}
- when: current_size.stdout|int < vol_size|int
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/0117987e/deployment/amazon-ec2/tasks/create-hosts.yml
----------------------------------------------------------------------
diff --git a/deployment/amazon-ec2/tasks/create-hosts.yml b/deployment/amazon-ec2/tasks/create-hosts.yml
deleted file mode 100644
index 39bae3a..0000000
--- a/deployment/amazon-ec2/tasks/create-hosts.yml
+++ /dev/null
@@ -1,54 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
-- name: "{{ env }}: Instantiate {{ host_count }} host(s) as {{ host_type }}"
- ec2:
- region: "{{ region }}"
- instance_type: "{{ instance_type }}"
- image: "{{ image }}"
- key_name: "{{ env }}-{{ key_name }}"
- assign_public_ip: True
- group: ["{{ env }}-vpc-all-inbound","{{ env }}-vpc-all-outbound"]
- vpc_subnet_id: "{{ vpc.subnets[0].id }}"
- instance_tags:
- Name: "[{{ env }}] {{ host_type }}"
- type: "{{ host_type }}"
- env: "{{ env }}"
- exact_count: "{{ host_count }}"
- count_tag:
- type: "{{ host_type }}"
- env: "{{ env }}"
- volumes:
- - device_name: /dev/sda1
- volume_type: "{{ volume_type }}"
- volume_size: "{{ xvda_vol_size }}"
- delete_on_termination: true
- - device_name: /dev/xvdb
- volume_type: "{{ volume_type }}"
- volume_size: "{{ xvdb_vol_size }}"
- delete_on_termination: true
- - device_name: /dev/xvdc
- volume_type: "{{ volume_type }}"
- volume_size: "{{ xvdc_vol_size }}"
- delete_on_termination: true
- wait: yes
- register: ec2
-
-- name: Add host(s) to a hostgroup
- add_host: hostname={{ item.public_dns_name }} groups={{ host_type }}
- with_items: "{{ ec2.tagged_instances }}"
- when: item.public_dns_name is defined
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/0117987e/deployment/amazon-ec2/tasks/create-keypair.yml
----------------------------------------------------------------------
diff --git a/deployment/amazon-ec2/tasks/create-keypair.yml b/deployment/amazon-ec2/tasks/create-keypair.yml
deleted file mode 100644
index 693039e..0000000
--- a/deployment/amazon-ec2/tasks/create-keypair.yml
+++ /dev/null
@@ -1,29 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
-- set_fact:
- the_key_file: "{{ key_file | default('~/.ssh/id_rsa.pub') }}"
-
-- name: Define keypair
- ec2_key:
- name: "{{ env }}-{{ key_name }}"
- region: "{{ region }}"
- key_material: "{{ item }}"
- with_file: "{{ the_key_file }}"
-
-- debug: msg="Created keypair '{{ env }}-{{ key_name }}' from '{{ the_key_file }}'"
-
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/0117987e/deployment/amazon-ec2/tasks/create-open-inbound-security-group.yml
----------------------------------------------------------------------
diff --git a/deployment/amazon-ec2/tasks/create-open-inbound-security-group.yml b/deployment/amazon-ec2/tasks/create-open-inbound-security-group.yml
deleted file mode 100644
index 67e89c8..0000000
--- a/deployment/amazon-ec2/tasks/create-open-inbound-security-group.yml
+++ /dev/null
@@ -1,26 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
-- name: "{{ env }}: Define open inbound security group"
- ec2_group:
- name: "{{ env }}-vpc-all-inbound"
- description: WARNING allow all inbound connections from the internet
- region: "{{ region }}"
- vpc_id: "{{ vpc_id }}"
- rules:
- - proto: all
- cidr_ip: 0.0.0.0/0
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/0117987e/deployment/amazon-ec2/tasks/create-open-outbound-security-group.yml
----------------------------------------------------------------------
diff --git a/deployment/amazon-ec2/tasks/create-open-outbound-security-group.yml b/deployment/amazon-ec2/tasks/create-open-outbound-security-group.yml
deleted file mode 100644
index 53f505f..0000000
--- a/deployment/amazon-ec2/tasks/create-open-outbound-security-group.yml
+++ /dev/null
@@ -1,26 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
-- name: "{{ env }}: Define open outbound security group"
- ec2_group:
- name: "{{ env }}-vpc-all-outbound"
- description: allow all outbound connections to the internet
- region: "{{ region }}"
- vpc_id: "{{ vpc_id }}"
- rules_egress:
- - proto: all
- cidr_ip: 0.0.0.0/0
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/0117987e/deployment/amazon-ec2/tasks/create-security-group.yml
----------------------------------------------------------------------
diff --git a/deployment/amazon-ec2/tasks/create-security-group.yml b/deployment/amazon-ec2/tasks/create-security-group.yml
deleted file mode 100644
index 1c9b909..0000000
--- a/deployment/amazon-ec2/tasks/create-security-group.yml
+++ /dev/null
@@ -1,28 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
-- name: "{{ env }}: Define the {{ name }} security group"
- ec2_group:
- name: "{{ env }}-{{ name }}"
- region: "{{ region }}"
- description: "[{{env}}] {{ name }}/{{ proto }}/{{ port }}"
- vpc_id: "{{ vpc_id }}"
- rules:
- - proto: "{{ proto }}"
- from_port: "{{ port }}"
- to_port: "{{ port }}"
- cidr_ip: 0.0.0.0/0
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/0117987e/deployment/amazon-ec2/tasks/create-vpc.yml
----------------------------------------------------------------------
diff --git a/deployment/amazon-ec2/tasks/create-vpc.yml b/deployment/amazon-ec2/tasks/create-vpc.yml
deleted file mode 100644
index 7fc31e7..0000000
--- a/deployment/amazon-ec2/tasks/create-vpc.yml
+++ /dev/null
@@ -1,50 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
- - name: "{{ env }}: Create virtual private cloud"
- ec2_vpc:
- region: "{{ region }}"
- internet_gateway: True
- resource_tags:
- Name: "{{ env }}-virtual-private-cloud"
- env: "{{ env }}"
- cidr_block: 10.0.0.0/16
- dns_hostnames: yes
- dns_support: yes
- subnets:
- - cidr: 10.0.0.0/24
- resource_tags:
- tier: web
- - cidr: 10.0.1.0/24
- resource_tags:
- tier: hdp
- - cidr: 10.0.2.0/24
- resource_tags:
- tier: sensors
- route_tables:
- - subnets:
- - 10.0.0.0/24
- - 10.0.1.0/24
- - 10.0.2.0/24
- routes:
- - dest: 0.0.0.0/0
- gw: igw
- register: vpc
-
- - name: "[{{ env }}] Created vpc with id={{ vpc.vpc_id }}"
- set_fact:
- vpc_id: "{{ vpc.vpc_id }}"
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/0117987e/deployment/amazon-ec2/tasks/expand-volume.yml
----------------------------------------------------------------------
diff --git a/deployment/amazon-ec2/tasks/expand-volume.yml b/deployment/amazon-ec2/tasks/expand-volume.yml
deleted file mode 100644
index 1e25e27..0000000
--- a/deployment/amazon-ec2/tasks/expand-volume.yml
+++ /dev/null
@@ -1,30 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
-- name: "Expand {{ vol_src }} volume"
-# sectors delete 1 new primary first past mbr to end bootable write and exit
- shell: "(echo u s; echo d 1; echo n; echo p; echo 1; echo 2048 ; echo ;echo a; echo 1; echo w) | fdisk {{ vol_src }} || true"
- args:
- executable: /bin/bash
-
-- name: Restart host(s)
- command: shutdown -r now "Trigger volume changes"
- async: 0
- poll: 0
- ignore_errors: True
-
-- include: tasks/check-hosts.yml
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/0117987e/deployment/amazon-ec2/tasks/mount-volume.yml
----------------------------------------------------------------------
diff --git a/deployment/amazon-ec2/tasks/mount-volume.yml b/deployment/amazon-ec2/tasks/mount-volume.yml
deleted file mode 100644
index 11259a5..0000000
--- a/deployment/amazon-ec2/tasks/mount-volume.yml
+++ /dev/null
@@ -1,32 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
-- name: Install xfsprogs
- yum:
- name: xfsprogs
- state: present
- update_cache: yes
- register: result
- until: result.rc == 0
- retries: 5
- delay: 10
-
-- name: Format data volume(s)
- filesystem: fstype=xfs dev={{ vol_src }}
-
-- name: Mount the volume
- mount: name={{ vol_mnt }} src={{ vol_src }} opts=noatime fstype=xfs state=mounted
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/0117987e/deployment/amazon-ec2/tasks/provisioning-report.yml
----------------------------------------------------------------------
diff --git a/deployment/amazon-ec2/tasks/provisioning-report.yml b/deployment/amazon-ec2/tasks/provisioning-report.yml
deleted file mode 100644
index d2abec0..0000000
--- a/deployment/amazon-ec2/tasks/provisioning-report.yml
+++ /dev/null
@@ -1,35 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the 'License'); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an 'AS IS' BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
-- name: Known hosts groups
- debug: var=groups
-
-- name: Sanity check Metron web
- local_action: wait_for host="{{ groups.web[0] }}" port=5000 timeout=20
-
-- name: Sanity check Ambari web
- local_action: wait_for host="{{ groups.ambari_master[0] }}" port="{{ ambari_port }}" timeout=20
-
-- set_fact:
- Success:
- - "Apache Metron deployed successfully"
- - " Metron @ http://{{ groups.web[0] }}:5000"
- - " Ambari @ http://{{ groups.ambari_master[0] }}:{{ ambari_port }}"
- - " Sensors @ {{ groups.sensors[0] }} on {{ sniff_interface }}"
- - For additional information, see https://metron.incubator.apache.org/'
-
-- debug: var=Success
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/0117987e/deployment/ansible.cfg
----------------------------------------------------------------------
diff --git a/deployment/ansible.cfg b/deployment/ansible.cfg
deleted file mode 100644
index 9b3916b..0000000
--- a/deployment/ansible.cfg
+++ /dev/null
@@ -1,23 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-[defaults]
-host_key_checking = false
-library = extra_modules
-roles_path = ./roles
-
-[ssh_connection]
-control_path = %(directory)s/%%h-%%p-%%r
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/0117987e/deployment/extra_modules/ambari_cluster_state.py
----------------------------------------------------------------------
diff --git a/deployment/extra_modules/ambari_cluster_state.py b/deployment/extra_modules/ambari_cluster_state.py
deleted file mode 100644
index 14c2004..0000000
--- a/deployment/extra_modules/ambari_cluster_state.py
+++ /dev/null
@@ -1,392 +0,0 @@
-#!/usr/bin/python
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-DOCUMENTATION = '''
----
-module: ambari_cluster_state
-version_added: "2.1"
-author: Mark Bittmann (https://github.com/mbittmann)
-short_description: Create, delete, start or stop an ambari cluster
-description:
- - Create, delete, start or stop an ambari cluster
-options:
- host:
- description:
- The hostname for the ambari web server
- port:
- description:
- The port for the ambari web server
- username:
- description:
- The username for the ambari web server
- password:
- description:
- The name of the cluster in web server
- required: yes
- cluster_name:
- description:
- The name of the cluster in ambari
- required: yes
- cluster_state:
- description:
- The desired state for the ambari cluster ['present', 'absent', 'started', 'stopped']. Setting the cluster
- state to absent will first stop the cluster.
- required: yes
- blueprint_var:
- description:
- The path to the file defining the cluster blueprint and host mapping. Required when state == 'present'
- required: no
- blueprint_name:
- description:
- The name of the blueprint. Required when state == 'present'
- required: no
- wait_for_complete:
- description:
- Whether to wait for the request to complete before returning. Default is False.
- required: no
- requirements: [ 'requests']
-'''
-
-EXAMPLES = '''
-# must use full relative path to any files in stored in roles/role_name/files/
-- name: Create a new ambari cluster
- ambari_cluster_state:
- host: localhost
- port: 8080
- username: admin
- password: admin
- cluster_name: my_cluster
- cluster_state: present
- blueprint_var: roles/my_role/files/blueprint.yml
- blueprint_name: hadoop
- wait_for_complete: True
-- name: Start the ambari cluster
- ambari_cluster_state:
- host: localhost
- port: 8080
- username: admin
- password: admin
- cluster_name: my_cluster
- cluster_state: started
- wait_for_complete: True
-- name: Stop the ambari cluster
- ambari_cluster_state:
- host: localhost
- port: 8080
- username: admin
- password: admin
- cluster_name: my_cluster
- cluster_state: stopped
- wait_for_complete: True
-- name: Delete the ambari cluster
- ambari_cluster_state:
- host: localhost
- port: 8080
- username: admin
- password: admin
- cluster_name: my_cluster
- cluster_state: absent
-'''
-
-RETURN = '''
-results:
- description: The content of the requests object returned from the RESTful call
- returned: success
- type: string
-created_blueprint:
- description: Whether a blueprint was created
- returned: success
- type: boolean
-status:
- description: The status of the blueprint creation process
- returned: success
- type: string
-'''
-
-__author__ = 'mbittmann'
-
-import json
-try:
- import requests
-except ImportError:
- REQUESTS_FOUND = False
-else:
- REQUESTS_FOUND = True
-
-
-def main():
-
- argument_spec = dict(
- host=dict(type='str', default=None, required=True),
- port=dict(type='int', default=None, required=True),
- username=dict(type='str', default=None, required=True),
- password=dict(type='str', default=None, required=True),
- cluster_name=dict(type='str', default=None, required=True),
- cluster_state=dict(type='str', default=None, required=True,
- choices=['present', 'absent', 'started', 'stopped']),
- blueprint_var=dict(type='dict', required=False),
- blueprint_name=dict(type='str', default=None, required=False),
- configurations=dict(type='list', default=None, required=False),
- wait_for_complete=dict(default=False, required=False, choices=BOOLEANS),
- )
-
- required_together = ['blueprint_var', 'blueprint_name']
-
- module = AnsibleModule(
- argument_spec=argument_spec,
- required_together=required_together
- )
-
- if not REQUESTS_FOUND:
- module.fail_json(
- msg='requests library is required for this module')
-
- p = module.params
-
- host = p.get('host')
- port = p.get('port')
- username = p.get('password')
- password = p.get('password')
- cluster_name = p.get('cluster_name')
- cluster_state = p.get('cluster_state')
- blueprint_name = p.get('blueprint_name')
- wait_for_complete = p.get('wait_for_complete')
-
- ambari_url = 'http://{0}:{1}'.format(host, port)
-
- try:
- if cluster_state in ['started', 'stopped']:
- if not cluster_exists(ambari_url, username, password, cluster_name):
- module.fail_json(msg="Cluster name {0} does not exist".format(cluster_name))
- state = ''
- if cluster_state == 'started':
- state = 'STARTED'
- elif cluster_state == 'stopped':
- state = 'INSTALLED'
-
- request = set_cluster_state(ambari_url, username, password, cluster_name, state)
- if wait_for_complete:
- request_id = json.loads(request.content)['Requests']['id']
- status = wait_for_request_complete(ambari_url, username, password, cluster_name, request_id, 2)
- if status != 'COMPLETED':
- module.fail_json(msg="Request failed with status {0}".format(status))
- module.exit_json(changed=True, results=request.content)
- elif cluster_state == 'absent':
- if not cluster_exists(ambari_url, username, password, cluster_name):
- module.exit_json(changed=False, msg='Skipping. Cluster does not exist')
- if not can_delete_cluster(ambari_url, username, password, cluster_name):
- request = set_cluster_state(ambari_url, username, password, cluster_name, 'INSTALLED')
- request_id = json.loads(request.content)['Requests']['id']
- status = wait_for_request_complete(ambari_url, username, password, cluster_name, request_id, 2)
- if status != 'COMPLETED':
- module.fail_json(msg="Request failed with status {0}".format(status))
- request = delete_cluster(ambari_url, username, password, cluster_name)
- module.exit_json(changed=True, results=request.content)
- elif cluster_state == 'present':
- if not p.get('blueprint_var') or not blueprint_name: # have neither name nor file
- module.fail_json(msg="Must provide blueprint_var and blueprint_name when cluster_state=='present'")
-
- blueprint_var = p.get('blueprint_var')
- blueprint, host_map = blueprint_var_to_ambari_converter(blueprint_var)
- created_blueprint = False
-
- if not blueprint_exists(ambari_url, username, password, blueprint_name):
- create_blueprint(ambari_url, username, password, blueprint_name, blueprint)
- created_blueprint = True
-
- if cluster_exists(ambari_url, username, password, cluster_name):
- module.exit_json(changed=False, msg='Cluster {0} already exists'.format(cluster_name),
- created_blueprint=created_blueprint)
-
- configurations = p.get('configurations')
- request = create_cluster(ambari_url, username, password, cluster_name, blueprint_name, configurations, host_map)
- request_id = json.loads(request.content)['Requests']['id']
- if wait_for_complete:
- status = wait_for_request_complete(ambari_url, username, password, cluster_name, request_id, 2)
- if status != 'COMPLETED':
- module.fail_json(msg="Request failed with status {0}".format(status))
- request_status = get_request_status(ambari_url, username, password, cluster_name, request_id)
- module.exit_json(changed=True, results=request.content,
- created_blueprint=created_blueprint, status=request_status)
-
- except requests.ConnectionError, e:
- module.fail_json(msg="Could not connect to Ambari client: " + str(e.message))
- except Exception, e:
- module.fail_json(msg="Ambari client exception occurred: " + str(e.message))
-
-
-def get_clusters(ambari_url, user, password):
- r = get(ambari_url, user, password, '/api/v1/clusters')
- if r.status_code != 200:
- msg = 'Coud not get cluster list: request code {0}, \
- request message {1}'.format(r.status_code, r.content)
- raise Exception(msg)
- clusters = json.loads(r.content)
- return clusters['items']
-
-
-def cluster_exists(ambari_url, user, password, cluster_name):
- clusters = get_clusters(ambari_url, user, password)
- return cluster_name in [item['Clusters']['cluster_name'] for item in clusters]
-
-
-def set_cluster_state(ambari_url, user, password, cluster_name, cluster_state):
- path = '/api/v1/clusters/{0}/services'.format(cluster_name)
- request = {"RequestInfo": {"context": "Setting cluster state"},
- "Body": {"ServiceInfo": {"state": "{0}".format(cluster_state)}}}
- payload = json.dumps(request)
- r = put(ambari_url, user, password, path, payload)
- if r.status_code not in [202, 200]:
- msg = 'Coud not set cluster state: request code {0}, \
- request message {1}'.format(r.status_code, r.content)
- raise Exception(msg)
- return r
-
-
-def create_cluster(ambari_url, user, password, cluster_name, blueprint_name, configurations, hosts_json):
- path = '/api/v1/clusters/{0}'.format(cluster_name)
- data = json.dumps({'blueprint': blueprint_name, 'configurations': configurations, 'host_groups': hosts_json})
- f = open('cluster.log', 'w')
- f.write(data)
- f.close()
- r = post(ambari_url, user, password, path, data)
- if r.status_code != 202:
- msg = 'Coud not create cluster: request code {0}, \
- request message {1}'.format(r.status_code, r.content)
- raise Exception(msg)
- return r
-
-
-def get_request_status(ambari_url, user, password, cluster_name, request_id):
- path = '/api/v1/clusters/{0}/requests/{1}'.format(cluster_name, request_id)
- r = get(ambari_url, user, password, path)
- if r.status_code != 200:
- msg = 'Coud not get cluster request status: request code {0}, \
- request message {1}'.format(r.status_code, r.content)
- raise Exception(msg)
- service = json.loads(r.content)
- return service['Requests']['request_status']
-
-
-def wait_for_request_complete(ambari_url, user, password, cluster_name, request_id, sleep_time):
- while True:
- status = get_request_status(ambari_url, user, password, cluster_name, request_id)
- if status == 'COMPLETED':
- return status
- elif status in ['FAILED', 'TIMEDOUT', 'ABORTED', 'SKIPPED_FAILED']:
- return status
- else:
- time.sleep(sleep_time)
-
-
-def can_delete_cluster(ambari_url, user, password, cluster_name):
- path = '/api/v1/clusters/{0}/services?ServiceInfo/state=STARTED'.format(cluster_name)
- r = get(ambari_url, user, password, path)
- items = json.loads(r.content)['items']
- return len(items) > 0
-
-
-def get_blueprints(ambari_url, user, password):
- path = '/api/v1/blueprints'
- r = get(ambari_url, user, password, path)
- if r.status_code != 200:
- msg = 'Coud not get blueprint list: request code {0}, \
- request message {1}'.format(r.status_code, r.content)
- raise Exception(msg)
-
- services = json.loads(r.content)
- return services['items']
-
-
-def create_blueprint(ambari_url, user, password, blueprint_name, blueprint_data):
- data = json.dumps(blueprint_data)
- f = open('blueprint.log', 'w')
- f.write(data)
- f.close()
- path = "/api/v1/blueprints/" + blueprint_name
- r = post(ambari_url, user, password, path, data)
- if r.status_code != 201:
- msg = 'Coud not create blueprint: request code {0}, \
- request message {1}'.format(r.status_code, r.content)
- raise Exception(msg)
- return r
-
-
-def blueprint_exists(ambari_url, user, password, blueprint_name):
- blueprints = get_blueprints(ambari_url, user, password)
- return blueprint_name in [item['Blueprints']['blueprint_name'] for item in blueprints]
-
-
-def delete_cluster(ambari_url, user, password, cluster_name):
- path = '/api/v1/clusters/{0}'.format(cluster_name)
- r = delete(ambari_url, user, password, path)
- if r.status_code != 200:
- msg = 'Coud not delete cluster: request code {0}, \
- request message {1}'.format(r.status_code, r.content)
- raise Exception(msg)
- return r
-
-
-def get(ambari_url, user, password, path):
- r = requests.get(ambari_url + path, auth=(user, password))
- return r
-
-
-def put(ambari_url, user, password, path, data):
- headers = {'X-Requested-By': 'ambari'}
- r = requests.put(ambari_url + path, data=data, auth=(user, password), headers=headers)
- return r
-
-
-def post(ambari_url, user, password, path, data):
- headers = {'X-Requested-By': 'ambari'}
- r = requests.post(ambari_url + path, data=data, auth=(user, password), headers=headers)
- return r
-
-
-def delete(ambari_url, user, password, path):
- headers = {'X-Requested-By': 'ambari'}
- r = requests.delete(ambari_url + path, auth=(user, password), headers=headers)
- return r
-
-
-def blueprint_var_to_ambari_converter(blueprint_var):
- groups = blueprint_var['groups']
- new_groups = []
- host_map = []
- for group in groups:
- components = []
- for component in group['components']:
- components.append({'name': component})
- group['components'] = components
- hosts = group.pop('hosts')
- new_groups.append(group)
- this_host_map = dict()
- this_host_map['name'] = group['name']
- this_host_list = [{'fqdn': host} for host in hosts]
- this_host_map['hosts'] = this_host_list
- host_map.append(this_host_map)
- blueprint = dict()
- blueprint['host_groups'] = new_groups
- blueprint['Blueprints'] = {'stack_name': blueprint_var['stack_name'], 'stack_version': blueprint_var['stack_version']}
-
- return blueprint, host_map
-
-from ansible.module_utils.basic import *
-if __name__ == '__main__':
- main()
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/0117987e/deployment/inventory/metron_example/group_vars/all
----------------------------------------------------------------------
diff --git a/deployment/inventory/metron_example/group_vars/all b/deployment/inventory/metron_example/group_vars/all
deleted file mode 100644
index 097516d..0000000
--- a/deployment/inventory/metron_example/group_vars/all
+++ /dev/null
@@ -1,77 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#Ansible Variables
-ansible_ssh_private_key_file: /Path/to/private/key/file #Change This
-ansible_ssh_user: root
-
-#Ambari variables
-ambari_host: "{{ groups.ambari_master[0] }}"
-ambari_port: 8080
-ambari_user: admin
-ambari_password: admin
-cluster_type: small_cluster
-
-# hbase
-pcap_hbase_table: pcap
-tracker_hbase_table: access_tracker
-threatintel_hbase_table: threatintel
-enrichment_hbase_table: enrichment
-
-# metron variables
-metron_version: 0.1BETA
-java_home: /usr/jdk64/jdk1.8.0_40
-pcapservice_port: 8081
-
-# sensors
-sensor_test_mode: True
-sniff_interface: eth0
-bro_version: "2.4.1"
-fixbuf_version: "1.7.1"
-yaf_version: "2.8.0"
-daq_version: "2.0.6-1"
-iface: "eth0"
-pycapa_repo: "https://github.com/OpenSOC/pycapa.git"
-pycapa_home: "/opt/pycapa"
-snort_version: "2.9.8.0-1"
-snort_alert_csv_path: "/var/log/snort/alert.csv"
-
-#PCAP Replay
-pcap_replay: True
-pcap_replay_interface: eth1
-
-#data directories - only required to override defaults
-#zookeeper_data_dir: "/newdir/hadoop/zookeeper"
-#namenode_checkpoint_dir: "/newdir/hadoop/hdfs/namesecondary"
-#namenode_name_dir: "/newdir/hadoop/hdfs/namenode"
-#datanode_data_dir: "/newdir/hadoop/hdfs/data"
-#journalnode_edits_dir: "/newdir/hadoop/hdfs/journalnode"
-#nodemanager_local_dirs: "/newdir/hadoop/yarn/local"
-#timeline_ldb_store_path: "/newdir/hadoop/yarn/timeline"
-#timeline_ldb_state_path: "/newdir/hadoop/yarn/timeline"
-#nodemanager_log_dirs: "/newdir/hadoop/yarn/log"
-#jhs_recovery_store_ldb_path: "/newdir/hadoop/mapreduce/jhs"
-#storm_local_dir: "/newdir/hadoop/storm"
-#kafka_log_dirs: "/newdir/kafka-log"
-#elasticsearch_data_dir: "/newdir1/elasticsearch"
-
-#Search
-install_elasticsearch: True
-install_solr: False
-elasticsearch_transport_port: 9300
-elasticsearch_network_interface: eth1
-elasticsearch_web_port: 9200
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/0117987e/deployment/inventory/metron_example/hosts
----------------------------------------------------------------------
diff --git a/deployment/inventory/metron_example/hosts b/deployment/inventory/metron_example/hosts
deleted file mode 100644
index 0d01327..0000000
--- a/deployment/inventory/metron_example/hosts
+++ /dev/null
@@ -1,63 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-[ambari_master]
-node1
-
-#minimum of 3 - 6 from 12 node cluser
-[ambari_slave]
-node2
-node3
-node4
-node5
-node6
-node7
-node8
-
-#last ambari_slave
-[hadoop_client]
-node9
-
-#3rd ambari_slave
-[enrichment]
-node1
-
-#1 or more
-[search]
-node10
-node11
-node12
-
-#1 only
-[sensors]
-node1
-
-#same as mysql in 12 node topology
-[web]
-node12
-
-[mysql]
-node12
-
-[metron:children]
-enrichment
-search
-web
-sensors
-mysql
-hadoop_client
-
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/0117987e/deployment/inventory/multinode-vagrant/group_vars/all
----------------------------------------------------------------------
diff --git a/deployment/inventory/multinode-vagrant/group_vars/all b/deployment/inventory/multinode-vagrant/group_vars/all
deleted file mode 100644
index a4a6af5..0000000
--- a/deployment/inventory/multinode-vagrant/group_vars/all
+++ /dev/null
@@ -1,75 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#Ambari variables
-ambari_host: "{{ groups.ambari_master[0] }}"
-hdp_host_group: "{{ groups.ambari_slave }}"
-ambari_port: 8080
-ambari_user: admin
-ambari_password: admin
-cluster_type: multi_vagrant_cluster
-
-# hbase
-pcap_hbase_table: pcap
-tracker_hbase_table: access_tracker
-threatintel_hbase_table: threatintel
-enrichment_hbase_table: enrichment
-
-#elasticsearch
-elasticsearch_transport_port: 9300
-elasticsearch_network_interface: eth1
-elasticsearch_web_port: 9200
-
-# metron variables
-metron_version: 0.1BETA
-java_home: /usr/jdk64/jdk1.8.0_40
-pcapservice_port: 8081
-
-# sensors
-sensor_test_mode: True
-sniff_interface: eth1
-bro_version: "2.4.1"
-fixbuf_version: "1.7.1"
-yaf_version: "2.8.0"
-daq_version: "2.0.6-1"
-iface: "eth0"
-pycapa_repo: "https://github.com/OpenSOC/pycapa.git"
-pycapa_home: "/opt/pycapa"
-snort_version: "2.9.8.0-1"
-snort_alert_csv_path: "/var/log/snort/alert.csv"
-
-#data directories
-#zookeeper_data_dir: "/newdir/hadoop/zookeeper"
-#namenode_checkpoint_dir: "/newdir/hadoop/hdfs/namesecondary"
-#namenode_name_dir: "/newdir/hadoop/hdfs/namenode"
-#datanode_data_dir: "/newdir/hadoop/hdfs/data"
-#journalnode_edits_dir: "/newdir/hadoop/hdfs/journalnode"
-#nodemanager_local_dirs: "/newdir/hadoop/yarn/local"
-#timeline_ldb_store_path: "/newdir/hadoop/yarn/timeline"
-#timeline_ldb_state_path: "/newdir/hadoop/yarn/timeline"
-#nodemanager_log_dirs: "/newdir/hadoop/yarn/log"
-#jhs_recovery_store_ldb_path: "/newdir/hadoop/mapreduce/jhs"
-#storm_local_dir: "/newdir/hadoop/storm"
-#kafka_log_dirs: "/newdir/kafka-log"
-#elasticsearch_data_dir: "/newdir1/elasticsearch"
-
-#Search
-install_elasticsearch: True
-install_solr: False
-elasticsearch_transport_port: 9300
-elasticsearch_network_interface: eth1
-elasticsearch_web_port: 9200
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/0117987e/deployment/inventory/multinode-vagrant/hosts
----------------------------------------------------------------------
diff --git a/deployment/inventory/multinode-vagrant/hosts b/deployment/inventory/multinode-vagrant/hosts
deleted file mode 100644
index d84ab1e..0000000
--- a/deployment/inventory/multinode-vagrant/hosts
+++ /dev/null
@@ -1,59 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-[ambari_master]
-node1
-
-#minimum of 3 - 6 from 12 node cluser
-[ambari_slave]
-node2
-node3
-node4
-
-#last ambari_slave
-[hadoop_client]
-node4
-
-[enrichment]
-node4
-
-#1 or more
-[search]
-node1
-
-#1 only
-[sensors]
-node1
-
-#same as mysql in 12 node topology
-[web]
-node3
-
-[mysql]
-node3
-
-[metron:children]
-enrichment
-search
-web
-sensors
-mysql
-hadoop_client
-
-
-
-
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/0117987e/deployment/inventory/singlenode-vagrant/group_vars/all
----------------------------------------------------------------------
diff --git a/deployment/inventory/singlenode-vagrant/group_vars/all b/deployment/inventory/singlenode-vagrant/group_vars/all
deleted file mode 100644
index 54dc168..0000000
--- a/deployment/inventory/singlenode-vagrant/group_vars/all
+++ /dev/null
@@ -1,87 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#Ambari variables
-ambari_host: "{{ groups.ambari_master[0] }}"
-hdp_host_group: "{{ groups.ambari_slave }}"
-ambari_port: 8080
-ambari_user: admin
-ambari_password: admin
-cluster_type: single_node_vm
-
-# hbase
-pcap_hbase_table: pcap
-tracker_hbase_table: access_tracker
-threatintel_hbase_table: threatintel
-enrichment_hbase_table: enrichment
-
-# metron variables
-metron_version: 0.1BETA
-metron_directory: /usr/metron/{{ metron_version }}
-java_home: /usr/jdk64/jdk1.8.0_40
-bro_version: "2.4.1"
-fixbuf_version: "1.7.1"
-yaf_version: "2.8.0"
-daq_version: "2.0.6-1"
-pycapa_repo: "https://github.com/OpenSOC/pycapa.git"
-pycapa_home: "/opt/pycapa"
-snort_version: "2.9.8.0-1"
-snort_alert_csv_path: "/var/log/snort/alert.csv"
-
-#data directories - only required to override defaults
-zookeeper_data_dir: "/data1/hadoop/zookeeper"
-namenode_checkpoint_dir: "/data1/hadoop/hdfs/namesecondary"
-namenode_name_dir: "/data1/hadoop/hdfs/namenode"
-datanode_data_dir: "/data1/hadoop/hdfs/data,/data2/hadoop/hdfs/data"
-journalnode_edits_dir: "/data1/hadoop/hdfs/journalnode"
-nodemanager_local_dirs: "/data1/hadoop/yarn/local"
-timeline_ldb_store_path: "/data1/hadoop/yarn/timeline"
-timeline_ldb_state_path: "/data1/hadoop/yarn/timeline"
-nodemanager_log_dirs: "/data1/hadoop/yarn/log"
-jhs_recovery_store_ldb_path: "/data1/hadoop/mapreduce/jhs"
-storm_local_dir: "/data1/hadoop/storm"
-kafka_log_dirs: "/data1/kafka-log"
-elasticsearch_data_dir: "/data1/elasticsearch,/data2/elasticsearch"
-
-ambari_server_mem: 512
-threat_intel_bulk_load: False
-
-# sensors
-sensor_test_mode: True
-install_pycapa: False
-install_bro: True
-install_snort: True
-install_yaf: True
-pcap_replay: True
-sniff_interface: eth1
-pcap_replay_interface: "{{ sniff_interface }}"
-storm_topologies:
- - "{{ metron_directory }}/config/topologies/bro/remote.yaml"
- - "{{ metron_directory }}/config/topologies/snort/remote.yaml"
- - "{{ metron_directory }}/config/topologies/yaf/remote.yaml"
- - "{{ metron_directory }}/config/topologies/enrichment/remote.yaml"
-pcapservice_port: 8081
-
-#Search
-install_elasticsearch: True
-install_solr: False
-solr_collection_name: Metron
-solr_number_shards: 1
-solr_replication_factor: 1
-elasticsearch_transport_port: 9300
-elasticsearch_network_interface: eth1
-elasticsearch_web_port: 9200
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/0117987e/deployment/inventory/singlenode-vagrant/hosts
----------------------------------------------------------------------
diff --git a/deployment/inventory/singlenode-vagrant/hosts b/deployment/inventory/singlenode-vagrant/hosts
deleted file mode 100644
index 6fd8b18..0000000
--- a/deployment/inventory/singlenode-vagrant/hosts
+++ /dev/null
@@ -1,48 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-[ambari_master]
-node1
-
-[ambari_slave]
-node1
-
-[hadoop_client]
-node1
-
-[enrichment]
-node1
-
-[search]
-node1
-
-[web]
-node1
-
-[sensors]
-node1
-
-[mysql]
-node1
-
-[metron:children]
-enrichment
-search
-web
-sensors
-mysql
-hadoop_client
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/0117987e/deployment/playbooks/ambari_install.yml
----------------------------------------------------------------------
diff --git a/deployment/playbooks/ambari_install.yml b/deployment/playbooks/ambari_install.yml
deleted file mode 100644
index 685753c..0000000
--- a/deployment/playbooks/ambari_install.yml
+++ /dev/null
@@ -1,55 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
-- hosts: ec2
- become: true
- tasks:
- - include_vars: ../amazon-ec2/conf/defaults.yml
- tags:
- - ec2
-
-- hosts: ambari_*
- become: true
- roles:
- - role: ambari_common
- tags:
- - ambari-prereqs
- - hdp-install
-
-- hosts: ambari_master
- become: true
- roles:
- - role: ambari_master
- tags:
- - ambari-server
- - hdp-install
-
-- hosts: ambari_slave
- become: true
- roles:
- - role: ambari_slave
- tags:
- - ambari-agent
- - hdp-install
-
-- hosts: ambari_master
- become: true
- roles:
- - role: ambari_config
- tags:
- - hdp-install
- - hdp-deploy
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/0117987e/deployment/playbooks/metron_full_install.yml
----------------------------------------------------------------------
diff --git a/deployment/playbooks/metron_full_install.yml b/deployment/playbooks/metron_full_install.yml
deleted file mode 100644
index 26ffd62..0000000
--- a/deployment/playbooks/metron_full_install.yml
+++ /dev/null
@@ -1,23 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
-- include: ambari_install.yml
- tags:
- - ambari
-- include: metron_install.yml
- tags:
- - metron
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/0117987e/deployment/playbooks/metron_install.yml
----------------------------------------------------------------------
diff --git a/deployment/playbooks/metron_install.yml b/deployment/playbooks/metron_install.yml
deleted file mode 100644
index f6bc492..0000000
--- a/deployment/playbooks/metron_install.yml
+++ /dev/null
@@ -1,96 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
-- hosts: ec2
- become: true
- tasks:
- - include_vars: ../amazon-ec2/conf/defaults.yml
- tags:
- - ec2
-
-- hosts: metron
- become: true
- roles:
- - role: metron_common
- tags:
- - metron-prereqs
-
-- hosts: hadoop_client
- become: true
- roles:
- - role: hadoop_setup
- tags:
- - metron-prereqs
-
-- hosts: search
- become: true
- vars:
- es_hosts: "{% set comma = joiner(',') %}{% for host in groups['search'] -%}{{ comma() }}{{ host }}{%- endfor %}"
- roles:
- - { role: elasticsearch, when: install_elasticsearch | default(True) == True }
- tags:
- - search
-
-- hosts: search
- become: true
- roles:
- - { role: solr, when: install_solr | default(False) == True }
- tags:
- - search
-
-- hosts: mysql
- become: true
- roles:
- - role: mysql_server
- tags:
- - mysql-server
-
-- hosts: ambari_slave
- become: true
- roles:
- - role: mysql_client
- tags:
- - mysql-client
-
-- hosts: sensors
- become: true
- roles:
- - { role: tap_interface, when: install_tap | default(False) == True }
- - { role: pycapa, when: install_pycapa | default(True) == True }
- - { role: bro, when: install_bro | default(True) == True }
- - { role: flume, when: install_snort | default(True) == True }
- - { role: snort , when: install_snort | default(True) == True }
- - { role: yaf, when: install_yaf | default(True) == True }
- - { role: pcap_replay , when: (pcap_replay | default(False)) or (sensor_test_mode | default(False)) == True }
- - { role: sensor-test-mode, when: sensor_test_mode | default(False) == True }
- tags:
- - sensors
-
-- hosts: enrichment
- become: true
- roles:
- - role: metron_streaming
- tags:
- - enrichment
-
-- hosts: web
- become: true
- roles:
- - { role: metron_ui, when: install_elasticsearch | default(True) == True }
- - { role: metron_pcapservice, when: install_elasticsearch | default(True) == True }
- tags:
- - web
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/0117987e/deployment/roles/ambari_common/defaults/main.yml
----------------------------------------------------------------------
diff --git a/deployment/roles/ambari_common/defaults/main.yml b/deployment/roles/ambari_common/defaults/main.yml
deleted file mode 100644
index 65c83d9..0000000
--- a/deployment/roles/ambari_common/defaults/main.yml
+++ /dev/null
@@ -1,19 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
-hadoop_logrotate_frequency: daily
-hadoop_logrotate_retention: 30
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/0117987e/deployment/roles/ambari_common/meta/main.yml
----------------------------------------------------------------------
diff --git a/deployment/roles/ambari_common/meta/main.yml b/deployment/roles/ambari_common/meta/main.yml
deleted file mode 100644
index 8992ac1..0000000
--- a/deployment/roles/ambari_common/meta/main.yml
+++ /dev/null
@@ -1,22 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
-dependencies:
- - libselinux-python
- - yum-update
- - epel
- - ntp
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/0117987e/deployment/roles/ambari_common/tasks/main.yml
----------------------------------------------------------------------
diff --git a/deployment/roles/ambari_common/tasks/main.yml b/deployment/roles/ambari_common/tasks/main.yml
deleted file mode 100644
index 2526923..0000000
--- a/deployment/roles/ambari_common/tasks/main.yml
+++ /dev/null
@@ -1,52 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
-- name: Check OS Version
- fail: msg="Ambari HDP deployment supports CentOS 6 only."
- when: (ansible_distribution != "CentOS" or ansible_distribution_major_version != "6")
-
-- include: passwd_less_ssh.yml
-
-- name: Ensure iptables is stopped and is not running at boot time.
- ignore_errors: yes
- service: name=iptables state=stopped enabled=no
-
-#
-# ambari uses socket.getfqdn() to find the hostname. with 'localhost.localdomain'
-# in '/etc/hosts' this function will report the hostname as 'localhost.localdomain'
-# rather than 'node1' as would be expected. other functions like socket.gethostname()
-# will always return 'node1' as expected. ambari needs to see 'node1' to be able to
-# communicate between the master and agents.
-
-- name: Remove ipv4 'localhost.localdomain' from /etc/hosts
- lineinfile: dest=/etc/hosts state=absent regexp="^127.0.0.1(.*)localdomain(.*)$"
-
-- name: Remove ipv6 'localhost.localdomain' from /etc/hosts
- lineinfile: dest=/etc/hosts state=absent regexp="^::1(.*)localdomain(.*)$"
-
-- name: Add localhost to /etc/hosts
- lineinfile: dest=/etc/hosts line="127.0.0.1 localhost"
-
-- name: Download Ambari repo
- get_url: url="{{ rhel_ambari_install_url }}" dest=/etc/yum.repos.d/ambari.repo
-
-- name: Create Logrotate Script for Hadoop Services
- template:
- src: "metron-hadoop-logrotate.yml"
- dest: "/etc/logrotate.d/metron-ambari"
- mode: 0644
-
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/0117987e/deployment/roles/ambari_common/tasks/passwd_less_ssh.yml
----------------------------------------------------------------------
diff --git a/deployment/roles/ambari_common/tasks/passwd_less_ssh.yml b/deployment/roles/ambari_common/tasks/passwd_less_ssh.yml
deleted file mode 100644
index 0928e34..0000000
--- a/deployment/roles/ambari_common/tasks/passwd_less_ssh.yml
+++ /dev/null
@@ -1,32 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
----
-- name: Generate ssh key pair for "{{ ambari_user }}"
- user: name={{ ambari_user }} generate_ssh_key=yes
-
-- name: Fetch the generated public key
- fetch: src=~{{ ambari_user }}/.ssh/id_rsa.pub dest=/tmp/keys/{{ inventory_hostname }}.pub flat=yes
-
-- name: Add key pairs to existing authorized_keys
- authorized_key: user={{ ambari_user }} key="{{ lookup('file', '/tmp/keys/{{ item }}.pub') }}"
- with_items:
- - "{{ play_hosts }}"
-
-- name: Remove local copy of ssh keys
- local_action: file path=/tmp/keys/{{ inventory_hostname }}.pub state=absent
- become: False
-