You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2016/08/31 17:01:21 UTC
[jira] [Commented] (METRON-400) Deploy Probes to running Docker
Container
[ https://issues.apache.org/jira/browse/METRON-400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15452755#comment-15452755 ]
ASF GitHub Bot commented on METRON-400:
---------------------------------------
GitHub user dlyle65535 opened a pull request:
https://github.com/apache/incubator-metron/pull/238
METRON-400 Deploy Probes to running Docker Container
This was tested in the following ways:
## Regression Testing ##
- [X] Full Dev - Worked as expected
- [X] EC2 - Worked as expected
## Functional Testing ##
- [X] Run on docker HDP instance from [docker-ambari](https://github.com/sequenceiq/docker-ambari) Note: used custom (dlyle65535/ambari-agent:2.4.0.0-1130-jdk8 and dlyle65535/ambari-server:2.4.0.0-1130-jdk8)
## Steps to Stand-Alone Test ##
**Note:**
You'll need docker.py installed and Ansible > 2, we still recommend 2.0.0.2.
Ansible will error without docker.py installed and give you a chance to:
```
pip install 'docker-py>=1.7.0'
```
### Provision Container ###
Start the container:
```
docker run -d --hostname amb-server --privileged --name amb-server -it dlyle65535/ambari-server:2.4.0.0-1130-jdk8
```
Run a shell on the container:
```
docker exec -it amb-server bash
```
Change nameserver to 8.8.8.8 (change to nameserver 8.8.8.8, delete 2nd line)
```
vi /etc/resolv.conf
```
Get HDP repo:
```
wget -nv http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.4.2.0/hdp.repo -O /etc/yum.repos.d/hdp.repo
```
Install Zookeeper:
```
yum install -y zookeeper-server
```
Setup and Start Zookeeper:
```
export ZOOKEEPER_CONF_DIR=/etc/zookeeper/conf
export ZOOKEEPER_HOME=/usr/hdp/current/zookeeper-server
export ZOO_LOG_DIR=/var/log/zookeeper
export ZOOPIDFILE=/var/run/zookeeper/zookeeper_server.pid
export SERVER_JVMFLAGS=-Xmx1024m
export JAVA=$JAVA_HOME/bin/java
export CLASSPATH=$CLASSPATH:$ZOOKEEPER_HOME/*
export ZOOCFGDIR=$ZOOKEEPER_CONF_DIR
export ZOOCFG=zoo.cfg
source $ZOOKEEPER_CONF_DIR/zookeeper-env.sh
/usr/hdp/current/zookeeper-server/bin/zkServer.sh start
```
Test Zookeeper:
```
/usr/hdp/current/zookeeper-server/bin/zkCli.sh -server localhost:2181 ls /
```
The command should return:
> Connecting to localhost:2181
>
> WATCHER::
>
> WatchedEvent state:SyncConnected type:None path:null
> [zookeeper]
>
Install and Start Kafka:
```
yum install -y kafka
/usr/hdp/current/kafka-broker/bin/kafka start
```
Test Kafka:
```
/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic test --replication-factor 1 --partitions 1
/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
```
Add some test data and hit ctrl-c.
```
/usr/hdp/current/kafka-broker//bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
```
You should see your test data - ctrl-c to exit.
Create Probe Data Topics:
```
/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic bro --replication-factor 1 --partitions 1
/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic snort --replication-factor 1 --partitions 1
/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic yaf --replication-factor 1 --partitions 1
```
Exit Container Shell:
```
exit
```
Install Probes using Ansible:
1. cd <metron-home>/metron-deployment/playbooks
2. In playbooks/docker_probe_install.yml change kafka_broker_url to amb-server:9092 (kafka_broker_url: amb-server:9092)
3. export DOCKER_VERSION=<docker -version> e.g. 1.12.1 (omit any trailing rc stuff)
ansible-playbook docker_probe_install.yml
4. run ansible-playbook
```
ansible-playbook docker_probe_install.yml
```
It should complete with:
> PLAY RECAP *********************************************************************
> amb-server : ok=100 changed=60 unreachable=0 failed=0
> localhost : ok=2 changed=0 unreachable=0 failed=0
> docker exec -it amb-server bash
Start Sensor Probes:
```
service pcap-replay start
/usr/local/bro/bin/broctl start
service yaf start
service snortd start
/usr/hdp/current/flume-server/bin/flume-ng agent -f /etc/flume/conf/flume-snort.conf -n snort > /dev/null 2>&1 &
```
Check for Data
```
/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic bro
/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic snort
/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic yaf
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dlyle65535/incubator-metron docker-probes-master-merge
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/238.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #238
----
commit f1f89271a426358c1ae45b079b1bfcdc48742db3
Author: David Lyle <dl...@gmail.com>
Date: 2016-08-31T16:29:53Z
METRON-400 Deploy Probes to running Docker Container
----
> Deploy Probes to running Docker Container
> -----------------------------------------
>
> Key: METRON-400
> URL: https://issues.apache.org/jira/browse/METRON-400
> Project: Metron
> Issue Type: New Feature
> Reporter: David M. Lyle
> Assignee: David M. Lyle
>
> Sometimes we test using kafka containers. Create playbook to deploy to docker container.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)