You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by dlyle65535 <gi...@git.apache.org> on 2016/08/31 17:00:31 UTC
[GitHub] incubator-metron pull request #238: METRON-400 Deploy Probes to running Dock...
GitHub user dlyle65535 opened a pull request:
https://github.com/apache/incubator-metron/pull/238
METRON-400 Deploy Probes to running Docker Container
This was tested in the following ways:
## Regression Testing ##
- [X] Full Dev - Worked as expected
- [X] EC2 - Worked as expected
## Functional Testing ##
- [X] Run on docker HDP instance from [docker-ambari](https://github.com/sequenceiq/docker-ambari) Note: used custom (dlyle65535/ambari-agent:2.4.0.0-1130-jdk8 and dlyle65535/ambari-server:2.4.0.0-1130-jdk8)
## Steps to Stand-Alone Test ##
**Note:**
You'll need docker.py installed and Ansible > 2, we still recommend 2.0.0.2.
Ansible will error without docker.py installed and give you a chance to:
```
pip install 'docker-py>=1.7.0'
```
### Provision Container ###
Start the container:
```
docker run -d --hostname amb-server --privileged --name amb-server -it dlyle65535/ambari-server:2.4.0.0-1130-jdk8
```
Run a shell on the container:
```
docker exec -it amb-server bash
```
Change nameserver to 8.8.8.8 (change to nameserver 8.8.8.8, delete 2nd line)
```
vi /etc/resolv.conf
```
Get HDP repo:
```
wget -nv http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.4.2.0/hdp.repo -O /etc/yum.repos.d/hdp.repo
```
Install Zookeeper:
```
yum install -y zookeeper-server
```
Setup and Start Zookeeper:
```
export ZOOKEEPER_CONF_DIR=/etc/zookeeper/conf
export ZOOKEEPER_HOME=/usr/hdp/current/zookeeper-server
export ZOO_LOG_DIR=/var/log/zookeeper
export ZOOPIDFILE=/var/run/zookeeper/zookeeper_server.pid
export SERVER_JVMFLAGS=-Xmx1024m
export JAVA=$JAVA_HOME/bin/java
export CLASSPATH=$CLASSPATH:$ZOOKEEPER_HOME/*
export ZOOCFGDIR=$ZOOKEEPER_CONF_DIR
export ZOOCFG=zoo.cfg
source $ZOOKEEPER_CONF_DIR/zookeeper-env.sh
/usr/hdp/current/zookeeper-server/bin/zkServer.sh start
```
Test Zookeeper:
```
/usr/hdp/current/zookeeper-server/bin/zkCli.sh -server localhost:2181 ls /
```
The command should return:
> Connecting to localhost:2181
>
> WATCHER::
>
> WatchedEvent state:SyncConnected type:None path:null
> [zookeeper]
>
Install and Start Kafka:
```
yum install -y kafka
/usr/hdp/current/kafka-broker/bin/kafka start
```
Test Kafka:
```
/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic test --replication-factor 1 --partitions 1
/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
```
Add some test data and hit ctrl-c.
```
/usr/hdp/current/kafka-broker//bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
```
You should see your test data - ctrl-c to exit.
Create Probe Data Topics:
```
/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic bro --replication-factor 1 --partitions 1
/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic snort --replication-factor 1 --partitions 1
/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic yaf --replication-factor 1 --partitions 1
```
Exit Container Shell:
```
exit
```
Install Probes using Ansible:
1. cd <metron-home>/metron-deployment/playbooks
2. In playbooks/docker_probe_install.yml change kafka_broker_url to amb-server:9092 (kafka_broker_url: amb-server:9092)
3. export DOCKER_VERSION=<docker -version> e.g. 1.12.1 (omit any trailing rc stuff)
ansible-playbook docker_probe_install.yml
4. run ansible-playbook
```
ansible-playbook docker_probe_install.yml
```
It should complete with:
> PLAY RECAP *********************************************************************
> amb-server : ok=100 changed=60 unreachable=0 failed=0
> localhost : ok=2 changed=0 unreachable=0 failed=0
> docker exec -it amb-server bash
Start Sensor Probes:
```
service pcap-replay start
/usr/local/bro/bin/broctl start
service yaf start
service snortd start
/usr/hdp/current/flume-server/bin/flume-ng agent -f /etc/flume/conf/flume-snort.conf -n snort > /dev/null 2>&1 &
```
Check for Data
```
/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic bro
/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic snort
/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic yaf
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dlyle65535/incubator-metron docker-probes-master-merge
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/238.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #238
----
commit f1f89271a426358c1ae45b079b1bfcdc48742db3
Author: David Lyle <dl...@gmail.com>
Date: 2016-08-31T16:29:53Z
METRON-400 Deploy Probes to running Docker Container
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #238: METRON-400 Deploy Probes to running Dock...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/238
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #238: METRON-400 Deploy Probes to running Docker Cont...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on the issue:
https://github.com/apache/incubator-metron/pull/238
Apologies- I mistakenly merged this in thinking it had been reviewed. Could use a hand pulling it back out.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #238: METRON-400 Deploy Probes to running Docker Cont...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on the issue:
https://github.com/apache/incubator-metron/pull/238
Nevermind, I sorted it. Sorry about that. I'll open a new PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---