You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by dlyle65535 <gi...@git.apache.org> on 2016/09/05 00:38:36 UTC
[GitHub] incubator-metron pull request #241: METRON-400 Deploy Probes to running Dock...
GitHub user dlyle65535 opened a pull request:
https://github.com/apache/incubator-metron/pull/241
METRON-400 Deploy Probes to running Docker Container
This was tested in the following ways:
Regression Testing ##
[X] Full Dev - Worked as expected
[X] EC2 - Worked as expected
Functional Testing ##
[X] Run on docker HDP instance from [docker-ambari](https://github.com/sequenceiq/docker-ambari) Note: used custom (dlyle65535/ambari-agent:2.4.0.0-1130-jdk8 and dlyle65535/ambari-server:2.4.0.0-1130-jdk8)
Steps to Stand-Alone Test ##
*Note:*
You'll need docker.py installed and Ansible > 2, we still recommend 2.0.0.2.
Ansible will error without docker.py installed and give you a chance to:
```
pip install 'docker-py>=1.7.0'
```
Provision Container ###
Start the container:
```
docker run -d --hostname amb-server --privileged --name amb-server -it dlyle65535/ambari-server:2.4.0.0-1130-jdk8
```
Run a shell on the container:
```
docker exec -it amb-server bash
```
Change nameserver to 8.8.8.8 (change to nameserver 8.8.8.8, delete 2nd line)
```
vi /etc/resolv.conf
```
Get HDP repo:
```
wget -nv http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.4.2.0/hdp.repo -O /etc/yum.repos.d/hdp.repo
```
Install Zookeeper:
```
yum install -y zookeeper-server
```
Setup and Start Zookeeper:
```
export ZOOKEEPER_CONF_DIR=/etc/zookeeper/conf
export ZOOKEEPER_HOME=/usr/hdp/current/zookeeper-server
export ZOO_LOG_DIR=/var/log/zookeeper
export ZOOPIDFILE=/var/run/zookeeper/zookeeper_server.pid
export SERVER_JVMFLAGS=-Xmx1024m
export JAVA=$JAVA_HOME/bin/java
export CLASSPATH=$CLASSPATH:$ZOOKEEPER_HOME/*
export ZOOCFGDIR=$ZOOKEEPER_CONF_DIR
export ZOOCFG=zoo.cfg
source $ZOOKEEPER_CONF_DIR/zookeeper-env.sh
/usr/hdp/current/zookeeper-server/bin/zkServer.sh start
```
Test Zookeeper:
```
/usr/hdp/current/zookeeper-server/bin/zkCli.sh -server localhost:2181 ls /
```
The command should return:
> Connecting to localhost:2181
>
> WATCHER::
>
> WatchedEvent state:SyncConnected type:None path:null
> [zookeeper]
>
Install and Start Kafka:
```
yum install -y kafka
/usr/hdp/current/kafka-broker/bin/kafka start
```
Test Kafka:
```
/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic test --replication-factor 1 --partitions 1
/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
```
Add some test data and hit ctrl-c.
```
/usr/hdp/current/kafka-broker//bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
```
You should see your test data - ctrl-c to exit.
Create Probe Data Topics:
```
/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic bro --replication-factor 1 --partitions 1
/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic snort --replication-factor 1 --partitions 1
/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic yaf --replication-factor 1 --partitions 1
```
Exit Container Shell:
```
exit
```
Install Probes using Ansible:
1. cd <metron-home>/metron-deployment/playbooks
2. In playbooks/docker_probe_install.yml change kafka_broker_url to amb-server:9092 (kafka_broker_url: amb-server:9092)
3. export DOCKER_VERSION=<docker -version> e.g. 1.12.1 (omit any trailing rc stuff)
ansible-playbook docker_probe_install.yml
4. run ansible-playbook
```
ansible-playbook docker_probe_install.yml
```
It should complete with:
> PLAY RECAP *********************************************************************
> amb-server : ok=100 changed=60 unreachable=0 failed=0
> localhost : ok=2 changed=0 unreachable=0 failed=0
> docker exec -it amb-server bash
Start Sensor Probes:
```
service pcap-replay start
/usr/local/bro/bin/broctl start
service yaf start eth0
service snortd start
/usr/hdp/current/flume-server/bin/flume-ng agent -f /etc/flume/conf/flume-snort.conf -n snort > /dev/null 2>&1 &
```
Check for Data
```
/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic bro
/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic snort
/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic yaf
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dlyle65535/incubator-metron docker-probes-master-merge
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/241.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #241
----
commit f1f89271a426358c1ae45b079b1bfcdc48742db3
Author: David Lyle <dl...@gmail.com>
Date: 2016-08-31T16:29:53Z
METRON-400 Deploy Probes to running Docker Container
commit 26d0561748da28d3a6e562334816dd6ec04d4e46
Author: David Lyle <dl...@gmail.com>
Date: 2016-09-04T17:41:03Z
Only use amb4 for Kafka broker
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #241: METRON-400 Deploy Probes to running Dock...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/241
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #241: METRON-400 Deploy Probes to running Docker Cont...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/241
There are 5 or 6 individual tests that are part of the profiler integration tests. The entire spin-up/spin-down of Kafka/Zookeeper occurs between each test. I believe the other integration tests each only have a single test, so the infrastructure is only spun-up/down once. That is at least one difference.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #241: METRON-400 Deploy Probes to running Docker Cont...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/241
@nickwallen The failing test in the profiler appears to be intermittent. It hit me once earlier and I couldn't recreate it, would you please take a look?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #241: METRON-400 Deploy Probes to running Docker Cont...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/241
I will admit to being quite confused as to why the EnrichmentIntegrationTest is not intermittently failing or any of the ParserIntegrationTests, considering they should have this same issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #241: METRON-400 Deploy Probes to running Docker Cont...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/241
So, one solution would be to try to read the configurations back out of zookeeper in the ConfigUploadComponent.start() in a loop until you can read them back (with a sleep between the attempt).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #241: METRON-400 Deploy Probes to running Docker Cont...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/241
@cestella Did you ever see it fail in any branch without the changes from #233 that was recently merged into master?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #241: METRON-400 Deploy Probes to running Docker Cont...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/241
+1 by inspection
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #241: METRON-400 Deploy Probes to running Docker Cont...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/241
Ah, I thought this just started to fail once the latest from master was merged.
It seems that in some cases the configs are not loaded in Zookeeper by the time messages start flowing through Storm in the integration test. The profiler then doesn't know what to do and freaks out. Hmm.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #241: METRON-400 Deploy Probes to running Dock...
Posted by dlyle65535 <gi...@git.apache.org>.
GitHub user dlyle65535 reopened a pull request:
https://github.com/apache/incubator-metron/pull/241
METRON-400 Deploy Probes to running Docker Container
This was tested in the following ways:
Regression Testing ##
[X] Full Dev - Worked as expected
[X] EC2 - Worked as expected
Functional Testing ##
[X] Run on docker HDP instance from [docker-ambari](https://github.com/sequenceiq/docker-ambari) Note: used custom (dlyle65535/ambari-agent:2.4.0.0-1130-jdk8 and dlyle65535/ambari-server:2.4.0.0-1130-jdk8)
Steps to Stand-Alone Test ##
*Note:*
You'll need docker.py installed and Ansible > 2, we still recommend 2.0.0.2.
Ansible will error without docker.py installed and give you a chance to:
```
pip install 'docker-py>=1.7.0'
```
Provision Container ###
Start the container:
```
docker run -d --hostname amb-server --privileged --name amb-server -it dlyle65535/ambari-server:2.4.0.0-1130-jdk8
```
Run a shell on the container:
```
docker exec -it amb-server bash
```
Change nameserver to 8.8.8.8 (change to nameserver 8.8.8.8, delete 2nd line)
```
vi /etc/resolv.conf
```
Get HDP repo:
```
wget -nv http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.4.2.0/hdp.repo -O /etc/yum.repos.d/hdp.repo
```
Install Zookeeper:
```
yum install -y zookeeper-server
```
Setup and Start Zookeeper:
```
export ZOOKEEPER_CONF_DIR=/etc/zookeeper/conf
export ZOOKEEPER_HOME=/usr/hdp/current/zookeeper-server
export ZOO_LOG_DIR=/var/log/zookeeper
export ZOOPIDFILE=/var/run/zookeeper/zookeeper_server.pid
export SERVER_JVMFLAGS=-Xmx1024m
export JAVA=$JAVA_HOME/bin/java
export CLASSPATH=$CLASSPATH:$ZOOKEEPER_HOME/*
export ZOOCFGDIR=$ZOOKEEPER_CONF_DIR
export ZOOCFG=zoo.cfg
source $ZOOKEEPER_CONF_DIR/zookeeper-env.sh
/usr/hdp/current/zookeeper-server/bin/zkServer.sh start
```
Test Zookeeper:
```
/usr/hdp/current/zookeeper-server/bin/zkCli.sh -server localhost:2181 ls /
```
The command should return:
> Connecting to localhost:2181
>
> WATCHER::
>
> WatchedEvent state:SyncConnected type:None path:null
> [zookeeper]
>
Install and Start Kafka:
```
yum install -y kafka
/usr/hdp/current/kafka-broker/bin/kafka start
```
Test Kafka:
```
/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic test --replication-factor 1 --partitions 1
/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
```
Add some test data and hit ctrl-c.
```
/usr/hdp/current/kafka-broker//bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
```
You should see your test data - ctrl-c to exit.
Create Probe Data Topics:
```
/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic bro --replication-factor 1 --partitions 1
/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic snort --replication-factor 1 --partitions 1
/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic yaf --replication-factor 1 --partitions 1
```
Exit Container Shell:
```
exit
```
Install Probes using Ansible:
1. cd <metron-home>/metron-deployment/playbooks
2. In playbooks/docker_probe_install.yml change kafka_broker_url to amb-server:9092 (kafka_broker_url: amb-server:9092)
3. export DOCKER_VERSION=<docker -version> e.g. 1.12.1 (omit any trailing rc stuff)
ansible-playbook docker_probe_install.yml
4. run ansible-playbook
```
ansible-playbook docker_probe_install.yml
```
It should complete with:
> PLAY RECAP *********************************************************************
> amb-server : ok=100 changed=60 unreachable=0 failed=0
> localhost : ok=2 changed=0 unreachable=0 failed=0
> docker exec -it amb-server bash
Start Sensor Probes:
```
service pcap-replay start
/usr/local/bro/bin/broctl start
service yaf start eth0
service snortd start
/usr/hdp/current/flume-server/bin/flume-ng agent -f /etc/flume/conf/flume-snort.conf -n snort > /dev/null 2>&1 &
```
Check for Data
```
/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic bro
/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic snort
/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic yaf
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dlyle65535/incubator-metron docker-probes-master-merge
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/241.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #241
----
commit f1f89271a426358c1ae45b079b1bfcdc48742db3
Author: David Lyle <dl...@gmail.com>
Date: 2016-08-31T16:29:53Z
METRON-400 Deploy Probes to running Docker Container
commit 26d0561748da28d3a6e562334816dd6ec04d4e46
Author: David Lyle <dl...@gmail.com>
Date: 2016-09-04T17:41:03Z
Only use amb4 for Kafka broker
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #241: METRON-400 Deploy Probes to running Dock...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 closed the pull request at:
https://github.com/apache/incubator-metron/pull/241
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #241: METRON-400 Deploy Probes to running Docker Cont...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/241
Well, this branch does not have #233 merged in and it's failing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---