You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by nickwallen <gi...@git.apache.org> on 2016/05/24 02:59:59 UTC
[GitHub] incubator-metron pull request: METRON-180 Enable each component to...
GitHub user nickwallen opened a pull request:
https://github.com/apache/incubator-metron/pull/133
METRON-180 Enable each component to be installed or not using tags
To allow users to make the most use of the deployment scripts, they should be able to choose whether each component is installed or not. This is important very highly customized environments where Metron will be installed. This is currently possible in most cases, but there are a few scenarios that are exceptions.
For example, the PCAP service will only be installed if `install_elasticsearch` is set to `true`. This was a good coupling back when PCAP searches were hosted in the index. Now that PCAP searches come from HDFS, we need to be able to install PCAP service separately.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/incubator-metron METRON-180
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/133.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #133
----
commit ec1e99b755fc922cae26256dcd0c22f36c3828c8
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-05-24T02:49:37Z
METRON-180 Enable each component to be installed or not using tags
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-180 Enable each component to be instal...
Posted by james-sirota <gi...@git.apache.org>.
Github user james-sirota commented on the pull request:
https://github.com/apache/incubator-metron/pull/133
@nickwallen what set of commands should i run to validate this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #133: METRON-180 Granular Control of Component...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/133#discussion_r66055730
--- Diff: metron-deployment/playbooks/metron_install.yml ---
@@ -74,47 +83,115 @@
tags:
- mysql-client
-- hosts: sensors
- become: true
- roles:
- - ambari_gather_facts
- - { role: tap_interface, when: install_tap | default(False) == True }
- - { role: pycapa, when: install_pycapa | default(True) == True }
- - { role: bro, when: install_bro | default(True) == True }
- - { role: flume, when: install_snort | default(True) == True }
- - { role: snort , when: install_snort | default(True) == True }
- - { role: yaf, when: install_yaf | default(True) == True }
- - { role: pcap_replay, when: install_pcap_replay | default(False) == True }
- - { role: sensor-test-mode, when: sensor_test_mode | default(False) == True }
- tags:
- - sensors
-
+#
+# parsers, enrichment, and indexing topologies
+#
- hosts: enrichment
become: true
roles:
- role: metron_streaming
tags:
- enrichment
+#
+# user interface
+#
- hosts: pcap_server
become: true
roles:
- - { role: metron_pcapservice, when: install_elasticsearch | default(True) == True }
+ - role: metron_pcapservice
tags:
- - pcap_service
+ - pcap-service
- hosts: web
become: true
roles:
- - { role: kibana, when: install_elasticsearch | default(True) == True }
+ - role: kibana
tags:
- - web
+ - kibana
+#
+# sensors
+#
+- hosts: sensors
+ become: true
+ roles:
+ - role: ambari_gather_facts
+ - role: tap_interface
+ tags:
+ - tap
+
+- hosts: sensors
+ become: true
+ roles:
+ - role: ambari_gather_facts
+ - role: pycapa
+ tags:
+ - pycapa
+
+- hosts: sensors
+ become: true
+ roles:
+ - role: ambari_gather_facts
+ - role: bro
+ tags:
+ - bro
+
+- hosts: sensors
+ become: true
+ roles:
+ - role: ambari_gather_facts
+ - role: flume
+ tags:
+ - flume
+
+- hosts: sensors
+ become: true
+ roles:
+ - role: ambari_gather_facts
+ - role: snort
+ tags:
+ - snort
+
+- hosts: sensors
+ become: true
+ roles:
+ - role: ambari_gather_facts
+ - role: yaf
+ tags:
+ - yaf
+
+- hosts: sensors
+ become: true
+ roles:
+ - role: ambari_gather_facts
+ - role: pcap_replay
+ tags:
+ - pcap-replay
+
+- hosts: sensors
+ become: true
+ roles:
+ - role: ambari_gather_facts
+ - role: sensor-test-mode
+ tags:
+ - sensor-test-mode
+
--- End diff --
Yes, that is nice. I like it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-180 Enable each component to...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the pull request:
https://github.com/apache/incubator-metron/pull/133#issuecomment-221577078
Yeah, this makes sense. Do we need to adjust the `run_ansible_role.sh` to pass in the `--skip-tags` arg as well?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-180 Enable each component to...
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the pull request:
https://github.com/apache/incubator-metron/pull/133#issuecomment-221616780
Looks to me like ec2.py never made its way into source control. These queries return no results:
`git log --all --diff-filter=D --summary | grep delete|grep ec2`
`git log --all -- "**ec2.py"`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-180 Enable each component to...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the pull request:
https://github.com/apache/incubator-metron/pull/133#issuecomment-222159692
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-180 Enable each component to...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the pull request:
https://github.com/apache/incubator-metron/pull/133#issuecomment-221640736
I couldn't resist. :)
TLDR; The `ec2.py` that you are seeing refers to a file distributed with Ansible. It will not exist in our repo.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-180 Enable each component to...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the pull request:
https://github.com/apache/incubator-metron/pull/133#issuecomment-221611276
Also, is there an ec2.py somewhere that didn't get checked in?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-180 Enable each component to...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the pull request:
https://github.com/apache/incubator-metron/pull/133#issuecomment-221676182
I do not think we need any changes to `run_ansible_role.sh` because we are explicitly defining which tags to run in that script.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #133: METRON-180 Granular Control of Component Instal...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on the issue:
https://github.com/apache/incubator-metron/pull/133
\U0001f44d looks good, ship it! :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #133: METRON-180 Granular Control of Component...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/133#discussion_r66062033
--- Diff: metron-deployment/playbooks/metron_install.yml ---
@@ -74,47 +83,115 @@
tags:
- mysql-client
-- hosts: sensors
- become: true
- roles:
- - ambari_gather_facts
- - { role: tap_interface, when: install_tap | default(False) == True }
- - { role: pycapa, when: install_pycapa | default(True) == True }
- - { role: bro, when: install_bro | default(True) == True }
- - { role: flume, when: install_snort | default(True) == True }
- - { role: snort , when: install_snort | default(True) == True }
- - { role: yaf, when: install_yaf | default(True) == True }
- - { role: pcap_replay, when: install_pcap_replay | default(False) == True }
- - { role: sensor-test-mode, when: sensor_test_mode | default(False) == True }
- tags:
- - sensors
-
+#
+# parsers, enrichment, and indexing topologies
+#
- hosts: enrichment
become: true
roles:
- role: metron_streaming
tags:
- enrichment
+#
+# user interface
+#
- hosts: pcap_server
become: true
roles:
- - { role: metron_pcapservice, when: install_elasticsearch | default(True) == True }
+ - role: metron_pcapservice
tags:
- - pcap_service
+ - pcap-service
- hosts: web
become: true
roles:
- - { role: kibana, when: install_elasticsearch | default(True) == True }
+ - role: kibana
tags:
- - web
+ - kibana
+#
+# sensors
+#
+- hosts: sensors
+ become: true
+ roles:
+ - role: ambari_gather_facts
+ - role: tap_interface
+ tags:
+ - tap
+
+- hosts: sensors
+ become: true
+ roles:
+ - role: ambari_gather_facts
+ - role: pycapa
+ tags:
+ - pycapa
+
+- hosts: sensors
+ become: true
+ roles:
+ - role: ambari_gather_facts
+ - role: bro
+ tags:
+ - bro
+
+- hosts: sensors
+ become: true
+ roles:
+ - role: ambari_gather_facts
+ - role: flume
+ tags:
+ - flume
+
+- hosts: sensors
+ become: true
+ roles:
+ - role: ambari_gather_facts
+ - role: snort
+ tags:
+ - snort
+
+- hosts: sensors
+ become: true
+ roles:
+ - role: ambari_gather_facts
+ - role: yaf
+ tags:
+ - yaf
+
+- hosts: sensors
+ become: true
+ roles:
+ - role: ambari_gather_facts
+ - role: pcap_replay
+ tags:
+ - pcap-replay
+
+- hosts: sensors
+ become: true
+ roles:
+ - role: ambari_gather_facts
+ - role: sensor-test-mode
+ tags:
+ - sensor-test-mode
+
--- End diff --
Those changes tested successfully. Added those in. Good suggestion. Much better looking.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-180 Enable each component to...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the pull request:
https://github.com/apache/incubator-metron/pull/133#issuecomment-221640103
https://community.hortonworks.com/comments/30932/view.html
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-180 Enable each component to...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the pull request:
https://github.com/apache/incubator-metron/pull/133#issuecomment-221925003
@cestella @mmiklavc Did I answer your question? Any other ways I can make this better?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #133: METRON-180 Granular Control of Component...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/133
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #133: METRON-180 Granular Control of Component...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/133#discussion_r65987913
--- Diff: metron-deployment/playbooks/metron_install.yml ---
@@ -74,47 +83,115 @@
tags:
- mysql-client
-- hosts: sensors
- become: true
- roles:
- - ambari_gather_facts
- - { role: tap_interface, when: install_tap | default(False) == True }
- - { role: pycapa, when: install_pycapa | default(True) == True }
- - { role: bro, when: install_bro | default(True) == True }
- - { role: flume, when: install_snort | default(True) == True }
- - { role: snort , when: install_snort | default(True) == True }
- - { role: yaf, when: install_yaf | default(True) == True }
- - { role: pcap_replay, when: install_pcap_replay | default(False) == True }
- - { role: sensor-test-mode, when: sensor_test_mode | default(False) == True }
- tags:
- - sensors
-
+#
+# parsers, enrichment, and indexing topologies
+#
- hosts: enrichment
become: true
roles:
- role: metron_streaming
tags:
- enrichment
+#
+# user interface
+#
- hosts: pcap_server
become: true
roles:
- - { role: metron_pcapservice, when: install_elasticsearch | default(True) == True }
+ - role: metron_pcapservice
tags:
- - pcap_service
+ - pcap-service
- hosts: web
become: true
roles:
- - { role: kibana, when: install_elasticsearch | default(True) == True }
+ - role: kibana
tags:
- - web
+ - kibana
+#
+# sensors
+#
+- hosts: sensors
+ become: true
+ roles:
+ - role: ambari_gather_facts
+ - role: tap_interface
+ tags:
+ - tap
+
+- hosts: sensors
+ become: true
+ roles:
+ - role: ambari_gather_facts
+ - role: pycapa
+ tags:
+ - pycapa
+
+- hosts: sensors
+ become: true
+ roles:
+ - role: ambari_gather_facts
+ - role: bro
+ tags:
+ - bro
+
+- hosts: sensors
+ become: true
+ roles:
+ - role: ambari_gather_facts
+ - role: flume
+ tags:
+ - flume
+
+- hosts: sensors
+ become: true
+ roles:
+ - role: ambari_gather_facts
+ - role: snort
+ tags:
+ - snort
+
+- hosts: sensors
+ become: true
+ roles:
+ - role: ambari_gather_facts
+ - role: yaf
+ tags:
+ - yaf
+
+- hosts: sensors
+ become: true
+ roles:
+ - role: ambari_gather_facts
+ - role: pcap_replay
+ tags:
+ - pcap-replay
+
+- hosts: sensors
+ become: true
+ roles:
+ - role: ambari_gather_facts
+ - role: sensor-test-mode
+ tags:
+ - sensor-test-mode
+
--- End diff --
I think I remembered my comment from last week- you may be able to combine these into a single hosts/roles group by using the following construct:
```yaml
hosts: sensors
become: true
tags: sensors
roles:
-{ role: ambari_gather_facts, tags: [ 'always'] }
- { role: tap_interface, tags: ['tap'] }
- { role: pycapa, tags: ['pycapa'] }
- { role: bro, tags: ['bro'] }
- { role: flume, tags: ['snort','flume'] }
- { role: snort , tags: ['snort'] }
- { role: yaf, tags: ['yaf'] }
- { role: pcap_replay, tags: ['pcap-replay'] }
- { role: sensor-test-mode, tags: ['sensor-test-mode'] }
```
This is per the docs, so ymmv. But if it works as advertised, I think it's a little bit more readable. Otherwise do what you got to do.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #133: METRON-180 Enable each component to be installe...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/133
You can test this by, for example, choosing to install only one of the 'sensor' components and not the rest. Before you had to install all 'sensor' components.
Should you want to explore packet capture in your test environment this command will only install pcap-replay and pycapa, but not the rest of the sensor components.
```
cd metron-deployment/vagrant/quick-dev-platform
vagrant --ansible-tags="hdp-deploy,metron" --ansible-skip-tags="solr,bro,flume,snort,yaf,sensor-test-mode" up
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---