You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by dlyle65535 <gi...@git.apache.org> on 2016/09/27 19:10:58 UTC
[GitHub] incubator-metron pull request #279: METRON-466: Full Dev Platform build some...
GitHub user dlyle65535 opened a pull request:
https://github.com/apache/incubator-metron/pull/279
METRON-466: Full Dev Platform build sometimes fails
Tested on Full Dev and Quick Dev.
Ambari seems to have changed their response in 2.4.0.1-1, if you request "START ALL" on a cluster where all services are already started, it doesn't actually make a request, so there is no request ID reported back.
I also suppressed starting of the Yaf probe (because we don't run the parser topology for Yaf) and bumped java_home so Snort would work.
I needed to separate the inventory directories for quick and full dev certain properties (in this case java_home) can lag between quick and full dev.
Note- Full dev still doesn't come up cleanly after the build successfully completes due to what looks like insufficient memory. I've opened [METRON-467](https://issues.apache.org/jira/browse/METRON-467).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dlyle65535/incubator-metron METRON-466
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/279.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #279
----
commit 64f2256bb549db1ee8bbf1870f628ca6a649a422
Author: David Lyle <dl...@gmail.com>
Date: 2016-09-27T17:29:04Z
METRON-466: Full Dev Platform build sometimes fails
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #279: METRON-466: Full Dev Platform build sometimes f...
Posted by james-sirota <gi...@git.apache.org>.
Github user james-sirota commented on the issue:
https://github.com/apache/incubator-metron/pull/279
+1 to this. Here is why. If fulldev can only support 2 running parser topologies I think it makes sense to let a user pick which 2 they want. The intention of vagrant dev images is to spin them up to test a PR and then destroy them. We generally only need a few topologies for that and it is not the intention to use vagrant images to simulate a full running system. By having only 2 topologies started by default we will avoid 90% of problems people have when they spin up the vagrant images and run out of memory. With that said I will also document what needs to be done to spin up the image with 4 topologies, but put a disclaimer to start at their own risk.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #279: METRON-466: Full Dev Platform build some...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/279#discussion_r80998612
--- Diff: metron-deployment/inventory/quick-dev-platform/group_vars/all ---
@@ -0,0 +1,95 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# which services should be started?
+services_to_start:
+ - mysql
+ - elasticsearch
+ - pcap-service
+ - kibana
+ - snort
+ - snort-logs
+ - bro
+ - pcap-replay
+ - bro-parser
+ - snort-parser
+ - enrichment
+ - indexing
--- End diff --
Don't have a firm opinion either way. Maybe solicit some opinions and respond to them as a separate issue?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #279: METRON-466: Full Dev Platform build sometimes f...
Posted by james-sirota <gi...@git.apache.org>.
Github user james-sirota commented on the issue:
https://github.com/apache/incubator-metron/pull/279
What is our preferred way to manually add sensors that were not installed by default? How would I add YAF back if I needed it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #279: METRON-466: Full Dev Platform build some...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/279
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #279: METRON-466: Full Dev Platform build sometimes f...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on the issue:
https://github.com/apache/incubator-metron/pull/279
Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #279: METRON-466: Full Dev Platform build some...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/279#discussion_r81005015
--- Diff: metron-deployment/inventory/full-dev-platform/group_vars/all ---
@@ -78,7 +79,7 @@ sensor_test_mode: True
install_pycapa: False
install_bro: True
install_snort: True
-install_yaf: True
+install_yaf: False
--- End diff --
If you just want to not start YAF, then not having it in the 'services_to_start' list should be sufficient. The change here causes YAF not to be installed.
On a side note, I would expect Snort and Bro to be much larger resource hogs than YAF.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #279: METRON-466: Full Dev Platform build sometimes f...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/279
So, just recapping and making sure I understand; the issue is as follows: Yaf being started tips us over the edge memory-wise in full-dev, correct?
What we're debating here is whether we should not install yaf in `quick-dev` and `full-dev` or whether we should install it, but not start it, right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #279: METRON-466: Full Dev Platform build some...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/279#discussion_r81008359
--- Diff: metron-deployment/vagrant/full-dev-platform/run.sh ---
@@ -17,4 +17,4 @@
# limitations under the License.
#
-vagrant --ansible-skip-tags="solr" up
+vagrant --ansible-skip-tags="solr,yaf" up
--- End diff --
That was my intention. No point in waiting for download and compilation if we're not using it. Can always override/rerun after.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #279: METRON-466: Full Dev Platform build some...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/279#discussion_r81005106
--- Diff: metron-deployment/vagrant/full-dev-platform/run.sh ---
@@ -17,4 +17,4 @@
# limitations under the License.
#
-vagrant --ansible-skip-tags="solr" up
+vagrant --ansible-skip-tags="solr,yaf" up
--- End diff --
This causes YAF to not be installed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #279: METRON-466: Full Dev Platform build sometimes f...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/279
Ok, just to clarify a bit; I think what is happening is that the yaf topology (which does exist) does not run by default (this was changed a while back because of resource constraints), but that does not prevent it from installing the yaf probe that pushes data to kafka. It's the yaf probe that @dlyle65535 is stopping by not installing it.
In general, just my $0.02, but I tend think the proper behavior is as follows:
* All sensors get installed by default; bro, snort, yaf, etc.
* None of them get started by default in either full-dev or quick-dev. If you want them to start, turn them on via monit
I'm not sure this should be within the scope of this PR, which is why @dlyle65535 made METRON-467. As for what we do in the interrim, I'd say that either:
* we either are ok with not installing yaf
* we make the sensors in the `services_to_start` not start, not just the topologies.
Thoughts and corrections would be greatly appreciated. :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #279: METRON-466: Full Dev Platform build some...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/279#discussion_r80998367
--- Diff: metron-deployment/inventory/full-dev-platform/group_vars/all ---
@@ -78,7 +79,7 @@ sensor_test_mode: True
install_pycapa: False
install_bro: True
install_snort: True
-install_yaf: True
+install_yaf: False
--- End diff --
Are some missing? I did intend to have the probe off as the topology isn't started so it's a waste of memory and I couldn't get the cluster service started with it running.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #279: METRON-466: Full Dev Platform build sometimes f...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on the issue:
https://github.com/apache/incubator-metron/pull/279
@nickwallen - I think the discussion has died down. Are you good with me committing this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #279: METRON-466: Full Dev Platform build some...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/279#discussion_r81008608
--- Diff: metron-deployment/inventory/full-dev-platform/group_vars/all ---
@@ -78,7 +79,7 @@ sensor_test_mode: True
install_pycapa: False
install_bro: True
install_snort: True
-install_yaf: True
+install_yaf: False
--- End diff --
I don't know. We had decided months ago to run bro and snort automatically.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #279: METRON-466: Full Dev Platform build sometimes f...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/279
just by the way, the above comments are just my opinions. More discussion about the proper way to deal with the resource constraints long-term can and should happen either on the dev list or on METRON-467
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #279: METRON-466: Full Dev Platform build some...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/279#discussion_r81035594
--- Diff: metron-deployment/vagrant/quick-dev-platform/run.sh ---
@@ -19,5 +19,5 @@
vagrant \
--ansible-tags="hdp-deploy,metron" \
- --ansible-skip-tags="solr" \
+ --ansible-skip-tags="solr,yaf" \
--- End diff --
Yes, I didn't want to install a probe that is not run and has no parser topology.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #279: METRON-466: Full Dev Platform build sometimes f...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/279
Yep, I'm good. We just need to be sure to update all docs around this at some point.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #279: METRON-466: Full Dev Platform build some...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/279#discussion_r80999009
--- Diff: metron-deployment/inventory/quick-dev-platform/group_vars/all ---
@@ -0,0 +1,95 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# which services should be started?
+services_to_start:
+ - mysql
+ - elasticsearch
+ - pcap-service
+ - kibana
+ - snort
+ - snort-logs
+ - bro
+ - pcap-replay
+ - bro-parser
+ - snort-parser
+ - enrichment
+ - indexing
--- End diff --
For what it's worth that is a good idea. Maybe there could be some 'special' scripts or tools available for that configuration ( things you would want while developing a parser and stellar rules etc )
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #279: METRON-466: Full Dev Platform build some...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/279#discussion_r80924532
--- Diff: metron-deployment/inventory/quick-dev-platform/group_vars/all ---
@@ -0,0 +1,95 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# which services should be started?
+services_to_start:
+ - mysql
+ - elasticsearch
+ - pcap-service
+ - kibana
+ - snort
+ - snort-logs
+ - bro
+ - pcap-replay
+ - bro-parser
+ - snort-parser
+ - enrichment
+ - indexing
--- End diff --
Yes! We finally split the config for quick-dev from full-dev. What if we didn't start anything by default for quick-dev? When you're developing with quick-dev, you just go in and manually start just what you need.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #279: METRON-466: Full Dev Platform build some...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/279#discussion_r81014011
--- Diff: metron-deployment/inventory/full-dev-platform/group_vars/all ---
@@ -78,7 +79,7 @@ sensor_test_mode: True
install_pycapa: False
install_bro: True
install_snort: True
-install_yaf: True
+install_yaf: False
--- End diff --
Before this change, Snort, Bro and YAF were all installed and started by default. That is my understanding of how it has been for months.
This PR changes that behavior so that YAF is neither installed nor started. The comments in your PR only say that you don't want YAF started. You don't mention anything about YAF not being installed.
Since there was this gap between what the PR says and what the PR does, I wanted to make sure we bring it to light.
There is a fair amount of documentation that describes to users that when they deploy Metron, they are going to see YAF, Snort and Bro data in the Metron Dashboard. This PR will change that. The user will not see YAF data.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #279: METRON-466: Full Dev Platform build sometimes f...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/279
@james-sirota sounds good to me. If we're going down this road, we might consider providing an argument to run.sh to specify up to 2 topologies as a follow-on JIRA.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #279: METRON-466: Full Dev Platform build sometimes f...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on the issue:
https://github.com/apache/incubator-metron/pull/279
@james-sirota - a couple of ways:
Pre-deploy:
1) Decide which parser/probe pair you wish to trade.
2) Toggle it off by setting install_(probe): false and add it to ansible-skip-tags in run.sh in place of 'yaf'.
Post-Deploy
1) Decide which parser/probe pair you wish to trade.
2) Kill the parsing topology and stop the associated sensor.
3) Toggle install_yaf to true.
4) Add yaf-parser to services to start.
4) ./run_ansible_role "sensors,enrichment,monit,start"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #279: METRON-466: Full Dev Platform build some...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/279#discussion_r81005130
--- Diff: metron-deployment/vagrant/quick-dev-platform/run.sh ---
@@ -19,5 +19,5 @@
vagrant \
--ansible-tags="hdp-deploy,metron" \
- --ansible-skip-tags="solr" \
+ --ansible-skip-tags="solr,yaf" \
--- End diff --
This causes YAF to not be installed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #279: METRON-466: Full Dev Platform build some...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/279#discussion_r80923536
--- Diff: metron-deployment/inventory/full-dev-platform/group_vars/all ---
@@ -78,7 +79,7 @@ sensor_test_mode: True
install_pycapa: False
install_bro: True
install_snort: True
-install_yaf: True
+install_yaf: False
--- End diff --
There are a few changes that turn off YAF installation. Did you intend for those to get into this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---