You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by merrimanr <gi...@git.apache.org> on 2018/10/03 21:52:20 UTC
[GitHub] metron pull request #1221: METRON-1805: Provide a default value for the Stor...
GitHub user merrimanr opened a pull request:
https://github.com/apache/metron/pull/1221
METRON-1805: Provide a default value for the Storm topology.max.spout.pending setting
## Contributor Comments
Users have reported problems in the random and batch indexing topologies when this setting is not set. The primary purpose of this PR is to start a discussion around providing a default value for the Storm topology.max.spout.pending setting in our topologies and implement the changes if we decide to do it.
1. Should we provide defaults or continue to defer to the Storm default where a maximum isn't enforced?
2. Which topologies should we provide defaults for?
3. What should the default values be for the topologies we include?
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
```
mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh
```
- [ ] Have you written or updated unit tests and or integration tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
```
cd site-book
mvn site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/merrimanr/incubator-metron METRON-1805
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/1221.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1221
----
commit 6988e0607afc1cd402063e0696a1220cc134d9d1
Author: merrimanr <me...@...>
Date: 2018-10-03T21:45:32Z
initial commit
----
---
[GitHub] metron issue #1221: METRON-1805: Provide a default value for the Storm topol...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/1221
For context, I set the default for Enrichments to 500. There was some [discussion here](https://github.com/apache/metron/pull/984#discussion_r180150453).
I would echo my comments from that discussion here. I think our defaults should focus on day 1 user experience. Defining a default in the 300 - 700 range tends to be reasonable on most platforms for day 1. What we don't want is to start Metron up and the topology to immediately die with an OOM because there is no back pressure.
---
[GitHub] metron issue #1221: METRON-1805: Provide a default value for the Storm topol...
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/1221
Anything else needed here @merrimanr?
---
[GitHub] metron issue #1221: METRON-1805: Provide a default value for the Storm topol...
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/1221
This seems reasonable to me all the way around. +1 by inspection.
---
[GitHub] metron pull request #1221: METRON-1805: Provide a default value for the Stor...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/metron/pull/1221
---
[GitHub] metron issue #1221: METRON-1805: Provide a default value for the Storm topol...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/metron/pull/1221
As far as I know there was not an issue with other topologies. Writing data tends to cause back pressure so that's why these changes were suggested for indexing topologies.
---
[GitHub] metron issue #1221: METRON-1805: Provide a default value for the Storm topol...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/1221
> @merrimanr: Which topologies should we provide defaults for?
To follow-up on the other topologies, we do not set a [default for the Profiler](https://github.com/apache/metron/blob/2ee6cc7e0b448d8d27f56f873e2c15a603c53917/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-profiler-env.xml#L168), which we probably should.
I am not sure about Parsers.
---
[GitHub] metron issue #1221: METRON-1805: Provide a default value for the Storm topol...
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/1221
Outside of pcap, I've yet to see issues where parsers would have any issues handling inbound throughput. The profiler default seems reasonable to me as well. Still +1 from me.
---
[GitHub] metron issue #1221: METRON-1805: Provide a default value for the Storm topol...
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/1221
I think this is fine. This is just the backpressure setting and having it set to the default means that the spigot is fully open and will potentially flood the topology with a high velocity of data. We recommend tuning all topologies, so starting from a default that doesn't crush the system seems reasonable to me. Since this change is only being made to indexing, can you clarify if there is/was any observable issues in the other topologies?
---
[GitHub] metron issue #1221: METRON-1805: Provide a default value for the Storm topol...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/metron/pull/1221
I added a default to the profiler topology. We're probably less likely to run into a problem with the parser topologies I would think. Should we provide defaults for those?
---