You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by jjonez <gi...@git.apache.org> on 2016/05/11 01:35:21 UTC
[GitHub] incubator-metron pull request: Metron 146 topology workers
GitHub user jjonez opened a pull request:
https://github.com/apache/incubator-metron/pull/116
Metron 146 topology workers
This allows you to configure the number of workers in the enrichment topology. The variable name is pretty long, for the sake of being verbose (storm.topology.enrichment.workers, or in Ansible it is storm_topology_enrichment_workers).
I tested this using the vagrant/quick-dev-platform image. After starting, I ran the ./run_enrichment_role.sh script to start the enrichment topology. I verified that the file has the configured value and the topology started correctly. You can see in the ansible output that it tried to replaced the variable here but the default value is the same since as the original value. I can only run this on my laptop:
'''
TASK [metron_streaming : Configure Metron Solr topology] ***********************
<!--- SNIP -->
ok: [node1] => (item={u'regexp': u'storm.topology.enrichment.workers=', u'line': u'storm.topology.enrichment.workers=1'})
'''
If someone could test changing the value on a real cluster that would be great. I don't have the hardware for it.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jjonez/incubator-metron METRON-146_topology_workers
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/116.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #116
----
commit 0e3f59b7a12322f491f72f99d019323d5443cddd
Author: Jim Jones <we...@gmail.com>
Date: 2016-05-09T21:38:41Z
initial stuff
commit 0153e35cbb23a097cea110d66aff4501d29329c1
Author: Jim Jones <we...@gmail.com>
Date: 2016-05-11T01:05:25Z
flux...workers=int, not string; Also, ansible defaults undefined variable.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #116: Metron 146 topology workers
Posted by jjonez <gi...@git.apache.org>.
Github user jjonez closed the pull request at:
https://github.com/apache/incubator-metron/pull/116
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #116: Metron 146 topology workers
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/116
I am unable to. You should be able to close it though. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #116: Metron 146 topology workers
Posted by jjonez <gi...@git.apache.org>.
Github user jjonez commented on the issue:
https://github.com/apache/incubator-metron/pull/116
Sorry. I'll take care of it.
> On Sep 21, 2016, at 1:24 PM, Nick Allen <no...@github.com> wrote:
>
> I am unable to. You should be able to close it though. Thanks.
>
> \u2014
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or mute the thread.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: Metron 146 topology workers
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/116#discussion_r63290512
--- Diff: metron-deployment/roles/metron_streaming/tasks/main.yml ---
@@ -130,6 +130,8 @@
- { regexp: "bolt.hdfs.rotation.policy=", line: "bolt.hdfs.rotation.policy={{ metron_hdfs_rotation_policy }}" }
- { regexp: "bolt.hdfs.rotation.policy.count=", line: "bolt.hdfs.rotation.policy.count={{ metron_hdfs_rotation_policy_count}}" }
- { regexp: "bolt.hdfs.rotation.policy.units=", line: "bolt.hdfs.rotation.policy.units={{ metron_hdfs_rotation_policy_units }}" }
+ - { regexp: "storm.topology.enrichment.workers=",
+ line: "storm.topology.enrichment.workers={{ storm_topology_enrichment_workers if storm_topology_enrichment_workers is defined else 1 }}" }
--- End diff --
Rather that use ```is defined``` here, you can set the default in metron_streaming/defaults/main.yml by adding ```storm.topology.enrichment.workers: 1``` somewhere in the file.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: Metron 146 topology workers
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/116#discussion_r63290446
--- Diff: metron-platform/metron-integration-test/src/main/java/org/apache/metron/integration/EnrichmentIntegrationTest.java ---
@@ -164,6 +164,7 @@ public void test() throws Exception {
setProperty("es.ip", "localhost");
setProperty("index.date.format", dateFormat);
setProperty("index.hdfs.output", hdfsDir);
+ setProperty("storm.topology.enrichment.workers", 1);
--- End diff --
I think this is what's failing the test. IIRC, that's setProperty(string, string).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: Metron 146 topology workers
Posted by jjonez <gi...@git.apache.org>.
Github user jjonez commented on the pull request:
https://github.com/apache/incubator-metron/pull/116#issuecomment-219264231
I had set the value in those files. The problem was that I set it as String in the Integration test rather than as an integer.
I also had merged with master, but the build failed on something that I *think* is totally unrelated to my changes, so I reverted the merge. [Here](https://travis-ci.org/apache/incubator-metron/builds/130322731) is the failed build.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: Metron 146 topology workers
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the pull request:
https://github.com/apache/incubator-metron/pull/116#issuecomment-218777995
Ah, ok, so this is failing the integration tests because you're not setting the topology.workers property. You can either set it
* In [this](https://github.com/apache/incubator-metron/blob/master/metron-platform/metron-integration-test/src/main/java/org/apache/metron/integration/EnrichmentIntegrationTest.java#L151) block of code
* In [this](https://github.com/apache/incubator-metron/blob/master/metron-platform/metron-elasticsearch/src/main/config/elasticsearch.properties) and [this](https://github.com/apache/incubator-metron/blob/master/metron-platform/metron-solr/src/main/config/solr.properties) file
The first will just fix it for the integration test. The second will provide a default for the deployed properties. That being said, it should be being set via ansible, so providing a default isn't strictly required.
I don't have a strong opinion either way.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #116: Metron 146 topology workers
Posted by jjonez <gi...@git.apache.org>.
Github user jjonez commented on the issue:
https://github.com/apache/incubator-metron/pull/116
Yes. Please close it. Thanks
> On Sep 21, 2016, at 1:20 PM, Nick Allen <no...@github.com> wrote:
>
> @jjonez Are you good with closing this PR? It has been a good amount of time since its been touched. Thanks
>
> \u2014
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or mute the thread.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #116: Metron 146 topology workers
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/116
@jjonez Are you good with closing this PR? It has been a good amount of time since its been touched. Thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---