You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by cestella <gi...@git.apache.org> on 2016/05/27 19:24:56 UTC
[GitHub] incubator-metron pull request: METRON-190: Make start_parser_topol...
GitHub user cestella opened a pull request:
https://github.com/apache/incubator-metron/pull/139
METRON-190: Make start_parser_topology.sh more adaptable regarding storm topology configuration
Add the ability to specify the message timeout, the max task parallelism, the number of ackers, the number of workers. Also allow the user to pass in JSON file containing a map of extra storm configs.
I added the following extra options to `start_parser_topology.sh`:
```
-e,--extra_options <JSON_FILE> Extra options in the form of a
JSON file with a map for
content.
-mt,--message_timeout <TIMEOUT_IN_SECS> Message Timeout in Seconds
-mtp,--max_task_parallelism <MAX_TASK> Max task parallelism
-na,--num_ackers <NUM_ACKERS> Number of Ackers
-nw,--num_workers <NUM_WORKERS> Number of Workers
```
For instance, if you wanted to set some storm property on
the config called `topology.ticks.tuple.freq.secs` to 1000 and `storm.local.dir` to `/opt/my/path`
you could create a file called `custom_config.json` containing
```
{
"topology.ticks.tuple.freq.secs" : 1000,
"storm.local.dir" : "/opt/my/path"
}
```
and pass `--extra_options custom_config.json` to `start_parser_topology.sh`.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cestella/incubator-metron parser_parallelism_expose
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/139.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #139
----
commit 33815e0382b54fb6931bae4579223535e7abdc90
Author: cstella <ce...@gmail.com>
Date: 2016-05-27T19:18:12Z
Exposed more storm topology config options.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-190: Make start_parser_topol...
Posted by james-sirota <gi...@git.apache.org>.
Github user james-sirota commented on the pull request:
https://github.com/apache/incubator-metron/pull/139#issuecomment-222345898
+1. The only thing I would suggest would be to name the variable parallelism hint and not just parallelism because that name has meaning for someone who would be tuning storm.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-190: Make start_parser_topol...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the pull request:
https://github.com/apache/incubator-metron/pull/139#issuecomment-222241649
Just watch out for monit's use of this script. Let me know if you need specific pointers.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-190: Make start_parser_topol...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/139
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-190: Make start_parser_topol...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the pull request:
https://github.com/apache/incubator-metron/pull/139#issuecomment-222363512
good catch, I'll make those changes now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-190: Make start_parser_topol...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the pull request:
https://github.com/apache/incubator-metron/pull/139#issuecomment-222253339
Ok, I verified this on full dev vagrant.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-190: Make start_parser_topol...
Posted by james-sirota <gi...@git.apache.org>.
Github user james-sirota commented on the pull request:
https://github.com/apache/incubator-metron/pull/139#issuecomment-222293764
The settings work, but I think you forgot to expose the parallelism hint. See this blog entry: https://storm.apache.org/releases/1.0.0/Understanding-the-parallelism-of-a-Storm-topology.html
Notice how they are setting
topologyBuilder.setBolt("green-bolt", new GreenBolt(), 2)
.setNumTasks(4)
.shuffleGrouping("blue-spout");
This will run two executors and four associated tasks (two tasks per executor)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-190: Make start_parser_topol...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the pull request:
https://github.com/apache/incubator-metron/pull/139#issuecomment-222313782
@james-sirota Right you are! Just updated exposing the numTasks for the spout and the parser bolt.
```
-pnt,--parser_num_tasks <PARSER_NUM_TASKS> Parser Num Tasks
-pp,--parser_p <PARSER_PARALLELISM> Parser Parallelism
-snt,--spout_num_tasks <NUM_TASKS> Spout Num Tasks
-sp,--spout_p <SPOUT_PARALLELISM> Spout Parallelism
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---