You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by "Sokolewicz, Michael A" <Mi...@searshc.com> on 2015/05/08 17:42:09 UTC
Storm message passing
Can anyone explain to me how Storm decides when to pass a tuple internally to a local bolt, vs. when to do a network hop to pass the tuple to a bolt on a different Storm server in the cluster?
If anyone knows of some documentation which explains this process, please let me know.
--Michael Sokolewicz
This message, including any attachments, is the property of Sears Holdings Corporation and/or one of its subsidiaries. It is confidential and may contain proprietary or legally privileged information. If you are not the intended recipient, please delete it without reading the contents. Thank you.
Re: Storm message passing
Posted by Nathan Leung <nc...@gmail.com>.
To expound, most fields send locally when their semantics allow it; fields
grouping and shuffle grouping will send locally iff the target bolt task is
local.
Local or shuffle grouping will send locally if possible; otherwise it will
use shuffle grouping.
On May 8, 2015 11:55 AM, "Bobby Evans" <ev...@yahoo-inc.com.invalid> wrote:
> This is all determined by the grouping code.
>
>
> https://github.com/apache/storm/blob/4f4f2c48d068b0f115ed635030eefe15baf897ef/storm-core/src/clj/backtype/storm/daemon/executor.clj#L44-L103
> Each grouping makes the decision of where to send a message. It is also
> pluggable so if you want to modify how it works you can implement
>
> https://github.com/apache/storm/blob/4f4f2c48d068b0f115ed635030eefe15baf897ef/storm-core/src/jvm/backtype/storm/grouping/CustomStreamGrouping.java
> and include it with your topology, with a custom grouping. - Bobby
>
>
>
> On Friday, May 8, 2015 10:44 AM, "Sokolewicz, Michael A" <
> MichaelA.Sokolewicz@searshc.com> wrote:
>
>
> Can anyone explain to me how Storm decides when to pass a tuple
> internally to a local bolt, vs. when to do a network hop to pass the tuple
> to a bolt on a different Storm server in the cluster?
>
> If anyone knows of some documentation which explains this process, please
> let me know.
>
> --Michael Sokolewicz
>
> This message, including any attachments, is the property of Sears Holdings
> Corporation and/or one of its subsidiaries. It is confidential and may
> contain proprietary or legally privileged information. If you are not the
> intended recipient, please delete it without reading the contents. Thank
> you.
>
>
Re: Storm message passing
Posted by Bobby Evans <ev...@yahoo-inc.com.INVALID>.
This is all determined by the grouping code.
https://github.com/apache/storm/blob/4f4f2c48d068b0f115ed635030eefe15baf897ef/storm-core/src/clj/backtype/storm/daemon/executor.clj#L44-L103
Each grouping makes the decision of where to send a message. It is also pluggable so if you want to modify how it works you can implement
https://github.com/apache/storm/blob/4f4f2c48d068b0f115ed635030eefe15baf897ef/storm-core/src/jvm/backtype/storm/grouping/CustomStreamGrouping.java
and include it with your topology, with a custom grouping. - Bobby
On Friday, May 8, 2015 10:44 AM, "Sokolewicz, Michael A" <Mi...@searshc.com> wrote:
Can anyone explain to me how Storm decides when to pass a tuple internally to a local bolt, vs. when to do a network hop to pass the tuple to a bolt on a different Storm server in the cluster?
If anyone knows of some documentation which explains this process, please let me know.
--Michael Sokolewicz
This message, including any attachments, is the property of Sears Holdings Corporation and/or one of its subsidiaries. It is confidential and may contain proprietary or legally privileged information. If you are not the intended recipient, please delete it without reading the contents. Thank you.