Avoiding bottleneck in fields grouping

[Jayant Sharma <sh...@gmail.com>]

Hi,

Is there a standard way of avoiding bottleneck which arises due to fields
grouping from one bolt to another. I have a use case where half a million
tuples have the same field and go to the same bolt task because of field
grouping. I cannot use shuffle grouping here because it is important that
these get processed in a sequence. Has anyone faced such an issue in Storm,
how were you able to resolve it?

Thank you,
Jayant

Re: Avoiding bottleneck in fields grouping

[Rui Abreu <ru...@gmail.com>]

If  maintaining the order of the messages is a requirement, fields grouping
seems to be the  only strategy that ensures that all tuples of the same
partition will be sent to the same task ID.


https://storm.apache.org/releases/current/Concepts.html

Stream groupings
> Part of defining a topology is specifying for each bolt which streams it
> should receive as input. A stream grouping defines how that stream should
> be partitioned among the bolt's tasks.
> There are eight built-in stream groupings in Storm, and you can implement
> a custom stream grouping by implementing the CustomStreamGrouping
> <https://storm.apache.org/releases/current/javadocs/org/apache/storm/grouping/CustomStreamGrouping.html>
>  interface:
>
>    1. Shuffle grouping: Tuples are randomly distributed across the bolt's
>    tasks in a way such that each bolt is guaranteed to get an equal number of
>    tuples.
>
>
>    1. Fields grouping: The stream is partitioned by the fields specified
>    in the grouping. For example, if the stream is grouped by the "user-id"
>    field, tuples with the same "user-id" will always go to the same task, but
>    tuples with different "user-id"'s may go to different tasks.
>
>
>    1. Partial Key grouping: The stream is partitioned by the fields
>    specified in the grouping, like the Fields grouping, but are load balanced
>    between two downstream bolts, which provides better utilization of
>    resources when the incoming data is skewed. This paper
>    <https://melmeric.files.wordpress.com/2014/11/the-power-of-both-choices-practical-load-balancing-for-distributed-stream-processing-engines.pdf> provides
>    a good explanation of how it works and the advantages it provides.
>
>
>    1. All grouping: The stream is replicated across all the bolt's tasks.
>    Use this grouping with care.
>
>
>    1. Global grouping: The entire stream goes to a single one of the
>    bolt's tasks. Specifically, it goes to the task with the lowest id.
>
>
>    1. None grouping: This grouping specifies that you don't care how the
>    stream is grouped. Currently, none groupings are equivalent to shuffle
>    groupings. Eventually though, Storm will push down bolts with none
>    groupings to execute in the same thread as the bolt or spout they subscribe
>    from (when possible).
>
>
>    1. Direct grouping: This is a special kind of grouping. A stream
>    grouped this way means that the producer of the tuple decides which
>    task of the consumer will receive this tuple. Direct groupings can only be
>    declared on streams that have been declared as direct streams. Tuples
>    emitted to a direct stream must be emitted using one of the emitDirect
>    <https://storm.apache.org/releases/current/javadocs/org/apache/storm/task/OutputCollector.html#emitDirect-int-java.util.Collection-java.util.List-> methods.
>    A bolt can get the task ids of its consumers by either using the provided
>    TopologyContext
>    <https://storm.apache.org/releases/current/javadocs/org/apache/storm/task/TopologyContext.html> or
>    by keeping track of the output of the emit method in OutputCollector
>    <https://storm.apache.org/releases/current/javadocs/org/apache/storm/task/OutputCollector.html> (which
>    returns the task ids that the tuple was sent to).
>
>
>    1. Local or shuffle grouping: If the target bolt has one or more tasks
>    in the same worker process, tuples will be shuffled to just those
>    in-process tasks. Otherwise, this acts like a normal shuffle grouping.
>
>
On Thu, 13 Aug 2020 at 12:05, Jayant Sharma <sh...@gmail.com>
wrote:

> Hi,
>
> Is there a standard way of avoiding bottleneck which arises due to fields
> grouping from one bolt to another. I have a use case where half a million
> tuples have the same field and go to the same bolt task because of field
> grouping. I cannot use shuffle grouping here because it is important that
> these get processed in a sequence. Has anyone faced such an issue in Storm,
> how were you able to resolve it?
>
> Thank you,
> Jayant
>