Use Partitioner to forward messages to subtask by index

[Joshua Griffith <jo...@gmail.com>]

Is it possible to use a custom Partitioner to forward messages to a
downstream substask by the subtask's index? I realize that it would not be
safe to rely upon this across job restarts but this does not effect my
particular application. I attempted to return a partition number identical
to the subtask number but this did not work. For example, assigning a
message to partition 1 would send it to subtask 0.

Thanks,
Josh

Re: Use Partitioner to forward messages to subtask by index

[Ken Krugler <kk...@transpac.com>]

Hi Joshua,

It is possible, but fragile, as it depends on the internal code that Flink uses to calculate a hash for a key, and the max operator parallelism, etc.

See makeKeyForOperatorIndex <https://github.com/ScaleUnlimited/flink-crawler/blob/master/src/main/java/com/scaleunlimited/flinkcrawler/utils/FlinkUtils.java#L185>, which will generate a String that can be used for partitioning to a specific subtask (operator index)

— Ken

> On Jun 21, 2019, at 10:15 AM, Joshua Griffith <jo...@gmail.com> wrote:
> 
> Is it possible to use a custom Partitioner to forward messages to a downstream substask by the subtask's index? I realize that it would not be safe to rely upon this across job restarts but this does not effect my particular application. I attempted to return a partition number identical to the subtask number but this did not work. For example, assigning a message to partition 1 would send it to subtask 0.
> 
> Thanks,
> Josh

--------------------------
Ken Krugler
+1 530-210-6378
http://www.scaleunlimited.com
Custom big data solutions & training
Flink, Solr, Hadoop, Cascading & Cassandra