[jira] [Comment Edited] (KAFKA-13602) Allow to broadcast a result record

["Sagar Rao (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17558753#comment-17558753 ] 

Sagar Rao edited comment on KAFKA-13602 at 6/25/22 10:43 AM:
-------------------------------------------------------------

Hey @florin , I have already sent out a Kip for this. Here is the link : [https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211883356.|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211883356]

Review is awaited still. You can also share your review comments on the discussion thread!

 


was (Author: sagarrao):
Hey @florin , I have already sent out a Kip for this. Here is the link : [https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211883356.|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211883356]


Review is awaited still. 

 

> Allow to broadcast a result record
> ----------------------------------
>
>                 Key: KAFKA-13602
>                 URL: https://issues.apache.org/jira/browse/KAFKA-13602
>             Project: Kafka
>          Issue Type: New Feature
>          Components: streams
>            Reporter: Matthias J. Sax
>            Assignee: Sagar Rao
>            Priority: Major
>              Labels: needs-kip, newbie++
>
> From time to time, users ask how they can send a record to more than one partition in a sink topic. Currently, this is only possible by replicate the message N times before the sink and use a custom partitioner to write the N messages into the N different partitions.
> It might be worth to make this easier and add a new feature for it. There are multiple options:
>  * extend `to()` / `addSink()` with a "broadcast" option/config
>  * add `toAllPartitions()` / `addBroadcastSink()` methods
>  * allow StreamPartitioner to return `-1` for "all partitions"
>  * extend `StreamPartitioner` to allow returning more than one partition (ie a list/collection of integers instead of a single int)
> The first three options imply that a "full broadcast" is supported only, so it's less flexible. On the other hand, it's easier to use (especially the first two options are easy as they do not require to implement a custom partitioner).
> The last option would be most flexible and also allow for a "partial broadcast" (aka multi-cast) pattern. It might also be possible to combine two options, or maye even a totally different idea.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)