[jira] [Commented] (KAFKA-3561) Auto create through topic for KStream aggregation and join

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-3561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15310298#comment-15310298 ] 

ASF GitHub Bot commented on KAFKA-3561:
---------------------------------------

GitHub user dguy opened a pull request:

    https://github.com/apache/kafka/pull/1453

    KAFKA-3561: Auto create through topic for KStream aggregation and join [WIP]

    @guozhangwang can you please take a look at this? It is not close to being done but i'd like some feedback to ensure i'm heading down the right path and understand what this JIRA is asking. The main things to look at are the `KStreamImpl.map(...)` method I added and the change made to `TopologyBuilder.copartitionGroups`
    
    There is also a test, `KStreamRepartitionMappedKeyTest`, that passes with these changes.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/dguy/kafka KAFKA-3561

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/kafka/pull/1453.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1453
    
----
commit 053c809b5aa322327988909d53027d4682df6825
Author: Damian Guy <da...@gmail.com>
Date:   2016-06-01T13:03:23Z

    repartition through internal topic on KStream.map()

----


> Auto create through topic for KStream aggregation and join
> ----------------------------------------------------------
>
>                 Key: KAFKA-3561
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3561
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>            Reporter: Guozhang Wang
>            Assignee: Damian Guy
>              Labels: api
>             Fix For: 0.10.1.0
>
>
> For KStream.join / aggregateByKey operations that requires the streams to be partitioned on the record key, today users should repartition themselves through the "through" call:
> {code}
> stream1 = builder.stream("topic1");
> stream2 = builder.stream("topic2");
> stream3 = stream1.map(/* set the right key for join*/).through("topic3");
> stream4 = stream2.map(/* set the right key for join*/).through("topic4");
> stream3.join(stream4, ..)
> {code}
> This pattern can actually be done by the Streams DSL itself instead of requiring users to specify themselves, i.e. users can just set the right key like (see KAFKA-3430) and then call join, which will be translated by adding the "internal topic for repartition".
> Another thing is that today if user do not call "through" after setting a new key, the aggregation result would not be correct as the aggregation is based on key B while the source partitions is partitioned by key A and hence each task will only get a partial aggregation for all keys. But this is not validated in the DSL today. We should do both the auto-translation and validation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)