[jira] [Commented] (BEAM-315) GroupByKey/CoGroupByKey doesn't group correctly in batch mode of FlinkPipelineRunner

["Aljoscha Krettek (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15309976#comment-15309976 ] 

Aljoscha Krettek commented on BEAM-315:
---------------------------------------

I played around this morning and finally found the problem thanks to [~frances]'s remark.

The problem is that the Flink {{KvCoderComparator}} (this is used to do comparisons/hasing for shuffle and sorting) uses the hash of the key instead of the hash of the encoded key while using the encoded key for doing comparisons. I already fixed it, but I want to include a test for this as well. [~pawelszczur@gmail.com], can I include a modified version of your program as a test?

> GroupByKey/CoGroupByKey doesn't group correctly in batch mode of FlinkPipelineRunner
> ------------------------------------------------------------------------------------
>
>                 Key: BEAM-315
>                 URL: https://issues.apache.org/jira/browse/BEAM-315
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-flink
>    Affects Versions: 0.1.0-incubating
>            Reporter: Pawel Szczur
>         Attachments: CoGroupPipelineStringKey.java
>
>
> Same keys are processed multiple times.
> A repo to reproduce the bug:
> https://github.com/orian/cogroup-wrong-grouping
> Discussion:
> http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201605.mbox/%3CCAB2uKkG2xHsWpLFUkYnt8eEzdxU%3DB_nu6crTwVi-ZuUpugxkPQ%40mail.gmail.com%3E
> Notice: I haven't tested other runners (didn't manage to configure Spark).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)