You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "Pawel Szczur (JIRA)" <ji...@apache.org> on 2016/06/02 08:34:59 UTC
[jira] [Comment Edited] (BEAM-315) Flink Runner compares keys
unencoded which may produce incorrect results
[ https://issues.apache.org/jira/browse/BEAM-315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15311940#comment-15311940 ]
Pawel Szczur edited comment on BEAM-315 at 6/2/16 8:34 AM:
-----------------------------------------------------------
I thought about it. From your description I guess this should not be a real problem. As long as the corresponding values are compared. In fact I think this was done on purpose to speed up the execution.
First the values are shuffled according to hash function of the key and all the KV ends up in a correct buckets, then we need to sort them. I guess we compare encoded keys because there's no guarantee that pure keys have comparator defined and we have predefined bytes comparator.
was (Author: pawelszczur@gmail.com):
I thought about it. From your description I guess this should not be a real problem. As long as the corresponding values are compared. In fact I think this was done on purpose to speed up the execution. First the values are shuffle according to hash function of the key and all the KV ends up in a correct buckets, then we need to sort.
> Flink Runner compares keys unencoded which may produce incorrect results
> ------------------------------------------------------------------------
>
> Key: BEAM-315
> URL: https://issues.apache.org/jira/browse/BEAM-315
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
> Affects Versions: 0.1.0-incubating
> Reporter: Pawel Szczur
> Assignee: Aljoscha Krettek
> Attachments: CoGroupPipelineStringKey.java
>
>
> Same keys are processed multiple times.
> A repo to reproduce the bug:
> https://github.com/orian/cogroup-wrong-grouping
> Discussion:
> http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201605.mbox/%3CCAB2uKkG2xHsWpLFUkYnt8eEzdxU%3DB_nu6crTwVi-ZuUpugxkPQ%40mail.gmail.com%3E
> Notice: I haven't tested other runners (didn't manage to configure Spark).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)