You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@crunch.apache.org by "Josh Wills (JIRA)" <ji...@apache.org> on 2015/01/08 03:55:36 UTC
[jira] [Updated] (CRUNCH-485) groupByKey on Spark incorrect if key
is Avro record with defined sort order
[ https://issues.apache.org/jira/browse/CRUNCH-485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Wills updated CRUNCH-485:
------------------------------
Attachment: CRUNCH-485.patch
Here's a first cut at a fix-- not super pretty, but I think it will at least do the right thing.
> groupByKey on Spark incorrect if key is Avro record with defined sort order
> ---------------------------------------------------------------------------
>
> Key: CRUNCH-485
> URL: https://issues.apache.org/jira/browse/CRUNCH-485
> Project: Crunch
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.11.0
> Reporter: Tycho Lamerigts
> Assignee: Josh Wills
> Attachments: CRUNCH-485.patch
>
>
> GroupByKey on Spark is incorrect if the key type is an Avro record with defined sort order (http://avro.apache.org/docs/1.7.7/spec.html#order).
> Instead, it serializes the entire avro record to a binary blob (byte array) and groups identical blobs. This is wrong. By contrast, groupByKey on MapReduce works as expected, so it does take Avro's sort order into account.
> The culprit is probably the following code from org.apache.crunch.impl.spark.collect.PGroupedTableImpl#getJavaRDDLikeInternal
> {code}
> groupedRDD = parentRDD.map(new PairMapFunction(ptype.getOutputMapFn(), runtime.getRuntimeContext()))
> .mapToPair(new MapOutputFunction(keySerde, valueSerde))
> .groupByKey(numPartitions);
> {code}
> where MapOutputFunction simply converts the entire key object to a binary blob, without taking sort order into account.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)