[jira] [Commented] (CRUNCH-485) groupByKey on Spark incorrect if key is Avro record with defined sort order

["Josh Wills (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CRUNCH-485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267294#comment-14267294 ] 

Josh Wills commented on CRUNCH-485:
-----------------------------------

Strictly speaking, groupByKey doesn't impose sorting unless you pass a GroupingOptions object with the requireSortedKeys() option enabled; if you pass that in to your example, does it work as expected? I suppose what I'm saying here is that it doesn't seem like Crunch-on-Spark should do something that Spark wouldn't do by default, but I'm happy to debate that topic on the list.

> groupByKey on Spark incorrect if key is Avro record with defined sort order
> ---------------------------------------------------------------------------
>
>                 Key: CRUNCH-485
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-485
>             Project: Crunch
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.11.0
>            Reporter: Tycho Lamerigts
>            Assignee: Josh Wills
>
> GroupByKey on Spark is incorrect if the key type is an Avro record with defined sort order (http://avro.apache.org/docs/1.7.7/spec.html#order).
> Instead, it serializes the entire avro record to a binary blob (byte array) and groups identical blobs. This is wrong. By contrast, groupByKey on MapReduce works as expected, so it does take Avro's sort order into account.
> The culprit is probably the following code from org.apache.crunch.impl.spark.collect.PGroupedTableImpl#getJavaRDDLikeInternal
> {code}
> groupedRDD = parentRDD.map(new PairMapFunction(ptype.getOutputMapFn(), runtime.getRuntimeContext()))
>           .mapToPair(new MapOutputFunction(keySerde, valueSerde))
>           .groupByKey(numPartitions);
> {code}
> where MapOutputFunction simply converts the entire key object to a binary blob, without taking sort order into account.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)