[jira] [Commented] (CRUNCH-455) Sort.sort doesn't work with ReverseAvroComparator in MemPipeline

["Gabriel Reid (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CRUNCH-455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093121#comment-14093121 ] 

Gabriel Reid commented on CRUNCH-455:
-------------------------------------

The removal of the re-use of the single AvroKey instance AvroKeyConverter worries me a (really tiny) little bit, but that's just me fearing the unknown. I'd love to try some benchmarking to see if this makes a real performance difference at all -- I'm pretty convinced it doesn't, but the fact that this stuff is done everywhere in MR code makes me wonder.

I was looking at Shuffler to see if there's any other way to do it, but the only thing I see is having a special case for handling Avro where we make a copy of the AvroKey object before putting it in the map, which wouldn't be too cool either.

> Sort.sort doesn't work with ReverseAvroComparator in MemPipeline
> ----------------------------------------------------------------
>
>                 Key: CRUNCH-455
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-455
>             Project: Crunch
>          Issue Type: Bug
>          Components: Core
>            Reporter: David Whiting
>            Assignee: Josh Wills
>            Priority: Minor
>         Attachments: CRUNCH-455.patch
>
>
> The mem Shuffler class discards the config that arrives with the GroupingOptions and only uses the unmodified Conifguration from the pipeline object, which means that "crunch.schema" is not set and causes a NullPointerException when you try and execute it.



--
This message was sent by Atlassian JIRA
(v6.2#6252)