You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@crunch.apache.org by "Chao Shi (JIRA)" <ji...@apache.org> on 2013/08/21 05:47:54 UTC
[jira] [Resolved] (CRUNCH-251) HFileUtils#sortAndPartition does not
work when two instances exist in the same pipeline
[ https://issues.apache.org/jira/browse/CRUNCH-251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chao Shi resolved CRUNCH-251.
-----------------------------
Resolution: Fixed
> HFileUtils#sortAndPartition does not work when two instances exist in the same pipeline
> ---------------------------------------------------------------------------------------
>
> Key: CRUNCH-251
> URL: https://issues.apache.org/jira/browse/CRUNCH-251
> Project: Crunch
> Issue Type: Bug
> Components: IO
> Reporter: Chao Shi
> Assignee: Chao Shi
> Fix For: 0.8.0
>
> Attachments: crunch-251.patch
>
>
> This bug happens when generating some data and write them to two HTables (which different number of regions) in a single pipeline. The job fails because of "Wrong number of partitions in keyset".
> This is because it is using TotalOrderPartitioner#setPartitionFile, which sets the path to partition file to a global conf. The correct approach should be only change the reducer's conf (as it is done in Sort#configureReducers). So the fix is straightforward.
> This has been tested on a real HBase cluster. I will make a patch soon.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira