Presplitting table regions - #Reducers different from #Regions

[Ioakim Perros <im...@gmail.com>]

By generalizing my question, in a M/R job, reading/writing from/to the 
same Hbase table,

is there any way to work with any desirable number of reducers importing 
data to a table,

which has its regions presplitted & specified?



Thanks in advance and excuse me for re-asking for help.



-------- Original Message --------
Subject: 	Re: Bulk load - #Reducers different from #Regions
Date: 	Tue, 07 Aug 2012 20:02:27 +0300
From: 	Ioakim Perros <im...@gmail.com>
To: 	user@hbase.apache.org



Excuse me for not well-defining.

I am bulk updating my hbase table through code, using 
configureIncrementalLoad function of HFileOutputFormat. At the 
respective documentation, I read that this function " Sets the number of 
reduce tasks to match the current number of regions" ,but I was 
wondering if I could explicitly avoid it, perhaps by another way of bulk 
importing data.

PS: I try to insist on bulk importing, because I have understood (I hope 
that this is correct), that it is much more efficient than going with 
the traditional Hbase API. And as I require my job to be of iterative 
nature, this way hopefully would end up giving a good boost-up, as 
opposed to the Hbase API.

Thank you for responding.



On 08/07/2012 07:53 PM, Subir S wrote:
> Bulk load using
> ImportTsv with pre-splitted regions for target table?
>
> Do u mean to set number of reducers that ImportTsv must use?
>
> On 8/7/12, Ioakim Perros<im...@gmail.com>  wrote:
>> HI,
>>
>> I am bulk importing (updating) data iteratively and I would like to be
>> able to set the number of reducers at a M/R task, to be different from
>> the number of regions of the table to which I am updating data.
>>
>> I tried it through job.setNumReduceTasks(#reducers), but the job ignored
>> it.
>>
>> Is there a way to avoid an intermediary job and to set the number of
>> reducers explicitly ?
>> I would be grateful if anyone could shed a light to this.
>>
>> Thanks and regards,
>> Ioakim
>>
>>