You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@hbase.apache.org by Kiru Pakkirisamy <ki...@yahoo.com> on 2014/01/13 23:37:26 UTC

Large Puts in MR job


We are creating a Put with large number (50000) of key/vals in our map task. 
This is causing an out-of-memory error. Is there a way to set the autoFlush on when using the TableMapReduceUtil in the map task ?

Thanks,
 
Regards,
- kiru

Re: Large Puts in MR job

Posted by Dhaval Shah <pr...@yahoo.co.in>.

If you are creating 1 big put object, how would auto flush help you? In theory you would run out of memory before you do a table.put() anyways. Am I missing something?

Why don't you split your put into smaller puts and let the deferred flush do its job? Do you need all the kv's to be flushed at the same time?

Technically you can create your own hbase client in setup() but I don't know if that's going to solve your issue

Sent from Yahoo Mail on Android