You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@hbase.apache.org by Billy Pearson <sa...@pearsonwholesale.com> on 2009/04/14 01:58:08 UTC

Skipping Bad Records

I am running a job that pulls data from hbase but I getting heap errors on 
some of the records because there to large to fit in the heap of the task

I enabled I thought so the skip option in the site conf file and I also 
added these options to my job conf

    conf.setMaxMapAttempts(10);
    SkipBadRecords.setMapperMaxSkipRecords(conf, 1);
    SkipBadRecords.setAttemptsToStartSkipping(conf, 1);

>From the MR docs it seams the task is split and ran as two different task 
how would this be handled in hbase
I read somewhere that someone is working on getting the scanners from MR 
jobs ability to run more then one task per region is this still pending or 
is it done?
and do we have an open issue for supporting this hadoop function for hbase?