You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by edward choi <mp...@gmail.com> on 2011/06/05 13:04:20 UTC

How to split a specified number of rows per Map

Hi,

I am using HBase as a source of my MapReduce jobs.

I recently found out that TableInputFormat automatically splits the input
table so that each region of the table will be assigned to a single Map job.

But what I want to do is to split the input table so that user-specified
lines of row will be assigned to each Mapper.

For example, if I set a certain parameter to 100, then each Mapper will get
100 lines from the input Table.

Is there a method for this kind of operation?
Or do I have to modify the getSplits() of
org.apache.hadoop.hbase.mapreduce.TableInputFormatBase?

Any answer or opinion will be much appreciated!!

Ed

Re: How to split a specified number of rows per Map

Posted by edward choi <mp...@gmail.com>.
I guess there are no other options.

Thanks for the info Ted.

Ed.

2011/6/5 Ted Yu <yu...@gmail.com>

> You need to modify getSplits().
>
> On Sun, Jun 5, 2011 at 4:04 AM, edward choi <mp...@gmail.com> wrote:
>
> > Hi,
> >
> > I am using HBase as a source of my MapReduce jobs.
> >
> > I recently found out that TableInputFormat automatically splits the input
> > table so that each region of the table will be assigned to a single Map
> > job.
> >
> > But what I want to do is to split the input table so that user-specified
> > lines of row will be assigned to each Mapper.
> >
> > For example, if I set a certain parameter to 100, then each Mapper will
> get
> > 100 lines from the input Table.
> >
> > Is there a method for this kind of operation?
> > Or do I have to modify the getSplits() of
> > org.apache.hadoop.hbase.mapreduce.TableInputFormatBase?
> >
> > Any answer or opinion will be much appreciated!!
> >
> > Ed
> >
>

Re: How to split a specified number of rows per Map

Posted by Ted Yu <yu...@gmail.com>.
You need to modify getSplits().

On Sun, Jun 5, 2011 at 4:04 AM, edward choi <mp...@gmail.com> wrote:

> Hi,
>
> I am using HBase as a source of my MapReduce jobs.
>
> I recently found out that TableInputFormat automatically splits the input
> table so that each region of the table will be assigned to a single Map
> job.
>
> But what I want to do is to split the input table so that user-specified
> lines of row will be assigned to each Mapper.
>
> For example, if I set a certain parameter to 100, then each Mapper will get
> 100 lines from the input Table.
>
> Is there a method for this kind of operation?
> Or do I have to modify the getSplits() of
> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase?
>
> Any answer or opinion will be much appreciated!!
>
> Ed
>