You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Mark question <ma...@gmail.com> on 2011/06/07 04:31:17 UTC

Reducing Mapper InputSplit size

Hi,

Does anyone have a way to reduce InputSplit size in general ?

By default, the minimum size chunk that map input should be split into is
set to 0 (ie.mapred.min.split.size). Can I change dfs.block.size or some
other configuration to reduce the split size and spawn many mappers?

Thanks,
Mark

Re: Reducing Mapper InputSplit size

Posted by Mark question <ma...@gmail.com>.
Great! Thanks guys :)
Mark

2011/6/6 Panayotis Antonopoulos <an...@hotmail.com>

>
> Hi Mark,
>
> Check:
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.html
>
> I think that setMaxInputSplitSize(Job job,
>                     long size)
>
>
> will do what you need.
>
> Regards,
> P.A.
>
> > Date: Mon, 6 Jun 2011 19:31:17 -0700
> > Subject: Reducing Mapper InputSplit size
> > From: markq2011@gmail.com
> > To: common-user@hadoop.apache.org
> >
> > Hi,
> >
> > Does anyone have a way to reduce InputSplit size in general ?
> >
> > By default, the minimum size chunk that map input should be split into is
> > set to 0 (ie.mapred.min.split.size). Can I change dfs.block.size or some
> > other configuration to reduce the split size and spawn many mappers?
> >
> > Thanks,
> > Mark
>
>

RE: Reducing Mapper InputSplit size

Posted by Panayotis Antonopoulos <an...@hotmail.com>.
Hi Mark,

Check: http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.html

I think that setMaxInputSplitSize(Job job,
                     long size)


will do what you need.

Regards,
P.A.

> Date: Mon, 6 Jun 2011 19:31:17 -0700
> Subject: Reducing Mapper InputSplit size
> From: markq2011@gmail.com
> To: common-user@hadoop.apache.org
> 
> Hi,
> 
> Does anyone have a way to reduce InputSplit size in general ?
> 
> By default, the minimum size chunk that map input should be split into is
> set to 0 (ie.mapred.min.split.size). Can I change dfs.block.size or some
> other configuration to reduce the split size and spawn many mappers?
> 
> Thanks,
> Mark
 		 	   		  

Re: Reducing Mapper InputSplit size

Posted by jagaran das <ja...@yahoo.co.in>.
Correct reduce the dfs.block.size to increase the number of mappers.

- Jagaran 



________________________________
From: Mark question <ma...@gmail.com>
To: common-user <co...@hadoop.apache.org>
Sent: Mon, 6 June, 2011 7:31:17 PM
Subject: Reducing Mapper InputSplit size

Hi,

Does anyone have a way to reduce InputSplit size in general ?

By default, the minimum size chunk that map input should be split into is
set to 0 (ie.mapred.min.split.size). Can I change dfs.block.size or some
other configuration to reduce the split size and spawn many mappers?

Thanks,
Mark