NLineInputFormat and very high number of maptasks

[Saptarshi Guha <sa...@gmail.com>]

Hello,
When I use NLIneInputFormat, when I output:
	System.out.println("mapred.map.tasks:"+jobConf.get("mapred.map.tasks"));
I see 51, but on the jobtracker site, the number is 18114. Yet with
TextInputFormat it shows 51.
I'm using Hadoop - 0.19

Any ideas why?
Regards
Saptarshi

-- 
Saptarshi Guha - saptarshi.guha@gmail.com

Re: NLineInputFormat and very high number of maptasks

[Amareshwari Sriramadasu <am...@yahoo-inc.com>]

Saptarshi Guha wrote:
> Sorry, i see - every line is now a maptask - one split,one task.(in 
> this case N=1 line per split)
> Is that correct?
> Saptarshi
>
You are right. NLineInputFormat splits N lines of input as one split and 
each split is given to a map task.
By default, N is 1. N can configured through the property 
"mapred.line.input.format.linespermap".
> On Jan 20, 2009, at 11:39 AM, Saptarshi Guha wrote:
>
>> Hello,
>> When I use NLIneInputFormat, when I output:
>>     System.out.println("mapred.map.tasks:"+jobConf.get("mapred.map.tasks")); 
>>
Where are you printing this statement? Looks like the JobConf, that you 
are looking at, is not set with the correct value of number of map tasks 
yet.
>> I see 51, but on the jobtracker site, the number is 18114. Yet with
>> TextInputFormat it shows 51.
>> I'm using Hadoop - 0.19
>>
>> Any ideas why?
>> Regards
>> Saptarshi
>>
>> --Saptarshi Guha - saptarshi.guha@gmail.com
>
> Saptarshi Guha | saptarshi.guha@gmail.com | 
> http://www.stat.purdue.edu/~sguha
> If the church put in half the time on covetousness that it does on lust,
> this would be a better world.
>         -- Garrison Keillor, "Lake Wobegon Days"
>

-Amareshwari

Re: NLineInputFormat and very high number of maptasks

[Saptarshi Guha <sa...@gmail.com>]

Sorry, i see - every line is now a maptask - one split,one task.(in  
this case N=1 line per split)
Is that correct?
Saptarshi

On Jan 20, 2009, at 11:39 AM, Saptarshi Guha wrote:

> Hello,
> When I use NLIneInputFormat, when I output:
> 	 
> System 
> .out.println("mapred.map.tasks:"+jobConf.get("mapred.map.tasks"));
> I see 51, but on the jobtracker site, the number is 18114. Yet with
> TextInputFormat it shows 51.
> I'm using Hadoop - 0.19
>
> Any ideas why?
> Regards
> Saptarshi
>
> -- 
> Saptarshi Guha - saptarshi.guha@gmail.com

Saptarshi Guha | saptarshi.guha@gmail.com | http://www.stat.purdue.edu/~sguha
If the church put in half the time on covetousness that it does on lust,
this would be a better world.
		-- Garrison Keillor, "Lake Wobegon Days"