Question about MultiFileInputFormat and conf.get( "map.input.file"), and the stability of the result

[Jason Venner <ja...@attributor.com>]

We have a lot of jobs that check the map.input.file in the configure 
method, and cause the mapper to act differently based on the file name.
We haven't started using the MultiFileInputFormat, but are wondering if 
anyone has a plan for our the map.input.file will appear in the config 
method, and or if this this becomes something you have to check in the 
mapper, rather than the configure method.


Thanks again

RE: Question about MultiFileInputFormat and conf.get( "map.input.file"), and the stability of the result

[Joydeep Sen Sarma <js...@facebook.com>]

Looking at 0.15 code - map.input.file would not work at all (it's only
set once per map task and only when 'if (instantiatedSplit instanceof
FileSplit)')

one of the things that stands out trying to do more complicated stuff in
map reduce is that the application ends up needing a global session
object (per map or reduce task). I wish there was a clean hook to set
one up - but essentially u would have to do ur own inputformat (well u
have to extend multifileinputformat anyway) and instantiate a singleton
and store the current file name in it. the map class can get the current
file name from the same singleton per map invocation ..


-----Original Message-----
From: Jason Venner [mailto:jason@attributor.com] 
Sent: Thursday, May 29, 2008 9:56 PM
To: core-user@hadoop.apache.org
Subject: Question about MultiFileInputFormat and conf.get(
"map.input.file"), and the stability of the result

We have a lot of jobs that check the map.input.file in the configure 
method, and cause the mapper to act differently based on the file name.
We haven't started using the MultiFileInputFormat, but are wondering if 
anyone has a plan for our the map.input.file will appear in the config 
method, and or if this this becomes something you have to check in the 
mapper, rather than the configure method.


Thanks again