You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by maha <ma...@umail.ucsb.edu> on 2011/03/03 03:44:38 UTC
Specify File Name to mappers
Hi,
If FileInputFormat is used with File.splitable(false) then each mapper will be getting a full file. I want the mapper to also know the path or at least name of the file it's assigned to.
Please help, any ideas are appreciated.
Thank you,
Maha
Re: Specify File Name to mappers
Posted by maha <ma...@umail.ucsb.edu>.
Actually there is a simpler way that to implement your own RecordReader which is adding the following function to your Mapper:
public void configure(JonConf job) {
mapTaskId = job.get("mapred.task.id");
inputFile = job.get("map.input.file");
}
:)
Maha
On Mar 2, 2011, at 9:03 PM, maha wrote:
> Thanks Harsh!!! but you don't think there is another way for the mappers to get configuration and access 'map.input.file' ???? because I didn't write my recordReader.
>
> I truly appreciate it.
>
> Maha
>
> On Mar 2, 2011, at 8:09 PM, Harsh J wrote:
>
>> The property 'map.input.file' should be what you're looking for. If
>> you have your own RecordReader, then you can get the Path from the
>> FileSplit object.
>>
>> On Thu, Mar 3, 2011 at 8:14 AM, maha <ma...@umail.ucsb.edu> wrote:
>>> Hi,
>>>
>>> If FileInputFormat is used with File.splitable(false) then each mapper will be getting a full file. I want the mapper to also know the path or at least name of the file it's assigned to.
>>>
>>> Please help, any ideas are appreciated.
>>>
>>> Thank you,
>>> Maha
>>>
>>>
>>
>>
>>
>> --
>> Harsh J
>> www.harshj.com
>
Re: Specify File Name to mappers
Posted by maha <ma...@umail.ucsb.edu>.
Thanks Harsh!!! but you don't think there is another way for the mappers to get configuration and access 'map.input.file' ???? because I didn't write my recordReader.
I truly appreciate it.
Maha
On Mar 2, 2011, at 8:09 PM, Harsh J wrote:
> The property 'map.input.file' should be what you're looking for. If
> you have your own RecordReader, then you can get the Path from the
> FileSplit object.
>
> On Thu, Mar 3, 2011 at 8:14 AM, maha <ma...@umail.ucsb.edu> wrote:
>> Hi,
>>
>> If FileInputFormat is used with File.splitable(false) then each mapper will be getting a full file. I want the mapper to also know the path or at least name of the file it's assigned to.
>>
>> Please help, any ideas are appreciated.
>>
>> Thank you,
>> Maha
>>
>>
>
>
>
> --
> Harsh J
> www.harshj.com
Re: Specify File Name to mappers
Posted by Harsh J <qw...@gmail.com>.
The property 'map.input.file' should be what you're looking for. If
you have your own RecordReader, then you can get the Path from the
FileSplit object.
On Thu, Mar 3, 2011 at 8:14 AM, maha <ma...@umail.ucsb.edu> wrote:
> Hi,
>
> If FileInputFormat is used with File.splitable(false) then each mapper will be getting a full file. I want the mapper to also know the path or at least name of the file it's assigned to.
>
> Please help, any ideas are appreciated.
>
> Thank you,
> Maha
>
>
--
Harsh J
www.harshj.com