You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Benjamin Hiller <Be...@urz.uni-heidelberg.de> on 2011/02/16 22:32:32 UTC
Is it possible to determine the source of a value in the Mapper?
Hi,
is it possible to determine the source (the filename for example) of a key-value pair in the mapper? What I need to do is to differentiate between two different sources, although the records of each source are of the same kind (so I can't differentiate between the sources by looking at the records). I guess I could do this by injecting some kind of tag in the RecordReader or elsewhere (which I haven't figured out yet either) but I hope there is an easier way to do this, preferably right there in the mapper.
As additional information: I haven't changed anything in the RecordReader or InputSplit yet, because I am working with text-files and it works just fine without any modification. So, if I have just missed something basic regarding the above question, it would be nice if you could point me to some information about it.
Thanks,
Ben
Re: Is it possible to determine the source of a value in the Mapper?
Posted by Benjamin Hiller <Be...@urz.uni-heidelberg.de>.
Thank you, that works fine. =)
----- Original Message -----
From: Alex Kozlov
To: mapreduce-user@hadoop.apache.org
Cc: Benjamin Hiller
Sent: Wednesday, February 16, 2011 10:37 PM
Subject: Re: Is it possible to determine the source of a value in the Mapper?
There is a way to get the file name in the new mapreduce API:
fileName = ((FileSplit) context.getInputSplit()).getPath().toString();
You usually do it in the setup() method.
On Wed, Feb 16, 2011 at 1:32 PM, Benjamin Hiller <Be...@urz.uni-heidelberg.de> wrote:
Hi,
is it possible to determine the source (the filename for example) of a key-value pair in the mapper? What I need to do is to differentiate between two different sources, although the records of each source are of the same kind (so I can't differentiate between the sources by looking at the records). I guess I could do this by injecting some kind of tag in the RecordReader or elsewhere (which I haven't figured out yet either) but I hope there is an easier way to do this, preferably right there in the mapper.
As additional information: I haven't changed anything in the RecordReader or InputSplit yet, because I am working with text-files and it works just fine without any modification. So, if I have just missed something basic regarding the above question, it would be nice if you could point me to some information about it.
Thanks,
Ben
Re: Is it possible to determine the source of a value in the Mapper?
Posted by Alex Kozlov <al...@cloudera.com>.
There is a way to get the file name in the new mapreduce API:
fileName = ((FileSplit) context.getInputSplit()).getPath().toString();
You usually do it in the setup() method.
On Wed, Feb 16, 2011 at 1:32 PM, Benjamin Hiller <
Benjamin.Hiller@urz.uni-heidelberg.de> wrote:
> Hi,
>
> is it possible to determine the source (the filename for example) of a
> key-value pair in the mapper? What I need to do is to differentiate between
> two different sources, although the records of each source are of the same
> kind (so I can't differentiate between the sources by looking at the
> records). I guess I could do this by injecting some kind of tag in the
> RecordReader or elsewhere (which I haven't figured out yet either) but I
> hope there is an easier way to do this, preferably right there in the
> mapper.
>
> As additional information: I haven't changed anything in the RecordReader
> or InputSplit yet, because I am working with text-files and it works just
> fine without any modification. So, if I have just missed something basic
> regarding the above question, it would be nice if you could point me to some
> information about it.
>
> Thanks,
> Ben
>
Re: Is it possible to determine the source of a value in the Mapper?
Posted by Harold Lim <ro...@yahoo.com>.
Hi Ben,
You can do something like this:
((FileSplit) context.getInputSplit()).getPath()
-Harold
--- On Wed, 2/16/11, Benjamin Hiller <Be...@urz.uni-heidelberg.de> wrote:
From: Benjamin Hiller <Be...@urz.uni-heidelberg.de>
Subject: Is it possible to determine the source of a value in the Mapper?
To: mapreduce-user@hadoop.apache.org
Date: Wednesday, February 16, 2011, 4:32 PM
_filtered #yiv1779397731 {
font-family:Calibri;}
_filtered #yiv1779397731 {
font-family:Tahoma;}
_filtered #yiv1779397731 {margin:70.85pt 70.85pt 70.85pt 70.85pt;}
#yiv1779397731 P.yiv1779397731MsoNormal {
MARGIN:0cm 0cm 0pt;FONT-FAMILY:"serif";FONT-SIZE:12pt;}
#yiv1779397731 LI.yiv1779397731MsoNormal {
MARGIN:0cm 0cm 0pt;FONT-FAMILY:"serif";FONT-SIZE:12pt;}
#yiv1779397731 DIV.yiv1779397731MsoNormal {
MARGIN:0cm 0cm 0pt;FONT-FAMILY:"serif";FONT-SIZE:12pt;}
#yiv1779397731 A:link {
COLOR:blue;TEXT-DECORATION:underline;}
#yiv1779397731 SPAN.yiv1779397731MsoHyperlink {
COLOR:blue;TEXT-DECORATION:underline;}
#yiv1779397731 A:visited {
COLOR:purple;TEXT-DECORATION:underline;}
#yiv1779397731 SPAN.yiv1779397731MsoHyperlinkFollowed {
COLOR:purple;TEXT-DECORATION:underline;}
#yiv1779397731 SPAN.yiv1779397731EmailStyle17 {
FONT-FAMILY:"sans-serif";COLOR:#1f497d;}
#yiv1779397731 .yiv1779397731MsoChpDefault {
}
#yiv1779397731 DIV.yiv1779397731Section1 {
}
Hi,
is it possible to determine the source (the
filename for example) of a key-value pair in the mapper? What I need to do is to
differentiate between two different sources, although the records of each source
are of the same kind (so I can't differentiate between the sources by looking at
the records). I guess I could do this by injecting some kind of tag in the
RecordReader or elsewhere (which I haven't figured out yet either) but I hope
there is an easier way to do this, preferably right there in the
mapper.
As additional information: I haven't changed
anything in the RecordReader or InputSplit yet, because I am working with
text-files and it works just fine without any modification. So, if I have just
missed something basic regarding the above question, it would be nice if you
could point me to some information about it.
Thanks,
Ben