You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by xeonmailinglist-gmail <xe...@gmail.com> on 2015/03/09 16:39:48 UTC
which file map and reduce in reading or writing?
Hi,
I am looking to YARN MapReduce internals, and I would like know if it
possible to know which file a map/reduce function is reading or writing
from inside a map or reduce function defined by the user, or simply by
the client?
Thanks,
--
--
Re: which file map and reduce in reading or writing?
Posted by xeonmailinglist-gmail <xe...@gmail.com>.
Reducer doesn't have the ((FileSplit) context.getInputSplit()). Just the
mapper.
I would like to know from java which files a reducer is reading, and to
where it is writing. Can I do this?
Thanks,
On 09-03-2015 15:44, Kai Voigt wrote:
> Hi,
>
> the context object (passed to map(), reduce() and setup()) contains
> information about the input split, such as the file name.
>
> From the top of my head: String fileName = ((FileSplit)
> context.getInputSplit()).getPath().getName();
>
> Kai
>
>> Am 09.03.2015 um 16:39 schrieb xeonmailinglist-gmail
>> <xeonmailinglist@gmail.com <ma...@gmail.com>>:
>>
>> Hi,
>>
>> I am looking to YARN MapReduce internals, and I would like know if it
>> possible to know which file a map/reduce function is reading or
>> writing from inside a map or reduce function defined by the user, or
>> simply by the client?
>>
>>
>> Thanks,
>>
>> --
>> --
>>
>>
>
> ------------------------------------------------------------------------
> *Kai Voigt*Am Germaniahafen 1k@123.org <ma...@123.org>
> 24143 Kiel+49 160 96683050
> Germany@KaiVoigt
>
--
--
Re: which file map and reduce in reading or writing?
Posted by xeonmailinglist-gmail <xe...@gmail.com>.
Reducer doesn't have the ((FileSplit) context.getInputSplit()). Just the
mapper.
I would like to know from java which files a reducer is reading, and to
where it is writing. Can I do this?
Thanks,
On 09-03-2015 15:44, Kai Voigt wrote:
> Hi,
>
> the context object (passed to map(), reduce() and setup()) contains
> information about the input split, such as the file name.
>
> From the top of my head: String fileName = ((FileSplit)
> context.getInputSplit()).getPath().getName();
>
> Kai
>
>> Am 09.03.2015 um 16:39 schrieb xeonmailinglist-gmail
>> <xeonmailinglist@gmail.com <ma...@gmail.com>>:
>>
>> Hi,
>>
>> I am looking to YARN MapReduce internals, and I would like know if it
>> possible to know which file a map/reduce function is reading or
>> writing from inside a map or reduce function defined by the user, or
>> simply by the client?
>>
>>
>> Thanks,
>>
>> --
>> --
>>
>>
>
> ------------------------------------------------------------------------
> *Kai Voigt*Am Germaniahafen 1k@123.org <ma...@123.org>
> 24143 Kiel+49 160 96683050
> Germany@KaiVoigt
>
--
--
Re: which file map and reduce in reading or writing?
Posted by xeonmailinglist-gmail <xe...@gmail.com>.
Reducer doesn't have the ((FileSplit) context.getInputSplit()). Just the
mapper.
I would like to know from java which files a reducer is reading, and to
where it is writing. Can I do this?
Thanks,
On 09-03-2015 15:44, Kai Voigt wrote:
> Hi,
>
> the context object (passed to map(), reduce() and setup()) contains
> information about the input split, such as the file name.
>
> From the top of my head: String fileName = ((FileSplit)
> context.getInputSplit()).getPath().getName();
>
> Kai
>
>> Am 09.03.2015 um 16:39 schrieb xeonmailinglist-gmail
>> <xeonmailinglist@gmail.com <ma...@gmail.com>>:
>>
>> Hi,
>>
>> I am looking to YARN MapReduce internals, and I would like know if it
>> possible to know which file a map/reduce function is reading or
>> writing from inside a map or reduce function defined by the user, or
>> simply by the client?
>>
>>
>> Thanks,
>>
>> --
>> --
>>
>>
>
> ------------------------------------------------------------------------
> *Kai Voigt*Am Germaniahafen 1k@123.org <ma...@123.org>
> 24143 Kiel+49 160 96683050
> Germany@KaiVoigt
>
--
--
Re: which file map and reduce in reading or writing?
Posted by xeonmailinglist-gmail <xe...@gmail.com>.
Reducer doesn't have the ((FileSplit) context.getInputSplit()). Just the
mapper.
I would like to know from java which files a reducer is reading, and to
where it is writing. Can I do this?
Thanks,
On 09-03-2015 15:44, Kai Voigt wrote:
> Hi,
>
> the context object (passed to map(), reduce() and setup()) contains
> information about the input split, such as the file name.
>
> From the top of my head: String fileName = ((FileSplit)
> context.getInputSplit()).getPath().getName();
>
> Kai
>
>> Am 09.03.2015 um 16:39 schrieb xeonmailinglist-gmail
>> <xeonmailinglist@gmail.com <ma...@gmail.com>>:
>>
>> Hi,
>>
>> I am looking to YARN MapReduce internals, and I would like know if it
>> possible to know which file a map/reduce function is reading or
>> writing from inside a map or reduce function defined by the user, or
>> simply by the client?
>>
>>
>> Thanks,
>>
>> --
>> --
>>
>>
>
> ------------------------------------------------------------------------
> *Kai Voigt*Am Germaniahafen 1k@123.org <ma...@123.org>
> 24143 Kiel+49 160 96683050
> Germany@KaiVoigt
>
--
--
Re: which file map and reduce in reading or writing?
Posted by Kai Voigt <k...@123.org>.
Hi,
the context object (passed to map(), reduce() and setup()) contains information about the input split, such as the file name.
From the top of my head: String fileName = ((FileSplit) context.getInputSplit()).getPath().getName();
Kai
> Am 09.03.2015 um 16:39 schrieb xeonmailinglist-gmail <xe...@gmail.com>:
>
> Hi,
>
> I am looking to YARN MapReduce internals, and I would like know if it possible to know which file a map/reduce function is reading or writing from inside a map or reduce function defined by the user, or simply by the client?
>
>
> Thanks,
>
> --
> --
>
>
Kai Voigt Am Germaniahafen 1 k@123.org
24143 Kiel +49 160 96683050
Germany @KaiVoigt
Re: which file map and reduce in reading or writing?
Posted by Kai Voigt <k...@123.org>.
Hi,
the context object (passed to map(), reduce() and setup()) contains information about the input split, such as the file name.
From the top of my head: String fileName = ((FileSplit) context.getInputSplit()).getPath().getName();
Kai
> Am 09.03.2015 um 16:39 schrieb xeonmailinglist-gmail <xe...@gmail.com>:
>
> Hi,
>
> I am looking to YARN MapReduce internals, and I would like know if it possible to know which file a map/reduce function is reading or writing from inside a map or reduce function defined by the user, or simply by the client?
>
>
> Thanks,
>
> --
> --
>
>
Kai Voigt Am Germaniahafen 1 k@123.org
24143 Kiel +49 160 96683050
Germany @KaiVoigt
Re: which file map and reduce in reading or writing?
Posted by Kai Voigt <k...@123.org>.
Hi,
the context object (passed to map(), reduce() and setup()) contains information about the input split, such as the file name.
From the top of my head: String fileName = ((FileSplit) context.getInputSplit()).getPath().getName();
Kai
> Am 09.03.2015 um 16:39 schrieb xeonmailinglist-gmail <xe...@gmail.com>:
>
> Hi,
>
> I am looking to YARN MapReduce internals, and I would like know if it possible to know which file a map/reduce function is reading or writing from inside a map or reduce function defined by the user, or simply by the client?
>
>
> Thanks,
>
> --
> --
>
>
Kai Voigt Am Germaniahafen 1 k@123.org
24143 Kiel +49 160 96683050
Germany @KaiVoigt
Re: which file map and reduce in reading or writing?
Posted by Kai Voigt <k...@123.org>.
Hi,
the context object (passed to map(), reduce() and setup()) contains information about the input split, such as the file name.
From the top of my head: String fileName = ((FileSplit) context.getInputSplit()).getPath().getName();
Kai
> Am 09.03.2015 um 16:39 schrieb xeonmailinglist-gmail <xe...@gmail.com>:
>
> Hi,
>
> I am looking to YARN MapReduce internals, and I would like know if it possible to know which file a map/reduce function is reading or writing from inside a map or reduce function defined by the user, or simply by the client?
>
>
> Thanks,
>
> --
> --
>
>
Kai Voigt Am Germaniahafen 1 k@123.org
24143 Kiel +49 160 96683050
Germany @KaiVoigt