You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Mike Spreitzer <ms...@us.ibm.com> on 2012/02/12 21:58:43 UTC
Can caller mutate value after calling OutputCollector.collect(key,value) ?
I have a question about the contract in the org.apache.hadoop.mapred
.OutputCollector interface. If it matters, let us say we are talking
about Hadoop-1.0.0. In my map or reduce method, after it calls
output.collect(key,value), is it allowed to mutate (side effect) either
the key or the value?
If the answer is "it depends on the OutputFormat", then what is the answer
for the more prominent ones, such as SequenceFileFormat and
TextOutputFormat?
Thanks,
Mike
Re: What happes if the caller mutates key or value after calling
OutputCollector.collect(key,value) ?
Posted by Harsh J <ha...@cloudera.com>.
Mike,
On Mon, Feb 13, 2012 at 10:31 AM, Mike Spreitzer <ms...@us.ibm.com> wrote:
> Ah, so my question was not clear. Let me try again. Suppose my map or
> reduce method invokes output.collect(key, value) and then, after that
> returns, mutates (side effects) either the key or the value --- what
> happens? Is this specified, forbidden, or unspecified? Is there a general
> answer, or is this up to the OutputFormat?
This is upto your RecordWriter. If you accumulate all passed key/value
objects as references to serialize later, you will run into issues
doing what you're talking about. The better thing would be to clone.
But all of the Hadoop provided implementations serialize, or create a
new object immediately upon call, and hence remain unaffected from
key/value object mutations at source.
> I think you are saying that for SequenceFileOutputFormat and
> TextOutputFormat, the scenario I outlined is allowed and the original values
> appear in the output. Have I got that right? If so, what remains to be
> answered then is whether this is something I have to answer for each
> OutputFormat or there is a general rule here.
OutputFormat and RecordWriters are mere interfaces to prepare and
connect you to the FS. You are free to write their implementations as
you want to -- the framework does not get in your way here.
> Suppose, in particular, that my map or reduce method calls
> output.collect(key,value) several times in series --- each time passing the
> same object reference for key, and each time passing the same object
> reference for value, but modifying those objects between calls on
> output.collect. I would like to know if this is a supported scenario, with
> the semantics that what is output is the contents of the key and value
> objects at the moment output.collect(key,value) is called.
My first para explains this - every call to your record writer gets
serialized with the values passed in - so yes, you can call that a
supported scenario.
> Have I overlooked some documentation that answers my question?
Perhaps. Could you please identify the spot where docs can carry this
for programmers (RecordWriter API) and file an issue on JIRA with that
knowledge (and possibly a patch)? Thanks! :)
--
Harsh J
Customer Ops. Engineer
Cloudera | http://tiny.cloudera.com/about
What happes if the caller mutates key or value after calling
OutputCollector.collect(key,value) ?
Posted by Mike Spreitzer <ms...@us.ibm.com>.
Ah, so my question was not clear. Let me try again. Suppose my map or
reduce method invokes output.collect(key, value) and then, after that
returns, mutates (side effects) either the key or the value --- what
happens? Is this specified, forbidden, or unspecified? Is there a
general answer, or is this up to the OutputFormat?
I think you are saying that for SequenceFileOutputFormat and
TextOutputFormat, the scenario I outlined is allowed and the original
values appear in the output. Have I got that right? If so, what remains
to be answered then is whether this is something I have to answer for each
OutputFormat or there is a general rule here.
Suppose, in particular, that my map or reduce method calls
output.collect(key,value) several times in series --- each time passing
the same object reference for key, and each time passing the same object
reference for value, but modifying those objects between calls on
output.collect. I would like to know if this is a supported scenario,
with the semantics that what is output is the contents of the key and
value objects at the moment output.collect(key,value) is called.
Have I overlooked some documentation that answers my question?
Thanks,
Mike
From: Brock Noland <br...@cloudera.com>
To: mapreduce-user@hadoop.apache.org
Date: 02/12/2012 07:16 PM
Subject: Re: Can caller mutate value after calling
OutputCollector.collect(key,value) ?
Hi
On Sun, Feb 12, 2012 at 8:58 PM, Mike Spreitzer <ms...@us.ibm.com>
wrote:
> I have a question about the contract in the
> org.apache.hadoop.mapred.OutputCollector interface. If it matters, let
us
> say we are talking about Hadoop-1.0.0. In my map or reduce method,
after it
> calls output.collect(key,value), is it allowed to mutate (side effect)
> either the key or the value?
>
> If the answer is "it depends on the OutputFormat", then what is the
answer
> for the more prominent ones, such as SequenceFileFormat and
> TextOutputFormat?
In both cases the key is serialized immediately so the caller cannot
mutate the key after calling collect.
http://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0/src/mapred/org/apache/hadoop/mapred/TextOutputFormat.java
http://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0/src/mapred/org/apache/hadoop/mapred/SequenceFileOutputFormat.java
http://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0/src/core/org/apache/hadoop/io/SequenceFile.java
--
Apache MRUnit - Unit testing MapReduce -
http://incubator.apache.org/mrunit/
Re: Can caller mutate value after calling OutputCollector.collect(key,value)
?
Posted by Brock Noland <br...@cloudera.com>.
Hi
On Sun, Feb 12, 2012 at 8:58 PM, Mike Spreitzer <ms...@us.ibm.com> wrote:
> I have a question about the contract in the
> org.apache.hadoop.mapred.OutputCollector interface. If it matters, let us
> say we are talking about Hadoop-1.0.0. In my map or reduce method, after it
> calls output.collect(key,value), is it allowed to mutate (side effect)
> either the key or the value?
>
> If the answer is "it depends on the OutputFormat", then what is the answer
> for the more prominent ones, such as SequenceFileFormat and
> TextOutputFormat?
In both cases the key is serialized immediately so the caller cannot
mutate the key after calling collect.
http://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0/src/mapred/org/apache/hadoop/mapred/TextOutputFormat.java
http://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0/src/mapred/org/apache/hadoop/mapred/SequenceFileOutputFormat.java
http://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0/src/core/org/apache/hadoop/io/SequenceFile.java
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/