You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Matthieu Labour <ma...@actionx.com> on 2013/07/30 22:08:11 UTC
Can a mapper forward its input text or does it have to copy it?
Hi
When a mapper does nothing, can it forward the input text or does it have
to allocate / deep copy it to a new Text?
Which one is the recommended between example#1 and example#2?
>From reading the documentation, I am under the impression that example#1 is
NOT correct as the input text reference is not immutable and can change. Am
I wrong?
Thank you for your help.
example#1:
static public class MySimpleMapper extends
Mapper<LongWritable, Text, Text, NullWritable> {
private final static Text MYNEWINPUTTEXT = new Text();
@Override
protected void map(LongWritable offset, Text text, Context context)
throws IOException, InterruptedException {
MYNEWINPUTTEXT.set(text);
context.write(MYNEWINPUTTEXT, NullWritable.get());
}
}
example#2:
static public class MySimpleMapper extends
Mapper<LongWritable, Text, Text, NullWritable> {
private final static Text MYNEWINPUTTEXT = new Text();
@Override
protected void map(LongWritable offset, Text text, Context context)
throws IOException, InterruptedException {
context.write(text, NullWritable.get());
}
}
Re: Can a mapper forward its input text or does it have to copy it?
Posted by Harsh J <ha...@cloudera.com>.
You can simply do the latter (example#2). The example#1 would work
too, but seems unnecessary.
On Wed, Jul 31, 2013 at 1:38 AM, Matthieu Labour <ma...@actionx.com> wrote:
>
> Hi
>
> When a mapper does nothing, can it forward the input text or does it have to
> allocate / deep copy it to a new Text?
>
> Which one is the recommended between example#1 and example#2?
>
> From reading the documentation, I am under the impression that example#1 is
> NOT correct as the input text reference is not immutable and can change. Am
> I wrong?
>
> Thank you for your help.
>
> example#1:
> static public class MySimpleMapper extends
> Mapper<LongWritable, Text, Text, NullWritable> {
> private final static Text MYNEWINPUTTEXT = new Text();
> @Override
> protected void map(LongWritable offset, Text text, Context context)
> throws IOException, InterruptedException {
> MYNEWINPUTTEXT.set(text);
> context.write(MYNEWINPUTTEXT, NullWritable.get());
> }
> }
>
> example#2:
> static public class MySimpleMapper extends
> Mapper<LongWritable, Text, Text, NullWritable> {
> private final static Text MYNEWINPUTTEXT = new Text();
> @Override
> protected void map(LongWritable offset, Text text, Context context)
> throws IOException, InterruptedException {
> context.write(text, NullWritable.get());
> }
> }
--
Harsh J
Re: Can a mapper forward its input text or does it have to copy it?
Posted by Harsh J <ha...@cloudera.com>.
You can simply do the latter (example#2). The example#1 would work
too, but seems unnecessary.
On Wed, Jul 31, 2013 at 1:38 AM, Matthieu Labour <ma...@actionx.com> wrote:
>
> Hi
>
> When a mapper does nothing, can it forward the input text or does it have to
> allocate / deep copy it to a new Text?
>
> Which one is the recommended between example#1 and example#2?
>
> From reading the documentation, I am under the impression that example#1 is
> NOT correct as the input text reference is not immutable and can change. Am
> I wrong?
>
> Thank you for your help.
>
> example#1:
> static public class MySimpleMapper extends
> Mapper<LongWritable, Text, Text, NullWritable> {
> private final static Text MYNEWINPUTTEXT = new Text();
> @Override
> protected void map(LongWritable offset, Text text, Context context)
> throws IOException, InterruptedException {
> MYNEWINPUTTEXT.set(text);
> context.write(MYNEWINPUTTEXT, NullWritable.get());
> }
> }
>
> example#2:
> static public class MySimpleMapper extends
> Mapper<LongWritable, Text, Text, NullWritable> {
> private final static Text MYNEWINPUTTEXT = new Text();
> @Override
> protected void map(LongWritable offset, Text text, Context context)
> throws IOException, InterruptedException {
> context.write(text, NullWritable.get());
> }
> }
--
Harsh J
Re: Can a mapper forward its input text or does it have to copy it?
Posted by Harsh J <ha...@cloudera.com>.
You can simply do the latter (example#2). The example#1 would work
too, but seems unnecessary.
On Wed, Jul 31, 2013 at 1:38 AM, Matthieu Labour <ma...@actionx.com> wrote:
>
> Hi
>
> When a mapper does nothing, can it forward the input text or does it have to
> allocate / deep copy it to a new Text?
>
> Which one is the recommended between example#1 and example#2?
>
> From reading the documentation, I am under the impression that example#1 is
> NOT correct as the input text reference is not immutable and can change. Am
> I wrong?
>
> Thank you for your help.
>
> example#1:
> static public class MySimpleMapper extends
> Mapper<LongWritable, Text, Text, NullWritable> {
> private final static Text MYNEWINPUTTEXT = new Text();
> @Override
> protected void map(LongWritable offset, Text text, Context context)
> throws IOException, InterruptedException {
> MYNEWINPUTTEXT.set(text);
> context.write(MYNEWINPUTTEXT, NullWritable.get());
> }
> }
>
> example#2:
> static public class MySimpleMapper extends
> Mapper<LongWritable, Text, Text, NullWritable> {
> private final static Text MYNEWINPUTTEXT = new Text();
> @Override
> protected void map(LongWritable offset, Text text, Context context)
> throws IOException, InterruptedException {
> context.write(text, NullWritable.get());
> }
> }
--
Harsh J
Re: Can a mapper forward its input text or does it have to copy it?
Posted by Harsh J <ha...@cloudera.com>.
You can simply do the latter (example#2). The example#1 would work
too, but seems unnecessary.
On Wed, Jul 31, 2013 at 1:38 AM, Matthieu Labour <ma...@actionx.com> wrote:
>
> Hi
>
> When a mapper does nothing, can it forward the input text or does it have to
> allocate / deep copy it to a new Text?
>
> Which one is the recommended between example#1 and example#2?
>
> From reading the documentation, I am under the impression that example#1 is
> NOT correct as the input text reference is not immutable and can change. Am
> I wrong?
>
> Thank you for your help.
>
> example#1:
> static public class MySimpleMapper extends
> Mapper<LongWritable, Text, Text, NullWritable> {
> private final static Text MYNEWINPUTTEXT = new Text();
> @Override
> protected void map(LongWritable offset, Text text, Context context)
> throws IOException, InterruptedException {
> MYNEWINPUTTEXT.set(text);
> context.write(MYNEWINPUTTEXT, NullWritable.get());
> }
> }
>
> example#2:
> static public class MySimpleMapper extends
> Mapper<LongWritable, Text, Text, NullWritable> {
> private final static Text MYNEWINPUTTEXT = new Text();
> @Override
> protected void map(LongWritable offset, Text text, Context context)
> throws IOException, InterruptedException {
> context.write(text, NullWritable.get());
> }
> }
--
Harsh J