You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by "Berry, Matt" <mw...@amazon.com> on 2012/07/03 23:16:16 UTC
Null Pointer Excpetion while writing Key on Mapper emit
While emitting a record from my mapper, I am receiving a NullPointerException. The stack trace seems to indicate there is a problem serializing the Key. The key contains a few strings and a few longs. It is a string that it is having trouble serializing. I've checked the contents of the string post-mortem and it looks like a valid string (at least, other strings like it work just fine). Could this be a problem not with the string itself but the destination of the arraycopy()?
The Key.write(DataOutput out) function:
@Override
public void write(DataOutput out) throws IOException {
if(StringOne.equals(null) | StringTwo.equals(null)) {
System.out.println("This is not hit");
}
out.writeUTF(StringOne); //Line 125
out.writeUTF(StringTwo);
out.writeLong(LongOne);
out.writeLong(LongTwo);
}
The Stack Trace:
java.lang.NullPointerException
at java.lang.System.arraycopy(Native Method)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$Buffer.write(MapTask.java:1108)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at java.io.DataOutputStream.writeUTF(DataOutputStream.java:384)
at java.io.DataOutputStream.writeUTF(DataOutputStream.java:306)
at xxxxxx.xxxxxxxxxxxx.xxxx.xxxxx.xxxxxKey.write(xxxxxKey.java:125)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:90)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:77)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:892)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:541)
at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at xxxxxx.xxxxxxxxxxx.xxxx.xxxxxxxx.xxx.xxxxxxxxxxxxxxxxxxxxxxx.map(xxxxxxxxxxxxxxxxxMapper.java:82)
at xxxxxx.xxxxxxxxxxx.xxxx.xxxxxxxx.xxx.xxxxxxxxxxxxxxxxxxxxxxx.map(xxxxxxxxxxxxxxxxxMapper.java:63)
at xxxxxx.xxxxxxxxxxx.xxxx.xxxxxxx.xxxxxxxxxxxxxxxxxxxxxxxxxxxx.xxxxxxxxxx(xxxxxxxxxxxxxxxxxxxxxxxxxxxx.java:131)
at xxxxxx.xxxxxxxxxxx.xxxx.xxxxx.xxxxxxxxxxxxxx.xxxxxxxxxxxxxxx(xxxxxxxxxxxxxx.java:282)
at xxxxxx.xxxxxxxxxxx.xxxx.xxxxx.xxxxxxxxxxxxxx.access$000(xxxxxxxxxxxxxx.java:42)
at xxxxxx.xxxxxxxxxxx.xxxx.xxxxx.xxxxxxxxxxxxxx$xxxxxxxxxxxx.run(xxxxxxxxxxxxx.java:74)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
RE: Null Pointer Excpetion while writing Key on Mapper emit
Posted by "Berry, Matt" <mw...@amazon.com>.
Thanks for the response Anand,
The NullPointerException was by design, I wanted to illustrate that it was not null. Your suspicion about having multiple threads was correct. I traced the error down to the threadpool terminating prior to the last mapper completing its operations.
Sincerely,
Matthew Berry
-----Original Message-----
From: Anand Srivastava [mailto:Anand.Srivastava@guavus.com]
Sent: Tuesday, July 03, 2012 8:31 PM
To: <ma...@hadoop.apache.org>
Subject: Re: Null Pointer Excpetion while writing Key on Mapper emit
Hi Matt,
You StringOne is clearly not null else it would have thrown an exception in the if. You should not be calling equals on String unless you are sure that it is not null; use == instead for nullity checks. This means that one of 'kvbuffer' or 'b' (byte[] argument) to be null.
Mind telling us the hadoop version you are using?
Any chance that there are two or more threads in your mapper class?
Regards,
Anand
On 04-Jul-2012, at 2:46 AM, Berry, Matt wrote:
> While emitting a record from my mapper, I am receiving a NullPointerException. The stack trace seems to indicate there is a problem serializing the Key. The key contains a few strings and a few longs. It is a string that it is having trouble serializing. I've checked the contents of the string post-mortem and it looks like a valid string (at least, other strings like it work just fine). Could this be a problem not with the string itself but the destination of the arraycopy()?
>
> The Key.write(DataOutput out) function:
>
> @Override
> public void write(DataOutput out) throws IOException {
> if(StringOne.equals(null) | StringTwo.equals(null)) {
> System.out.println("This is not hit");
> }
> out.writeUTF(StringOne); //Line 125
> out.writeUTF(StringTwo);
> out.writeLong(LongOne);
> out.writeLong(LongTwo);
> }
>
>
> The Stack Trace:
>
> java.lang.NullPointerException
> at java.lang.System.arraycopy(Native Method)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$Buffer.write(MapTask.java:1108)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at java.io.DataOutputStream.writeUTF(DataOutputStream.java:384)
> at java.io.DataOutputStream.writeUTF(DataOutputStream.java:306)
> at xxxxxx.xxxxxxxxxxxx.xxxx.xxxxx.xxxxxKey.write(xxxxxKey.java:125)
> at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:90)
> at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:77)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:892)
> at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:541)
> at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> at xxxxxx.xxxxxxxxxxx.xxxx.xxxxxxxx.xxx.xxxxxxxxxxxxxxxxxxxxxxx.map(xxxxxxxxxxxxxxxxxMapper.java:82)
> at xxxxxx.xxxxxxxxxxx.xxxx.xxxxxxxx.xxx.xxxxxxxxxxxxxxxxxxxxxxx.map(xxxxxxxxxxxxxxxxxMapper.java:63)
> at xxxxxx.xxxxxxxxxxx.xxxx.xxxxxxx.xxxxxxxxxxxxxxxxxxxxxxxxxxxx.xxxxxxxxxx(xxxxxxxxxxxxxxxxxxxxxxxxxxxx.java:131)
> at xxxxxx.xxxxxxxxxxx.xxxx.xxxxx.xxxxxxxxxxxxxx.xxxxxxxxxxxxxxx(xxxxxxxxxxxxxx.java:282)
> at xxxxxx.xxxxxxxxxxx.xxxx.xxxxx.xxxxxxxxxxxxxx.access$000(xxxxxxxxxxxxxx.java:42)
> at xxxxxx.xxxxxxxxxxx.xxxx.xxxxx.xxxxxxxxxxxxxx$xxxxxxxxxxxx.run(xxxxxxxxxxxxx.java:74)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
Re: Null Pointer Excpetion while writing Key on Mapper emit
Posted by Anand Srivastava <An...@guavus.com>.
Hi Matt,
You StringOne is clearly not null else it would have thrown an exception in the if. You should not be calling equals on String unless you are sure that it is not null; use == instead for nullity checks. This means that one of 'kvbuffer' or 'b' (byte[] argument) to be null.
Mind telling us the hadoop version you are using?
Any chance that there are two or more threads in your mapper class?
Regards,
Anand
On 04-Jul-2012, at 2:46 AM, Berry, Matt wrote:
> While emitting a record from my mapper, I am receiving a NullPointerException. The stack trace seems to indicate there is a problem serializing the Key. The key contains a few strings and a few longs. It is a string that it is having trouble serializing. I've checked the contents of the string post-mortem and it looks like a valid string (at least, other strings like it work just fine). Could this be a problem not with the string itself but the destination of the arraycopy()?
>
> The Key.write(DataOutput out) function:
>
> @Override
> public void write(DataOutput out) throws IOException {
> if(StringOne.equals(null) | StringTwo.equals(null)) {
> System.out.println("This is not hit");
> }
> out.writeUTF(StringOne); //Line 125
> out.writeUTF(StringTwo);
> out.writeLong(LongOne);
> out.writeLong(LongTwo);
> }
>
>
> The Stack Trace:
>
> java.lang.NullPointerException
> at java.lang.System.arraycopy(Native Method)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$Buffer.write(MapTask.java:1108)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at java.io.DataOutputStream.writeUTF(DataOutputStream.java:384)
> at java.io.DataOutputStream.writeUTF(DataOutputStream.java:306)
> at xxxxxx.xxxxxxxxxxxx.xxxx.xxxxx.xxxxxKey.write(xxxxxKey.java:125)
> at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:90)
> at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:77)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:892)
> at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:541)
> at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> at xxxxxx.xxxxxxxxxxx.xxxx.xxxxxxxx.xxx.xxxxxxxxxxxxxxxxxxxxxxx.map(xxxxxxxxxxxxxxxxxMapper.java:82)
> at xxxxxx.xxxxxxxxxxx.xxxx.xxxxxxxx.xxx.xxxxxxxxxxxxxxxxxxxxxxx.map(xxxxxxxxxxxxxxxxxMapper.java:63)
> at xxxxxx.xxxxxxxxxxx.xxxx.xxxxxxx.xxxxxxxxxxxxxxxxxxxxxxxxxxxx.xxxxxxxxxx(xxxxxxxxxxxxxxxxxxxxxxxxxxxx.java:131)
> at xxxxxx.xxxxxxxxxxx.xxxx.xxxxx.xxxxxxxxxxxxxx.xxxxxxxxxxxxxxx(xxxxxxxxxxxxxx.java:282)
> at xxxxxx.xxxxxxxxxxx.xxxx.xxxxx.xxxxxxxxxxxxxx.access$000(xxxxxxxxxxxxxx.java:42)
> at xxxxxx.xxxxxxxxxxx.xxxx.xxxxx.xxxxxxxxxxxxxx$xxxxxxxxxxxx.run(xxxxxxxxxxxxx.java:74)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)