You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Frank Du <fr...@riskmetrics.com> on 2010/02/03 21:34:33 UTC
Encountered NullPointerException with the WordCount example, hadoop
common v0.20.1
Dear All,
Please help with the NullPointerException in the WordCount example. Sorry it’s the simple code because I am new to hadoop. :-)
I am running v0.20.1 in Ubuntu 9.10. The map task works perfectly. But there is NPE with the reduce task. Is there anything wrong with the configuration or the Reduce.java code?
Below are the error messages from job tracker, and the attached are the source code. Thank you so much!
Error: java.lang.NullPointerException
at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2683)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2605)
Error: java.lang.NullPointerException
at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2683)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2605)
Error: java.lang.NullPointerException
at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2683)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2605)
Error: java.lang.NullPointerException
at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2683)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2605)
- Frank
This email message and any attachments are for the sole use of the intended recipients and may contain proprietary and/or confidential information which may be privileged or otherwise protected from disclosure. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not an intended recipient, please contact the sender by reply email and destroy the original message and any copies of the message as well as any attachments to the original message.
Re: configuration file
Posted by Eric Arenas <ea...@rocketmail.com>.
Hi Gang,
You have to load the XML config file in your M/R code.
Something like this:
FSDataInputStream inS = fs.open(in);
conf.addResource(inS);
Where "conf" is your Configuration.
This will in effect read all the parameters from that XML and override anything that you have previously set with:
conf.set("parameter",parameterValue);
regards,
Eric Arenas
----- Original Message ----
From: Gang Luo <lg...@yahoo.com.cn>
To: common-user@hadoop.apache.org
Sent: Thu, February 4, 2010 6:14:54 AM
Subject: Re: configuration file
I give the path to that xml file in that command. Do I need to add that path to classpath? I try to give a wrong path, there is no error reported.
Aren't those parameters all configurable? like io.sort.mb, mapred.reduce.tasks, io.sort.factor, etc.
Thanks.
-Gang
----- 原始邮件 ----
发件人： Amogh Vasekar <am...@yahoo-inc.com>
收件人： "common-user@hadoop.apache.org" <co...@hadoop.apache.org>
发送日期： 2010/2/4 (周四) 6:09:04 上午
主 题： Re: configuration file
Hi,
A shot in the dark, is the conf file in your classpath? If yes, are the parameters you are trying to override marked final?
Amogh
On 2/4/10 3:18 AM, "Gang Luo" <lg...@yahoo.com.cn> wrote:
Hi,
I am writing script to run whole bunch of jobs automatically. But the configuration file doesn't seems working. I think there is something wrong in my command.
The command is my script is like:
bin/hadoop jar myJarFile myClass -conf myConfigurationFilr.xml arg1 agr2 ....
I use conf.get() so show the value of some parameters. But the values are not what I define in that xml file. Is there something wrong?
Thanks.
-Gang
___________________________________________________________
好玩贺卡等你发，邮箱贺卡全新上线！
http://card.mail.cn.yahoo.com/
Re: configuration file
Posted by Gang Luo <lg...@yahoo.com.cn>.
I give the path to that xml file in that command. Do I need to add that path to classpath? I try to give a wrong path, there is no error reported.
Aren't those parameters all configurable? like io.sort.mb, mapred.reduce.tasks, io.sort.factor, etc.
Thanks.
-Gang
----- 原始邮件 ----
发件人： Amogh Vasekar <am...@yahoo-inc.com>
收件人： "common-user@hadoop.apache.org" <co...@hadoop.apache.org>
发送日期： 2010/2/4 (周四) 6:09:04 上午
主 题： Re: configuration file
Hi,
A shot in the dark, is the conf file in your classpath? If yes, are the parameters you are trying to override marked final?
Amogh
On 2/4/10 3:18 AM, "Gang Luo" <lg...@yahoo.com.cn> wrote:
Hi,
I am writing script to run whole bunch of jobs automatically. But the configuration file doesn't seems working. I think there is something wrong in my command.
The command is my script is like:
bin/hadoop jar myJarFile myClass -conf myConfigurationFilr.xml arg1 agr2 ....
I use conf.get() so show the value of some parameters. But the values are not what I define in that xml file. Is there something wrong?
Thanks.
-Gang
___________________________________________________________
好玩贺卡等你发，邮箱贺卡全新上线！
http://card.mail.cn.yahoo.com/
Re: configuration file
Posted by Amogh Vasekar <am...@yahoo-inc.com>.
Hi,
A shot in the dark, is the conf file in your classpath? If yes, are the parameters you are trying to override marked final?
Amogh
On 2/4/10 3:18 AM, "Gang Luo" <lg...@yahoo.com.cn> wrote:
Hi,
I am writing script to run whole bunch of jobs automatically. But the configuration file doesn't seems working. I think there is something wrong in my command.
The command is my script is like:
bin/hadoop jar myJarFile myClass -conf myConfigurationFilr.xml arg1 agr2 ....
I use conf.get() so show the value of some parameters. But the values are not what I define in that xml file. Is there something wrong?
Thanks.
-Gang
configuration file
Posted by Gang Luo <lg...@yahoo.com.cn>.
Hi,
I am writing script to run whole bunch of jobs automatically. But the configuration file doesn't seems working. I think there is something wrong in my command.
The command is my script is like:
bin/hadoop jar myJarFile myClass -conf myConfigurationFilr.xml arg1 agr2 ....
I use conf.get() so show the value of some parameters. But the values are not what I define in that xml file. Is there something wrong?
Thanks.
-Gang
___________________________________________________________
好玩贺卡等你发，邮箱贺卡全新上线！
http://card.mail.cn.yahoo.com/