You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by "Chinni, Ravi" <rc...@syncsort.com> on 2010/07/16 17:56:11 UTC
Error: java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.get
I am trying to run the terasort example with a small input on a 4 node
cluster. I just did the minimal configuration (fs.default.name, master,
slaves etc.), but did not do anything specific to terasort. I am getting
the following java.lang.NullPointerException on running terasort
example:
10/07/16 11:48:29 INFO terasort.TeraSort: starting
10/07/16 11:48:29 INFO mapred.FileInputFormat: Total input paths to
process : 1
10/07/16 11:48:29 INFO util.NativeCodeLoader: Loaded the native-hadoop
library
10/07/16 11:48:29 INFO zlib.ZlibFactory: Successfully loaded &
initialized native-zlib library
10/07/16 11:48:29 INFO compress.CodecPool: Got brand-new compressor
Making 1 from 10 records
Step size is 10.0
10/07/16 11:48:30 INFO mapred.JobClient: Running job:
job_201006251254_0010
10/07/16 11:48:31 INFO mapred.JobClient: map 0% reduce 0%
10/07/16 11:48:38 INFO mapred.JobClient: map 50% reduce 0%
10/07/16 11:48:39 INFO mapred.JobClient: map 100% reduce 0%
10/07/16 11:48:43 INFO mapred.JobClient: Task Id :
attempt_201006251254_0010_r_000000_0, Status : FAILED
Error: java.lang.NullPointerException
at
java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getM
apCompletionEvents(ReduceTask.java:2683)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(
ReduceTask.java:2605)
10/07/16 11:48:49 INFO mapred.JobClient: Task Id :
attempt_201006251254_0010_r_000000_1, Status : FAILED
Error: java.lang.NullPointerException
at
java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getM
apCompletionEvents(ReduceTask.java:2683)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(
ReduceTask.java:2605)
10/07/16 11:48:57 INFO mapred.JobClient: Task Id :
attempt_201006251254_0010_r_000000_2, Status : FAILED
Error: java.lang.NullPointerException
at
java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getM
apCompletionEvents(ReduceTask.java:2683)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(
ReduceTask.java:2605)
Also, I got similar error from reduce jobs on running a streaming
map-reduce application. In this case also, the map tasks completed
successfully.
Any pointers are appreciated.
Thanks,
Ravi Chinni
_____________________________________________________________________________
ATTENTION:
The information contained in this message (including any files transmitted
with this message) may contain proprietary, trade secret or other
confidential and/or legally privileged information. Any pricing
information contained in this message or in any files transmitted with
this message is always confidential and cannot be shared with any third
parties without prior written approval from Syncsort. This message is
intended to be read only by the individual or entity to whom it is
addressed or by their designee. If the reader of this message is not the
intended recipient, you are on notice that any use, disclosure, copying or
distribution of this message, in any form, is strictly prohibited. If you
have received this message in error, please immediately notify the sender
and/or Syncsort and destroy all copies of this message in your possession,
custody or control.
RE: Error: java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.get
Posted by "Chinni, Ravi" <rc...@syncsort.com>.
The problem got fixed by using ip-address in all hadoop configuration
files and also in /etc/hosts file.
Anyway, thanks Ted. To partially answer your question - line
ReduceTask.java:2683 is 'List<MapOutputLocation> loc =
mapLocations.get(host);'
From: Ted Yu [mailto:yuzhihong@gmail.com]
Sent: Sunday, July 18, 2010 6:11 PM
To: mapreduce-user@hadoop.apache.org
Subject: Re: Error: java.lang.NullPointerException at
java.util.concurrent.ConcurrentHashMap.get
Have you looked at https://issues.apache.org/jira/browse/MAPREDUCE-969 ?
It points to https://issues.apache.org/jira/browse/HADOOP-4744
See if HADOOP-4744 is contained in the hadoop version you're running.
I am reading cdh3b2 source code. Please confirm that
ReduceTask.java:2683 is this line:
List<MapOutputLocation> loc = mapLocations.get(host);
If so, you can print the value of host, or value of URI u which host is
derived.
Cheers
On Fri, Jul 16, 2010 at 9:42 AM, Chinni, Ravi <rc...@syncsort.com>
wrote:
Hadoop version: 0.20.2
From: Ted Yu [mailto:yuzhihong@gmail.com]
Sent: Friday, July 16, 2010 12:34 PM
To: mapreduce-user@hadoop.apache.org
Subject: Re: Error: java.lang.NullPointerException at
java.util.concurrent.ConcurrentHashMap.get
What version of hadoop are you using ?
On Fri, Jul 16, 2010 at 8:56 AM, Chinni, Ravi <rc...@syncsort.com>
wrote:
I am trying to run the terasort example with a small input on a 4 node
cluster. I just did the minimal configuration (fs.default.name, master,
slaves etc.), but did not do anything specific to terasort. I am getting
the following java.lang.NullPointerException on running terasort
example:
10/07/16 11:48:29 INFO terasort.TeraSort: starting
10/07/16 11:48:29 INFO mapred.FileInputFormat: Total input paths to
process : 1
10/07/16 11:48:29 INFO util.NativeCodeLoader: Loaded the native-hadoop
library
10/07/16 11:48:29 INFO zlib.ZlibFactory: Successfully loaded &
initialized native-zlib library
10/07/16 11:48:29 INFO compress.CodecPool: Got brand-new compressor
Making 1 from 10 records
Step size is 10.0
10/07/16 11:48:30 INFO mapred.JobClient: Running job:
job_201006251254_0010
10/07/16 11:48:31 INFO mapred.JobClient: map 0% reduce 0%
10/07/16 11:48:38 INFO mapred.JobClient: map 50% reduce 0%
10/07/16 11:48:39 INFO mapred.JobClient: map 100% reduce 0%
10/07/16 11:48:43 INFO mapred.JobClient: Task Id :
attempt_201006251254_0010_r_000000_0, Status : FAILED
Error: java.lang.NullPointerException
at
java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getM
apCompletionEvents(ReduceTask.java:2683)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(
ReduceTask.java:2605)
10/07/16 11:48:49 INFO mapred.JobClient: Task Id :
attempt_201006251254_0010_r_000000_1, Status : FAILED
Error: java.lang.NullPointerException
at
java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getM
apCompletionEvents(ReduceTask.java:2683)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(
ReduceTask.java:2605)
10/07/16 11:48:57 INFO mapred.JobClient: Task Id :
attempt_201006251254_0010_r_000000_2, Status : FAILED
Error: java.lang.NullPointerException
at
java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getM
apCompletionEvents(ReduceTask.java:2683)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(
ReduceTask.java:2605)
Also, I got similar error from reduce jobs on running a streaming
map-reduce application. In this case also, the map tasks completed
successfully.
Any pointers are appreciated.
Thanks,
Ravi Chinni
________________________________________________________________________
_____
ATTENTION:
The information contained in this message (including any files
transmitted with this message) may contain proprietary, trade secret or
other confidential and/or legally privileged information. Any pricing
information contained in this message or in any files transmitted with
this message is always confidential and cannot be shared with any third
parties without prior written approval from Syncsort. This message is
intended to be read only by the individual or entity to whom it is
addressed or by their designee. If the reader of this message is not the
intended recipient, you are on notice that any use, disclosure, copying
or distribution of this message, in any form, is strictly prohibited. If
you have received this message in error, please immediately notify the
sender and/or Syncsort and destroy all copies of this message in your
possession, custody or control.
RE: Why does the MR framework sorts the mapper output?
Posted by "Chinni, Ravi" <rc...@syncsort.com>.
Thanks Alex and Ken.
My application does not do aggregation. It mainly does some data
cleansing and transformation. So I don't need a combiner. (Also, I don't
see why a combiner always needs sorted input; it should be optional and
user specified)
To take advantage of some optimizations, I need a partitioner - this
means most of my application logic is in the reducers and cannot set the
# of reducers to 0. Of course, I will be happy if there is a way to set
the # of mappers to 0.
Ravi
From: Ken Goodhope [mailto:kengoodhope@gmail.com]
Sent: Monday, July 26, 2010 8:00 PM
To: mapreduce-user@hadoop.apache.org
Subject: Re: Why does the MR framework sorts the mapper output?
The combiner needs sorted input.
On Mon, Jul 26, 2010 at 1:46 PM, Alex Kozlov <al...@cloudera.com>
wrote:
Hi Ravi,
Whether a sort is required is still a point of debate: the primary
reason is to collect the entries with the same key, but one can
implement MapReduce with hash deduping. The performance
advantages/disadvantages are still a subject of debate.
If you don't need sorting, you can always implement map-side aggregation
though and potentially set the # of reducers to 0. There is no
potential risk, but if you want to aggregate results across different
mappers you'll get back to the original problem.
Alex K
On Mon, Jul 26, 2010 at 1:32 PM, Chinni, Ravi <rc...@syncsort.com>
wrote:
I have an MR application that is running fine except for the
performance. Increasing the number of data nodes is not an option to me.
Looking at the source code of MR framework, I noticed that the
partitioned output of each mapper is sorted (MapTask.java), and on the
reduce side partitions from various mappers are merged (ReduceTask.java)
before running the reduce step. Functionally, reducers in my application
does not require data to be in sorted order and getting rid of the sort
and merge steps in the framework will help my application.
Does anyone know, why the sort and merge of intermediate data is being
done by the framework? Is there anything - MR functional concepts,
framework design etc. - that will need the sort and merge of
intermediate data? I want to give a shot in getting rid of the sort and
merge steps in the framework and want to know of any potential risks
involved.
Any input is appreciated.
Thanks,
Ravi
________________________________________________________________________
_____
ATTENTION:
The information contained in this message (including any files
transmitted with this message) may contain proprietary, trade secret or
other confidential and/or legally privileged information. Any pricing
information contained in this message or in any files transmitted with
this message is always confidential and cannot be shared with any third
parties without prior written approval from Syncsort. This message is
intended to be read only by the individual or entity to whom it is
addressed or by their designee. If the reader of this message is not the
intended recipient, you are on notice that any use, disclosure, copying
or distribution of this message, in any form, is strictly prohibited. If
you have received this message in error, please immediately notify the
sender and/or Syncsort and destroy all copies of this message in your
possession, custody or control.
Re: Why does the MR framework sorts the mapper output?
Posted by Ken Goodhope <ke...@gmail.com>.
The combiner needs sorted input.
On Mon, Jul 26, 2010 at 1:46 PM, Alex Kozlov <al...@cloudera.com> wrote:
> Hi Ravi,
>
> Whether a sort is required is still a point of debate: the primary reason
> is to collect the entries with the same key, but one can implement MapReduce
> with hash deduping. The performance advantages/disadvantages are still a
> subject of debate.
>
> If you don't need sorting, you can always implement map-side aggregation
> though and potentially set the # of reducers to 0. There is no potential
> risk, but if you want to aggregate results across different mappers you'll
> get back to the original problem.
>
> Alex K
>
> On Mon, Jul 26, 2010 at 1:32 PM, Chinni, Ravi <rc...@syncsort.com>wrote:
>
>> I have an MR application that is running fine except for the
>> performance. Increasing the number of data nodes is not an option to me.
>>
>>
>>
>> Looking at the source code of MR framework, I noticed that the partitioned
>> output of each mapper is sorted (MapTask.java), and on the reduce side
>> partitions from various mappers are merged (ReduceTask.java) before running
>> the reduce step. Functionally, reducers in my application does not require
>> data to be in sorted order and getting rid of the sort and merge steps in
>> the framework will help my application.
>>
>>
>>
>> Does anyone know, why the sort and merge of intermediate data is being
>> done by the framework? Is there anything - MR functional concepts, framework
>> design etc. - that will need the sort and merge of intermediate data? I want
>> to give a shot in getting rid of the sort and merge steps in the framework
>> and want to know of any potential risks involved.
>>
>>
>>
>> Any input is appreciated.
>>
>>
>>
>> Thanks,
>>
>> Ravi
>>
>>
>>
>>
>> _____________________________________________________________________________
>>
>> ATTENTION:
>>
>> The information contained in this message (including any files transmitted
>> with this message) may contain proprietary, trade secret or other
>> confidential and/or legally privileged information. Any pricing information
>> contained in this message or in any files transmitted with this message is
>> always confidential and cannot be shared with any third parties without
>> prior written approval from Syncsort. This message is intended to be read
>> only by the individual or entity to whom it is addressed or by their
>> designee. If the reader of this message is not the intended recipient, you
>> are on notice that any use, disclosure, copying or distribution of this
>> message, in any form, is strictly prohibited. If you have received this
>> message in error, please immediately notify the sender and/or Syncsort and
>> destroy all copies of this message in your possession, custody or control.
>>
>
>
Re: Why does the MR framework sorts the mapper output?
Posted by Alex Kozlov <al...@cloudera.com>.
Hi Ravi,
Whether a sort is required is still a point of debate: the primary reason is
to collect the entries with the same key, but one can implement MapReduce
with hash deduping. The performance advantages/disadvantages are still a
subject of debate.
If you don't need sorting, you can always implement map-side aggregation
though and potentially set the # of reducers to 0. There is no potential
risk, but if you want to aggregate results across different mappers you'll
get back to the original problem.
Alex K
On Mon, Jul 26, 2010 at 1:32 PM, Chinni, Ravi <rc...@syncsort.com> wrote:
> I have an MR application that is running fine except for the
> performance. Increasing the number of data nodes is not an option to me.
>
>
>
> Looking at the source code of MR framework, I noticed that the partitioned
> output of each mapper is sorted (MapTask.java), and on the reduce side
> partitions from various mappers are merged (ReduceTask.java) before running
> the reduce step. Functionally, reducers in my application does not require
> data to be in sorted order and getting rid of the sort and merge steps in
> the framework will help my application.
>
>
>
> Does anyone know, why the sort and merge of intermediate data is being done
> by the framework? Is there anything - MR functional concepts, framework
> design etc. - that will need the sort and merge of intermediate data? I want
> to give a shot in getting rid of the sort and merge steps in the framework
> and want to know of any potential risks involved.
>
>
>
> Any input is appreciated.
>
>
>
> Thanks,
>
> Ravi
>
>
>
>
> _____________________________________________________________________________
>
> ATTENTION:
>
> The information contained in this message (including any files transmitted
> with this message) may contain proprietary, trade secret or other
> confidential and/or legally privileged information. Any pricing information
> contained in this message or in any files transmitted with this message is
> always confidential and cannot be shared with any third parties without
> prior written approval from Syncsort. This message is intended to be read
> only by the individual or entity to whom it is addressed or by their
> designee. If the reader of this message is not the intended recipient, you
> are on notice that any use, disclosure, copying or distribution of this
> message, in any form, is strictly prohibited. If you have received this
> message in error, please immediately notify the sender and/or Syncsort and
> destroy all copies of this message in your possession, custody or control.
>
Why does the MR framework sorts the mapper output?
Posted by "Chinni, Ravi" <rc...@syncsort.com>.
I have an MR application that is running fine except for the
performance. Increasing the number of data nodes is not an option to me.
Looking at the source code of MR framework, I noticed that the
partitioned output of each mapper is sorted (MapTask.java), and on the
reduce side partitions from various mappers are merged (ReduceTask.java)
before running the reduce step. Functionally, reducers in my application
does not require data to be in sorted order and getting rid of the sort
and merge steps in the framework will help my application.
Does anyone know, why the sort and merge of intermediate data is being
done by the framework? Is there anything - MR functional concepts,
framework design etc. - that will need the sort and merge of
intermediate data? I want to give a shot in getting rid of the sort and
merge steps in the framework and want to know of any potential risks
involved.
Any input is appreciated.
Thanks,
Ravi
_____________________________________________________________________________
ATTENTION:
The information contained in this message (including any files transmitted
with this message) may contain proprietary, trade secret or other
confidential and/or legally privileged information. Any pricing
information contained in this message or in any files transmitted with
this message is always confidential and cannot be shared with any third
parties without prior written approval from Syncsort. This message is
intended to be read only by the individual or entity to whom it is
addressed or by their designee. If the reader of this message is not the
intended recipient, you are on notice that any use, disclosure, copying or
distribution of this message, in any form, is strictly prohibited. If you
have received this message in error, please immediately notify the sender
and/or Syncsort and destroy all copies of this message in your possession,
custody or control.
Re: Error: java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.get
Posted by Ted Yu <yu...@gmail.com>.
Have you looked at https://issues.apache.org/jira/browse/MAPREDUCE-969 ?
It points to https://issues.apache.org/jira/browse/HADOOP-4744
See if HADOOP-4744 is contained in the hadoop version you're running.
I am reading cdh3b2 source code. Please confirm that *ReduceTask.java:2683 *is
this line:
List<MapOutputLocation> loc = mapLocations.get(host);
If so, you can print the value of host, or value of URI u which host is
derived.
Cheers
On Fri, Jul 16, 2010 at 9:42 AM, Chinni, Ravi <rc...@syncsort.com> wrote:
> Hadoop version: 0.20.2
>
>
>
> *From:* Ted Yu [mailto:yuzhihong@gmail.com]
> *Sent:* Friday, July 16, 2010 12:34 PM
> *To:* mapreduce-user@hadoop.apache.org
> *Subject:* Re: Error: java.lang.NullPointerException at
> java.util.concurrent.ConcurrentHashMap.get
>
>
>
> What version of hadoop are you using ?
>
> On Fri, Jul 16, 2010 at 8:56 AM, Chinni, Ravi <rc...@syncsort.com>
> wrote:
>
> I am trying to run the terasort example with a small input on a 4 node
> cluster. I just did the minimal configuration (fs.default.name, master,
> slaves etc.), but did not do anything specific to terasort. I am getting the
> following java.lang.NullPointerException on running terasort example:
>
>
>
>
>
> 10/07/16 11:48:29 INFO terasort.TeraSort: starting
>
> 10/07/16 11:48:29 INFO mapred.FileInputFormat: Total input paths to process
> : 1
>
> 10/07/16 11:48:29 INFO util.NativeCodeLoader: Loaded the native-hadoop
> library
>
> 10/07/16 11:48:29 INFO zlib.ZlibFactory: Successfully loaded & initialized
> native-zlib library
>
> 10/07/16 11:48:29 INFO compress.CodecPool: Got brand-new compressor
>
> Making 1 from 10 records
>
> Step size is 10.0
>
> 10/07/16 11:48:30 INFO mapred.JobClient: Running job: job_201006251254_0010
>
> 10/07/16 11:48:31 INFO mapred.JobClient: map 0% reduce 0%
>
> 10/07/16 11:48:38 INFO mapred.JobClient: map 50% reduce 0%
>
> 10/07/16 11:48:39 INFO mapred.JobClient: map 100% reduce 0%
>
> 10/07/16 11:48:43 INFO mapred.JobClient: Task Id :
> attempt_201006251254_0010_r_000000_0, Status : FAILED
>
> *Error: java.lang.NullPointerException*
>
> * at
> java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)*
>
> * at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2683)
> *
>
> * at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2605)
> *
>
>
>
> 10/07/16 11:48:49 INFO mapred.JobClient: Task Id :
> attempt_201006251254_0010_r_000000_1, Status : FAILED
>
> Error: java.lang.NullPointerException
>
> at
> java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
>
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2683)
>
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2605)
>
>
>
> 10/07/16 11:48:57 INFO mapred.JobClient: Task Id :
> attempt_201006251254_0010_r_000000_2, Status : FAILED
>
> Error: java.lang.NullPointerException
>
> at
> java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
>
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2683)
>
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2605)
>
>
>
>
>
>
>
> Also, I got similar error from reduce jobs on running a streaming
> map-reduce application. In this case also, the map tasks completed
> successfully.
>
>
>
>
>
> Any pointers are appreciated.
>
>
>
> Thanks,
>
> *Ravi Chinni*
>
>
>
>
>
>
> _____________________________________________________________________________
>
>
>
> ATTENTION:
>
>
>
> The information contained in this message (including any files transmitted
> with this message) may contain proprietary, trade secret or other
> confidential and/or legally privileged information. Any pricing information
> contained in this message or in any files transmitted with this message is
> always confidential and cannot be shared with any third parties without
> prior written approval from Syncsort. This message is intended to be read
> only by the individual or entity to whom it is addressed or by their
> designee. If the reader of this message is not the intended recipient, you
> are on notice that any use, disclosure, copying or distribution of this
> message, in any form, is strictly prohibited. If you have received this
> message in error, please immediately notify the sender and/or Syncsort and
> destroy all copies of this message in your possession, custody or control.
>
>
>
RE: Error: java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.get
Posted by "Chinni, Ravi" <rc...@syncsort.com>.
Hadoop version: 0.20.2
From: Ted Yu [mailto:yuzhihong@gmail.com]
Sent: Friday, July 16, 2010 12:34 PM
To: mapreduce-user@hadoop.apache.org
Subject: Re: Error: java.lang.NullPointerException at
java.util.concurrent.ConcurrentHashMap.get
What version of hadoop are you using ?
On Fri, Jul 16, 2010 at 8:56 AM, Chinni, Ravi <rc...@syncsort.com>
wrote:
I am trying to run the terasort example with a small input on a 4 node
cluster. I just did the minimal configuration (fs.default.name, master,
slaves etc.), but did not do anything specific to terasort. I am getting
the following java.lang.NullPointerException on running terasort
example:
10/07/16 11:48:29 INFO terasort.TeraSort: starting
10/07/16 11:48:29 INFO mapred.FileInputFormat: Total input paths to
process : 1
10/07/16 11:48:29 INFO util.NativeCodeLoader: Loaded the native-hadoop
library
10/07/16 11:48:29 INFO zlib.ZlibFactory: Successfully loaded &
initialized native-zlib library
10/07/16 11:48:29 INFO compress.CodecPool: Got brand-new compressor
Making 1 from 10 records
Step size is 10.0
10/07/16 11:48:30 INFO mapred.JobClient: Running job:
job_201006251254_0010
10/07/16 11:48:31 INFO mapred.JobClient: map 0% reduce 0%
10/07/16 11:48:38 INFO mapred.JobClient: map 50% reduce 0%
10/07/16 11:48:39 INFO mapred.JobClient: map 100% reduce 0%
10/07/16 11:48:43 INFO mapred.JobClient: Task Id :
attempt_201006251254_0010_r_000000_0, Status : FAILED
Error: java.lang.NullPointerException
at
java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getM
apCompletionEvents(ReduceTask.java:2683)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(
ReduceTask.java:2605)
10/07/16 11:48:49 INFO mapred.JobClient: Task Id :
attempt_201006251254_0010_r_000000_1, Status : FAILED
Error: java.lang.NullPointerException
at
java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getM
apCompletionEvents(ReduceTask.java:2683)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(
ReduceTask.java:2605)
10/07/16 11:48:57 INFO mapred.JobClient: Task Id :
attempt_201006251254_0010_r_000000_2, Status : FAILED
Error: java.lang.NullPointerException
at
java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getM
apCompletionEvents(ReduceTask.java:2683)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(
ReduceTask.java:2605)
Also, I got similar error from reduce jobs on running a streaming
map-reduce application. In this case also, the map tasks completed
successfully.
Any pointers are appreciated.
Thanks,
Ravi Chinni
________________________________________________________________________
_____
ATTENTION:
The information contained in this message (including any files
transmitted with this message) may contain proprietary, trade secret or
other confidential and/or legally privileged information. Any pricing
information contained in this message or in any files transmitted with
this message is always confidential and cannot be shared with any third
parties without prior written approval from Syncsort. This message is
intended to be read only by the individual or entity to whom it is
addressed or by their designee. If the reader of this message is not the
intended recipient, you are on notice that any use, disclosure, copying
or distribution of this message, in any form, is strictly prohibited. If
you have received this message in error, please immediately notify the
sender and/or Syncsort and destroy all copies of this message in your
possession, custody or control.
Re: Error: java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.get
Posted by Ted Yu <yu...@gmail.com>.
What version of hadoop are you using ?
On Fri, Jul 16, 2010 at 8:56 AM, Chinni, Ravi <rc...@syncsort.com> wrote:
> I am trying to run the terasort example with a small input on a 4 node
> cluster. I just did the minimal configuration (fs.default.name, master,
> slaves etc.), but did not do anything specific to terasort. I am getting the
> following java.lang.NullPointerException on running terasort example:
>
>
>
>
>
> 10/07/16 11:48:29 INFO terasort.TeraSort: starting
>
> 10/07/16 11:48:29 INFO mapred.FileInputFormat: Total input paths to process
> : 1
>
> 10/07/16 11:48:29 INFO util.NativeCodeLoader: Loaded the native-hadoop
> library
>
> 10/07/16 11:48:29 INFO zlib.ZlibFactory: Successfully loaded & initialized
> native-zlib library
>
> 10/07/16 11:48:29 INFO compress.CodecPool: Got brand-new compressor
>
> Making 1 from 10 records
>
> Step size is 10.0
>
> 10/07/16 11:48:30 INFO mapred.JobClient: Running job: job_201006251254_0010
>
> 10/07/16 11:48:31 INFO mapred.JobClient: map 0% reduce 0%
>
> 10/07/16 11:48:38 INFO mapred.JobClient: map 50% reduce 0%
>
> 10/07/16 11:48:39 INFO mapred.JobClient: map 100% reduce 0%
>
> 10/07/16 11:48:43 INFO mapred.JobClient: Task Id :
> attempt_201006251254_0010_r_000000_0, Status : FAILED
>
> *Error: java.lang.NullPointerException*
>
> * at
> java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)*
>
> * at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2683)
> *
>
> * at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2605)
> *
>
>
>
> 10/07/16 11:48:49 INFO mapred.JobClient: Task Id :
> attempt_201006251254_0010_r_000000_1, Status : FAILED
>
> Error: java.lang.NullPointerException
>
> at
> java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
>
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2683)
>
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2605)
>
>
>
> 10/07/16 11:48:57 INFO mapred.JobClient: Task Id :
> attempt_201006251254_0010_r_000000_2, Status : FAILED
>
> Error: java.lang.NullPointerException
>
> at
> java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
>
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2683)
>
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2605)
>
>
>
>
>
>
>
> Also, I got similar error from reduce jobs on running a streaming
> map-reduce application. In this case also, the map tasks completed
> successfully.
>
>
>
>
>
> Any pointers are appreciated.
>
>
>
> Thanks,
>
> *Ravi Chinni***
>
>
>
>
> _____________________________________________________________________________
>
> ATTENTION:
>
> The information contained in this message (including any files transmitted
> with this message) may contain proprietary, trade secret or other
> confidential and/or legally privileged information. Any pricing information
> contained in this message or in any files transmitted with this message is
> always confidential and cannot be shared with any third parties without
> prior written approval from Syncsort. This message is intended to be read
> only by the individual or entity to whom it is addressed or by their
> designee. If the reader of this message is not the intended recipient, you
> are on notice that any use, disclosure, copying or distribution of this
> message, in any form, is strictly prohibited. If you have received this
> message in error, please immediately notify the sender and/or Syncsort and
> destroy all copies of this message in your possession, custody or control.
>