You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Marco Didonna <m....@gmail.com> on 2011/12/11 18:59:31 UTC
[EC2] Could not reserve enough space for object heap
Hello everyone,
I'm running a small toy cluster (3 nodes) on EC2 configured as follows:
* one node as JT+NN
* two nodes as DN+TT
I use whirr to build such cluster on demand (config file here
http://pastebin.com/JXHYvMNb). Since my jobs are memory intensive I'd
like to exploit the 8GB of ram the m1.large instance offers. Thus I
added mapred.map.child.java.opts=-Xmx1g and
mapred.reduce.child.java.opts=-Xmx1g (adding just
hadoop-mapreduce.mapred.child.java.opts=-Xmx1g produced no effect
since the map tasks were allocated the default 200MB). The problem is
that with these settings I cannot have any job running because I
always get
1/12/11 18:14:24 INFO mapred.JobClient: Running job: job_201112111644_0002
11/12/11 18:14:25 INFO mapred.JobClient: map 0% reduce 0%
11/12/11 18:14:28 INFO mapred.JobClient: Task Id :
attempt_201112111644_0002_m_000004_0, Status : FAILED
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:242)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:229)
11/12/11 18:14:28 WARN mapred.JobClient: Error reading task
outputip-10-87-1-170.ec2.internal
11/12/11 18:14:28 WARN mapred.JobClient: Error reading task
outputip-10-87-1-170.ec2.internal
11/12/11 18:14:30 INFO mapred.JobClient: Task Id :
attempt_201112111644_0002_r_000001_0, Status : FAILED
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:242)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:229)
And the only log I can get from the job_201112111644_0002 is the
stdout and the stderr whose combined output is
Could not create the Java virtual machine.
Error occurred during initialization of VM
Could not reserve enough space for object heap
I really cannot understand why the jvm cannot allocate enough space:
there's plenty of ram. I also tried to reduce the number of map slots
to two: nothing changed. I'm out of ideas. I hope you can shed some
light :)
FYI I use cloudera distribution for hadoop, latest stable release available.
Thanks for your attention.
MD
Re: [EC2] Could not reserve enough space for object heap
Posted by Marco Didonna <m....@gmail.com>.
Hi, thanks for your answer.
I am indeed using a 64bit jvm on all nodes. In fact I get the
following answer when executing java -version
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)
And I do have a looot of free RAM:
total used free shared buffers cached
Mem: 7700 930 6769 0 14 462
-/+ buffers/cache: 453 7246
Swap: 0 0 0
And still I cannot make use of such amount of ram. Again I tried to
set mapred.child.java.opts to -Xmx1g but the map task got only the
default amount of heap space (I detect this using the ps command).
Only If i use mapred.{map|reduce}.child.java.opts=-Xmx1g this setting
affects the job execution and I end up with the error we're trying to
debug.
The other non-default setting can be seen from the attached whirr
configuration file (previous message).
Thanks again,
Marco
On 11 December 2011 21:37, Harsh J <ha...@cloudera.com> wrote:
> Marco,
>
> Are you using a 64-bit JVM on your nodes or a 32-bit one?
>
> Sun JRE should say something like (for hadoop -version):
> "Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02-402, mixed mode)"
>
> If you are, could you post what 'free' says on your slave nodes?
>
> On Sun, Dec 11, 2011 at 11:29 PM, Marco Didonna <m....@gmail.com> wrote:
>> Hello everyone,
>> I'm running a small toy cluster (3 nodes) on EC2 configured as follows:
>>
>> * one node as JT+NN
>> * two nodes as DN+TT
>>
>> I use whirr to build such cluster on demand (config file here
>> http://pastebin.com/JXHYvMNb). Since my jobs are memory intensive I'd
>> like to exploit the 8GB of ram the m1.large instance offers. Thus I
>> added mapred.map.child.java.opts=-Xmx1g and
>> mapred.reduce.child.java.opts=-Xmx1g (adding just
>> hadoop-mapreduce.mapred.child.java.opts=-Xmx1g produced no effect
>> since the map tasks were allocated the default 200MB). The problem is
>> that with these settings I cannot have any job running because I
>> always get
>>
>> 1/12/11 18:14:24 INFO mapred.JobClient: Running job: job_201112111644_0002
>> 11/12/11 18:14:25 INFO mapred.JobClient: map 0% reduce 0%
>> 11/12/11 18:14:28 INFO mapred.JobClient: Task Id :
>> attempt_201112111644_0002_m_000004_0, Status : FAILED
>> java.lang.Throwable: Child Error
>> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:242)
>> Caused by: java.io.IOException: Task process exit with nonzero status of 1.
>> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:229)
>>
>> 11/12/11 18:14:28 WARN mapred.JobClient: Error reading task
>> outputip-10-87-1-170.ec2.internal
>> 11/12/11 18:14:28 WARN mapred.JobClient: Error reading task
>> outputip-10-87-1-170.ec2.internal
>> 11/12/11 18:14:30 INFO mapred.JobClient: Task Id :
>> attempt_201112111644_0002_r_000001_0, Status : FAILED
>> java.lang.Throwable: Child Error
>> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:242)
>> Caused by: java.io.IOException: Task process exit with nonzero status of 1.
>> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:229)
>>
>> And the only log I can get from the job_201112111644_0002 is the
>> stdout and the stderr whose combined output is
>>
>> Could not create the Java virtual machine.
>> Error occurred during initialization of VM
>> Could not reserve enough space for object heap
>>
>> I really cannot understand why the jvm cannot allocate enough space:
>> there's plenty of ram. I also tried to reduce the number of map slots
>> to two: nothing changed. I'm out of ideas. I hope you can shed some
>> light :)
>>
>> FYI I use cloudera distribution for hadoop, latest stable release available.
>>
>> Thanks for your attention.
>>
>> MD
>
>
>
> --
> Harsh J
Re: [EC2] Could not reserve enough space for object heap
Posted by Harsh J <ha...@cloudera.com>.
Marco,
Are you using a 64-bit JVM on your nodes or a 32-bit one?
Sun JRE should say something like (for hadoop -version):
"Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02-402, mixed mode)"
If you are, could you post what 'free' says on your slave nodes?
On Sun, Dec 11, 2011 at 11:29 PM, Marco Didonna <m....@gmail.com> wrote:
> Hello everyone,
> I'm running a small toy cluster (3 nodes) on EC2 configured as follows:
>
> * one node as JT+NN
> * two nodes as DN+TT
>
> I use whirr to build such cluster on demand (config file here
> http://pastebin.com/JXHYvMNb). Since my jobs are memory intensive I'd
> like to exploit the 8GB of ram the m1.large instance offers. Thus I
> added mapred.map.child.java.opts=-Xmx1g and
> mapred.reduce.child.java.opts=-Xmx1g (adding just
> hadoop-mapreduce.mapred.child.java.opts=-Xmx1g produced no effect
> since the map tasks were allocated the default 200MB). The problem is
> that with these settings I cannot have any job running because I
> always get
>
> 1/12/11 18:14:24 INFO mapred.JobClient: Running job: job_201112111644_0002
> 11/12/11 18:14:25 INFO mapred.JobClient: map 0% reduce 0%
> 11/12/11 18:14:28 INFO mapred.JobClient: Task Id :
> attempt_201112111644_0002_m_000004_0, Status : FAILED
> java.lang.Throwable: Child Error
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:242)
> Caused by: java.io.IOException: Task process exit with nonzero status of 1.
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:229)
>
> 11/12/11 18:14:28 WARN mapred.JobClient: Error reading task
> outputip-10-87-1-170.ec2.internal
> 11/12/11 18:14:28 WARN mapred.JobClient: Error reading task
> outputip-10-87-1-170.ec2.internal
> 11/12/11 18:14:30 INFO mapred.JobClient: Task Id :
> attempt_201112111644_0002_r_000001_0, Status : FAILED
> java.lang.Throwable: Child Error
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:242)
> Caused by: java.io.IOException: Task process exit with nonzero status of 1.
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:229)
>
> And the only log I can get from the job_201112111644_0002 is the
> stdout and the stderr whose combined output is
>
> Could not create the Java virtual machine.
> Error occurred during initialization of VM
> Could not reserve enough space for object heap
>
> I really cannot understand why the jvm cannot allocate enough space:
> there's plenty of ram. I also tried to reduce the number of map slots
> to two: nothing changed. I'm out of ideas. I hope you can shed some
> light :)
>
> FYI I use cloudera distribution for hadoop, latest stable release available.
>
> Thanks for your attention.
>
> MD
--
Harsh J