You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@giraph.apache.org by Francesco Sclano <sc...@gmail.com> on 2018/10/02 10:09:25 UTC
Giraph's estimated cluster heap xxxxMBs ask is greater than the
current available cluster heap of 0MB. Aborting Job
I'm using giraph-1.3.0-SNAPSHOT and hadoop-2.8.4 on Amazon EC2 cluster. My
cluster is composed by 4 t2.large machines each having 8 GB RAM and 2 cpus
(In future I'll have to use 20 c3.8xlarge machines, each having 60 GB RAM
and 32 CPU).
I'm blocked on this problem: "Giraph's estimated cluster heap xxxxMBs ask
is greater than the current available cluster heap of 0MB. Aborting Job". I
red this previous post
https://stackoverflow.com/questions/28977337/giraphs-estimated-cluster-heap-4096mb-ask-is-greater-than-the-current-available
but I didn't understand what caused the problem in my case since I have
configured yarn.resourcemanager.hostname (see below) and my security group
is open to all traffic. Maybe I miss some settings (or some ports)?
Furthermore, I have following questions:
- Since Giraph doesn't use reduce but only map, is it correct to assign
less memory to mapreduce.reduce.memory.mb than the memory assigned to
mapreduce.map.memory.mb? Maybe it could be right to assign even 0 MBs to
mapreduce.reduce.memory.mb since giraph doesn't use reduce?
- I red http://giraph.apache.org/quick_start.html that
mapred.tasktracker.map.tasks.maximum and mapred.map.tasks must be set to 4
since "by default hadoop allows 2 mappers to run at once. Giraph's code,
however, assumes that we can run 4 mappers at the same time." Hence 4 value
must be always set to these properties?
This is my configuration. I reported only mapred-site.xml and yarn-site.xml
because on others hadoop configuration files I'm quite sure that they are
correct.
mapred-site.xml
<configuration>
<property>
<name>mapreduce.jobtracker.address</name>
<value>{HOSTNAME}:54311</value>
</property>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<value>4</value>
</property>
<property>
<name>mapred.map.tasks</name>
<value>4</value>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>4608</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>512</value>
</property>
</configuration>
yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>{HOSTNAME}</value>
</property>
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>2</value>
</property>
<property>
<name>yarn.app.mapreduce.am.resource.mb</name>
<value>2048</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>6144</value>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>6144</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>512</value>
</property>
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>
</configuration>
--
Francesco Sclano
Re: Giraph's estimated cluster heap xxxxMBs ask is greater than the
current available cluster heap of 0MB. Aborting Job
Posted by Francesco Sclano <sc...@gmail.com>.
I tryed this other configuration in mapred-site.xml and now it works (I get
_SUCCESS file and expected output) but I think that it is not the optimal
configuration!
<property>
<name>mapreduce.map.memory.mb</name>
<value>1024</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>1024</value>
</property>
Il giorno mar 2 ott 2018 alle ore 12:09 Francesco Sclano <
sclano.francesco@gmail.com> ha scritto:
> I'm using giraph-1.3.0-SNAPSHOT and hadoop-2.8.4 on Amazon EC2 cluster. My
> cluster is composed by 4 t2.large machines each having 8 GB RAM and 2 cpus
> (In future I'll have to use 20 c3.8xlarge machines, each having 60 GB RAM
> and 32 CPU).
> I'm blocked on this problem: "Giraph's estimated cluster heap xxxxMBs ask
> is greater than the current available cluster heap of 0MB. Aborting Job". I
> red this previous post
> https://stackoverflow.com/questions/28977337/giraphs-estimated-cluster-heap-4096mb-ask-is-greater-than-the-current-available
> but I didn't understand what caused the problem in my case since I have
> configured yarn.resourcemanager.hostname (see below) and my security group
> is open to all traffic. Maybe I miss some settings (or some ports)?
>
>
> Furthermore, I have following questions:
> - Since Giraph doesn't use reduce but only map, is it correct to assign
> less memory to mapreduce.reduce.memory.mb than the memory assigned to
> mapreduce.map.memory.mb? Maybe it could be right to assign even 0 MBs to
> mapreduce.reduce.memory.mb since giraph doesn't use reduce?
> - I red http://giraph.apache.org/quick_start.html that
> mapred.tasktracker.map.tasks.maximum and mapred.map.tasks must be set to 4
> since "by default hadoop allows 2 mappers to run at once. Giraph's code,
> however, assumes that we can run 4 mappers at the same time." Hence 4 value
> must be always set to these properties?
>
>
>
> This is my configuration. I reported only mapred-site.xml and
> yarn-site.xml because on others hadoop configuration files I'm quite sure
> that they are correct.
>
>
> mapred-site.xml
>
>
> <configuration>
> <property>
> <name>mapreduce.jobtracker.address</name>
> <value>{HOSTNAME}:54311</value>
> </property>
> <property>
> <name>mapreduce.framework.name</name>
> <value>yarn</value>
> </property>
> <property>
> <name>mapred.tasktracker.map.tasks.maximum</name>
> <value>4</value>
> </property>
> <property>
> <name>mapred.map.tasks</name>
> <value>4</value>
> </property>
> <property>
> <name>mapreduce.map.memory.mb</name>
> <value>4608</value>
> </property>
> <property>
> <name>mapreduce.reduce.memory.mb</name>
> <value>512</value>
> </property>
> </configuration>
>
>
>
>
> yarn-site.xml
>
> <configuration>
> <property>
> <name>yarn.nodemanager.aux-services</name>
> <value>mapreduce_shuffle</value>
> </property>
> <property>
> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
> <value>org.apache.hadoop.mapred.ShuffleHandler</value>
> </property>
> <property>
> <name>yarn.resourcemanager.hostname</name>
> <value>{HOSTNAME}</value>
> </property>
> <property>
> <name>yarn.nodemanager.resource.cpu-vcores</name>
> <value>2</value>
> </property>
> <property>
> <name>yarn.app.mapreduce.am.resource.mb</name>
> <value>2048</value>
> </property>
> <property>
> <name>yarn.nodemanager.resource.memory-mb</name>
> <value>6144</value>
> </property>
> <property>
> <name>yarn.scheduler.maximum-allocation-mb</name>
> <value>6144</value>
> </property>
> <property>
> <name>yarn.scheduler.minimum-allocation-mb</name>
> <value>512</value>
> </property>
> <property>
> <name>yarn.nodemanager.vmem-check-enabled</name>
> <value>false</value>
> </property>
> </configuration>
>
> --
> Francesco Sclano
>
--
Francesco Sclano