You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Subash D'Souza <sd...@truecar.com> on 2012/10/21 15:18:43 UTC
Java heap space error
I'm running CDH 4 on a 4 node cluster each with 96 G of RAM. Up until last week the cluster was running until there was an error in the name node log file and I had to reformat it put the data back
Now when I run hive on YARN. I keep getting a Java heap space error. Based on the research I did. I upped the my mapred.child.java.opts first from 200m to 400 m to 800m and I still have the same issue. It seems to fail near the 100% mapper mark
I checked the log files and the only thing that it does output is java heap space error. Nothing more.
Any help would be appreciated.
Thanks
Subash
Re: Java heap space error
Posted by Michael Segel <mi...@hotmail.com>.
Try upping the child to 1.5GB or more.
On Oct 21, 2012, at 8:18 AM, Subash D'Souza <sd...@truecar.com> wrote:
> I'm running CDH 4 on a 4 node cluster each with 96 G of RAM. Up until last week the cluster was running until there was an error in the name node log file and I had to reformat it put the data back
>
> Now when I run hive on YARN. I keep getting a Java heap space error. Based on the research I did. I upped the my mapred.child.java.opts first from 200m to 400 m to 800m and I still have the same issue. It seems to fail near the 100% mapper mark
>
> I checked the log files and the only thing that it does output is java heap space error. Nothing more.
>
> Any help would be appreciated.
>
> Thanks
> Subash
>
Re: Java heap space error
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
Did this job ever run successfully for you? With 200m heap size?
Seems like your maps are failing. Can you paste your settings for the following:
- io.sort.factor
- io.sort.mb
- mapreduce.map.sort.spill.percent
Thanks,
+Vinod
On Oct 21, 2012, at 6:18 AM, Subash D'Souza wrote:
> I'm running CDH 4 on a 4 node cluster each with 96 G of RAM. Up until last week the cluster was running until there was an error in the name node log file and I had to reformat it put the data back
>
> Now when I run hive on YARN. I keep getting a Java heap space error. Based on the research I did. I upped the my mapred.child.java.opts first from 200m to 400 m to 800m and I still have the same issue. It seems to fail near the 100% mapper mark
>
> I checked the log files and the only thing that it does output is java heap space error. Nothing more.
>
> Any help would be appreciated.
>
> Thanks
> Subash
>
Re: Java heap space error
Posted by Michael Segel <mi...@hotmail.com>.
Try upping the child to 1.5GB or more.
On Oct 21, 2012, at 8:18 AM, Subash D'Souza <sd...@truecar.com> wrote:
> I'm running CDH 4 on a 4 node cluster each with 96 G of RAM. Up until last week the cluster was running until there was an error in the name node log file and I had to reformat it put the data back
>
> Now when I run hive on YARN. I keep getting a Java heap space error. Based on the research I did. I upped the my mapred.child.java.opts first from 200m to 400 m to 800m and I still have the same issue. It seems to fail near the 100% mapper mark
>
> I checked the log files and the only thing that it does output is java heap space error. Nothing more.
>
> Any help would be appreciated.
>
> Thanks
> Subash
>
Re: Java heap space error
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
Did this job ever run successfully for you? With 200m heap size?
Seems like your maps are failing. Can you paste your settings for the following:
- io.sort.factor
- io.sort.mb
- mapreduce.map.sort.spill.percent
Thanks,
+Vinod
On Oct 21, 2012, at 6:18 AM, Subash D'Souza wrote:
> I'm running CDH 4 on a 4 node cluster each with 96 G of RAM. Up until last week the cluster was running until there was an error in the name node log file and I had to reformat it put the data back
>
> Now when I run hive on YARN. I keep getting a Java heap space error. Based on the research I did. I upped the my mapred.child.java.opts first from 200m to 400 m to 800m and I still have the same issue. It seems to fail near the 100% mapper mark
>
> I checked the log files and the only thing that it does output is java heap space error. Nothing more.
>
> Any help would be appreciated.
>
> Thanks
> Subash
>
Re: Java heap space error
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
Did this job ever run successfully for you? With 200m heap size?
Seems like your maps are failing. Can you paste your settings for the following:
- io.sort.factor
- io.sort.mb
- mapreduce.map.sort.spill.percent
Thanks,
+Vinod
On Oct 21, 2012, at 6:18 AM, Subash D'Souza wrote:
> I'm running CDH 4 on a 4 node cluster each with 96 G of RAM. Up until last week the cluster was running until there was an error in the name node log file and I had to reformat it put the data back
>
> Now when I run hive on YARN. I keep getting a Java heap space error. Based on the research I did. I upped the my mapred.child.java.opts first from 200m to 400 m to 800m and I still have the same issue. It seems to fail near the 100% mapper mark
>
> I checked the log files and the only thing that it does output is java heap space error. Nothing more.
>
> Any help would be appreciated.
>
> Thanks
> Subash
>
Re: Java heap space error
Posted by Michael Segel <mi...@hotmail.com>.
Try upping the child to 1.5GB or more.
On Oct 21, 2012, at 8:18 AM, Subash D'Souza <sd...@truecar.com> wrote:
> I'm running CDH 4 on a 4 node cluster each with 96 G of RAM. Up until last week the cluster was running until there was an error in the name node log file and I had to reformat it put the data back
>
> Now when I run hive on YARN. I keep getting a Java heap space error. Based on the research I did. I upped the my mapred.child.java.opts first from 200m to 400 m to 800m and I still have the same issue. It seems to fail near the 100% mapper mark
>
> I checked the log files and the only thing that it does output is java heap space error. Nothing more.
>
> Any help would be appreciated.
>
> Thanks
> Subash
>
Re: Java heap space error
Posted by Michael Segel <mi...@hotmail.com>.
Try upping the child to 1.5GB or more.
On Oct 21, 2012, at 8:18 AM, Subash D'Souza <sd...@truecar.com> wrote:
> I'm running CDH 4 on a 4 node cluster each with 96 G of RAM. Up until last week the cluster was running until there was an error in the name node log file and I had to reformat it put the data back
>
> Now when I run hive on YARN. I keep getting a Java heap space error. Based on the research I did. I upped the my mapred.child.java.opts first from 200m to 400 m to 800m and I still have the same issue. It seems to fail near the 100% mapper mark
>
> I checked the log files and the only thing that it does output is java heap space error. Nothing more.
>
> Any help would be appreciated.
>
> Thanks
> Subash
>
Re: Java heap space error
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
Did this job ever run successfully for you? With 200m heap size?
Seems like your maps are failing. Can you paste your settings for the following:
- io.sort.factor
- io.sort.mb
- mapreduce.map.sort.spill.percent
Thanks,
+Vinod
On Oct 21, 2012, at 6:18 AM, Subash D'Souza wrote:
> I'm running CDH 4 on a 4 node cluster each with 96 G of RAM. Up until last week the cluster was running until there was an error in the name node log file and I had to reformat it put the data back
>
> Now when I run hive on YARN. I keep getting a Java heap space error. Based on the research I did. I upped the my mapred.child.java.opts first from 200m to 400 m to 800m and I still have the same issue. It seems to fail near the 100% mapper mark
>
> I checked the log files and the only thing that it does output is java heap space error. Nothing more.
>
> Any help would be appreciated.
>
> Thanks
> Subash
>
RE: Java heap space error
Posted by "Kartashov, Andy" <An...@mpac.ca>.
Subash,
I have been experiencing this type of an error at some point and no matter how much I played with heap size it didn't work. What I found out at the end is I was running out of physical memory. My output file was about 4Gb with only 2.5Gb of free space available. Check you space with "$hadop fs -df
From: Subash D'Souza [mailto:sdsouza@truecar.com]
Sent: Sunday, October 21, 2012 9:19 AM
To: user@hadoop.apache.org
Subject: Java heap space error
I'm running CDH 4 on a 4 node cluster each with 96 G of RAM. Up until last week the cluster was running until there was an error in the name node log file and I had to reformat it put the data back
Now when I run hive on YARN. I keep getting a Java heap space error. Based on the research I did. I upped the my mapred.child.java.opts first from 200m to 400 m to 800m and I still have the same issue. It seems to fail near the 100% mapper mark
I checked the log files and the only thing that it does output is java heap space error. Nothing more.
Any help would be appreciated.
Thanks
Subash
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le pr?sent courriel et toute pi?ce jointe qui l'accompagne sont confidentiels, prot?g?s par le droit d'auteur et peuvent ?tre couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autoris?e est interdite. Si vous n'?tes pas le destinataire pr?vu de ce courriel, supprimez-le et contactez imm?diatement l'exp?diteur. Veuillez penser ? l'environnement avant d'imprimer le pr?sent courriel
RE: Java heap space error
Posted by "Kartashov, Andy" <An...@mpac.ca>.
Subash,
I have been experiencing this type of an error at some point and no matter how much I played with heap size it didn't work. What I found out at the end is I was running out of physical memory. My output file was about 4Gb with only 2.5Gb of free space available. Check you space with "$hadop fs -df
From: Subash D'Souza [mailto:sdsouza@truecar.com]
Sent: Sunday, October 21, 2012 9:19 AM
To: user@hadoop.apache.org
Subject: Java heap space error
I'm running CDH 4 on a 4 node cluster each with 96 G of RAM. Up until last week the cluster was running until there was an error in the name node log file and I had to reformat it put the data back
Now when I run hive on YARN. I keep getting a Java heap space error. Based on the research I did. I upped the my mapred.child.java.opts first from 200m to 400 m to 800m and I still have the same issue. It seems to fail near the 100% mapper mark
I checked the log files and the only thing that it does output is java heap space error. Nothing more.
Any help would be appreciated.
Thanks
Subash
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le pr?sent courriel et toute pi?ce jointe qui l'accompagne sont confidentiels, prot?g?s par le droit d'auteur et peuvent ?tre couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autoris?e est interdite. Si vous n'?tes pas le destinataire pr?vu de ce courriel, supprimez-le et contactez imm?diatement l'exp?diteur. Veuillez penser ? l'environnement avant d'imprimer le pr?sent courriel
RE: Java heap space error
Posted by "Kartashov, Andy" <An...@mpac.ca>.
Subash,
I have been experiencing this type of an error at some point and no matter how much I played with heap size it didn't work. What I found out at the end is I was running out of physical memory. My output file was about 4Gb with only 2.5Gb of free space available. Check you space with "$hadop fs -df
From: Subash D'Souza [mailto:sdsouza@truecar.com]
Sent: Sunday, October 21, 2012 9:19 AM
To: user@hadoop.apache.org
Subject: Java heap space error
I'm running CDH 4 on a 4 node cluster each with 96 G of RAM. Up until last week the cluster was running until there was an error in the name node log file and I had to reformat it put the data back
Now when I run hive on YARN. I keep getting a Java heap space error. Based on the research I did. I upped the my mapred.child.java.opts first from 200m to 400 m to 800m and I still have the same issue. It seems to fail near the 100% mapper mark
I checked the log files and the only thing that it does output is java heap space error. Nothing more.
Any help would be appreciated.
Thanks
Subash
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le pr?sent courriel et toute pi?ce jointe qui l'accompagne sont confidentiels, prot?g?s par le droit d'auteur et peuvent ?tre couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autoris?e est interdite. Si vous n'?tes pas le destinataire pr?vu de ce courriel, supprimez-le et contactez imm?diatement l'exp?diteur. Veuillez penser ? l'environnement avant d'imprimer le pr?sent courriel
RE: Java heap space error
Posted by "Kartashov, Andy" <An...@mpac.ca>.
Subash,
I have been experiencing this type of an error at some point and no matter how much I played with heap size it didn't work. What I found out at the end is I was running out of physical memory. My output file was about 4Gb with only 2.5Gb of free space available. Check you space with "$hadop fs -df
From: Subash D'Souza [mailto:sdsouza@truecar.com]
Sent: Sunday, October 21, 2012 9:19 AM
To: user@hadoop.apache.org
Subject: Java heap space error
I'm running CDH 4 on a 4 node cluster each with 96 G of RAM. Up until last week the cluster was running until there was an error in the name node log file and I had to reformat it put the data back
Now when I run hive on YARN. I keep getting a Java heap space error. Based on the research I did. I upped the my mapred.child.java.opts first from 200m to 400 m to 800m and I still have the same issue. It seems to fail near the 100% mapper mark
I checked the log files and the only thing that it does output is java heap space error. Nothing more.
Any help would be appreciated.
Thanks
Subash
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le pr?sent courriel et toute pi?ce jointe qui l'accompagne sont confidentiels, prot?g?s par le droit d'auteur et peuvent ?tre couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autoris?e est interdite. Si vous n'?tes pas le destinataire pr?vu de ce courriel, supprimez-le et contactez imm?diatement l'exp?diteur. Veuillez penser ? l'environnement avant d'imprimer le pr?sent courriel
Re: Java heap space error
Posted by Subash D'Souza <sd...@truecar.com>.
Here's the mapped-site and yarn-site configs
Mapred-site.xml
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapred.job.tracker</name>
<value>hadoop1.rad.wc.truecarcorp.com:8021</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>hadoop1.rad.wc.truecarcorp.com:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>hadoop1.rad.wc.truecarcorp.com:19888</value>
</property>
<property>
<name>mapreduce.map.output.compress</name>
<value>true</value>
</property>
<property>
<name>mapred.map.output.compress.codec</name>
<value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>
<property>
<name>mapred.child.java.opts</name>
<value>-Xmx800m</value>
</property>
<property>
<name>mapred.map.child.java.opts</name>
<value>-Xmx800m</value>
</property>
<property>
<name>mapred.reduce.child.java.opts</name>
<value>-Xmx800m</value>
</property>
</configuration>
Yarn-site.xml
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce.shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>hadoop1.rad.wc.truecarcorp.com:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>hadoop1.rad.wc.truecarcorp.com:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>hadoop1.rad.wc.truecarcorp.com:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>hadoop1.rad.wc.truecarcorp.com:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>hadoop1.rad.wc.truecarcorp.com:8088</value>
</property>
<property>
<description>Where to aggregate logs to.</description>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/var/log/hadoop-yarn/apps</value>
</property>
<property>
<description>Classpath for typical applications.</description>
<name>yarn.application.classpath</name>
<value>
$HADOOP_CONF_DIR,
$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,
$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,
$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,
$YARN_HOME/*,$YARN_HOME/lib/*
</value>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/home/data/1/yarn/local,/home/data/2/yarn/local,/home/data/3/yarn/lo
cal</value>
</property>
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>/home/data/1/yarn/logs,/home/data/2/yarn/logs,/home/data/3/yarn/logs
</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>hadoop1.rad.wc.truecarcorp.com:19888</value>
</property>
<property>
<name>yarn.app.mapreduce.am.staging-dir</name>
<value>/home/data/tmp</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>84000</value>
</property>
</configuration>
On 10/21/12 7:22 AM, " Marcos Ortiz Valmaseda" <ml...@uci.cu> wrote:
>Regards, Subash.
>Can you share more information about your YARN cluster?
>
>----- Mensaje original -----
>De: Subash D'Souza <sd...@truecar.com>
>Para: user@hadoop.apache.org
>Enviado: Sun, 21 Oct 2012 09:18:43 -0400 (CDT)
>Asunto: Java heap space error
>
>I'm running CDH 4 on a 4 node cluster each with 96 G of RAM. Up until
>last week the cluster was running until there was an error in the name
>node log file and I had to reformat it put the data back
>
>Now when I run hive on YARN. I keep getting a Java heap space error.
>Based on the research I did. I upped the my mapred.child.java.opts first
>from 200m to 400 m to 800m and I still have the same issue. It seems to
>fail near the 100% mapper mark
>
>I checked the log files and the only thing that it does output is java
>heap space error. Nothing more.
>
>Any help would be appreciated.
>
>Thanks
>Subash
>
>
>
>10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS
>INFORMATICAS...
>CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
>
>http://www.uci.cu
>http://www.facebook.com/universidad.uci
>http://www.flickr.com/photos/universidad_uci
>
>
>
>10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS
>INFORMATICAS...
>CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
>
>http://www.uci.cu
>http://www.facebook.com/universidad.uci
>http://www.flickr.com/photos/universidad_uci
Re: Java heap space error
Posted by Subash D'Souza <sd...@truecar.com>.
Here's the mapped-site and yarn-site configs
Mapred-site.xml
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapred.job.tracker</name>
<value>hadoop1.rad.wc.truecarcorp.com:8021</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>hadoop1.rad.wc.truecarcorp.com:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>hadoop1.rad.wc.truecarcorp.com:19888</value>
</property>
<property>
<name>mapreduce.map.output.compress</name>
<value>true</value>
</property>
<property>
<name>mapred.map.output.compress.codec</name>
<value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>
<property>
<name>mapred.child.java.opts</name>
<value>-Xmx800m</value>
</property>
<property>
<name>mapred.map.child.java.opts</name>
<value>-Xmx800m</value>
</property>
<property>
<name>mapred.reduce.child.java.opts</name>
<value>-Xmx800m</value>
</property>
</configuration>
Yarn-site.xml
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce.shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>hadoop1.rad.wc.truecarcorp.com:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>hadoop1.rad.wc.truecarcorp.com:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>hadoop1.rad.wc.truecarcorp.com:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>hadoop1.rad.wc.truecarcorp.com:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>hadoop1.rad.wc.truecarcorp.com:8088</value>
</property>
<property>
<description>Where to aggregate logs to.</description>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/var/log/hadoop-yarn/apps</value>
</property>
<property>
<description>Classpath for typical applications.</description>
<name>yarn.application.classpath</name>
<value>
$HADOOP_CONF_DIR,
$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,
$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,
$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,
$YARN_HOME/*,$YARN_HOME/lib/*
</value>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/home/data/1/yarn/local,/home/data/2/yarn/local,/home/data/3/yarn/lo
cal</value>
</property>
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>/home/data/1/yarn/logs,/home/data/2/yarn/logs,/home/data/3/yarn/logs
</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>hadoop1.rad.wc.truecarcorp.com:19888</value>
</property>
<property>
<name>yarn.app.mapreduce.am.staging-dir</name>
<value>/home/data/tmp</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>84000</value>
</property>
</configuration>
On 10/21/12 7:22 AM, " Marcos Ortiz Valmaseda" <ml...@uci.cu> wrote:
>Regards, Subash.
>Can you share more information about your YARN cluster?
>
>----- Mensaje original -----
>De: Subash D'Souza <sd...@truecar.com>
>Para: user@hadoop.apache.org
>Enviado: Sun, 21 Oct 2012 09:18:43 -0400 (CDT)
>Asunto: Java heap space error
>
>I'm running CDH 4 on a 4 node cluster each with 96 G of RAM. Up until
>last week the cluster was running until there was an error in the name
>node log file and I had to reformat it put the data back
>
>Now when I run hive on YARN. I keep getting a Java heap space error.
>Based on the research I did. I upped the my mapred.child.java.opts first
>from 200m to 400 m to 800m and I still have the same issue. It seems to
>fail near the 100% mapper mark
>
>I checked the log files and the only thing that it does output is java
>heap space error. Nothing more.
>
>Any help would be appreciated.
>
>Thanks
>Subash
>
>
>
>10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS
>INFORMATICAS...
>CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
>
>http://www.uci.cu
>http://www.facebook.com/universidad.uci
>http://www.flickr.com/photos/universidad_uci
>
>
>
>10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS
>INFORMATICAS...
>CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
>
>http://www.uci.cu
>http://www.facebook.com/universidad.uci
>http://www.flickr.com/photos/universidad_uci
Re: Java heap space error
Posted by Subash D'Souza <sd...@truecar.com>.
Here's the mapped-site and yarn-site configs
Mapred-site.xml
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapred.job.tracker</name>
<value>hadoop1.rad.wc.truecarcorp.com:8021</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>hadoop1.rad.wc.truecarcorp.com:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>hadoop1.rad.wc.truecarcorp.com:19888</value>
</property>
<property>
<name>mapreduce.map.output.compress</name>
<value>true</value>
</property>
<property>
<name>mapred.map.output.compress.codec</name>
<value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>
<property>
<name>mapred.child.java.opts</name>
<value>-Xmx800m</value>
</property>
<property>
<name>mapred.map.child.java.opts</name>
<value>-Xmx800m</value>
</property>
<property>
<name>mapred.reduce.child.java.opts</name>
<value>-Xmx800m</value>
</property>
</configuration>
Yarn-site.xml
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce.shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>hadoop1.rad.wc.truecarcorp.com:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>hadoop1.rad.wc.truecarcorp.com:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>hadoop1.rad.wc.truecarcorp.com:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>hadoop1.rad.wc.truecarcorp.com:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>hadoop1.rad.wc.truecarcorp.com:8088</value>
</property>
<property>
<description>Where to aggregate logs to.</description>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/var/log/hadoop-yarn/apps</value>
</property>
<property>
<description>Classpath for typical applications.</description>
<name>yarn.application.classpath</name>
<value>
$HADOOP_CONF_DIR,
$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,
$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,
$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,
$YARN_HOME/*,$YARN_HOME/lib/*
</value>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/home/data/1/yarn/local,/home/data/2/yarn/local,/home/data/3/yarn/lo
cal</value>
</property>
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>/home/data/1/yarn/logs,/home/data/2/yarn/logs,/home/data/3/yarn/logs
</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>hadoop1.rad.wc.truecarcorp.com:19888</value>
</property>
<property>
<name>yarn.app.mapreduce.am.staging-dir</name>
<value>/home/data/tmp</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>84000</value>
</property>
</configuration>
On 10/21/12 7:22 AM, " Marcos Ortiz Valmaseda" <ml...@uci.cu> wrote:
>Regards, Subash.
>Can you share more information about your YARN cluster?
>
>----- Mensaje original -----
>De: Subash D'Souza <sd...@truecar.com>
>Para: user@hadoop.apache.org
>Enviado: Sun, 21 Oct 2012 09:18:43 -0400 (CDT)
>Asunto: Java heap space error
>
>I'm running CDH 4 on a 4 node cluster each with 96 G of RAM. Up until
>last week the cluster was running until there was an error in the name
>node log file and I had to reformat it put the data back
>
>Now when I run hive on YARN. I keep getting a Java heap space error.
>Based on the research I did. I upped the my mapred.child.java.opts first
>from 200m to 400 m to 800m and I still have the same issue. It seems to
>fail near the 100% mapper mark
>
>I checked the log files and the only thing that it does output is java
>heap space error. Nothing more.
>
>Any help would be appreciated.
>
>Thanks
>Subash
>
>
>
>10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS
>INFORMATICAS...
>CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
>
>http://www.uci.cu
>http://www.facebook.com/universidad.uci
>http://www.flickr.com/photos/universidad_uci
>
>
>
>10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS
>INFORMATICAS...
>CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
>
>http://www.uci.cu
>http://www.facebook.com/universidad.uci
>http://www.flickr.com/photos/universidad_uci
Re: Java heap space error
Posted by Subash D'Souza <sd...@truecar.com>.
Here's the mapped-site and yarn-site configs
Mapred-site.xml
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapred.job.tracker</name>
<value>hadoop1.rad.wc.truecarcorp.com:8021</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>hadoop1.rad.wc.truecarcorp.com:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>hadoop1.rad.wc.truecarcorp.com:19888</value>
</property>
<property>
<name>mapreduce.map.output.compress</name>
<value>true</value>
</property>
<property>
<name>mapred.map.output.compress.codec</name>
<value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>
<property>
<name>mapred.child.java.opts</name>
<value>-Xmx800m</value>
</property>
<property>
<name>mapred.map.child.java.opts</name>
<value>-Xmx800m</value>
</property>
<property>
<name>mapred.reduce.child.java.opts</name>
<value>-Xmx800m</value>
</property>
</configuration>
Yarn-site.xml
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce.shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>hadoop1.rad.wc.truecarcorp.com:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>hadoop1.rad.wc.truecarcorp.com:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>hadoop1.rad.wc.truecarcorp.com:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>hadoop1.rad.wc.truecarcorp.com:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>hadoop1.rad.wc.truecarcorp.com:8088</value>
</property>
<property>
<description>Where to aggregate logs to.</description>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/var/log/hadoop-yarn/apps</value>
</property>
<property>
<description>Classpath for typical applications.</description>
<name>yarn.application.classpath</name>
<value>
$HADOOP_CONF_DIR,
$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,
$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,
$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,
$YARN_HOME/*,$YARN_HOME/lib/*
</value>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/home/data/1/yarn/local,/home/data/2/yarn/local,/home/data/3/yarn/lo
cal</value>
</property>
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>/home/data/1/yarn/logs,/home/data/2/yarn/logs,/home/data/3/yarn/logs
</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>hadoop1.rad.wc.truecarcorp.com:19888</value>
</property>
<property>
<name>yarn.app.mapreduce.am.staging-dir</name>
<value>/home/data/tmp</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>84000</value>
</property>
</configuration>
On 10/21/12 7:22 AM, " Marcos Ortiz Valmaseda" <ml...@uci.cu> wrote:
>Regards, Subash.
>Can you share more information about your YARN cluster?
>
>----- Mensaje original -----
>De: Subash D'Souza <sd...@truecar.com>
>Para: user@hadoop.apache.org
>Enviado: Sun, 21 Oct 2012 09:18:43 -0400 (CDT)
>Asunto: Java heap space error
>
>I'm running CDH 4 on a 4 node cluster each with 96 G of RAM. Up until
>last week the cluster was running until there was an error in the name
>node log file and I had to reformat it put the data back
>
>Now when I run hive on YARN. I keep getting a Java heap space error.
>Based on the research I did. I upped the my mapred.child.java.opts first
>from 200m to 400 m to 800m and I still have the same issue. It seems to
>fail near the 100% mapper mark
>
>I checked the log files and the only thing that it does output is java
>heap space error. Nothing more.
>
>Any help would be appreciated.
>
>Thanks
>Subash
>
>
>
>10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS
>INFORMATICAS...
>CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
>
>http://www.uci.cu
>http://www.facebook.com/universidad.uci
>http://www.flickr.com/photos/universidad_uci
>
>
>
>10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS
>INFORMATICAS...
>CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
>
>http://www.uci.cu
>http://www.facebook.com/universidad.uci
>http://www.flickr.com/photos/universidad_uci
Re: Java heap space error
Posted by Marcos Ortiz Valmaseda <ml...@uci.cu>.
Regards, Subash.
Can you share more information about your YARN cluster?
----- Mensaje original -----
De: Subash D'Souza <sd...@truecar.com>
Para: user@hadoop.apache.org
Enviado: Sun, 21 Oct 2012 09:18:43 -0400 (CDT)
Asunto: Java heap space error
I'm running CDH 4 on a 4 node cluster each with 96 G of RAM. Up until last week the cluster was running until there was an error in the name node log file and I had to reformat it put the data back
Now when I run hive on YARN. I keep getting a Java heap space error. Based on the research I did. I upped the my mapred.child.java.opts first from 200m to 400 m to 800m and I still have the same issue. It seems to fail near the 100% mapper mark
I checked the log files and the only thing that it does output is java heap space error. Nothing more.
Any help would be appreciated.
Thanks
Subash
10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci
10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci
Re: Java heap space error
Posted by Marcos Ortiz Valmaseda <ml...@uci.cu>.
Regards, Subash.
Can you share more information about your YARN cluster?
----- Mensaje original -----
De: Subash D'Souza <sd...@truecar.com>
Para: user@hadoop.apache.org
Enviado: Sun, 21 Oct 2012 09:18:43 -0400 (CDT)
Asunto: Java heap space error
I'm running CDH 4 on a 4 node cluster each with 96 G of RAM. Up until last week the cluster was running until there was an error in the name node log file and I had to reformat it put the data back
Now when I run hive on YARN. I keep getting a Java heap space error. Based on the research I did. I upped the my mapred.child.java.opts first from 200m to 400 m to 800m and I still have the same issue. It seems to fail near the 100% mapper mark
I checked the log files and the only thing that it does output is java heap space error. Nothing more.
Any help would be appreciated.
Thanks
Subash
10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci
10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci
Re: Java heap space error
Posted by Marcos Ortiz Valmaseda <ml...@uci.cu>.
Regards, Subash.
Can you share more information about your YARN cluster?
----- Mensaje original -----
De: Subash D'Souza <sd...@truecar.com>
Para: user@hadoop.apache.org
Enviado: Sun, 21 Oct 2012 09:18:43 -0400 (CDT)
Asunto: Java heap space error
I'm running CDH 4 on a 4 node cluster each with 96 G of RAM. Up until last week the cluster was running until there was an error in the name node log file and I had to reformat it put the data back
Now when I run hive on YARN. I keep getting a Java heap space error. Based on the research I did. I upped the my mapred.child.java.opts first from 200m to 400 m to 800m and I still have the same issue. It seems to fail near the 100% mapper mark
I checked the log files and the only thing that it does output is java heap space error. Nothing more.
Any help would be appreciated.
Thanks
Subash
10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci
10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci
Re: Java heap space error
Posted by Marcos Ortiz Valmaseda <ml...@uci.cu>.
Regards, Subash.
Can you share more information about your YARN cluster?
----- Mensaje original -----
De: Subash D'Souza <sd...@truecar.com>
Para: user@hadoop.apache.org
Enviado: Sun, 21 Oct 2012 09:18:43 -0400 (CDT)
Asunto: Java heap space error
I'm running CDH 4 on a 4 node cluster each with 96 G of RAM. Up until last week the cluster was running until there was an error in the name node log file and I had to reformat it put the data back
Now when I run hive on YARN. I keep getting a Java heap space error. Based on the research I did. I upped the my mapred.child.java.opts first from 200m to 400 m to 800m and I still have the same issue. It seems to fail near the 100% mapper mark
I checked the log files and the only thing that it does output is java heap space error. Nothing more.
Any help would be appreciated.
Thanks
Subash
10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci
10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci