You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by "Hammerman, Joseph" <Jo...@clearchannel.com> on 2012/08/02 20:52:02 UTC

Unable to execute reduce job on hive 0.8.1 against Yarn CDH4 Hadoop

Hello Hive users,

I am attempting to execute a reduce job against a Yarn enabled Hadoop
cluster. Plain map jobs run without error. I am able to use the Hadoop
streaming jars to execute simple map and reduce jobs, so it seems that
something must be wrong (most likely misconfigured) with my Hive
integration.

Package versions:
hive-0.8.1+61-1.cdh4.0.1.p0.1.el6.noarch
hadoop-2.0.0+91-1.cdh4.0.1.p0.1.el6.x86_64
hadoop-hdfs-secondarynamenode-2.0.0+91-1.cdh4.0.1.p0.1.el6.x86_64
hadoop-yarn-proxyserver-2.0.0+91-1.cdh4.0.1.p0.1.el6.x86_64
hadoop-yarn-2.0.0+91-1.cdh4.0.1.p0.1.el6.x86_64
hadoop-hdfs-2.0.0+91-1.cdh4.0.1.p0.1.el6.x86_64
hadoop-0.20-mapreduce-0.20.2+1216-1.cdh4.0.1.p0.1.el6.x86_64
hadoop-client-2.0.0+91-1.cdh4.0.1.p0.1.el6.x86_64
hadoop-mapreduce-historyserver-2.0.0+91-1.cdh4.0.1.p0.1.el6.x86_64
hadoop-yarn-resourcemanager-2.0.0+91-1.cdh4.0.1.p0.1.el6.x86_64
hadoop-mapreduce-2.0.0+91-1.cdh4.0.1.p0.1.el6.x86_64


The error output is only the process registering that it cannot log it's
job output with the JobHistory server (I think this is known issue with
Hive / Yarn?). Here is a link to pastebin with the full error output:
http://pastebin.com/DH9auij7

My yarn-site.xml and hive-site.xml are here:http://pastebin.com/LK3rnrgb

A side note: if I set HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce, the
hive M/R process will hang indefinitely outputting:

2012-08-02 15:15:22,032 null map = 0%,  reduce = 0%


Any assistance would be greatly appreciated.

This is a test cluster, so we can pursue any troubleshooting steps that
may illuminate the issue.

Regards,
Joseph Hammerman