You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Jim Colestock <jc...@ramblingredneck.com> on 2013/08/22 14:16:57 UTC
RE: Hive 0.11 with Cloudera CHD4.3 MR v1
Hello All,
Sorry for not replying to the thread, but I never received any of your responses.. Hopefully this info will help:
@kulkarni.swarnim
Here is the error I'm getting on task tracker for the mapper:
2013-08-21 15:34:32,491 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
2013-08-21 15:34:32,494 WARN org.apache.hadoop.mapred.Child: Error running child
java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:230)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:381)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:374)
at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:394)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:215)
... 11 more
2013-08-21 15:34:32,498 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task
@Srinivas Surasani
Hopefully the above helps..
@Lars Francke
Thanks, I'll take a peek at those 2 patches.. But I've just recently pulled down fresh code and recompiled everything and I'm still getting the above error.
And for reference, below is my original email to the list.. Thanks all for you help!
JC
Hello All,
Has anyone been successful at running hive 0.11 with Cloudera CDH 4.3?
I've been able to get hive to connect to my metadb (which is in Postgres). Verified by doing a show tables.. I can run explain and describes on tables, but when I try to run anything that fires off an M/R job, I get the following error:
hive>select count(*) from tableA;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Starting Job = job_201307112247_13816, Tracking URL = http://master:50030/jobdetails.jsp?jobid=job_201307112247_13816
Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_201307112247_13816
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2013-07-12 02:11:42,829 Stage-1 map = 0%, reduce = 0%
2013-07-12 02:12:08,173 Stage-1 map = 100%, reduce = 100%
Ended Job = job_201307112247_13816 with errors
Error during job, obtaining debugging information...
Job Tracking URL: http://master:50030/jobdetails.jsp?jobid=job_201307112247_13816
Examining task ID: task_201307112247_13816_m_000002 (and more) from job job_201307112247_13816
Exception in thread "Thread-19" java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/util/HostUtil
at org.apache.hadoop.hive.shims.Hadoop23Shims.getTaskAttemptLogUrl(Hadoop23Shims.java:61)
at org.apache.hadoop.hive.ql.exec.JobDebugger$TaskInfoGrabber.getTaskInfos(JobDebugger.java:186)
at org.apache.hadoop.hive.ql.exec.JobDebugger$TaskInfoGrabber.run(JobDebugger.java:142)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.util.HostUtil
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
... 4 more
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
MapReduce Jobs Launched:
Job 0: Map: 1 Reduce: 1 HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec
I'm using my configs from hive 0.10, which works with no issues and this was pretty much a drop in replacement on the machine that hadoop 0.10 was running on..
I've done a bunch of googling around and have found a bunch of other folks that have have had the same issue, but no solid answers..
Thanks in advance for any help..
JC
Re: Hive 0.11 with Cloudera CHD4.3 MR v1
Posted by Justin Workman <ju...@gmail.com>.
I am not using yarn either, but this value MUST be set. It is one of the 2
issues I ran into. Something to do with the path to the intermediate files
not being correct for MR1, this sets that value to a bogus value. I have
mine set as
<property>
<name>yarn.resourcemanager.address</name>
<value>foobar</value>
</property>
In my hive-site.xml.
On Fri, Aug 23, 2013 at 11:36 AM, Jim Colestock <jc...@ramblingredneck.com>wrote:
> I'm not using Yarn, But I'll give that a shot..
>
> Is hive 11 dependent on Yarn?
>
> JC
>
>
> On Aug 23, 2013, at 1:19 PM, Justin Workman <ju...@gmail.com>
> wrote:
>
> This looks similar to the problem you get when you don't set
> yarn.resourcemanager.address.
>
> Sent from my iPhone
>
> On Aug 23, 2013, at 11:12 AM, Jim Colestock <jc...@ramblingredneck.com>
> wrote:
>
> Hello Justin,
>
> Thanks so much for the write up.. I'm compiling now.. I did find one
> error in your instructions:
>
>
> patch -p0 > HIVE-4619.D10971.1.path
>
>
> I believe it should be:
>
> patch -p0 *<* HIVE-4619.D10971.1.patch
>
>
>
> But alas, I still doesn't work for me. I managed to get it to compile
> after using maven to install a zookeeper dependency.. But when I try to do
> anything in hive that requires an M/R job, I get the following:
>
> 2013-08-23 13:07:07,865 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
> Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
> 2013-08-23 13:07:07,867 WARN org.apache.hadoop.mapred.Child: Error running
> child
> java.lang.RuntimeException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:230)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:381)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:374)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:394)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> at org.apache.hadoop.mapred.Child.main(Child.java:262)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:215)
> ... 11 more
>
> 2013-08-23 13:07:07,871 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task
>
>
> Any thoughts?
>
> Thanks again,
>
> JC
>
>
>
>
>
>
>
> On Aug 22, 2013, at 11:15 PM, Justin Workman <ju...@gmail.com>
> wrote:
>
> For anyone interested, I have finished my blog on getting Hive-0.11.0
> working with CDH4.3.0. It is a hack and can be made better but works like a
> charm.
>
> http://www.justinjworkman.com/big-data/hive-0-11-0-on-cloudera/
>
>
>
> On Thu, Aug 22, 2013 at 8:29 AM, Jim Colestock <jc...@ramblingredneck.com>wrote:
>
>> Justin,
>>
>> That would be much appreciated!
>>
>> Thanks Again,
>>
>> JC
>>
>> On Aug 22, 2013, at 9:18 AM, Justin Workman <ju...@gmail.com>
>> wrote:
>>
>> I do have this working on CHD4.3.0 with MR1 and Kerberos. I am finishing
>> up a blog today explaining what was needed to get it working. I will reply
>> with the link later today.
>>
>> Justin
>>
>> Sent from my iPhone
>>
>> On Aug 22, 2013, at 6:17 AM, Jim Colestock <jc...@ramblingredneck.com>
>> wrote:
>>
>> Hello All,
>>
>> Sorry for not replying to the thread, but I never received any of your
>> responses.. Hopefully this info will help:
>>
>> @kulkarni.swarnim<http://search.gmane.org/?author=kulkarni.swarnim-Re5JQEeQqe8AvxtiuMwx3w%40public.gmane.org&sort=date>
>>
>> Here is the error I'm getting on task tracker for the mapper:
>>
>> 2013-08-21 15:34:32,491 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
>> 2013-08-21 15:34:32,494 WARN org.apache.hadoop.mapred.Child: Error running child
>> java.lang.RuntimeException: java.lang.NullPointerException
>> at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:230)
>> at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255)
>> at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:381)
>> at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:374)
>> at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540)
>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:394)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
>> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>> at org.apache.hadoop.mapred.Child.main(Child.java:262)
>> Caused by: java.lang.NullPointerException
>> at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:215)
>> ... 11 more
>> 2013-08-21 15:34:32,498 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task
>>
>>
>> @Srinivas Surasani<http://search.gmane.org/?author=Srinivas+Surasani&sort=date>
>> Hopefully the above helps..
>>
>> @Lars Francke <http://search.gmane.org/?author=Lars+Francke&sort=date>
>> Thanks, I'll take a peek at those 2 patches.. But I've just recently
>> pulled down fresh code and recompiled everything and I'm still getting the
>> above error.
>>
>> And for reference, below is my original email to the list.. Thanks all
>> for you help!
>>
>>
>> JC
>>
>>
>>
>> Hello All,
>>
>> Has anyone been successful at running hive 0.11 with Cloudera CDH 4.3?
>>
>> I've been able to get hive to connect to my metadb (which is in
>> Postgres). Verified by doing a show tables.. I can run explain and
>> describes on tables, but when I try to run anything that fires off an M/R
>> job, I get the following error:
>>
>> hive>select count(*) from tableA;
>> Total MapReduce jobs = 1
>> Launching Job 1 out of 1
>> Number of reduce tasks determined at compile time: 1
>> In order to change the average load for a reducer (in bytes):
>> set hive.exec.reducers.bytes.per.reducer=<number>
>> In order to limit the maximum number of reducers:
>> set hive.exec.reducers.max=<number>
>> In order to set a constant number of reducers:
>> set mapred.reduce.tasks=<number>
>> Starting Job = job_201307112247_13816, Tracking URL =
>> http://master:50030/jobdetails.jsp?jobid=job_201307112247_13816
>> Kill Command = /usr/lib/hadoop/bin/hadoop job -kill
>> job_201307112247_13816
>> Hadoop job information for Stage-1: number of mappers: 1; number of
>> reducers: 1
>> 2013-07-12 02:11:42,829 Stage-1 map = 0%, reduce = 0%
>> 2013-07-12 02:12:08,173 Stage-1 map = 100%, reduce = 100%
>> Ended Job = job_201307112247_13816 with errors
>> Error during job, obtaining debugging information...
>> Job Tracking URL:
>> http://master:50030/jobdetails.jsp?jobid=job_201307112247_13816
>> Examining task ID: task_201307112247_13816_m_000002 (and more) from job
>> job_201307112247_13816
>> Exception in thread "Thread-19" java.lang.NoClassDefFoundError:
>> org/apache/hadoop/mapreduce/util/HostUtil
>> at
>> org.apache.hadoop.hive.shims.Hadoop23Shims.getTaskAttemptLogUrl(Hadoop23Shims.java:61)
>> at
>> org.apache.hadoop.hive.ql.exec.JobDebugger$TaskInfoGrabber.getTaskInfos(JobDebugger.java:186)
>> at
>> org.apache.hadoop.hive.ql.exec.JobDebugger$TaskInfoGrabber.run(JobDebugger.java:142)
>> at java.lang.Thread.run(Thread.java:619)
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.hadoop.mapreduce.util.HostUtil
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>> ... 4 more
>> FAILED: Execution Error, return code 2 from
>> org.apache.hadoop.hive.ql.exec.MapRedTask
>> MapReduce Jobs Launched:
>> Job 0: Map: 1 Reduce: 1 HDFS Read: 0 HDFS Write: 0 FAIL
>> Total MapReduce CPU Time Spent: 0 msec
>>
>>
>> I'm using my configs from hive 0.10, which works with no issues and this
>> was pretty much a drop in replacement on the machine that hadoop 0.10 was
>> running on..
>>
>> I've done a bunch of googling around and have found a bunch of other
>> folks that have have had the same issue, but no solid answers..
>>
>> Thanks in advance for any help..
>>
>> JC
>>
>>
>>
>>
>
>
>
Re: Hive 0.11 with Cloudera CHD4.3 MR v1
Posted by Jim Colestock <jc...@ramblingredneck.com>.
I'm not using Yarn, But I'll give that a shot..
Is hive 11 dependent on Yarn?
JC
On Aug 23, 2013, at 1:19 PM, Justin Workman <ju...@gmail.com> wrote:
> This looks similar to the problem you get when you don't set yarn.resourcemanager.address.
>
> Sent from my iPhone
>
> On Aug 23, 2013, at 11:12 AM, Jim Colestock <jc...@ramblingredneck.com> wrote:
>
>> Hello Justin,
>>
>> Thanks so much for the write up.. I'm compiling now.. I did find one error in your instructions:
>>
>> patch -p0 > HIVE-4619.D10971.1.path
>>
>> I believe it should be:
>>
>> patch -p0 < HIVE-4619.D10971.1.patch
>>
>>
>> But alas, I still doesn't work for me. I managed to get it to compile after using maven to install a zookeeper dependency.. But when I try to do anything in hive that requires an M/R job, I get the following:
>>
>> 2013-08-23 13:07:07,865 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
>> 2013-08-23 13:07:07,867 WARN org.apache.hadoop.mapred.Child: Error running child
>> java.lang.RuntimeException: java.lang.NullPointerException
>> at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:230)
>> at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255)
>> at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:381)
>> at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:374)
>> at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540)
>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:394)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
>> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>> at org.apache.hadoop.mapred.Child.main(Child.java:262)
>> Caused by: java.lang.NullPointerException
>> at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:215)
>> ... 11 more
>> 2013-08-23 13:07:07,871 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task
>>
>> Any thoughts?
>>
>> Thanks again,
>>
>> JC
>>
>>
>>
>>
>>
>>
>>
>> On Aug 22, 2013, at 11:15 PM, Justin Workman <ju...@gmail.com> wrote:
>>
>>> For anyone interested, I have finished my blog on getting Hive-0.11.0 working with CDH4.3.0. It is a hack and can be made better but works like a charm.
>>>
>>> http://www.justinjworkman.com/big-data/hive-0-11-0-on-cloudera/
>>>
>>>
>>>
>>> On Thu, Aug 22, 2013 at 8:29 AM, Jim Colestock <jc...@ramblingredneck.com> wrote:
>>> Justin,
>>>
>>> That would be much appreciated!
>>>
>>> Thanks Again,
>>>
>>> JC
>>>
>>> On Aug 22, 2013, at 9:18 AM, Justin Workman <ju...@gmail.com> wrote:
>>>
>>>> I do have this working on CHD4.3.0 with MR1 and Kerberos. I am finishing up a blog today explaining what was needed to get it working. I will reply with the link later today.
>>>>
>>>> Justin
>>>>
>>>> Sent from my iPhone
>>>>
>>>> On Aug 22, 2013, at 6:17 AM, Jim Colestock <jc...@ramblingredneck.com> wrote:
>>>>
>>>>> Hello All,
>>>>>
>>>>> Sorry for not replying to the thread, but I never received any of your responses.. Hopefully this info will help:
>>>>>
>>>>> @kulkarni.swarnim
>>>>>
>>>>> Here is the error I'm getting on task tracker for the mapper:
>>>>>
>>>>> 2013-08-21 15:34:32,491 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
>>>>> 2013-08-21 15:34:32,494 WARN org.apache.hadoop.mapred.Child: Error running child
>>>>> java.lang.RuntimeException: java.lang.NullPointerException
>>>>> at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:230)
>>>>> at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255)
>>>>> at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:381)
>>>>> at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:374)
>>>>> at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540)
>>>>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:394)
>>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
>>>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>>>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>>>>> at org.apache.hadoop.mapred.Child.main(Child.java:262)
>>>>> Caused by: java.lang.NullPointerException
>>>>> at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:215)
>>>>> ... 11 more
>>>>> 2013-08-21 15:34:32,498 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task
>>>>>
>>>>> @Srinivas Surasani
>>>>> Hopefully the above helps..
>>>>>
>>>>> @Lars Francke
>>>>> Thanks, I'll take a peek at those 2 patches.. But I've just recently pulled down fresh code and recompiled everything and I'm still getting the above error.
>>>>>
>>>>> And for reference, below is my original email to the list.. Thanks all for you help!
>>>>>
>>>>>
>>>>> JC
>>>>>
>>>>>
>>>>>
>>>>> Hello All,
>>>>>
>>>>> Has anyone been successful at running hive 0.11 with Cloudera CDH 4.3?
>>>>>
>>>>> I've been able to get hive to connect to my metadb (which is in Postgres). Verified by doing a show tables.. I can run explain and describes on tables, but when I try to run anything that fires off an M/R job, I get the following error:
>>>>>
>>>>> hive>select count(*) from tableA;
>>>>> Total MapReduce jobs = 1
>>>>> Launching Job 1 out of 1
>>>>> Number of reduce tasks determined at compile time: 1
>>>>> In order to change the average load for a reducer (in bytes):
>>>>> set hive.exec.reducers.bytes.per.reducer=<number>
>>>>> In order to limit the maximum number of reducers:
>>>>> set hive.exec.reducers.max=<number>
>>>>> In order to set a constant number of reducers:
>>>>> set mapred.reduce.tasks=<number>
>>>>> Starting Job = job_201307112247_13816, Tracking URL = http://master:50030/jobdetails.jsp?jobid=job_201307112247_13816
>>>>> Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_201307112247_13816
>>>>> Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
>>>>> 2013-07-12 02:11:42,829 Stage-1 map = 0%, reduce = 0%
>>>>> 2013-07-12 02:12:08,173 Stage-1 map = 100%, reduce = 100%
>>>>> Ended Job = job_201307112247_13816 with errors
>>>>> Error during job, obtaining debugging information...
>>>>> Job Tracking URL: http://master:50030/jobdetails.jsp?jobid=job_201307112247_13816
>>>>> Examining task ID: task_201307112247_13816_m_000002 (and more) from job job_201307112247_13816
>>>>> Exception in thread "Thread-19" java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/util/HostUtil
>>>>> at org.apache.hadoop.hive.shims.Hadoop23Shims.getTaskAttemptLogUrl(Hadoop23Shims.java:61)
>>>>> at org.apache.hadoop.hive.ql.exec.JobDebugger$TaskInfoGrabber.getTaskInfos(JobDebugger.java:186)
>>>>> at org.apache.hadoop.hive.ql.exec.JobDebugger$TaskInfoGrabber.run(JobDebugger.java:142)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.util.HostUtil
>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>>>> ... 4 more
>>>>> FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
>>>>> MapReduce Jobs Launched:
>>>>> Job 0: Map: 1 Reduce: 1 HDFS Read: 0 HDFS Write: 0 FAIL
>>>>> Total MapReduce CPU Time Spent: 0 msec
>>>>>
>>>>> I'm using my configs from hive 0.10, which works with no issues and this was pretty much a drop in replacement on the machine that hadoop 0.10 was running on..
>>>>>
>>>>> I've done a bunch of googling around and have found a bunch of other folks that have have had the same issue, but no solid answers..
>>>>>
>>>>> Thanks in advance for any help..
>>>>>
>>>>> JC
>>>>>
>>>>>
>>>
>>>
>>
Re: Hive 0.11 with Cloudera CHD4.3 MR v1
Posted by Justin Workman <ju...@gmail.com>.
This looks similar to the problem you get when you don't set
yarn.resourcemanager.address.
Sent from my iPhone
On Aug 23, 2013, at 11:12 AM, Jim Colestock <jc...@ramblingredneck.com> wrote:
Hello Justin,
Thanks so much for the write up.. I'm compiling now.. I did find one error
in your instructions:
patch -p0 > HIVE-4619.D10971.1.path
I believe it should be:
patch -p0 *<* HIVE-4619.D10971.1.patch
But alas, I still doesn't work for me. I managed to get it to compile
after using maven to install a zookeeper dependency.. But when I try to do
anything in hive that requires an M/R job, I get the following:
2013-08-23 13:07:07,865 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
2013-08-23 13:07:07,867 WARN org.apache.hadoop.mapred.Child: Error running
child
java.lang.RuntimeException: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:230)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:381)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:374)
at
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:394)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:215)
... 11 more
2013-08-23 13:07:07,871 INFO org.apache.hadoop.mapred.Task: Runnning
cleanup for the task
Any thoughts?
Thanks again,
JC
On Aug 22, 2013, at 11:15 PM, Justin Workman <ju...@gmail.com>
wrote:
For anyone interested, I have finished my blog on getting Hive-0.11.0
working with CDH4.3.0. It is a hack and can be made better but works like a
charm.
http://www.justinjworkman.com/big-data/hive-0-11-0-on-cloudera/
On Thu, Aug 22, 2013 at 8:29 AM, Jim Colestock <jc...@ramblingredneck.com>wrote:
> Justin,
>
> That would be much appreciated!
>
> Thanks Again,
>
> JC
>
> On Aug 22, 2013, at 9:18 AM, Justin Workman <ju...@gmail.com>
> wrote:
>
> I do have this working on CHD4.3.0 with MR1 and Kerberos. I am finishing
> up a blog today explaining what was needed to get it working. I will reply
> with the link later today.
>
> Justin
>
> Sent from my iPhone
>
> On Aug 22, 2013, at 6:17 AM, Jim Colestock <jc...@ramblingredneck.com> wrote:
>
> Hello All,
>
> Sorry for not replying to the thread, but I never received any of your
> responses.. Hopefully this info will help:
>
> @kulkarni.swarnim<http://search.gmane.org/?author=kulkarni.swarnim-Re5JQEeQqe8AvxtiuMwx3w%40public.gmane.org&sort=date>
>
> Here is the error I'm getting on task tracker for the mapper:
>
> 2013-08-21 15:34:32,491 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
> 2013-08-21 15:34:32,494 WARN org.apache.hadoop.mapred.Child: Error running child
> java.lang.RuntimeException: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:230)
> at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255)
> at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:381)
> at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:374)
> at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:394)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> at org.apache.hadoop.mapred.Child.main(Child.java:262)
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:215)
> ... 11 more
> 2013-08-21 15:34:32,498 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task
>
>
> @Srinivas Surasani<http://search.gmane.org/?author=Srinivas+Surasani&sort=date>
> Hopefully the above helps..
>
> @Lars Francke <http://search.gmane.org/?author=Lars+Francke&sort=date>
> Thanks, I'll take a peek at those 2 patches.. But I've just recently
> pulled down fresh code and recompiled everything and I'm still getting the
> above error.
>
> And for reference, below is my original email to the list.. Thanks all
> for you help!
>
>
> JC
>
>
>
> Hello All,
>
> Has anyone been successful at running hive 0.11 with Cloudera CDH 4.3?
>
> I've been able to get hive to connect to my metadb (which is in Postgres).
> Verified by doing a show tables.. I can run explain and describes on
> tables, but when I try to run anything that fires off an M/R job, I get the
> following error:
>
> hive>select count(*) from tableA;
> Total MapReduce jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks determined at compile time: 1
> In order to change the average load for a reducer (in bytes):
> set hive.exec.reducers.bytes.per.reducer=<number>
> In order to limit the maximum number of reducers:
> set hive.exec.reducers.max=<number>
> In order to set a constant number of reducers:
> set mapred.reduce.tasks=<number>
> Starting Job = job_201307112247_13816, Tracking URL =
> http://master:50030/jobdetails.jsp?jobid=job_201307112247_13816
> Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_201307112247_13816
> Hadoop job information for Stage-1: number of mappers: 1; number of
> reducers: 1
> 2013-07-12 02:11:42,829 Stage-1 map = 0%, reduce = 0%
> 2013-07-12 02:12:08,173 Stage-1 map = 100%, reduce = 100%
> Ended Job = job_201307112247_13816 with errors
> Error during job, obtaining debugging information...
> Job Tracking URL:
> http://master:50030/jobdetails.jsp?jobid=job_201307112247_13816
> Examining task ID: task_201307112247_13816_m_000002 (and more) from job
> job_201307112247_13816
> Exception in thread "Thread-19" java.lang.NoClassDefFoundError:
> org/apache/hadoop/mapreduce/util/HostUtil
> at
> org.apache.hadoop.hive.shims.Hadoop23Shims.getTaskAttemptLogUrl(Hadoop23Shims.java:61)
> at
> org.apache.hadoop.hive.ql.exec.JobDebugger$TaskInfoGrabber.getTaskInfos(JobDebugger.java:186)
> at
> org.apache.hadoop.hive.ql.exec.JobDebugger$TaskInfoGrabber.run(JobDebugger.java:142)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.mapreduce.util.HostUtil
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> ... 4 more
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.MapRedTask
> MapReduce Jobs Launched:
> Job 0: Map: 1 Reduce: 1 HDFS Read: 0 HDFS Write: 0 FAIL
> Total MapReduce CPU Time Spent: 0 msec
>
>
> I'm using my configs from hive 0.10, which works with no issues and this
> was pretty much a drop in replacement on the machine that hadoop 0.10 was
> running on..
>
> I've done a bunch of googling around and have found a bunch of other folks
> that have have had the same issue, but no solid answers..
>
> Thanks in advance for any help..
>
> JC
>
>
>
>
Re: Hive 0.11 with Cloudera CHD4.3 MR v1
Posted by Jim Colestock <jc...@ramblingredneck.com>.
I miss the bit in the middle about adding the value to hive-site.xml (did the skip ahead bit).. So after adding:
<property>
<name>yarn.resourcemanager.address</name>
<value>foobar</value>
</property>
to hive-site.xml, It's working! Thanks so much for your help..
have a great weekend..
JC
On Aug 23, 2013, at 1:37 PM, Justin Workman <ju...@gmail.com> wrote:
> I just went through this same process again and deployed to another 5 node cluster, and it is also working as expected. I have ran anything very intensive yet, but did spawn several mapreduce jobs.
>
>
> On Fri, Aug 23, 2013 at 11:11 AM, Jim Colestock <jc...@ramblingredneck.com> wrote:
> Hello Justin,
>
> Thanks so much for the write up.. I'm compiling now.. I did find one error in your instructions:
>
> patch -p0 > HIVE-4619.D10971.1.path
>
> I believe it should be:
>
> patch -p0 < HIVE-4619.D10971.1.patch
>
>
> But alas, I still doesn't work for me. I managed to get it to compile after using maven to install a zookeeper dependency.. But when I try to do anything in hive that requires an M/R job, I get the following:
>
> 2013-08-23 13:07:07,865 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
> 2013-08-23 13:07:07,867 WARN org.apache.hadoop.mapred.Child: Error running child
>
> java.lang.RuntimeException: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:230)
> at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255)
> at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:381)
> at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:374)
> at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:394)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> at org.apache.hadoop.mapred.Child.main(Child.java:262)
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:215)
> ... 11 more
> 2013-08-23 13:07:07,871 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task
>
> Any thoughts?
>
> Thanks again,
>
> JC
>
>
>
>
>
>
>
> On Aug 22, 2013, at 11:15 PM, Justin Workman <ju...@gmail.com> wrote:
>
>> For anyone interested, I have finished my blog on getting Hive-0.11.0 working with CDH4.3.0. It is a hack and can be made better but works like a charm.
>>
>> http://www.justinjworkman.com/big-data/hive-0-11-0-on-cloudera/
>>
>>
>>
>> On Thu, Aug 22, 2013 at 8:29 AM, Jim Colestock <jc...@ramblingredneck.com> wrote:
>> Justin,
>>
>> That would be much appreciated!
>>
>> Thanks Again,
>>
>> JC
>>
>> On Aug 22, 2013, at 9:18 AM, Justin Workman <ju...@gmail.com> wrote:
>>
>>> I do have this working on CHD4.3.0 with MR1 and Kerberos. I am finishing up a blog today explaining what was needed to get it working. I will reply with the link later today.
>>>
>>> Justin
>>>
>>> Sent from my iPhone
>>>
>>> On Aug 22, 2013, at 6:17 AM, Jim Colestock <jc...@ramblingredneck.com> wrote:
>>>
>>>> Hello All,
>>>>
>>>> Sorry for not replying to the thread, but I never received any of your responses.. Hopefully this info will help:
>>>>
>>>> @kulkarni.swarnim
>>>>
>>>> Here is the error I'm getting on task tracker for the mapper:
>>>>
>>>> 2013-08-21 15:34:32,491 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
>>>> 2013-08-21 15:34:32,494 WARN org.apache.hadoop.mapred.Child: Error running child
>>>> java.lang.RuntimeException: java.lang.NullPointerException
>>>> at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:230)
>>>> at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255)
>>>> at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:381)
>>>> at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:374)
>>>> at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540)
>>>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:394)
>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
>>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>>>> at org.apache.hadoop.mapred.Child.main(Child.java:262)
>>>> Caused by: java.lang.NullPointerException
>>>> at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:215)
>>>> ... 11 more
>>>> 2013-08-21 15:34:32,498 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task
>>>>
>>>> @Srinivas Surasani
>>>> Hopefully the above helps..
>>>>
>>>> @Lars Francke
>>>> Thanks, I'll take a peek at those 2 patches.. But I've just recently pulled down fresh code and recompiled everything and I'm still getting the above error.
>>>>
>>>> And for reference, below is my original email to the list.. Thanks all for you help!
>>>>
>>>>
>>>> JC
>>>>
>>>>
>>>>
>>>> Hello All,
>>>>
>>>> Has anyone been successful at running hive 0.11 with Cloudera CDH 4.3?
>>>>
>>>> I've been able to get hive to connect to my metadb (which is in Postgres). Verified by doing a show tables.. I can run explain and describes on tables, but when I try to run anything that fires off an M/R job, I get the following error:
>>>>
>>>> hive>select count(*) from tableA;
>>>> Total MapReduce jobs = 1
>>>> Launching Job 1 out of 1
>>>> Number of reduce tasks determined at compile time: 1
>>>> In order to change the average load for a reducer (in bytes):
>>>> set hive.exec.reducers.bytes.per.reducer=<number>
>>>> In order to limit the maximum number of reducers:
>>>> set hive.exec.reducers.max=<number>
>>>> In order to set a constant number of reducers:
>>>> set mapred.reduce.tasks=<number>
>>>> Starting Job = job_201307112247_13816, Tracking URL = http://master:50030/jobdetails.jsp?jobid=job_201307112247_13816
>>>> Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_201307112247_13816
>>>> Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
>>>> 2013-07-12 02:11:42,829 Stage-1 map = 0%, reduce = 0%
>>>> 2013-07-12 02:12:08,173 Stage-1 map = 100%, reduce = 100%
>>>> Ended Job = job_201307112247_13816 with errors
>>>> Error during job, obtaining debugging information...
>>>> Job Tracking URL: http://master:50030/jobdetails.jsp?jobid=job_201307112247_13816
>>>> Examining task ID: task_201307112247_13816_m_000002 (and more) from job job_201307112247_13816
>>>> Exception in thread "Thread-19" java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/util/HostUtil
>>>> at org.apache.hadoop.hive.shims.Hadoop23Shims.getTaskAttemptLogUrl(Hadoop23Shims.java:61)
>>>> at org.apache.hadoop.hive.ql.exec.JobDebugger$TaskInfoGrabber.getTaskInfos(JobDebugger.java:186)
>>>> at org.apache.hadoop.hive.ql.exec.JobDebugger$TaskInfoGrabber.run(JobDebugger.java:142)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.util.HostUtil
>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>>> ... 4 more
>>>> FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
>>>> MapReduce Jobs Launched:
>>>> Job 0: Map: 1 Reduce: 1 HDFS Read: 0 HDFS Write: 0 FAIL
>>>> Total MapReduce CPU Time Spent: 0 msec
>>>>
>>>> I'm using my configs from hive 0.10, which works with no issues and this was pretty much a drop in replacement on the machine that hadoop 0.10 was running on..
>>>>
>>>> I've done a bunch of googling around and have found a bunch of other folks that have have had the same issue, but no solid answers..
>>>>
>>>> Thanks in advance for any help..
>>>>
>>>> JC
>>>>
>>>>
>>
>>
>
>
Re: Hive 0.11 with Cloudera CHD4.3 MR v1
Posted by Justin Workman <ju...@gmail.com>.
I just went through this same process again and deployed to another 5 node
cluster, and it is also working as expected. I have ran anything very
intensive yet, but did spawn several mapreduce jobs.
On Fri, Aug 23, 2013 at 11:11 AM, Jim Colestock <jc...@ramblingredneck.com>wrote:
> Hello Justin,
>
> Thanks so much for the write up.. I'm compiling now.. I did find one
> error in your instructions:
>
>
> patch -p0 > HIVE-4619.D10971.1.path
>
>
> I believe it should be:
>
> patch -p0 *<* HIVE-4619.D10971.1.patch
>
>
>
> But alas, I still doesn't work for me. I managed to get it to compile
> after using maven to install a zookeeper dependency.. But when I try to do
> anything in hive that requires an M/R job, I get the following:
>
> 2013-08-23 13:07:07,865 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
> Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
> 2013-08-23 13:07:07,867 WARN org.apache.hadoop.mapred.Child: Error running
> child
>
> java.lang.RuntimeException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:230)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:381)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:374)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:394)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> at org.apache.hadoop.mapred.Child.main(Child.java:262)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:215)
> ... 11 more
>
> 2013-08-23 13:07:07,871 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task
>
>
> Any thoughts?
>
> Thanks again,
>
> JC
>
>
>
>
>
>
>
> On Aug 22, 2013, at 11:15 PM, Justin Workman <ju...@gmail.com>
> wrote:
>
> For anyone interested, I have finished my blog on getting Hive-0.11.0
> working with CDH4.3.0. It is a hack and can be made better but works like a
> charm.
>
> http://www.justinjworkman.com/big-data/hive-0-11-0-on-cloudera/
>
>
>
> On Thu, Aug 22, 2013 at 8:29 AM, Jim Colestock <jc...@ramblingredneck.com>wrote:
>
>> Justin,
>>
>> That would be much appreciated!
>>
>> Thanks Again,
>>
>> JC
>>
>> On Aug 22, 2013, at 9:18 AM, Justin Workman <ju...@gmail.com>
>> wrote:
>>
>> I do have this working on CHD4.3.0 with MR1 and Kerberos. I am finishing
>> up a blog today explaining what was needed to get it working. I will reply
>> with the link later today.
>>
>> Justin
>>
>> Sent from my iPhone
>>
>> On Aug 22, 2013, at 6:17 AM, Jim Colestock <jc...@ramblingredneck.com>
>> wrote:
>>
>> Hello All,
>>
>> Sorry for not replying to the thread, but I never received any of your
>> responses.. Hopefully this info will help:
>>
>> @kulkarni.swarnim<http://search.gmane.org/?author=kulkarni.swarnim-Re5JQEeQqe8AvxtiuMwx3w%40public.gmane.org&sort=date>
>>
>> Here is the error I'm getting on task tracker for the mapper:
>>
>> 2013-08-21 15:34:32,491 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
>> 2013-08-21 15:34:32,494 WARN org.apache.hadoop.mapred.Child: Error running child
>> java.lang.RuntimeException: java.lang.NullPointerException
>> at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:230)
>> at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255)
>> at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:381)
>> at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:374)
>> at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540)
>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:394)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
>> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>> at org.apache.hadoop.mapred.Child.main(Child.java:262)
>> Caused by: java.lang.NullPointerException
>> at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:215)
>> ... 11 more
>> 2013-08-21 15:34:32,498 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task
>>
>>
>> @Srinivas Surasani<http://search.gmane.org/?author=Srinivas+Surasani&sort=date>
>> Hopefully the above helps..
>>
>> @Lars Francke <http://search.gmane.org/?author=Lars+Francke&sort=date>
>> Thanks, I'll take a peek at those 2 patches.. But I've just recently
>> pulled down fresh code and recompiled everything and I'm still getting the
>> above error.
>>
>> And for reference, below is my original email to the list.. Thanks all
>> for you help!
>>
>>
>> JC
>>
>>
>>
>> Hello All,
>>
>> Has anyone been successful at running hive 0.11 with Cloudera CDH 4.3?
>>
>> I've been able to get hive to connect to my metadb (which is in
>> Postgres). Verified by doing a show tables.. I can run explain and
>> describes on tables, but when I try to run anything that fires off an M/R
>> job, I get the following error:
>>
>> hive>select count(*) from tableA;
>> Total MapReduce jobs = 1
>> Launching Job 1 out of 1
>> Number of reduce tasks determined at compile time: 1
>> In order to change the average load for a reducer (in bytes):
>> set hive.exec.reducers.bytes.per.reducer=<number>
>> In order to limit the maximum number of reducers:
>> set hive.exec.reducers.max=<number>
>> In order to set a constant number of reducers:
>> set mapred.reduce.tasks=<number>
>> Starting Job = job_201307112247_13816, Tracking URL =
>> http://master:50030/jobdetails.jsp?jobid=job_201307112247_13816
>> Kill Command = /usr/lib/hadoop/bin/hadoop job -kill
>> job_201307112247_13816
>> Hadoop job information for Stage-1: number of mappers: 1; number of
>> reducers: 1
>> 2013-07-12 02:11:42,829 Stage-1 map = 0%, reduce = 0%
>> 2013-07-12 02:12:08,173 Stage-1 map = 100%, reduce = 100%
>> Ended Job = job_201307112247_13816 with errors
>> Error during job, obtaining debugging information...
>> Job Tracking URL:
>> http://master:50030/jobdetails.jsp?jobid=job_201307112247_13816
>> Examining task ID: task_201307112247_13816_m_000002 (and more) from job
>> job_201307112247_13816
>> Exception in thread "Thread-19" java.lang.NoClassDefFoundError:
>> org/apache/hadoop/mapreduce/util/HostUtil
>> at
>> org.apache.hadoop.hive.shims.Hadoop23Shims.getTaskAttemptLogUrl(Hadoop23Shims.java:61)
>> at
>> org.apache.hadoop.hive.ql.exec.JobDebugger$TaskInfoGrabber.getTaskInfos(JobDebugger.java:186)
>> at
>> org.apache.hadoop.hive.ql.exec.JobDebugger$TaskInfoGrabber.run(JobDebugger.java:142)
>> at java.lang.Thread.run(Thread.java:619)
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.hadoop.mapreduce.util.HostUtil
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>> ... 4 more
>> FAILED: Execution Error, return code 2 from
>> org.apache.hadoop.hive.ql.exec.MapRedTask
>> MapReduce Jobs Launched:
>> Job 0: Map: 1 Reduce: 1 HDFS Read: 0 HDFS Write: 0 FAIL
>> Total MapReduce CPU Time Spent: 0 msec
>>
>>
>> I'm using my configs from hive 0.10, which works with no issues and this
>> was pretty much a drop in replacement on the machine that hadoop 0.10 was
>> running on..
>>
>> I've done a bunch of googling around and have found a bunch of other
>> folks that have have had the same issue, but no solid answers..
>>
>> Thanks in advance for any help..
>>
>> JC
>>
>>
>>
>>
>
>
Re: Hive 0.11 with Cloudera CHD4.3 MR v1
Posted by Jim Colestock <jc...@ramblingredneck.com>.
Hello Justin,
Thanks so much for the write up.. I'm compiling now.. I did find one error in your instructions:
patch -p0 > HIVE-4619.D10971.1.path
I believe it should be:
patch -p0 < HIVE-4619.D10971.1.patch
But alas, I still doesn't work for me. I managed to get it to compile after using maven to install a zookeeper dependency.. But when I try to do anything in hive that requires an M/R job, I get the following:
2013-08-23 13:07:07,865 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
2013-08-23 13:07:07,867 WARN org.apache.hadoop.mapred.Child: Error running child
java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:230)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:381)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:374)
at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:394)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:215)
... 11 more
2013-08-23 13:07:07,871 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task
Any thoughts?
Thanks again,
JC
On Aug 22, 2013, at 11:15 PM, Justin Workman <ju...@gmail.com> wrote:
> For anyone interested, I have finished my blog on getting Hive-0.11.0 working with CDH4.3.0. It is a hack and can be made better but works like a charm.
>
> http://www.justinjworkman.com/big-data/hive-0-11-0-on-cloudera/
>
>
>
> On Thu, Aug 22, 2013 at 8:29 AM, Jim Colestock <jc...@ramblingredneck.com> wrote:
> Justin,
>
> That would be much appreciated!
>
> Thanks Again,
>
> JC
>
> On Aug 22, 2013, at 9:18 AM, Justin Workman <ju...@gmail.com> wrote:
>
>> I do have this working on CHD4.3.0 with MR1 and Kerberos. I am finishing up a blog today explaining what was needed to get it working. I will reply with the link later today.
>>
>> Justin
>>
>> Sent from my iPhone
>>
>> On Aug 22, 2013, at 6:17 AM, Jim Colestock <jc...@ramblingredneck.com> wrote:
>>
>>> Hello All,
>>>
>>> Sorry for not replying to the thread, but I never received any of your responses.. Hopefully this info will help:
>>>
>>> @kulkarni.swarnim
>>>
>>> Here is the error I'm getting on task tracker for the mapper:
>>>
>>> 2013-08-21 15:34:32,491 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
>>> 2013-08-21 15:34:32,494 WARN org.apache.hadoop.mapred.Child: Error running child
>>> java.lang.RuntimeException: java.lang.NullPointerException
>>> at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:230)
>>> at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255)
>>> at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:381)
>>> at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:374)
>>> at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540)
>>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:394)
>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>>> at org.apache.hadoop.mapred.Child.main(Child.java:262)
>>> Caused by: java.lang.NullPointerException
>>> at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:215)
>>> ... 11 more
>>> 2013-08-21 15:34:32,498 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task
>>>
>>> @Srinivas Surasani
>>> Hopefully the above helps..
>>>
>>> @Lars Francke
>>> Thanks, I'll take a peek at those 2 patches.. But I've just recently pulled down fresh code and recompiled everything and I'm still getting the above error.
>>>
>>> And for reference, below is my original email to the list.. Thanks all for you help!
>>>
>>>
>>> JC
>>>
>>>
>>>
>>> Hello All,
>>>
>>> Has anyone been successful at running hive 0.11 with Cloudera CDH 4.3?
>>>
>>> I've been able to get hive to connect to my metadb (which is in Postgres). Verified by doing a show tables.. I can run explain and describes on tables, but when I try to run anything that fires off an M/R job, I get the following error:
>>>
>>> hive>select count(*) from tableA;
>>> Total MapReduce jobs = 1
>>> Launching Job 1 out of 1
>>> Number of reduce tasks determined at compile time: 1
>>> In order to change the average load for a reducer (in bytes):
>>> set hive.exec.reducers.bytes.per.reducer=<number>
>>> In order to limit the maximum number of reducers:
>>> set hive.exec.reducers.max=<number>
>>> In order to set a constant number of reducers:
>>> set mapred.reduce.tasks=<number>
>>> Starting Job = job_201307112247_13816, Tracking URL = http://master:50030/jobdetails.jsp?jobid=job_201307112247_13816
>>> Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_201307112247_13816
>>> Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
>>> 2013-07-12 02:11:42,829 Stage-1 map = 0%, reduce = 0%
>>> 2013-07-12 02:12:08,173 Stage-1 map = 100%, reduce = 100%
>>> Ended Job = job_201307112247_13816 with errors
>>> Error during job, obtaining debugging information...
>>> Job Tracking URL: http://master:50030/jobdetails.jsp?jobid=job_201307112247_13816
>>> Examining task ID: task_201307112247_13816_m_000002 (and more) from job job_201307112247_13816
>>> Exception in thread "Thread-19" java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/util/HostUtil
>>> at org.apache.hadoop.hive.shims.Hadoop23Shims.getTaskAttemptLogUrl(Hadoop23Shims.java:61)
>>> at org.apache.hadoop.hive.ql.exec.JobDebugger$TaskInfoGrabber.getTaskInfos(JobDebugger.java:186)
>>> at org.apache.hadoop.hive.ql.exec.JobDebugger$TaskInfoGrabber.run(JobDebugger.java:142)
>>> at java.lang.Thread.run(Thread.java:619)
>>> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.util.HostUtil
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>> ... 4 more
>>> FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
>>> MapReduce Jobs Launched:
>>> Job 0: Map: 1 Reduce: 1 HDFS Read: 0 HDFS Write: 0 FAIL
>>> Total MapReduce CPU Time Spent: 0 msec
>>>
>>> I'm using my configs from hive 0.10, which works with no issues and this was pretty much a drop in replacement on the machine that hadoop 0.10 was running on..
>>>
>>> I've done a bunch of googling around and have found a bunch of other folks that have have had the same issue, but no solid answers..
>>>
>>> Thanks in advance for any help..
>>>
>>> JC
>>>
>>>
>
>
Re: Hive 0.11 with Cloudera CHD4.3 MR v1
Posted by Justin Workman <ju...@gmail.com>.
For anyone interested, I have finished my blog on getting Hive-0.11.0
working with CDH4.3.0. It is a hack and can be made better but works like a
charm.
http://www.justinjworkman.com/big-data/hive-0-11-0-on-cloudera/
On Thu, Aug 22, 2013 at 8:29 AM, Jim Colestock <jc...@ramblingredneck.com>wrote:
> Justin,
>
> That would be much appreciated!
>
> Thanks Again,
>
> JC
>
> On Aug 22, 2013, at 9:18 AM, Justin Workman <ju...@gmail.com>
> wrote:
>
> I do have this working on CHD4.3.0 with MR1 and Kerberos. I am finishing
> up a blog today explaining what was needed to get it working. I will reply
> with the link later today.
>
> Justin
>
> Sent from my iPhone
>
> On Aug 22, 2013, at 6:17 AM, Jim Colestock <jc...@ramblingredneck.com> wrote:
>
> Hello All,
>
> Sorry for not replying to the thread, but I never received any of your
> responses.. Hopefully this info will help:
>
> @kulkarni.swarnim<http://search.gmane.org/?author=kulkarni.swarnim-Re5JQEeQqe8AvxtiuMwx3w%40public.gmane.org&sort=date>
>
> Here is the error I'm getting on task tracker for the mapper:
>
> 2013-08-21 15:34:32,491 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
> 2013-08-21 15:34:32,494 WARN org.apache.hadoop.mapred.Child: Error running child
> java.lang.RuntimeException: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:230)
> at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255)
> at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:381)
> at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:374)
> at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:394)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> at org.apache.hadoop.mapred.Child.main(Child.java:262)
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:215)
> ... 11 more
> 2013-08-21 15:34:32,498 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task
>
>
> @Srinivas Surasani<http://search.gmane.org/?author=Srinivas+Surasani&sort=date>
> Hopefully the above helps..
>
> @Lars Francke <http://search.gmane.org/?author=Lars+Francke&sort=date>
> Thanks, I'll take a peek at those 2 patches.. But I've just recently
> pulled down fresh code and recompiled everything and I'm still getting the
> above error.
>
> And for reference, below is my original email to the list.. Thanks all
> for you help!
>
>
> JC
>
>
>
> Hello All,
>
> Has anyone been successful at running hive 0.11 with Cloudera CDH 4.3?
>
> I've been able to get hive to connect to my metadb (which is in Postgres).
> Verified by doing a show tables.. I can run explain and describes on
> tables, but when I try to run anything that fires off an M/R job, I get the
> following error:
>
> hive>select count(*) from tableA;
> Total MapReduce jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks determined at compile time: 1
> In order to change the average load for a reducer (in bytes):
> set hive.exec.reducers.bytes.per.reducer=<number>
> In order to limit the maximum number of reducers:
> set hive.exec.reducers.max=<number>
> In order to set a constant number of reducers:
> set mapred.reduce.tasks=<number>
> Starting Job = job_201307112247_13816, Tracking URL =
> http://master:50030/jobdetails.jsp?jobid=job_201307112247_13816
> Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_201307112247_13816
> Hadoop job information for Stage-1: number of mappers: 1; number of
> reducers: 1
> 2013-07-12 02:11:42,829 Stage-1 map = 0%, reduce = 0%
> 2013-07-12 02:12:08,173 Stage-1 map = 100%, reduce = 100%
> Ended Job = job_201307112247_13816 with errors
> Error during job, obtaining debugging information...
> Job Tracking URL:
> http://master:50030/jobdetails.jsp?jobid=job_201307112247_13816
> Examining task ID: task_201307112247_13816_m_000002 (and more) from job
> job_201307112247_13816
> Exception in thread "Thread-19" java.lang.NoClassDefFoundError:
> org/apache/hadoop/mapreduce/util/HostUtil
> at
> org.apache.hadoop.hive.shims.Hadoop23Shims.getTaskAttemptLogUrl(Hadoop23Shims.java:61)
> at
> org.apache.hadoop.hive.ql.exec.JobDebugger$TaskInfoGrabber.getTaskInfos(JobDebugger.java:186)
> at
> org.apache.hadoop.hive.ql.exec.JobDebugger$TaskInfoGrabber.run(JobDebugger.java:142)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.mapreduce.util.HostUtil
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> ... 4 more
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.MapRedTask
> MapReduce Jobs Launched:
> Job 0: Map: 1 Reduce: 1 HDFS Read: 0 HDFS Write: 0 FAIL
> Total MapReduce CPU Time Spent: 0 msec
>
>
> I'm using my configs from hive 0.10, which works with no issues and this
> was pretty much a drop in replacement on the machine that hadoop 0.10 was
> running on..
>
> I've done a bunch of googling around and have found a bunch of other folks
> that have have had the same issue, but no solid answers..
>
> Thanks in advance for any help..
>
> JC
>
>
>
>
Re: Hive 0.11 with Cloudera CHD4.3 MR v1
Posted by Jim Colestock <jc...@ramblingredneck.com>.
Justin,
That would be much appreciated!
Thanks Again,
JC
On Aug 22, 2013, at 9:18 AM, Justin Workman <ju...@gmail.com> wrote:
> I do have this working on CHD4.3.0 with MR1 and Kerberos. I am finishing up a blog today explaining what was needed to get it working. I will reply with the link later today.
>
> Justin
>
> Sent from my iPhone
>
> On Aug 22, 2013, at 6:17 AM, Jim Colestock <jc...@ramblingredneck.com> wrote:
>
>> Hello All,
>>
>> Sorry for not replying to the thread, but I never received any of your responses.. Hopefully this info will help:
>>
>> @kulkarni.swarnim
>>
>> Here is the error I'm getting on task tracker for the mapper:
>>
>> 2013-08-21 15:34:32,491 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
>> 2013-08-21 15:34:32,494 WARN org.apache.hadoop.mapred.Child: Error running child
>> java.lang.RuntimeException: java.lang.NullPointerException
>> at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:230)
>> at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255)
>> at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:381)
>> at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:374)
>> at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540)
>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:394)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
>> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>> at org.apache.hadoop.mapred.Child.main(Child.java:262)
>> Caused by: java.lang.NullPointerException
>> at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:215)
>> ... 11 more
>> 2013-08-21 15:34:32,498 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task
>>
>> @Srinivas Surasani
>> Hopefully the above helps..
>>
>> @Lars Francke
>> Thanks, I'll take a peek at those 2 patches.. But I've just recently pulled down fresh code and recompiled everything and I'm still getting the above error.
>>
>> And for reference, below is my original email to the list.. Thanks all for you help!
>>
>>
>> JC
>>
>>
>>
>> Hello All,
>>
>> Has anyone been successful at running hive 0.11 with Cloudera CDH 4.3?
>>
>> I've been able to get hive to connect to my metadb (which is in Postgres). Verified by doing a show tables.. I can run explain and describes on tables, but when I try to run anything that fires off an M/R job, I get the following error:
>>
>> hive>select count(*) from tableA;
>> Total MapReduce jobs = 1
>> Launching Job 1 out of 1
>> Number of reduce tasks determined at compile time: 1
>> In order to change the average load for a reducer (in bytes):
>> set hive.exec.reducers.bytes.per.reducer=<number>
>> In order to limit the maximum number of reducers:
>> set hive.exec.reducers.max=<number>
>> In order to set a constant number of reducers:
>> set mapred.reduce.tasks=<number>
>> Starting Job = job_201307112247_13816, Tracking URL = http://master:50030/jobdetails.jsp?jobid=job_201307112247_13816
>> Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_201307112247_13816
>> Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
>> 2013-07-12 02:11:42,829 Stage-1 map = 0%, reduce = 0%
>> 2013-07-12 02:12:08,173 Stage-1 map = 100%, reduce = 100%
>> Ended Job = job_201307112247_13816 with errors
>> Error during job, obtaining debugging information...
>> Job Tracking URL: http://master:50030/jobdetails.jsp?jobid=job_201307112247_13816
>> Examining task ID: task_201307112247_13816_m_000002 (and more) from job job_201307112247_13816
>> Exception in thread "Thread-19" java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/util/HostUtil
>> at org.apache.hadoop.hive.shims.Hadoop23Shims.getTaskAttemptLogUrl(Hadoop23Shims.java:61)
>> at org.apache.hadoop.hive.ql.exec.JobDebugger$TaskInfoGrabber.getTaskInfos(JobDebugger.java:186)
>> at org.apache.hadoop.hive.ql.exec.JobDebugger$TaskInfoGrabber.run(JobDebugger.java:142)
>> at java.lang.Thread.run(Thread.java:619)
>> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.util.HostUtil
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>> ... 4 more
>> FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
>> MapReduce Jobs Launched:
>> Job 0: Map: 1 Reduce: 1 HDFS Read: 0 HDFS Write: 0 FAIL
>> Total MapReduce CPU Time Spent: 0 msec
>>
>> I'm using my configs from hive 0.10, which works with no issues and this was pretty much a drop in replacement on the machine that hadoop 0.10 was running on..
>>
>> I've done a bunch of googling around and have found a bunch of other folks that have have had the same issue, but no solid answers..
>>
>> Thanks in advance for any help..
>>
>> JC
>>
>>
Re: Hive 0.11 with Cloudera CHD4.3 MR v1
Posted by Justin Workman <ju...@gmail.com>.
I do have this working on CHD4.3.0 with MR1 and Kerberos. I am finishing
up a blog today explaining what was needed to get it working. I will reply
with the link later today.
Justin
Sent from my iPhone
On Aug 22, 2013, at 6:17 AM, Jim Colestock <jc...@ramblingredneck.com> wrote:
Hello All,
Sorry for not replying to the thread, but I never received any of your
responses.. Hopefully this info will help:
@kulkarni.swarnim<http://search.gmane.org/?author=kulkarni.swarnim-Re5JQEeQqe8AvxtiuMwx3w%40public.gmane.org&sort=date>
Here is the error I'm getting on task tracker for the mapper:
2013-08-21 15:34:32,491 INFO
org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs'
truncater with mapRetainSize=-1 and reduceRetainSize=-1
2013-08-21 15:34:32,494 WARN org.apache.hadoop.mapred.Child: Error running child
java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:230)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:381)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:374)
at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:394)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:215)
... 11 more
2013-08-21 15:34:32,498 INFO org.apache.hadoop.mapred.Task: Runnning
cleanup for the task
@Srinivas Surasani<http://search.gmane.org/?author=Srinivas+Surasani&sort=date>
Hopefully the above helps..
@Lars Francke <http://search.gmane.org/?author=Lars+Francke&sort=date>
Thanks, I'll take a peek at those 2 patches.. But I've just recently pulled
down fresh code and recompiled everything and I'm still getting the above
error.
And for reference, below is my original email to the list.. Thanks all for
you help!
JC
Hello All,
Has anyone been successful at running hive 0.11 with Cloudera CDH 4.3?
I've been able to get hive to connect to my metadb (which is in Postgres).
Verified by doing a show tables.. I can run explain and describes on
tables, but when I try to run anything that fires off an M/R job, I get the
following error:
hive>select count(*) from tableA;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Starting Job = job_201307112247_13816, Tracking URL =
http://master:50030/jobdetails.jsp?jobid=job_201307112247_13816
Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_201307112247_13816
Hadoop job information for Stage-1: number of mappers: 1; number of
reducers: 1
2013-07-12 02:11:42,829 Stage-1 map = 0%, reduce = 0%
2013-07-12 02:12:08,173 Stage-1 map = 100%, reduce = 100%
Ended Job = job_201307112247_13816 with errors
Error during job, obtaining debugging information...
Job Tracking URL:
http://master:50030/jobdetails.jsp?jobid=job_201307112247_13816
Examining task ID: task_201307112247_13816_m_000002 (and more) from job
job_201307112247_13816
Exception in thread "Thread-19" java.lang.NoClassDefFoundError:
org/apache/hadoop/mapreduce/util/HostUtil
at
org.apache.hadoop.hive.shims.Hadoop23Shims.getTaskAttemptLogUrl(Hadoop23Shims.java:61)
at
org.apache.hadoop.hive.ql.exec.JobDebugger$TaskInfoGrabber.getTaskInfos(JobDebugger.java:186)
at
org.apache.hadoop.hive.ql.exec.JobDebugger$TaskInfoGrabber.run(JobDebugger.java:142)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.mapreduce.util.HostUtil
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
... 4 more
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.MapRedTask
MapReduce Jobs Launched:
Job 0: Map: 1 Reduce: 1 HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec
I'm using my configs from hive 0.10, which works with no issues and this
was pretty much a drop in replacement on the machine that hadoop 0.10 was
running on..
I've done a bunch of googling around and have found a bunch of other folks
that have have had the same issue, but no solid answers..
Thanks in advance for any help..
JC