You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Marcin Cylke <mc...@touk.pl> on 2013/02/14 08:27:55 UTC
Re: [ hive ] Re: hive-0.10 and nullpointerexception
On 13/02/13 17:34, Mark Grover wrote:
> Hi Marcin,
> Sorry to hear that you ran into this.
>
> My guess is you are using Yarn and this is, in fact, a known issue.
>
> The culprit line here is
> https://github.com/apache/hive/blob/branch-0.10/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L209
>
> This issue is being tracked
> by https://issues.apache.org/jira/browse/HIVE-4003 which is assigned to
> me, I will try to get this in sooner than later.
>
> In the meanwhile, a workaround is to set yarn.resourcemanager.address
> property to some value other than "local" so Hive queries can run on
> your MR2 cluster.
Actually I'm not using YARN, just the classic MR. Perhaps the setting is
somewhere in my configuration? My whole cluster is running MR1. Since
I'm running Cloudera cdh4.1 I'm starting hive with the following line:
HADOOP_MAPRED_HOME=/usr/lib/hadoop-0.20-mapreduce
HADOOP_HOME=/usr/lib/hadoop bin/hive
Is there anything else I can do to debug this issue? I'd like to
confirm, that in my situation Hive is not trying to use YARN.
Regards
Marcin
Re: [ hive ] Re: hive-0.10 and nullpointerexception
Posted by Marcin Cylke <mc...@touk.pl>.
> I'll build the trunk and try to test it. Then I'll let you know if that
> helped.
I've just performed the test on current trunk. The problem is still
there, no changes.
Any ideas? Perhaps my config says something interesting? I've added it
to the GIST: https://gist.github.com/zygm0nt/5028591#file-hive-site-xml
The config consists some addidional parameters, but I don't think they
would influence MapReduce behavior.
Regards
Marcin
Re: hive-0.10 and nullpointerexception
Posted by Mark Grover <gr...@gmail.com>.
For the sake of readers, I just saw another manifestation of this problem.
The issue was that mapreduce.framework.name in mapred-site.xml was set
to "yarn" instead of "classic" when the intent was to use MR1.
Hope that helps,
Mark
On Mon, Mar 4, 2013 at 8:26 AM, Mark Grover <gr...@gmail.com> wrote:
> Sorry for the delayed response, Marcin, I was away.
>
> This error in the Gist means that the jar containing the class in
> question is not on your classpath.
> Exception in thread "Thread-39" java.lang.NoClassDefFoundError:
> org/apache/hadoop/mapreduce/util/HostUtil
> at org.apache.hadoop.hive.shims.Hadoop23Shims.getTaskAttemptLogUrl(Hadoop23Shims.java:53)
> at org.apache.hadoop.hive.ql.exec.JobDebugger$TaskInfoGrabber.getTaskInfos(JobDebugger.java:186)
> at org.apache.hadoop.hive.ql.exec.JobDebugger$TaskInfoGrabber.run(JobDebugger.java:142)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.mapreduce.util.HostUtil
> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
> ... 4 more
>
> So if you add the appropriate jar (hadoop-mapreduce jar?) to your
> classpath, that should take you further.Have you tried that?
>
> On Mon, Feb 25, 2013 at 4:01 AM, Marcin Cylke <mc...@touk.pl> wrote:
>> On 25/02/13 10:05, Marcin Cylke wrote:
>>
>>> I'll build the trunk and try to test it. Then I'll let you know if that
>>> helped.
>>
>> I've just performed the test on current trunk. The problem is still
>> there, no changes.
>>
>> Any ideas? Perhaps my config says something interesting? I've added it
>> to the GIST: https://gist.github.com/zygm0nt/5028591#file-hive-site-xml
>>
>> The config consists some addidional parameters, but I don't think they
>> would influence MapReduce behavior.
>>
>> Regards
>> Marcin
>>
Re: hive-0.10 and nullpointerexception
Posted by Mark Grover <gr...@gmail.com>.
Sorry for the delayed response, Marcin, I was away.
This error in the Gist means that the jar containing the class in
question is not on your classpath.
Exception in thread "Thread-39" java.lang.NoClassDefFoundError:
org/apache/hadoop/mapreduce/util/HostUtil
at org.apache.hadoop.hive.shims.Hadoop23Shims.getTaskAttemptLogUrl(Hadoop23Shims.java:53)
at org.apache.hadoop.hive.ql.exec.JobDebugger$TaskInfoGrabber.getTaskInfos(JobDebugger.java:186)
at org.apache.hadoop.hive.ql.exec.JobDebugger$TaskInfoGrabber.run(JobDebugger.java:142)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.mapreduce.util.HostUtil
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
... 4 more
So if you add the appropriate jar (hadoop-mapreduce jar?) to your
classpath, that should take you further.Have you tried that?
On Mon, Feb 25, 2013 at 4:01 AM, Marcin Cylke <mc...@touk.pl> wrote:
> On 25/02/13 10:05, Marcin Cylke wrote:
>
>> I'll build the trunk and try to test it. Then I'll let you know if that
>> helped.
>
> I've just performed the test on current trunk. The problem is still
> there, no changes.
>
> Any ideas? Perhaps my config says something interesting? I've added it
> to the GIST: https://gist.github.com/zygm0nt/5028591#file-hive-site-xml
>
> The config consists some addidional parameters, but I don't think they
> would influence MapReduce behavior.
>
> Regards
> Marcin
>
Re: hive-0.10 and nullpointerexception
Posted by Marcin Cylke <mc...@touk.pl>.
On 20/02/13 00:19, Mark Grover wrote:
> Could you try setting mapred.job.tracker property in your
> mapred-site.xml to your jobtracker and see if that fixes it?
I've been setting while in hive with:
set mapred.job.tracker = 'host:port'
but this didn't help. Is setting that property in mapred-site.xml any
different?
I've created a GIST with config and logs from my test:
https://gist.github.com/zygm0nt/5028591
What I did was just call the following commands:
use hdp_sand; -- to change schema
select count(*) from page_view;
page_view is a simple table, from hive "Getting started" guide. It has
partitions in my case, but doing the same on a non-partitioned table
produces the same error.
> Also, if you could provide me any information (your config, setup,
> which queries fail, etc.) that could help me in reproducing the
> problem on hive trunk would be greatly appreciated!
I'll build the trunk and try to test it. Then I'll let you know if that
helped.
Regards
Marcin
Re: hive-0.10 and nullpointerexception
Posted by Mark Grover <gr...@gmail.com>.
Hey Marcin,
Sorry I haven't gotten a chance to look into this further.
I am fairly certain that it's a bug in Utilities.java which HIVE-4003
is going to fix. I am have been trying to reproduce the behavior
locally with trunk but so far, have been unsuccessful in doing so.
In any case, so it boils down to the shim you are using. Here is the
Shim for Hadoop 0.20.
https://github.com/apache/hive/blob/trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java#L615
Could you try setting mapred.job.tracker property in your
mapred-site.xml to your jobtracker and see if that fixes it?
Also, if you could provide me any information (your config, setup,
which queries fail, etc.) that could help me in reproducing the
problem on hive trunk would be greatly appreciated!
Mark
On Tue, Feb 19, 2013 at 1:08 AM, Marcin Cylke <mc...@touk.pl> wrote:
>
> On 14/02/13 08:27, Marcin Cylke wrote:
>
> > Actually I'm not using YARN, just the classic MR. Perhaps the setting is
> > somewhere in my configuration? My whole cluster is running MR1. Since
> > I'm running Cloudera cdh4.1 I'm starting hive with the following line:
> >
> > HADOOP_MAPRED_HOME=/usr/lib/hadoop-0.20-mapreduce
> > HADOOP_HOME=/usr/lib/hadoop bin/hive
> >
> > Is there anything else I can do to debug this issue? I'd like to
> > confirm, that in my situation Hive is not trying to use YARN.
>
>
> Could You suggest what to do next with this particular problem?
>
> Regards
> Marcin
>
Re: hive-0.10 and nullpointerexception
Posted by Marcin Cylke <mc...@touk.pl>.
On 14/02/13 08:27, Marcin Cylke wrote:
> Actually I'm not using YARN, just the classic MR. Perhaps the setting is
> somewhere in my configuration? My whole cluster is running MR1. Since
> I'm running Cloudera cdh4.1 I'm starting hive with the following line:
>
> HADOOP_MAPRED_HOME=/usr/lib/hadoop-0.20-mapreduce
> HADOOP_HOME=/usr/lib/hadoop bin/hive
>
> Is there anything else I can do to debug this issue? I'd like to
> confirm, that in my situation Hive is not trying to use YARN.
Could You suggest what to do next with this particular problem?
Regards
Marcin