You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Massoud Mazar <Ma...@avg.com> on 2010/03/01 22:08:30 UTC
RE: hive 0.50 on hadoop 0.22
Zheng,
Thanks for answering.
I've decided to give it (hive 0.50 on hadoop 0.22) a try. I'm a developer, but not a Java developer, so with some initial help I can spend time and work on this.
Just to start, I modified the ShimLoader.java and copied the same HADOOP_SHIM_CLASSES and JETTY_SHIM_CLASSES from 0.20 to 0.22 to see where it breaks.
I built and deployed hive 0.50 to a running hadoop 0.22 and did "show tables;" in hive, and I got this:
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.security.UserGroupInformation: method <init>()V not found
at org.apache.hadoop.security.UnixUserGroupInformation.<init>(UnixUserGroupInformation.java:69)
at org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:271)
at org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:300)
at org.apache.hadoop.hive.ql.Driver.<init>(Driver.java:243)
at org.apache.hadoop.hive.ql.processors.CommandProcessorFactory.get(CommandProcessorFactory.java:40)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:116)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:181)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:287)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:187)
Now, when I look at the UserGroupInformation class in hadoop 0.22 source code, it does not have a parameter-less constructor, but documentation at http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/security/UserGroupInformation.html shows such a constructor.
Now, my question is: is this something that can be fixed by shims? Or it is a problem with hadoop?
-----Original Message-----
From: Zheng Shao [mailto:zshao9@gmail.com]
Sent: Saturday, February 27, 2010 4:24 AM
To: hive-user@hadoop.apache.org
Subject: Re: hive 0.50 on hadoop 0.22
Hi Mazar,
We have not tried Hive on Hadoop higher than 0.20 yet.
However, Hive has the shim infrastructure which makes it easy to port
to new Hadoop versions.
Please see the shim directory inside Hive.
Zheng
On Fri, Feb 26, 2010 at 1:59 PM, Massoud Mazar <Ma...@avg.com> wrote:
> Is it possible to run release-0.5.0-rc0 on top of hadoop 0.22.0 (trunk)?
>
--
Yours,
Zheng
Re: hive 0.50 on hadoop 0.22
Posted by Zheng Shao <zs...@gmail.com>.
Hi Massoud,
Great work!
Yes this is exactly the use of shims. When we see an API change across
hadoop versions, we add a new function to shims interface, and
implement it in each of the shim.
For this one, you probably want to wrap the logic in Driver.java into
a single shim interface function, and implement that function in all
shim versions.
Does that make sense?
Zheng
On Mon, Mar 1, 2010 at 1:08 PM, Massoud Mazar <Ma...@avg.com> wrote:
> Zheng,
>
> Thanks for answering.
> I've decided to give it (hive 0.50 on hadoop 0.22) a try. I'm a developer, but not a Java developer, so with some initial help I can spend time and work on this.
> Just to start, I modified the ShimLoader.java and copied the same HADOOP_SHIM_CLASSES and JETTY_SHIM_CLASSES from 0.20 to 0.22 to see where it breaks.
>
> I built and deployed hive 0.50 to a running hadoop 0.22 and did "show tables;" in hive, and I got this:
>
> Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.security.UserGroupInformation: method <init>()V not found
> at org.apache.hadoop.security.UnixUserGroupInformation.<init>(UnixUserGroupInformation.java:69)
> at org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:271)
> at org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:300)
> at org.apache.hadoop.hive.ql.Driver.<init>(Driver.java:243)
> at org.apache.hadoop.hive.ql.processors.CommandProcessorFactory.get(CommandProcessorFactory.java:40)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:116)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:181)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:287)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:187)
>
> Now, when I look at the UserGroupInformation class in hadoop 0.22 source code, it does not have a parameter-less constructor, but documentation at http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/security/UserGroupInformation.html shows such a constructor.
>
> Now, my question is: is this something that can be fixed by shims? Or it is a problem with hadoop?
>
> -----Original Message-----
> From: Zheng Shao [mailto:zshao9@gmail.com]
> Sent: Saturday, February 27, 2010 4:24 AM
> To: hive-user@hadoop.apache.org
> Subject: Re: hive 0.50 on hadoop 0.22
>
> Hi Mazar,
>
> We have not tried Hive on Hadoop higher than 0.20 yet.
>
> However, Hive has the shim infrastructure which makes it easy to port
> to new Hadoop versions.
> Please see the shim directory inside Hive.
>
> Zheng
>
> On Fri, Feb 26, 2010 at 1:59 PM, Massoud Mazar <Ma...@avg.com> wrote:
>> Is it possible to run release-0.5.0-rc0 on top of hadoop 0.22.0 (trunk)?
>>
>
>
>
> --
> Yours,
> Zheng
>
--
Yours,
Zheng