You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@tez.apache.org by Stephen Sprague <sp...@gmail.com> on 2016/03/09 06:20:25 UTC
hive 2.0.0 + tez
hey guys,
obvious question i suppose but i gotta ask. :) the tez jars i built for
hive 1.2.1 will not work with hive 2.0.0 i presume?
the reason ask is: i can switch back forth between hive clients 1.2.1 and
2.0.0 and run m/r jobs no problem. however i can not do this with tez.
I'm thinking this is because the tez jars on hdfs have been built (bound
to) hive 1.2.1?
ie. these jar files in hdfs
<property>
<name>tez.lib.uris</name>
<value>${fs.defaultFS}/apps/tez-0.8.2,${fs.defaultFS}/apps/tez-0.8.2/lib</value>
</property>
really should be identified as "tez-0.8.2-hive.1.2.1" and that i should
have another dir called "tez-0.8.2-hive.2.0.0" which another tez-site.xml
should refer to.
Am i making any sense here? :)
Cheers,
Stephen.
Re: hive 2.0.0 + tez
Posted by Stephen Sprague <sp...@gmail.com>.
just to update the group... i was trying to run hive 1.2.1 and hive 2.0.0
in parallel on my client box and didn't i get the configs right. basically
i forgot about the metastore service. once i got that off on its own port
for 2.0.0 and hs2 already on its own port things came together. now its
pretty easy to switch clients by just re-adapting HIVE_HOME.
thanks again!
Cheers,
Stephen.
On Wed, Mar 9, 2016 at 5:12 PM, Stephen Sprague <sp...@gmail.com> wrote:
> thanks as always Gopal. lemme use jdk8 for starters and eliminate that as
> a possibility.
>
> On Wed, Mar 9, 2016 at 10:30 AM, Gopal Vijayaraghavan <go...@apache.org>
> wrote:
>
>>
>> > ok, thanks. very strange then that i can run hive 1.2.1 and hive 2.0.0
>> >m/r jobs against the same hadoop cluster w/o issue
>> > (different HS2, different MetaStore, but same hdfs data) but yet once i
>> >switch to the HS2/2.0.0 tez chokes out - but works fine with the
>> >HS2/1.2.1.
>>
>> It is not particularly strange considering no development for MRv2 has
>> happened in the mean-time - there's no difference in version dependencies
>> between 2.0 & 1.2 for MRv2.
>>
>> Not so for Tez.
>>
>> > I'm thinking this is because the tez jars on hdfs have been built (bound
>> >to) hive 1.2.1?
>>
>> I suspect the tez jars inside hive's classpath rather than the ones on
>> HDFS.
>>
>> 0.8.2 is effectively a super-set of APIs in 0.7.4, so ending up with 0.7.x
>> jars in the local classpath would work fine for hive-1.2.1, while when you
>> run the same classpath with hive-2.0, it wouldn't work.
>>
>> Saw a PermGen error reported in a later email - the solution to all
>> permgen issues was to leave JDK7 behind (End-of-Life April 2015) & use the
>> Metaspace impl in JDK8, along with the G1GC to unload the garbage classes
>> very fast.
>>
>> Cheers,
>> Gopal
>>
>>
>>
>
Re: hive 2.0.0 + tez
Posted by Stephen Sprague <sp...@gmail.com>.
thanks as always Gopal. lemme use jdk8 for starters and eliminate that as
a possibility.
On Wed, Mar 9, 2016 at 10:30 AM, Gopal Vijayaraghavan <go...@apache.org>
wrote:
>
> > ok, thanks. very strange then that i can run hive 1.2.1 and hive 2.0.0
> >m/r jobs against the same hadoop cluster w/o issue
> > (different HS2, different MetaStore, but same hdfs data) but yet once i
> >switch to the HS2/2.0.0 tez chokes out - but works fine with the
> >HS2/1.2.1.
>
> It is not particularly strange considering no development for MRv2 has
> happened in the mean-time - there's no difference in version dependencies
> between 2.0 & 1.2 for MRv2.
>
> Not so for Tez.
>
> > I'm thinking this is because the tez jars on hdfs have been built (bound
> >to) hive 1.2.1?
>
> I suspect the tez jars inside hive's classpath rather than the ones on
> HDFS.
>
> 0.8.2 is effectively a super-set of APIs in 0.7.4, so ending up with 0.7.x
> jars in the local classpath would work fine for hive-1.2.1, while when you
> run the same classpath with hive-2.0, it wouldn't work.
>
> Saw a PermGen error reported in a later email - the solution to all
> permgen issues was to leave JDK7 behind (End-of-Life April 2015) & use the
> Metaspace impl in JDK8, along with the G1GC to unload the garbage classes
> very fast.
>
> Cheers,
> Gopal
>
>
>
Re: hive 2.0.0 + tez
Posted by Gopal Vijayaraghavan <go...@apache.org>.
> ok, thanks. very strange then that i can run hive 1.2.1 and hive 2.0.0
>m/r jobs against the same hadoop cluster w/o issue
> (different HS2, different MetaStore, but same hdfs data) but yet once i
>switch to the HS2/2.0.0 tez chokes out - but works fine with the
>HS2/1.2.1.
It is not particularly strange considering no development for MRv2 has
happened in the mean-time - there's no difference in version dependencies
between 2.0 & 1.2 for MRv2.
Not so for Tez.
> I'm thinking this is because the tez jars on hdfs have been built (bound
>to) hive 1.2.1?
I suspect the tez jars inside hive's classpath rather than the ones on
HDFS.
0.8.2 is effectively a super-set of APIs in 0.7.4, so ending up with 0.7.x
jars in the local classpath would work fine for hive-1.2.1, while when you
run the same classpath with hive-2.0, it wouldn't work.
Saw a PermGen error reported in a later email - the solution to all
permgen issues was to leave JDK7 behind (End-of-Life April 2015) & use the
Metaspace impl in JDK8, along with the G1GC to unload the garbage classes
very fast.
Cheers,
Gopal
Re: hive 2.0.0 + tez
Posted by Stephen Sprague <sp...@gmail.com>.
well i got a little more information now.
under 2.0.0
* running beeline-cli; engine=tez works (beeline -u jdbc:hive2://)
* running beeline-hs2; engine=tez fails (beeline -u
jdbc:hive2://host:10000/)
examining HS2 log file i find this:
java.lang.OutOfMemoryError: PermGen space
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at
org.apache.tez.mapreduce.protos.MRRuntimeProtos$MRInputUserPayloadProto.newBuilder(MRRuntimeProtos.java:1944)
at
org.apache.tez.mapreduce.hadoop.MRInputHelpers.createMRInputPayload(MRInputHelpers.java:699)
at
org.apache.tez.mapreduce.hadoop.MRInputHelpers.createMRInputPayloadWithGrouping(MRInputHelpers.java:681)
at
org.apache.tez.mapreduce.input.MRInput$MRInputHelpersInternal.createMRInputPayloadWithGrouping(MRInput.java:660)
at
org.apache.tez.mapreduce.input.MRInput$MRInputConfigBuilder.createCustomDataSource(MRInput.java:283)
at
org.apache.tez.mapreduce.input.MRInput$MRInputConfigBuilder.build(MRInput.java:232)
at
org.apache.hadoop.hive.ql.exec.tez.DagUtils.createVertex(DagUtils.java:605)
at
org.apache.hadoop.hive.ql.exec.tez.DagUtils.createVertex(DagUtils.java:1113)
at
org.apache.hadoop.hive.ql.exec.tez.TezTask.build(TezTask.java:389)
at
org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:175)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:158)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:101)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1840)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1584)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1361)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1184)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1177)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:175)
at
org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:73)
at
org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:228)
FAILED: Execution Error, return code -101 from
org.apache.hadoop.hive.ql.exec.tez.TezTask. PermGen space
Exception in thread
"374e076e-90ee-42a7-9dce-71f90f95f218HiveServer2-Handler-Pool: Thread-50"
java.lang.OutOfMemoryError: PermGen space
i also find that the HS2 process is still up but the HS2 Web UI port
(10002) is dead.
here's the settings HS2 is running as:
dwr 32434 32432 9 09:07 pts/7 00:01:41
/usr/lib/jvm/java-7-oracle/jre/bin/java -Xmx8192m
-Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/usr/lib/hadoop/logs
-Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/lib/hadoop
-Dhadoop.id.str= -Dhadoop.root.logger=INFO,console
-Djava.library.path=/usr/lib/hadoop/lib/native
-Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true
-Xmx268435456 *-Xmx4G* -Dlog4j.configurationFile=hive-log4j2.properties
-Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar
/usr/lib/hive-standalone-2.0.0/lib/hive-service-2.0.0.jar
org.apache.hive.service.server.HiveServer2
so my first question is where is the "TezTask PermGen space" defined?
Thanks,
Stephen.
On Wed, Mar 9, 2016 at 7:48 AM, Stephen Sprague <sp...@gmail.com> wrote:
> ok, thanks. very strange then that i can run hive 1.2.1 and hive 2.0.0
> m/r jobs against the same hadoop cluster w/o issue (different HS2,
> different MetaStore, but same hdfs data) but yet once i switch to the
> HS2/2.0.0 tez chokes out - but works fine with the HS2/1.2.1.
>
> given i built tez when HIVE_HOME was set to 1.2.1 i'm wondering... that's
> why its not having any part of working under 2.0.0.
>
> anyway, i'm going to review the tez installation instructions now that i'm
> testing hive 2.0.0 (and tez) and probably just start from scratch.
>
> I'll keep you all posted.
>
> Cheers,
> stephen.
>
>
>
> On Wed, Mar 9, 2016 at 2:51 AM, Andre Kelpe <ak...@concurrentinc.com>
> wrote:
>
>> tez has no dependency on hive, it is the other way around. I don't
>> think, that's the problem you are facing.
>>
>> - André
>>
>> On Wed, Mar 9, 2016 at 6:20 AM, Stephen Sprague <sp...@gmail.com>
>> wrote:
>> > hey guys,
>> >
>> > obvious question i suppose but i gotta ask. :) the tez jars i built
>> for
>> > hive 1.2.1 will not work with hive 2.0.0 i presume?
>> >
>> > the reason ask is: i can switch back forth between hive clients 1.2.1
>> and
>> > 2.0.0 and run m/r jobs no problem. however i can not do this with tez.
>> >
>> > I'm thinking this is because the tez jars on hdfs have been built
>> (bound to)
>> > hive 1.2.1?
>> >
>> > ie. these jar files in hdfs
>> >
>> > <property>
>> > <name>tez.lib.uris</name>
>> >
>> >
>> <value>${fs.defaultFS}/apps/tez-0.8.2,${fs.defaultFS}/apps/tez-0.8.2/lib</value>
>> > </property>
>> >
>> > really should be identified as "tez-0.8.2-hive.1.2.1" and that i should
>> > have another dir called "tez-0.8.2-hive.2.0.0" which another
>> tez-site.xml
>> > should refer to.
>> >
>> >
>> > Am i making any sense here? :)
>> >
>> > Cheers,
>> > Stephen.
>>
>>
>>
>> --
>> André Kelpe
>> andre@concurrentinc.com
>> http://concurrentinc.com
>>
>
>
Re: hive 2.0.0 + tez
Posted by Stephen Sprague <sp...@gmail.com>.
ok, thanks. very strange then that i can run hive 1.2.1 and hive 2.0.0 m/r
jobs against the same hadoop cluster w/o issue (different HS2, different
MetaStore, but same hdfs data) but yet once i switch to the HS2/2.0.0 tez
chokes out - but works fine with the HS2/1.2.1.
given i built tez when HIVE_HOME was set to 1.2.1 i'm wondering... that's
why its not having any part of working under 2.0.0.
anyway, i'm going to review the tez installation instructions now that i'm
testing hive 2.0.0 (and tez) and probably just start from scratch.
I'll keep you all posted.
Cheers,
stephen.
On Wed, Mar 9, 2016 at 2:51 AM, Andre Kelpe <ak...@concurrentinc.com>
wrote:
> tez has no dependency on hive, it is the other way around. I don't
> think, that's the problem you are facing.
>
> - André
>
> On Wed, Mar 9, 2016 at 6:20 AM, Stephen Sprague <sp...@gmail.com>
> wrote:
> > hey guys,
> >
> > obvious question i suppose but i gotta ask. :) the tez jars i built for
> > hive 1.2.1 will not work with hive 2.0.0 i presume?
> >
> > the reason ask is: i can switch back forth between hive clients 1.2.1 and
> > 2.0.0 and run m/r jobs no problem. however i can not do this with tez.
> >
> > I'm thinking this is because the tez jars on hdfs have been built (bound
> to)
> > hive 1.2.1?
> >
> > ie. these jar files in hdfs
> >
> > <property>
> > <name>tez.lib.uris</name>
> >
> >
> <value>${fs.defaultFS}/apps/tez-0.8.2,${fs.defaultFS}/apps/tez-0.8.2/lib</value>
> > </property>
> >
> > really should be identified as "tez-0.8.2-hive.1.2.1" and that i should
> > have another dir called "tez-0.8.2-hive.2.0.0" which another tez-site.xml
> > should refer to.
> >
> >
> > Am i making any sense here? :)
> >
> > Cheers,
> > Stephen.
>
>
>
> --
> André Kelpe
> andre@concurrentinc.com
> http://concurrentinc.com
>
Re: hive 2.0.0 + tez
Posted by Andre Kelpe <ak...@concurrentinc.com>.
tez has no dependency on hive, it is the other way around. I don't
think, that's the problem you are facing.
- André
On Wed, Mar 9, 2016 at 6:20 AM, Stephen Sprague <sp...@gmail.com> wrote:
> hey guys,
>
> obvious question i suppose but i gotta ask. :) the tez jars i built for
> hive 1.2.1 will not work with hive 2.0.0 i presume?
>
> the reason ask is: i can switch back forth between hive clients 1.2.1 and
> 2.0.0 and run m/r jobs no problem. however i can not do this with tez.
>
> I'm thinking this is because the tez jars on hdfs have been built (bound to)
> hive 1.2.1?
>
> ie. these jar files in hdfs
>
> <property>
> <name>tez.lib.uris</name>
>
> <value>${fs.defaultFS}/apps/tez-0.8.2,${fs.defaultFS}/apps/tez-0.8.2/lib</value>
> </property>
>
> really should be identified as "tez-0.8.2-hive.1.2.1" and that i should
> have another dir called "tez-0.8.2-hive.2.0.0" which another tez-site.xml
> should refer to.
>
>
> Am i making any sense here? :)
>
> Cheers,
> Stephen.
--
André Kelpe
andre@concurrentinc.com
http://concurrentinc.com