You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Eva Tse <et...@netflix.com> on 2009/07/09 00:45:06 UTC
Can't start hive after setting HIVE_AUX_JARS_PATH ...
We set the env variable HIVE_AUX_JARS_PATH to a local path on the master
node, which is on EC2 LS , which has a few jar files. When we start hive, it
fails to start
Hadoop/Hive version : Hadoop 0.20 unpatched and Hive trunk (r786648) w/
HIVE-487 patch
It looks like the invocation command line is changed/different. Or maybe we
have a setup issue? Any idea why this is happening?
Thanks in advance,
Eva.
Here is the command line that starts hive (extracted from ps):
/usr/java/default/bin/java -Xmx1000m -Dhadoop.log.dir=/var/log/hadoop-0.20
-Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/lib/hadoop-0.20
-Dhadoop.id.str= -Dhadoop.root.logger=INFO,console
-Dhadoop.policy.file=hadoop-policy.xml -classpath
/etc/hadoop-0.20/conf:/usr/java/default/lib/tools.jar:/usr/lib/hadoop-0.20:/
usr/lib/hadoop-0.20/hadoop-0.20.0-1.cloudera-core.jar:/usr/lib/hadoop-0.20/l
ib/commons-cli-2.0-SNAPSHOT.jar:/usr/lib/hadoop-0.20/lib/commons-codec-1.3.j
ar:/usr/lib/hadoop-0.20/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20/lib/comm
ons-httpclient-3.0.1.jar:/usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar:
/usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar:/usr/lib/hadoop-0.20/
lib/commons-net-1.4.1.jar:/usr/lib/hadoop-0.20/lib/core-3.1.1.jar:/usr/lib/h
adoop-0.20/lib/hadoop-0.20.0-1.cloudera-fairscheduler.jar:/usr/lib/hadoop-0.
20/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.j
ar:/usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar:/usr/lib/hadoop-0.20/l
ib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20/lib/jetty-6.1.14.jar:/usr/lib/hadoo
p-0.20/lib/jetty-util-6.1.14.jar:/usr/lib/hadoop-0.20/lib/junit-3.8.1.jar:/u
sr/lib/hadoop-0.20/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20/lib/log4j-1.2.15.j
ar:/usr/lib/hadoop-0.20/lib/mysql-connector-java-5.0.8-bin.jar:/usr/lib/hado
op-0.20/lib/oro-2.0.8.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.ja
r:/usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar:/usr/lib/hadoop-0.20/lib/slf4
j-log4j12-1.4.3.jar:/usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar:/usr/lib/hadoop
-0.20/lib/jsp-2.1/jsp-2.1.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.j
ar:/usr/lib/hive-r786648/build/dist/bin/../conf:/usr/lib/hive-r786648/build/
dist/bin/../lib/TestSerDe.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/an
tlr-runtime-3.0.1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/asm-3.1.ja
r:/usr/lib/hive-r786648/build/dist/bin/../lib/commons-cli-2.0-SNAPSHOT.jar:/
usr/lib/hive-r786648/build/dist/bin/../lib/commons-collections-3.2.1.jar:/us
r/lib/hive-r786648/build/dist/bin/../lib/commons-lang-2.4.jar:/usr/lib/hive-
r786648/build/dist/bin/../lib/commons-logging-1.0.4.jar:/usr/lib/hive-r78664
8/build/dist/bin/../lib/commons-logging-api-1.0.4.jar:/usr/lib/hive-r786648/
build/dist/bin/../lib/derby.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/
hive_cli.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_common.jar:/us
r/lib/hive-r786648/build/dist/bin/../lib/hive_exec.jar:/usr/lib/hive-r786648
/build/dist/bin/../lib/hive_jdbc.jar:/usr/lib/hive-r786648/build/dist/bin/..
/lib/hive_metastore.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_ser
de.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_service.jar:/usr/lib
/hive-r786648/build/dist/bin/../lib/jdo2-api-2.1.jar:/usr/lib/hive-r786648/b
uild/dist/bin/../lib/jline-0.9.94.jar:/usr/lib/hive-r786648/build/dist/bin/.
./lib/jpox-core-1.2.2.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/jpox-e
nhancer-1.2.2.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/jpox-rdbms-1.2
.2.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/json.jar:/usr/lib/hive-r7
86648/build/dist/bin/../lib/junit-3.8.1.jar:/usr/lib/hive-r786648/build/dist
/bin/../lib/libfb303.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/libthri
ft.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/log4j-1.2.15.jar:/usr/lib
/hive-r786648/build/dist/bin/../lib/stringtemplate-3.1b1.jar:/usr/lib/hive-r
786648/build/dist/bin/../lib/velocity-1.5.jar::/home/dataeng/nfdev/dist/Pars
eNCCP.jar:/home/dataeng/nfdev/dist/geoip.jar:/home/dataeng/nfdev/dist/hiveud
f.jar org.apache.hadoop.util.RunJar -libjars
file:///home/dataeng/nfdev/dist/ParseNCCP.jar,file:///home/dataeng/nfdev/dis
t/geoip.jar,file:///home/dataeng/nfdev/dist/hiveudf.jar
/usr/lib/hive-r786648/build/dist/bin/../lib/hive_cli.jar
org.apache.hadoop.hive.cli.CliDriver -hiveconf
hive.aux.jars.path=file:///home/dataeng/nfdev/dist/ParseNCCP.jar,file:///hom
e/dataeng/nfdev/dist/geoip.jar,file:///home/dataeng/nfdev/dist/hiveudf.jar
Error we get:
[dataeng@master nfdev]$ hive
Exception in thread "main" java.io.IOException: Error opening job jar:
-libjars
at org.apache.hadoop.util.RunJar.main(RunJar.java:90)
Caused by: java.util.zip.ZipException: error in opening zip file
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.<init>(ZipFile.java:114)
at java.util.jar.JarFile.<init>(JarFile.java:133)
at java.util.jar.JarFile.<init>(JarFile.java:70)
at org.apache.hadoop.util.RunJar.main(RunJar.java:88)
On another version of Hadoop/Hive combination (i.e, 0.19 Hadoop and 0.3
Hive) on a different cluster. Here is the last portion of the command line
when starting hive and it would work:
:/home/etse/nfdev/dist/geoip.jar:/home/etse/nfdev/dist/hiveudf.jar:/home/ets
e/nfdev/dist/ParseNCCP.jar org.apache.hadoop.mapred.JobShell -libjars
file:///home/etse/nfdev/dist/geoip.jar,file:///home/etse/nfdev/dist/hiveudf.
jar,file:///home/etse/nfdev/dist/ParseNCCP.jar
/apps/hadoop/hive/bin/../lib/hive_cli.jar
org.apache.hadoop.hive.cli.CliDriver -hiveconf
hive.aux.jars.path=file:///home/etse/nfdev/dist/geoip.jar,file:///home/etse/
nfdev/dist/hiveudf.jar,file:///home/etse/nfdev/dist/ParseNCCP.jar
Re: Can't start hive after setting HIVE_AUX_JARS_PATH ...
Posted by Eva Tse <et...@netflix.com>.
Thanks Joydeep & Zheng for the feedback.
We will apply the new 487 patch when it is ready.
Thanks!
Eva.
On 7/8/09 5:58 PM, "Joydeep Sen Sarma" <js...@facebook.com> wrote:
> Sorry - this is also needed as part of hive-487:
>
> In hadoop-20 - the -libjars has to come after the jar file/class
>
> Please try applying this patch to bin/ext/cli.sh
>
> --- cli.sh (revision 789726)
> +++ cli.sh (working copy)
> @@ -10,7 +10,7 @@
> exit 3;
> fi
>
> - exec $HADOOP jar $AUX_JARS_CMD_LINE ${HIVE_LIB}/hive_cli.jar $CLASS
> $HIVE_OPTS "$@"
> + exec $HADOOP jar ${HIVE_LIB}/hive_cli.jar $CLASS $AUX_JARS_CMD_LINE
> $HIVE_OPTS "$@"
> }
>
>
> I will add this to the 487 patch.
>
> -----Original Message-----
> From: Zheng Shao [mailto:zshao9@gmail.com]
> Sent: Wednesday, July 08, 2009 5:51 PM
> To: hive-user@hadoop.apache.org
> Subject: Re: Can't start hive after setting HIVE_AUX_JARS_PATH ...
>
> It seems hadoop 0.20 RunJar.java does not like the "-libjars" option.
> Can you try remove the "-libjars xxx" from the command line?
> It's added somewhere in bin/hive (or scripts that got called by bin/hive)
>
> Zheng
>
> On Wed, Jul 8, 2009 at 3:45 PM, Eva Tse<et...@netflix.com> wrote:
>>
>> We set the env variable HIVE_AUX_JARS_PATH to a local path on the master
>> node, which is on EC2 LS , which has a few jar files. When we start hive, it
>> fails to start
>>
>> Hadoop/Hive version : Hadoop 0.20 unpatched and Hive trunk (r786648) w/
>> HIVE-487 patch
>>
>> It looks like the invocation command line is changed/different. Or maybe we
>> have a setup issue? Any idea why this is happening?
>>
>> Thanks in advance,
>> Eva.
>>
>> Here is the command line that starts hive (extracted from ps):
>> /usr/java/default/bin/java -Xmx1000m -Dhadoop.log.dir=/var/log/hadoop-0.20
>> -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/lib/hadoop-0.20
>> -Dhadoop.id.str= -Dhadoop.root.logger=INFO,console
>> -Dhadoop.policy.file=hadoop-policy.xml -classpath
>> /etc/hadoop-0.20/conf:/usr/java/default/lib/tools.jar:/usr/lib/hadoop-0.20:/u
>> sr/lib/hadoop-0.20/hadoop-0.20.0-1.cloudera-core.jar:/usr/lib/hadoop-0.20/lib
>> /commons-cli-2.0-SNAPSHOT.jar:/usr/lib/hadoop-0.20/lib/commons-codec-1.3.jar:
>> /usr/lib/hadoop-0.20/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20/lib/commons-
>> httpclient-3.0.1.jar:/usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar:/usr/
>> lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar:/usr/lib/hadoop-0.20/lib/co
>> mmons-net-1.4.1.jar:/usr/lib/hadoop-0.20/lib/core-3.1.1.jar:/usr/lib/hadoop-0
>> .20/lib/hadoop-0.20.0-1.cloudera-fairscheduler.jar:/usr/lib/hadoop-0.20/lib/h
>> sqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar:/usr/l
>> ib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jets3t-
>> 0.6.1.jar:/usr/lib/hadoop-0.20/lib/jetty-6.1.14.jar:/usr/lib/hadoop-0.20/lib/
>> jetty-util-6.1.14.jar:/usr/lib/hadoop-0.20/lib/junit-3.8.1.jar:/usr/lib/hadoo
>> p-0.20/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar:/usr/lib/h
>> adoop-0.20/lib/mysql-connector-java-5.0.8-bin.jar:/usr/lib/hadoop-0.20/lib/or
>> o-2.0.8.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hado
>> op-0.20/lib/slf4j-api-1.4.3.jar:/usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.
>> jar:/usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1
>> /jsp-2.1.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar:/usr/lib/hive-r
>> 786648/build/dist/bin/../conf:/usr/lib/hive-r786648/build/dist/bin/../lib/Tes
>> tSerDe.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/antlr-runtime-3.0.1.ja
>> r:/usr/lib/hive-r786648/build/dist/bin/../lib/asm-3.1.jar:/usr/lib/hive-r7866
>> 48/build/dist/bin/../lib/commons-cli-2.0-SNAPSHOT.jar:/usr/lib/hive-r786648/b
>> uild/dist/bin/../lib/commons-collections-3.2.1.jar:/usr/lib/hive-r786648/buil
>> d/dist/bin/../lib/commons-lang-2.4.jar:/usr/lib/hive-r786648/build/dist/bin/.
>> ./lib/commons-logging-1.0.4.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/c
>> ommons-logging-api-1.0.4.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/derb
>> y.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_cli.jar:/usr/lib/hive-
>> r786648/build/dist/bin/../lib/hive_common.jar:/usr/lib/hive-r786648/build/dis
>> t/bin/../lib/hive_exec.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_j
>> dbc.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_metastore.jar:/usr/l
>> ib/hive-r786648/build/dist/bin/../lib/hive_serde.jar:/usr/lib/hive-r786648/bu
>> ild/dist/bin/../lib/hive_service.jar:/usr/lib/hive-r786648/build/dist/bin/../
>> lib/jdo2-api-2.1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/jline-0.9.94
>> .jar:/usr/lib/hive-r786648/build/dist/bin/../lib/jpox-core-1.2.2.jar:/usr/lib
>> /hive-r786648/build/dist/bin/../lib/jpox-enhancer-1.2.2.jar:/usr/lib/hive-r78
>> 6648/build/dist/bin/../lib/jpox-rdbms-1.2.2.jar:/usr/lib/hive-r786648/build/d
>> ist/bin/../lib/json.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/junit-3.8
>> .1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/libfb303.jar:/usr/lib/hive
>> -r786648/build/dist/bin/../lib/libthrift.jar:/usr/lib/hive-r786648/build/dist
>> /bin/../lib/log4j-1.2.15.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/stri
>> ngtemplate-3.1b1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/velocity-1.5
>> .jar::/home/dataeng/nfdev/dist/ParseNCCP.jar:/home/dataeng/nfdev/dist/geoip.j
>> ar:/home/dataeng/nfdev/dist/hiveudf.jar
>> org.apache.hadoop.util.RunJar -libjars
>> file:///home/dataeng/nfdev/dist/ParseNCCP.jar,file:///home/dataeng/nfdev/dist
>> /geoip.jar,file:///home/dataeng/nfdev/dist/hiveudf.jar
>> /usr/lib/hive-r786648/build/dist/bin/../lib/hive_cli.jar
>> org.apache.hadoop.hive.cli.CliDriver -hiveconf
>> hive.aux.jars.path=file:///home/dataeng/nfdev/dist/ParseNCCP.jar,file:///home
>> /dataeng/nfdev/dist/geoip.jar,file:///home/dataeng/nfdev/dist/hiveudf.jar
>>
>> Error we get:
>> [dataeng@master nfdev]$ hive
>> Exception in thread "main" java.io.IOException: Error opening job jar:
>> -libjars
>> at org.apache.hadoop.util.RunJar.main(RunJar.java:90)
>> Caused by: java.util.zip.ZipException: error in opening zip file
>> at java.util.zip.ZipFile.open(Native Method)
>> at java.util.zip.ZipFile.<init>(ZipFile.java:114)
>> at java.util.jar.JarFile.<init>(JarFile.java:133)
>> at java.util.jar.JarFile.<init>(JarFile.java:70)
>> at org.apache.hadoop.util.RunJar.main(RunJar.java:88)
>>
>>
>> On another version of Hadoop/Hive combination (i.e, 0.19 Hadoop and 0.3
>> Hive) on a different cluster. Here is the last portion of the command line
>> when starting hive and it would work:
>>
>> :/home/etse/nfdev/dist/geoip.jar:/home/etse/nfdev/dist/hiveudf.jar:/home/etse
>> /nfdev/dist/ParseNCCP.jar
>> org.apache.hadoop.mapred.JobShell -libjars
>> file:///home/etse/nfdev/dist/geoip.jar,file:///home/etse/nfdev/dist/hiveudf.j
>> ar,file:///home/etse/nfdev/dist/ParseNCCP.jar
>> /apps/hadoop/hive/bin/../lib/hive_cli.jar
>> org.apache.hadoop.hive.cli.CliDriver -hiveconf
>> hive.aux.jars.path=file:///home/etse/nfdev/dist/geoip.jar,file:///home/etse/n
>> fdev/dist/hiveudf.jar,file:///home/etse/nfdev/dist/ParseNCCP.jar
>
>
RE: Can't start hive after setting HIVE_AUX_JARS_PATH ...
Posted by Joydeep Sen Sarma <js...@facebook.com>.
Sorry - this is also needed as part of hive-487:
In hadoop-20 - the -libjars has to come after the jar file/class
Please try applying this patch to bin/ext/cli.sh
--- cli.sh (revision 789726)
+++ cli.sh (working copy)
@@ -10,7 +10,7 @@
exit 3;
fi
- exec $HADOOP jar $AUX_JARS_CMD_LINE ${HIVE_LIB}/hive_cli.jar $CLASS $HIVE_OPTS "$@"
+ exec $HADOOP jar ${HIVE_LIB}/hive_cli.jar $CLASS $AUX_JARS_CMD_LINE $HIVE_OPTS "$@"
}
I will add this to the 487 patch.
-----Original Message-----
From: Zheng Shao [mailto:zshao9@gmail.com]
Sent: Wednesday, July 08, 2009 5:51 PM
To: hive-user@hadoop.apache.org
Subject: Re: Can't start hive after setting HIVE_AUX_JARS_PATH ...
It seems hadoop 0.20 RunJar.java does not like the "-libjars" option.
Can you try remove the "-libjars xxx" from the command line?
It's added somewhere in bin/hive (or scripts that got called by bin/hive)
Zheng
On Wed, Jul 8, 2009 at 3:45 PM, Eva Tse<et...@netflix.com> wrote:
>
> We set the env variable HIVE_AUX_JARS_PATH to a local path on the master
> node, which is on EC2 LS , which has a few jar files. When we start hive, it
> fails to start
>
> Hadoop/Hive version : Hadoop 0.20 unpatched and Hive trunk (r786648) w/
> HIVE-487 patch
>
> It looks like the invocation command line is changed/different. Or maybe we
> have a setup issue? Any idea why this is happening?
>
> Thanks in advance,
> Eva.
>
> Here is the command line that starts hive (extracted from ps):
> /usr/java/default/bin/java -Xmx1000m -Dhadoop.log.dir=/var/log/hadoop-0.20
> -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/lib/hadoop-0.20
> -Dhadoop.id.str= -Dhadoop.root.logger=INFO,console
> -Dhadoop.policy.file=hadoop-policy.xml -classpath
> /etc/hadoop-0.20/conf:/usr/java/default/lib/tools.jar:/usr/lib/hadoop-0.20:/usr/lib/hadoop-0.20/hadoop-0.20.0-1.cloudera-core.jar:/usr/lib/hadoop-0.20/lib/commons-cli-2.0-SNAPSHOT.jar:/usr/lib/hadoop-0.20/lib/commons-codec-1.3.jar:/usr/lib/hadoop-0.20/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20/lib/commons-httpclient-3.0.1.jar:/usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-net-1.4.1.jar:/usr/lib/hadoop-0.20/lib/core-3.1.1.jar:/usr/lib/hadoop-0.20/lib/hadoop-0.20.0-1.cloudera-fairscheduler.jar:/usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20/lib/jetty-6.1.14.jar:/usr/lib/hadoop-0.20/lib/jetty-util-6.1.14.jar:/usr/lib/hadoop-0.20/lib/junit-3.8.1.jar:/usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar:/usr/lib/hadoop-0.20/lib/mysql-connector-java-5.0.8-bin.jar:/usr/lib/hadoop-0.20/lib/oro-2.0.8.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar:/usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar:/usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar:/usr/lib/hive-r786648/build/dist/bin/../conf:/usr/lib/hive-r786648/build/dist/bin/../lib/TestSerDe.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/antlr-runtime-3.0.1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/asm-3.1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/commons-cli-2.0-SNAPSHOT.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/commons-collections-3.2.1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/commons-lang-2.4.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/commons-logging-1.0.4.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/commons-logging-api-1.0.4.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/derby.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_cli.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_common.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_exec.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_jdbc.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_metastore.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_serde.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_service.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/jdo2-api-2.1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/jline-0.9.94.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/jpox-core-1.2.2.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/jpox-enhancer-1.2.2.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/jpox-rdbms-1.2.2.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/json.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/junit-3.8.1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/libfb303.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/libthrift.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/log4j-1.2.15.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/stringtemplate-3.1b1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/velocity-1.5.jar::/home/dataeng/nfdev/dist/ParseNCCP.jar:/home/dataeng/nfdev/dist/geoip.jar:/home/dataeng/nfdev/dist/hiveudf.jar
> org.apache.hadoop.util.RunJar -libjars
> file:///home/dataeng/nfdev/dist/ParseNCCP.jar,file:///home/dataeng/nfdev/dist/geoip.jar,file:///home/dataeng/nfdev/dist/hiveudf.jar
> /usr/lib/hive-r786648/build/dist/bin/../lib/hive_cli.jar
> org.apache.hadoop.hive.cli.CliDriver -hiveconf
> hive.aux.jars.path=file:///home/dataeng/nfdev/dist/ParseNCCP.jar,file:///home/dataeng/nfdev/dist/geoip.jar,file:///home/dataeng/nfdev/dist/hiveudf.jar
>
> Error we get:
> [dataeng@master nfdev]$ hive
> Exception in thread "main" java.io.IOException: Error opening job jar:
> -libjars
> at org.apache.hadoop.util.RunJar.main(RunJar.java:90)
> Caused by: java.util.zip.ZipException: error in opening zip file
> at java.util.zip.ZipFile.open(Native Method)
> at java.util.zip.ZipFile.<init>(ZipFile.java:114)
> at java.util.jar.JarFile.<init>(JarFile.java:133)
> at java.util.jar.JarFile.<init>(JarFile.java:70)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:88)
>
>
> On another version of Hadoop/Hive combination (i.e, 0.19 Hadoop and 0.3
> Hive) on a different cluster. Here is the last portion of the command line
> when starting hive and it would work:
>
> :/home/etse/nfdev/dist/geoip.jar:/home/etse/nfdev/dist/hiveudf.jar:/home/etse/nfdev/dist/ParseNCCP.jar
> org.apache.hadoop.mapred.JobShell -libjars
> file:///home/etse/nfdev/dist/geoip.jar,file:///home/etse/nfdev/dist/hiveudf.jar,file:///home/etse/nfdev/dist/ParseNCCP.jar
> /apps/hadoop/hive/bin/../lib/hive_cli.jar
> org.apache.hadoop.hive.cli.CliDriver -hiveconf
> hive.aux.jars.path=file:///home/etse/nfdev/dist/geoip.jar,file:///home/etse/nfdev/dist/hiveudf.jar,file:///home/etse/nfdev/dist/ParseNCCP.jar
--
Yours,
Zheng
Re: Can't start hive after setting HIVE_AUX_JARS_PATH ...
Posted by Zheng Shao <zs...@gmail.com>.
It seems hadoop 0.20 RunJar.java does not like the "-libjars" option.
Can you try remove the "-libjars xxx" from the command line?
It's added somewhere in bin/hive (or scripts that got called by bin/hive)
Zheng
On Wed, Jul 8, 2009 at 3:45 PM, Eva Tse<et...@netflix.com> wrote:
>
> We set the env variable HIVE_AUX_JARS_PATH to a local path on the master
> node, which is on EC2 LS , which has a few jar files. When we start hive, it
> fails to start
>
> Hadoop/Hive version : Hadoop 0.20 unpatched and Hive trunk (r786648) w/
> HIVE-487 patch
>
> It looks like the invocation command line is changed/different. Or maybe we
> have a setup issue? Any idea why this is happening?
>
> Thanks in advance,
> Eva.
>
> Here is the command line that starts hive (extracted from ps):
> /usr/java/default/bin/java -Xmx1000m -Dhadoop.log.dir=/var/log/hadoop-0.20
> -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/lib/hadoop-0.20
> -Dhadoop.id.str= -Dhadoop.root.logger=INFO,console
> -Dhadoop.policy.file=hadoop-policy.xml -classpath
> /etc/hadoop-0.20/conf:/usr/java/default/lib/tools.jar:/usr/lib/hadoop-0.20:/usr/lib/hadoop-0.20/hadoop-0.20.0-1.cloudera-core.jar:/usr/lib/hadoop-0.20/lib/commons-cli-2.0-SNAPSHOT.jar:/usr/lib/hadoop-0.20/lib/commons-codec-1.3.jar:/usr/lib/hadoop-0.20/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20/lib/commons-httpclient-3.0.1.jar:/usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-net-1.4.1.jar:/usr/lib/hadoop-0.20/lib/core-3.1.1.jar:/usr/lib/hadoop-0.20/lib/hadoop-0.20.0-1.cloudera-fairscheduler.jar:/usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20/lib/jetty-6.1.14.jar:/usr/lib/hadoop-0.20/lib/jetty-util-6.1.14.jar:/usr/lib/hadoop-0.20/lib/junit-3.8.1.jar:/usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar:/usr/lib/hadoop-0.20/lib/mysql-connector-java-5.0.8-bin.jar:/usr/lib/hadoop-0.20/lib/oro-2.0.8.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar:/usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar:/usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar:/usr/lib/hive-r786648/build/dist/bin/../conf:/usr/lib/hive-r786648/build/dist/bin/../lib/TestSerDe.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/antlr-runtime-3.0.1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/asm-3.1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/commons-cli-2.0-SNAPSHOT.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/commons-collections-3.2.1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/commons-lang-2.4.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/commons-logging-1.0.4.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/commons-logging-api-1.0.4.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/derby.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_cli.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_common.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_exec.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_jdbc.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_metastore.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_serde.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_service.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/jdo2-api-2.1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/jline-0.9.94.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/jpox-core-1.2.2.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/jpox-enhancer-1.2.2.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/jpox-rdbms-1.2.2.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/json.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/junit-3.8.1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/libfb303.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/libthrift.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/log4j-1.2.15.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/stringtemplate-3.1b1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/velocity-1.5.jar::/home/dataeng/nfdev/dist/ParseNCCP.jar:/home/dataeng/nfdev/dist/geoip.jar:/home/dataeng/nfdev/dist/hiveudf.jar
> org.apache.hadoop.util.RunJar -libjars
> file:///home/dataeng/nfdev/dist/ParseNCCP.jar,file:///home/dataeng/nfdev/dist/geoip.jar,file:///home/dataeng/nfdev/dist/hiveudf.jar
> /usr/lib/hive-r786648/build/dist/bin/../lib/hive_cli.jar
> org.apache.hadoop.hive.cli.CliDriver -hiveconf
> hive.aux.jars.path=file:///home/dataeng/nfdev/dist/ParseNCCP.jar,file:///home/dataeng/nfdev/dist/geoip.jar,file:///home/dataeng/nfdev/dist/hiveudf.jar
>
> Error we get:
> [dataeng@master nfdev]$ hive
> Exception in thread "main" java.io.IOException: Error opening job jar:
> -libjars
> at org.apache.hadoop.util.RunJar.main(RunJar.java:90)
> Caused by: java.util.zip.ZipException: error in opening zip file
> at java.util.zip.ZipFile.open(Native Method)
> at java.util.zip.ZipFile.<init>(ZipFile.java:114)
> at java.util.jar.JarFile.<init>(JarFile.java:133)
> at java.util.jar.JarFile.<init>(JarFile.java:70)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:88)
>
>
> On another version of Hadoop/Hive combination (i.e, 0.19 Hadoop and 0.3
> Hive) on a different cluster. Here is the last portion of the command line
> when starting hive and it would work:
>
> :/home/etse/nfdev/dist/geoip.jar:/home/etse/nfdev/dist/hiveudf.jar:/home/etse/nfdev/dist/ParseNCCP.jar
> org.apache.hadoop.mapred.JobShell -libjars
> file:///home/etse/nfdev/dist/geoip.jar,file:///home/etse/nfdev/dist/hiveudf.jar,file:///home/etse/nfdev/dist/ParseNCCP.jar
> /apps/hadoop/hive/bin/../lib/hive_cli.jar
> org.apache.hadoop.hive.cli.CliDriver -hiveconf
> hive.aux.jars.path=file:///home/etse/nfdev/dist/geoip.jar,file:///home/etse/nfdev/dist/hiveudf.jar,file:///home/etse/nfdev/dist/ParseNCCP.jar
--
Yours,
Zheng