You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Dilip Joseph <di...@gmail.com> on 2010/04/07 20:54:22 UTC

HIVE_AUX_JARS_PATH interfering with startup of Hive Web Interface

The Hive Webserver fails to startup with the following error message,
if HIVE_AUX_JARS_PATH environment variable is set (works fine if
unset).   Is this a bug?

$ build/dist/bin/hive --service hwi
Exception in thread "main" java.io.IOException: Error opening job jar: -libjars
        at org.apache.hadoop.util.RunJar.main(RunJar.java:90)
Caused by: java.util.zip.ZipException: error in opening zip file
        at java.util.zip.ZipFile.open(Native Method)
        at java.util.zip.ZipFile.<init>(ZipFile.java:114)
        at java.util.jar.JarFile.<init>(JarFile.java:133)
        at java.util.jar.JarFile.<init>(JarFile.java:70)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:88)

Slightly modifying the command line to launch hadoop in hwi.sh solves
the problem:

$ diff bin/ext/hwi.sh  /tmp/new-hwi.sh
28c28
<   exec $HADOOP jar $AUX_JARS_CMD_LINE ${HWI_JAR_FILE} $CLASS $HIVE_OPTS "$@"
---
>   exec $HADOOP jar ${HWI_JAR_FILE}  $CLASS $AUX_JARS_CMD_LINE $HIVE_OPTS "$@"


Dilip

Re: HIVE_AUX_JARS_PATH interfering with startup of Hive Web Interface

Posted by Edward Capriolo <ed...@gmail.com>.
On Thu, Apr 8, 2010 at 1:44 AM, Dilip Joseph
<di...@gmail.com>wrote:

> Created JIRA ticket https://issues.apache.org/jira/browse/HIVE-1294
>
> Dilip
>
> On Wed, Apr 7, 2010 at 12:08 PM, Edward Capriolo <ed...@gmail.com>
> wrote:
> >
> >
> > On Wed, Apr 7, 2010 at 2:54 PM, Dilip Joseph <
> dilip.antony.joseph@gmail.com>
> > wrote:
> >>
> >> The Hive Webserver fails to startup with the following error message,
> >> if HIVE_AUX_JARS_PATH environment variable is set (works fine if
> >> unset).   Is this a bug?
> >>
> >> $ build/dist/bin/hive --service hwi
> >> Exception in thread "main" java.io.IOException: Error opening job jar:
> >> -libjars
> >>        at org.apache.hadoop.util.RunJar.main(RunJar.java:90)
> >> Caused by: java.util.zip.ZipException: error in opening zip file
> >>        at java.util.zip.ZipFile.open(Native Method)
> >>        at java.util.zip.ZipFile.<init>(ZipFile.java:114)
> >>        at java.util.jar.JarFile.<init>(JarFile.java:133)
> >>        at java.util.jar.JarFile.<init>(JarFile.java:70)
> >>        at org.apache.hadoop.util.RunJar.main(RunJar.java:88)
> >>
> >> Slightly modifying the command line to launch hadoop in hwi.sh solves
> >> the problem:
> >>
> >> $ diff bin/ext/hwi.sh  /tmp/new-hwi.sh
> >> 28c28
> >> <   exec $HADOOP jar $AUX_JARS_CMD_LINE ${HWI_JAR_FILE} $CLASS
> $HIVE_OPTS
> >> "$@"
> >> ---
> >> >   exec $HADOOP jar ${HWI_JAR_FILE}  $CLASS $AUX_JARS_CMD_LINE
> $HIVE_OPTS
> >> > "$@"
> >>
> >>
> >> Dilip
> >
> > Dilip,
> >
> > From looking at the starup for the CLI
> >
> >   if [ $minor_ver -lt 20 ]; then
> >       exec $HADOOP jar $AUX_JARS_CMD_LINE ${HIVE_LIB}/hive-cli-*.jar
> $CLASS
> > $HIVE_OPTS "$@"
> >   else
> >       # hadoop 20 or newer - skip the aux_jars option. picked up from
> > hiveconf
> >       exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"
> >   fi
> >
> > HWI probably needs similar logic. We do not really have any direct code
> > coverage of the startup scripts.
> >
> > You can open a Jira ticket and assign it to me or tackle it if you would
> > like.
> >
> > Edward
> >
>

Dilip,

Great! consider re-factoring the entire process a bit. The idea is the main
shell script is supposed to carry all the common code. The ext/cli script is
supposed to carry the service dependent code.

This version detection is needed by all modules likely so it should be taken
out of the cli script and put in a shared place. Be careful to test your
changes against all the services by launching cli, web interface, hive
service. lineage, etc just for some confidence testing .

As I mentioned we have no unit testing on the kickoff scripts and it is easy
to forget that hive has more then just a cli.

Edward

Re: HIVE_AUX_JARS_PATH interfering with startup of Hive Web Interface

Posted by Dilip Joseph <di...@gmail.com>.
Created JIRA ticket https://issues.apache.org/jira/browse/HIVE-1294

Dilip

On Wed, Apr 7, 2010 at 12:08 PM, Edward Capriolo <ed...@gmail.com> wrote:
>
>
> On Wed, Apr 7, 2010 at 2:54 PM, Dilip Joseph <di...@gmail.com>
> wrote:
>>
>> The Hive Webserver fails to startup with the following error message,
>> if HIVE_AUX_JARS_PATH environment variable is set (works fine if
>> unset).   Is this a bug?
>>
>> $ build/dist/bin/hive --service hwi
>> Exception in thread "main" java.io.IOException: Error opening job jar:
>> -libjars
>>        at org.apache.hadoop.util.RunJar.main(RunJar.java:90)
>> Caused by: java.util.zip.ZipException: error in opening zip file
>>        at java.util.zip.ZipFile.open(Native Method)
>>        at java.util.zip.ZipFile.<init>(ZipFile.java:114)
>>        at java.util.jar.JarFile.<init>(JarFile.java:133)
>>        at java.util.jar.JarFile.<init>(JarFile.java:70)
>>        at org.apache.hadoop.util.RunJar.main(RunJar.java:88)
>>
>> Slightly modifying the command line to launch hadoop in hwi.sh solves
>> the problem:
>>
>> $ diff bin/ext/hwi.sh  /tmp/new-hwi.sh
>> 28c28
>> <   exec $HADOOP jar $AUX_JARS_CMD_LINE ${HWI_JAR_FILE} $CLASS $HIVE_OPTS
>> "$@"
>> ---
>> >   exec $HADOOP jar ${HWI_JAR_FILE}  $CLASS $AUX_JARS_CMD_LINE $HIVE_OPTS
>> > "$@"
>>
>>
>> Dilip
>
> Dilip,
>
> From looking at the starup for the CLI
>
>   if [ $minor_ver -lt 20 ]; then
>       exec $HADOOP jar $AUX_JARS_CMD_LINE ${HIVE_LIB}/hive-cli-*.jar $CLASS
> $HIVE_OPTS "$@"
>   else
>       # hadoop 20 or newer - skip the aux_jars option. picked up from
> hiveconf
>       exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"
>   fi
>
> HWI probably needs similar logic. We do not really have any direct code
> coverage of the startup scripts.
>
> You can open a Jira ticket and assign it to me or tackle it if you would
> like.
>
> Edward
>

Re: HIVE_AUX_JARS_PATH interfering with startup of Hive Web Interface

Posted by Edward Capriolo <ed...@gmail.com>.
On Wed, Apr 7, 2010 at 2:54 PM, Dilip Joseph
<di...@gmail.com>wrote:

> The Hive Webserver fails to startup with the following error message,
> if HIVE_AUX_JARS_PATH environment variable is set (works fine if
> unset).   Is this a bug?
>
> $ build/dist/bin/hive --service hwi
> Exception in thread "main" java.io.IOException: Error opening job jar:
> -libjars
>        at org.apache.hadoop.util.RunJar.main(RunJar.java:90)
> Caused by: java.util.zip.ZipException: error in opening zip file
>        at java.util.zip.ZipFile.open(Native Method)
>        at java.util.zip.ZipFile.<init>(ZipFile.java:114)
>        at java.util.jar.JarFile.<init>(JarFile.java:133)
>        at java.util.jar.JarFile.<init>(JarFile.java:70)
>        at org.apache.hadoop.util.RunJar.main(RunJar.java:88)
>
> Slightly modifying the command line to launch hadoop in hwi.sh solves
> the problem:
>
> $ diff bin/ext/hwi.sh  /tmp/new-hwi.sh
> 28c28
> <   exec $HADOOP jar $AUX_JARS_CMD_LINE ${HWI_JAR_FILE} $CLASS $HIVE_OPTS
> "$@"
> ---
> >   exec $HADOOP jar ${HWI_JAR_FILE}  $CLASS $AUX_JARS_CMD_LINE $HIVE_OPTS
> "$@"
>
>
> Dilip
>

Dilip,

>From looking at the starup for the CLI

  if [ $minor_ver -lt 20 ]; then
      exec $HADOOP jar $AUX_JARS_CMD_LINE ${HIVE_LIB}/hive-cli-*.jar $CLASS
$HIVE_OPTS "$@"
  else
      # hadoop 20 or newer - skip the aux_jars option. picked up from
hiveconf
      exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"
  fi

HWI probably needs similar logic. We do not really have any direct code
coverage of the startup scripts.

You can open a Jira ticket and assign it to me or tackle it if you would
like.

Edward