You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Ranjan Bagchi <ra...@powerreviews.com> on 2012/06/18 21:56:07 UTC

Migrating to hive 8.1 on EMR

Hi,

I've built a datastore using Hive 7.1 backed by S3 using persistent metadata.  Now that hive 8.1 is available, I'd like to migrate to the new version.

However, I'm having trouble reading tables with the persistent schema.  Looking in the logs, I'm getting stack traces like the following:
2012-06-18 19:51:23,176 ERROR org.apache.hadoop.hive.ql.exec.Task (SessionState.
java:printError(380)) - Failed with exception null
java.lang.NullPointerException
        at org.apache.hadoop.hive.ql.metadata.Table.checkValidity(Table.java:162
)
        at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:946)
        at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:860)
        at org.apache.hadoop.hive.ql.exec.DDLTask.describeTable(DDLTask.java:252
7)
        at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:298)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:133)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.ja
va:57)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1332)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1123)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:931)
        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:261)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:218)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:409)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:684)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:567)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

2012-06-18 19:51:23,180 ERROR org.apache.hadoop.hive.ql.Driver (SessionState.java:printError(380)) - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

Does anyone have experience with this?

Thanks,

Ranjan

Re: Migrating to hive 8.1 on EMR

Posted by Mark Grover <mg...@oanda.com>.
Sorry, missed the link:
https://forums.aws.amazon.com/thread.jspa?threadID=95988&tstart=25

----- Original Message -----
From: "Mark Grover" <mg...@oanda.com>
To: user@hive.apache.org
Sent: Tuesday, June 19, 2012 1:31:25 PM
Subject: Re: Migrating to hive 8.1 on EMR

Rajan,
On a somewhat related note, I had issues when using Amazon's MySQL server (on the master node) as metastore on a Hive 0.8.1 cluster.
Related thread:
----- Original Message -----
From: "Carl Steinbach" <ca...@cloudera.com>
To: user@hive.apache.org
Sent: Monday, June 18, 2012 5:07:12 PM
Subject: Re: Migrating to hive 8.1 on EMR

Hi Ranjan, 


Looks like the NPE is getting generated here: 



if (0 == getCols().size()) { 
throw new HiveException( 
"at least one column must be specified for the table"); 
} 


Which would seem to indicate that the table record which was fetched doesn't have any columns. 


Did you run the metastore upgrade scripts for 0.8.0 (located in metastore/scripts/upgrade)? Hive 0.8.0 includes HIVE-2246 which changed the name and schema of the metastore table used to store column information. 


Thanks. 


Carl 





On Mon, Jun 18, 2012 at 12:56 PM, Ranjan Bagchi < ranjan@powerreviews.com > wrote: 


Hi, 

I've built a datastore using Hive 7.1 backed by S3 using persistent metadata. Now that hive 8.1 is available, I'd like to migrate to the new version. 

However, I'm having trouble reading tables with the persistent schema. Looking in the logs, I'm getting stack traces like the following: 
2012-06-18 19:51:23,176 ERROR org.apache.hadoop.hive.ql.exec.Task (SessionState. 
java:printError(380)) - Failed with exception null 
java.lang.NullPointerException 
at org.apache.hadoop.hive.ql.metadata.Table.checkValidity(Table.java:162 
) 
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:946) 
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:860) 
at org.apache.hadoop.hive.ql.exec.DDLTask.describeTable(DDLTask.java:252 
7) 
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:298) 
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:133) 
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.ja 
va:57) 
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1332) 
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1123) 
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:931) 
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:261) 
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:218) 
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:409) 
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:684) 
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:567) 
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
at java.lang.reflect.Method.invoke(Method.java:597) 
at org.apache.hadoop.util.RunJar.main(RunJar.java:156) 

2012-06-18 19:51:23,180 ERROR org.apache.hadoop.hive.ql.Driver (SessionState.java:printError(380)) - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask 

Does anyone have experience with this? 

Thanks, 

Ranjan

Re: Migrating to hive 8.1 on EMR

Posted by Mark Grover <mg...@oanda.com>.
Rajan,
On a somewhat related note, I had issues when using Amazon's MySQL server (on the master node) as metastore on a Hive 0.8.1 cluster.
Related thread:
----- Original Message -----
From: "Carl Steinbach" <ca...@cloudera.com>
To: user@hive.apache.org
Sent: Monday, June 18, 2012 5:07:12 PM
Subject: Re: Migrating to hive 8.1 on EMR

Hi Ranjan, 


Looks like the NPE is getting generated here: 



if (0 == getCols().size()) { 
throw new HiveException( 
"at least one column must be specified for the table"); 
} 


Which would seem to indicate that the table record which was fetched doesn't have any columns. 


Did you run the metastore upgrade scripts for 0.8.0 (located in metastore/scripts/upgrade)? Hive 0.8.0 includes HIVE-2246 which changed the name and schema of the metastore table used to store column information. 


Thanks. 


Carl 





On Mon, Jun 18, 2012 at 12:56 PM, Ranjan Bagchi < ranjan@powerreviews.com > wrote: 


Hi, 

I've built a datastore using Hive 7.1 backed by S3 using persistent metadata. Now that hive 8.1 is available, I'd like to migrate to the new version. 

However, I'm having trouble reading tables with the persistent schema. Looking in the logs, I'm getting stack traces like the following: 
2012-06-18 19:51:23,176 ERROR org.apache.hadoop.hive.ql.exec.Task (SessionState. 
java:printError(380)) - Failed with exception null 
java.lang.NullPointerException 
at org.apache.hadoop.hive.ql.metadata.Table.checkValidity(Table.java:162 
) 
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:946) 
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:860) 
at org.apache.hadoop.hive.ql.exec.DDLTask.describeTable(DDLTask.java:252 
7) 
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:298) 
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:133) 
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.ja 
va:57) 
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1332) 
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1123) 
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:931) 
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:261) 
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:218) 
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:409) 
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:684) 
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:567) 
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
at java.lang.reflect.Method.invoke(Method.java:597) 
at org.apache.hadoop.util.RunJar.main(RunJar.java:156) 

2012-06-18 19:51:23,180 ERROR org.apache.hadoop.hive.ql.Driver (SessionState.java:printError(380)) - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask 

Does anyone have experience with this? 

Thanks, 

Ranjan

Re: Migrating to hive 8.1 on EMR

Posted by Carl Steinbach <ca...@cloudera.com>.
Hi Ranjan,

Looks like the NPE is getting generated here:

    if (0 == getCols().size()) {
      throw new HiveException(
          "at least one column must be specified for the table");
    }

Which would seem to indicate that the table record which was fetched
doesn't have any columns.

Did you run the metastore upgrade scripts for 0.8.0 (located in
metastore/scripts/upgrade)? Hive 0.8.0 includes HIVE-2246 which changed the
name and schema of the metastore table used to store column information.

Thanks.

Carl



On Mon, Jun 18, 2012 at 12:56 PM, Ranjan Bagchi <ra...@powerreviews.com>wrote:

> Hi,
>
> I've built a datastore using Hive 7.1 backed by S3 using persistent
> metadata.  Now that hive 8.1 is available, I'd like to migrate to the new
> version.
>
> However, I'm having trouble reading tables with the persistent schema.
>  Looking in the logs, I'm getting stack traces like the following:
> 2012-06-18 19:51:23,176 ERROR org.apache.hadoop.hive.ql.exec.Task
> (SessionState.
> java:printError(380)) - Failed with exception null
> java.lang.NullPointerException
>        at
> org.apache.hadoop.hive.ql.metadata.Table.checkValidity(Table.java:162
> )
>        at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:946)
>        at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:860)
>        at
> org.apache.hadoop.hive.ql.exec.DDLTask.describeTable(DDLTask.java:252
> 7)
>        at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:298)
>        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:133)
>        at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.ja
> va:57)
>        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1332)
>        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1123)
>        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:931)
>        at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:261)
>        at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:218)
>        at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:409)
>        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:684)
>        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:567)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>
> 2012-06-18 19:51:23,180 ERROR org.apache.hadoop.hive.ql.Driver
> (SessionState.java:printError(380)) - FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.DDLTask
>
> Does anyone have experience with this?
>
> Thanks,
>
> Ranjan
>
>