You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Ophir Cohen <op...@gmail.com> on 2015/09/29 14:20:37 UTC
Hive alter table is failing
Hi,
I'm using Spark on top of Hive.
As I want to keep old tables I store the DataFrame into tmp table in hive
and when finished successfully I rename the table.
In last few days I've upgrade to use Spark 1.4.1, and as I'm using aws emr
I got Hive 1.0.
Now when I try to rename the table I get the following error:
Caused by: InvalidOperationException(message:Unable to access old location
hdfs://ip-10-140-189-94.ec2.internal:8020/user/hive/warehouse/<table_name>_29092015_111704_tmp
for table default.<table_name>_29092015_111704_tmp)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_table_with_environment_context_result$alter_table_with_environment_context_resultStandardScheme.read(ThriftHiveMetastore.java:34066)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_table_with_environment_context_result$alter_table_with_environment_context_resultStandardScheme.read(ThriftHiveMetastore.java:34052)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_table_with_environment_context_result.read(ThriftHiveMetastore.java:33994)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_alter_table_with_environment_context(ThriftHiveMetastore.java:1163)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.alter_table_with_environment_context(ThriftHiveMetastore.java:1147)
I suspect that this is the bug:
https://issues.apache.org/jira/browse/HIVE-10719 but it strange cause it
works from Hive CLI.
Did anyone encounter that?
Do we have any workaround?
Thanks,
Ophir
Re: Hive alter table is failing
Posted by Ophir Cohen <op...@gmail.com>.
Nop, I'm checking it out thanks!
On Tue, Sep 29, 2015 at 3:30 PM, Ted Yu <yu...@gmail.com> wrote:
> Have you seen this thread ?
> http://search-hadoop.com/m/q3RTtGwP431AQ2B41
>
> Plugin metastore version for your deployment.
>
> Cheers
>
> On Sep 29, 2015, at 5:20 AM, Ophir Cohen <op...@gmail.com> wrote:
>
> Hi,
>
> I'm using Spark on top of Hive.
> As I want to keep old tables I store the DataFrame into tmp table in hive
> and when finished successfully I rename the table.
>
> In last few days I've upgrade to use Spark 1.4.1, and as I'm using aws emr
> I got Hive 1.0.
> Now when I try to rename the table I get the following error:
>
> Caused by: InvalidOperationException(message:Unable to access old location
> hdfs://ip-10-140-189-94.ec2.internal:8020/user/hive/warehouse/<table_name>_29092015_111704_tmp
> for table default.<table_name>_29092015_111704_tmp)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_table_with_environment_context_result$alter_table_with_environment_context_resultStandardScheme.read(ThriftHiveMetastore.java:34066)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_table_with_environment_context_result$alter_table_with_environment_context_resultStandardScheme.read(ThriftHiveMetastore.java:34052)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_table_with_environment_context_result.read(ThriftHiveMetastore.java:33994)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_alter_table_with_environment_context(ThriftHiveMetastore.java:1163)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.alter_table_with_environment_context(ThriftHiveMetastore.java:1147)
>
>
> I suspect that this is the bug:
> https://issues.apache.org/jira/browse/HIVE-10719 but it strange cause it
> works from Hive CLI.
>
> Did anyone encounter that?
> Do we have any workaround?
>
> Thanks,
> Ophir
>
>
Re: Hive alter table is failing
Posted by Ted Yu <yu...@gmail.com>.
Have you seen this thread ?
http://search-hadoop.com/m/q3RTtGwP431AQ2B41
Plugin metastore version for your deployment.
Cheers
> On Sep 29, 2015, at 5:20 AM, Ophir Cohen <op...@gmail.com> wrote:
>
> Hi,
>
> I'm using Spark on top of Hive.
> As I want to keep old tables I store the DataFrame into tmp table in hive and when finished successfully I rename the table.
>
> In last few days I've upgrade to use Spark 1.4.1, and as I'm using aws emr I got Hive 1.0.
> Now when I try to rename the table I get the following error:
>
> Caused by: InvalidOperationException(message:Unable to access old location hdfs://ip-10-140-189-94.ec2.internal:8020/user/hive/warehouse/<table_name>_29092015_111704_tmp for table default.<table_name>_29092015_111704_tmp)
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_table_with_environment_context_result$alter_table_with_environment_context_resultStandardScheme.read(ThriftHiveMetastore.java:34066)
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_table_with_environment_context_result$alter_table_with_environment_context_resultStandardScheme.read(ThriftHiveMetastore.java:34052)
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_table_with_environment_context_result.read(ThriftHiveMetastore.java:33994)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_alter_table_with_environment_context(ThriftHiveMetastore.java:1163)
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.alter_table_with_environment_context(ThriftHiveMetastore.java:1147)
>
>
> I suspect that this is the bug:
> https://issues.apache.org/jira/browse/HIVE-10719 but it strange cause it works from Hive CLI.
>
> Did anyone encounter that?
> Do we have any workaround?
>
> Thanks,
> Ophir