You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@impala.apache.org by Akila Wajirasena <ak...@gmail.com> on 2019/12/18 11:47:10 UTC
Using Apache Impala with Hive Meta store 3.1.0
Hi,
I am trying to build impala with hive 3.1 support by enabling
USE_CDP_HIVE=true.
Build finishes with any issues and impala starts and connect to the hive
3.1 standalone meta store without any issue.
However when I try to create a table using impala shell I get the following
error;
Has anyone encountered a similar issue?
Query: CREATE TABLE test_123(id INT, name STRING)
ERROR: ImpalaRuntimeException: Error making 'createTable' RPC to Hive
Metastore:
CAUSED BY: TApplicationException: Invalid method name: 'create_table_req'
I1218 11:10:13.229060 62868 jni-util.cc:288]
org.apache.impala.common.ImpalaRuntimeException: Error making 'createTable'
RPC to Hive Metastore:
at
org.apache.impala.service.CatalogOpExecutor.createTable(CatalogOpExecutor.java:2107)
at
org.apache.impala.service.CatalogOpExecutor.createTable(CatalogOpExecutor.java:1936)
at
org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:330)
at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:167)
Caused by: org.apache.thrift.TApplicationException: Invalid method name:
'create_table_req'
at
org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
at
org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_create_table_req(ThriftHiveMetastore.java:1698)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.create_table_req(ThriftHiveMetastore.java:1685)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.create_table_with_environment_context(HiveMetaStoreClient.java:3359)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:990)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:975)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:208)
at com.sun.proxy.$Proxy4.createTable(Unknown Source)
at
org.apache.impala.service.CatalogOpExecutor.createTable(CatalogOpExecutor.java:2086)
... 3 more
I1218 11:10:13.230059 62868 status.cc:124] ImpalaRuntimeException: Error
making 'createTable' RPC to Hive Metastore:
CAUSED BY: TApplicationException: Invalid method name: 'create_table_req'
@ 0xbc38a9
@ 0x129139e
@ 0xbb0317
@ 0xb9f0e9
@ 0xca58cc
@ 0xca3e4c
@ 0xb91059
@ 0x1028a10
@ 0x101e68e
@ 0x101f521
@ 0x131928f
@ 0x1319e39
@ 0x1a5f329
@ 0x7fc4313d0e24
@ 0x7fc42de2b34c
E1218 11:10:13.230078 62868 catalog-server.cc:112] ImpalaRuntimeException:
Error making 'createTable' RPC to Hive Metastore:
CAUSED BY: TApplicationException: Invalid method name: 'create_table_req'
I1218 11:10:13.230155 62868 catalog-server.cc:116] ExecDdl():
response=TDdlExecResponse {
01: result (struct) = TCatalogUpdateResult {
01: catalog_service_id (struct) = TUniqueId {
01: hi (i64) = 0,
02: lo (i64) = 0,
},
02: version (i64) = 0,
03: status (struct) = TStatus {
01: status_code (i32) = 2,
02: error_msgs (list) = list<string>[1] {
[0] = "ImpalaRuntimeException: Error making 'createTable' RPC to
Hive Metastore: \nCAUSED BY: TApplicationException: Invalid method name:
'create_table_req'",
},
},
04: is_invalidate (bool) = false,
},
}
--
Regards
Akila Wajirasena
Re: Using Apache Impala with Hive Meta store 3.1.0
Posted by Joe McDonnell <jo...@cloudera.com>.
Hi Akila,
I think getting Impala to compile against Hive 3.1 would be a modest amount
of effort. Most of it revolves around the IMPALA_HIVE_VERSION, and it would
require the Hive source (to get hive_metastore.thrift) and a maven
repository with the Hive 3.1 artifacts (which I think would be available in
the Apache maven repository).
So far, there has not been an effort to support vanilla versions of
Hadoop/Hive/etc. Contributions are definitely welcome. I think it would be
useful for it to be easy for Impala to be deployed against vanilla
deployments, but it would also a very large amount of effort to maintain.
Thanks,
Joe
On Wed, Dec 18, 2019 at 8:54 PM Akila Wajirasena <ak...@gmail.com>
wrote:
> Thanks you Joe. I was able to start get it work by using the Hive version
> that is downloaded when building Impala.
> How ever my other questions are;
> Is it possible the use the 3.1 Hive version to build Impala?
> Also is there any plans to support vanilla version of Hive and Hadoop for
> Impala?
> I saw most of the dependencies are downloaded from Cloudera repos.
>
> Thanks & Regards,
> Akila
>
>
> On Thu, Dec 19, 2019 at 6:37 AM Joe McDonnell <jo...@cloudera.com>
> wrote:
>
>> Hi Akila,
>>
>> USE_CDP_HIVE=true uses a Hive based off of Hive's master branch, which
>> has changed quite a bit since Hive 3.1. create_table_req was introduced in
>> HIVE-21838:
>>
>> https://github.com/apache/hive/commit/1fb0107ea522730196cd3bff7b1c4a7273f25291#diff-96182be0c29b7ceda2b60e931c70ecc4R1966
>> It looks like Hive's branch-3.1 does not have HIVE-21838, so it doesn't
>> understand create_table_req.
>>
>> Thanks,
>> Joe
>>
>>
>>
>>
>> On Wed, Dec 18, 2019 at 3:47 AM Akila Wajirasena <
>> akila.wajirasena@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I am trying to build impala with hive 3.1 support by enabling
>>> USE_CDP_HIVE=true.
>>> Build finishes with any issues and impala starts and connect to the hive
>>> 3.1 standalone meta store without any issue.
>>>
>>> However when I try to create a table using impala shell I get the
>>> following error;
>>> Has anyone encountered a similar issue?
>>>
>>> Query: CREATE TABLE test_123(id INT, name STRING)
>>> ERROR: ImpalaRuntimeException: Error making 'createTable' RPC to Hive
>>> Metastore:
>>> CAUSED BY: TApplicationException: Invalid method name: 'create_table_req'
>>>
>>> I1218 11:10:13.229060 62868 jni-util.cc:288]
>>> org.apache.impala.common.ImpalaRuntimeException: Error making 'createTable'
>>> RPC to Hive Metastore:
>>> at
>>> org.apache.impala.service.CatalogOpExecutor.createTable(CatalogOpExecutor.java:2107)
>>> at
>>> org.apache.impala.service.CatalogOpExecutor.createTable(CatalogOpExecutor.java:1936)
>>> at
>>> org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:330)
>>> at
>>> org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:167)
>>> Caused by: org.apache.thrift.TApplicationException: Invalid method name:
>>> 'create_table_req'
>>> at
>>> org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
>>> at
>>> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
>>> at
>>> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_create_table_req(ThriftHiveMetastore.java:1698)
>>> at
>>> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.create_table_req(ThriftHiveMetastore.java:1685)
>>> at
>>> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.create_table_with_environment_context(HiveMetaStoreClient.java:3359)
>>> at
>>> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:990)
>>> at
>>> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:975)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>> at
>>> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:208)
>>> at com.sun.proxy.$Proxy4.createTable(Unknown Source)
>>> at
>>> org.apache.impala.service.CatalogOpExecutor.createTable(CatalogOpExecutor.java:2086)
>>> ... 3 more
>>> I1218 11:10:13.230059 62868 status.cc:124] ImpalaRuntimeException: Error
>>> making 'createTable' RPC to Hive Metastore:
>>> CAUSED BY: TApplicationException: Invalid method name: 'create_table_req'
>>> @ 0xbc38a9
>>> @ 0x129139e
>>> @ 0xbb0317
>>> @ 0xb9f0e9
>>> @ 0xca58cc
>>> @ 0xca3e4c
>>> @ 0xb91059
>>> @ 0x1028a10
>>> @ 0x101e68e
>>> @ 0x101f521
>>> @ 0x131928f
>>> @ 0x1319e39
>>> @ 0x1a5f329
>>> @ 0x7fc4313d0e24
>>> @ 0x7fc42de2b34c
>>> E1218 11:10:13.230078 62868 catalog-server.cc:112]
>>> ImpalaRuntimeException: Error making 'createTable' RPC to Hive Metastore:
>>> CAUSED BY: TApplicationException: Invalid method name: 'create_table_req'
>>> I1218 11:10:13.230155 62868 catalog-server.cc:116] ExecDdl():
>>> response=TDdlExecResponse {
>>> 01: result (struct) = TCatalogUpdateResult {
>>> 01: catalog_service_id (struct) = TUniqueId {
>>> 01: hi (i64) = 0,
>>> 02: lo (i64) = 0,
>>> },
>>> 02: version (i64) = 0,
>>> 03: status (struct) = TStatus {
>>> 01: status_code (i32) = 2,
>>> 02: error_msgs (list) = list<string>[1] {
>>> [0] = "ImpalaRuntimeException: Error making 'createTable' RPC to
>>> Hive Metastore: \nCAUSED BY: TApplicationException: Invalid method name:
>>> 'create_table_req'",
>>> },
>>> },
>>> 04: is_invalidate (bool) = false,
>>> },
>>> }
>>>
>>>
>>> --
>>> Regards
>>> Akila Wajirasena
>>>
>>>
>>>
>>>
>
>
>
>
>
Re: Using Apache Impala with Hive Meta store 3.1.0
Posted by Akila Wajirasena <ak...@gmail.com>.
Thanks you Joe. I was able to start get it work by using the Hive version
that is downloaded when building Impala.
How ever my other questions are;
Is it possible the use the 3.1 Hive version to build Impala?
Also is there any plans to support vanilla version of Hive and Hadoop for
Impala?
I saw most of the dependencies are downloaded from Cloudera repos.
Thanks & Regards,
Akila
On Thu, Dec 19, 2019 at 6:37 AM Joe McDonnell <jo...@cloudera.com>
wrote:
> Hi Akila,
>
> USE_CDP_HIVE=true uses a Hive based off of Hive's master branch, which has
> changed quite a bit since Hive 3.1. create_table_req was introduced in
> HIVE-21838:
>
> https://github.com/apache/hive/commit/1fb0107ea522730196cd3bff7b1c4a7273f25291#diff-96182be0c29b7ceda2b60e931c70ecc4R1966
> It looks like Hive's branch-3.1 does not have HIVE-21838, so it doesn't
> understand create_table_req.
>
> Thanks,
> Joe
>
>
>
>
> On Wed, Dec 18, 2019 at 3:47 AM Akila Wajirasena <
> akila.wajirasena@gmail.com> wrote:
>
>> Hi,
>>
>> I am trying to build impala with hive 3.1 support by enabling
>> USE_CDP_HIVE=true.
>> Build finishes with any issues and impala starts and connect to the hive
>> 3.1 standalone meta store without any issue.
>>
>> However when I try to create a table using impala shell I get the
>> following error;
>> Has anyone encountered a similar issue?
>>
>> Query: CREATE TABLE test_123(id INT, name STRING)
>> ERROR: ImpalaRuntimeException: Error making 'createTable' RPC to Hive
>> Metastore:
>> CAUSED BY: TApplicationException: Invalid method name: 'create_table_req'
>>
>> I1218 11:10:13.229060 62868 jni-util.cc:288]
>> org.apache.impala.common.ImpalaRuntimeException: Error making 'createTable'
>> RPC to Hive Metastore:
>> at
>> org.apache.impala.service.CatalogOpExecutor.createTable(CatalogOpExecutor.java:2107)
>> at
>> org.apache.impala.service.CatalogOpExecutor.createTable(CatalogOpExecutor.java:1936)
>> at
>> org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:330)
>> at
>> org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:167)
>> Caused by: org.apache.thrift.TApplicationException: Invalid method name:
>> 'create_table_req'
>> at
>> org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
>> at
>> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
>> at
>> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_create_table_req(ThriftHiveMetastore.java:1698)
>> at
>> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.create_table_req(ThriftHiveMetastore.java:1685)
>> at
>> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.create_table_with_environment_context(HiveMetaStoreClient.java:3359)
>> at
>> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:990)
>> at
>> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:975)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at
>> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:208)
>> at com.sun.proxy.$Proxy4.createTable(Unknown Source)
>> at
>> org.apache.impala.service.CatalogOpExecutor.createTable(CatalogOpExecutor.java:2086)
>> ... 3 more
>> I1218 11:10:13.230059 62868 status.cc:124] ImpalaRuntimeException: Error
>> making 'createTable' RPC to Hive Metastore:
>> CAUSED BY: TApplicationException: Invalid method name: 'create_table_req'
>> @ 0xbc38a9
>> @ 0x129139e
>> @ 0xbb0317
>> @ 0xb9f0e9
>> @ 0xca58cc
>> @ 0xca3e4c
>> @ 0xb91059
>> @ 0x1028a10
>> @ 0x101e68e
>> @ 0x101f521
>> @ 0x131928f
>> @ 0x1319e39
>> @ 0x1a5f329
>> @ 0x7fc4313d0e24
>> @ 0x7fc42de2b34c
>> E1218 11:10:13.230078 62868 catalog-server.cc:112]
>> ImpalaRuntimeException: Error making 'createTable' RPC to Hive Metastore:
>> CAUSED BY: TApplicationException: Invalid method name: 'create_table_req'
>> I1218 11:10:13.230155 62868 catalog-server.cc:116] ExecDdl():
>> response=TDdlExecResponse {
>> 01: result (struct) = TCatalogUpdateResult {
>> 01: catalog_service_id (struct) = TUniqueId {
>> 01: hi (i64) = 0,
>> 02: lo (i64) = 0,
>> },
>> 02: version (i64) = 0,
>> 03: status (struct) = TStatus {
>> 01: status_code (i32) = 2,
>> 02: error_msgs (list) = list<string>[1] {
>> [0] = "ImpalaRuntimeException: Error making 'createTable' RPC to
>> Hive Metastore: \nCAUSED BY: TApplicationException: Invalid method name:
>> 'create_table_req'",
>> },
>> },
>> 04: is_invalidate (bool) = false,
>> },
>> }
>>
>>
>> --
>> Regards
>> Akila Wajirasena
>>
>>
>>
>>
Re: Using Apache Impala with Hive Meta store 3.1.0
Posted by Joe McDonnell <jo...@cloudera.com>.
Hi Akila,
USE_CDP_HIVE=true uses a Hive based off of Hive's master branch, which has
changed quite a bit since Hive 3.1. create_table_req was introduced in
HIVE-21838:
https://github.com/apache/hive/commit/1fb0107ea522730196cd3bff7b1c4a7273f25291#diff-96182be0c29b7ceda2b60e931c70ecc4R1966
It looks like Hive's branch-3.1 does not have HIVE-21838, so it doesn't
understand create_table_req.
Thanks,
Joe
On Wed, Dec 18, 2019 at 3:47 AM Akila Wajirasena <ak...@gmail.com>
wrote:
> Hi,
>
> I am trying to build impala with hive 3.1 support by enabling
> USE_CDP_HIVE=true.
> Build finishes with any issues and impala starts and connect to the hive
> 3.1 standalone meta store without any issue.
>
> However when I try to create a table using impala shell I get the
> following error;
> Has anyone encountered a similar issue?
>
> Query: CREATE TABLE test_123(id INT, name STRING)
> ERROR: ImpalaRuntimeException: Error making 'createTable' RPC to Hive
> Metastore:
> CAUSED BY: TApplicationException: Invalid method name: 'create_table_req'
>
> I1218 11:10:13.229060 62868 jni-util.cc:288]
> org.apache.impala.common.ImpalaRuntimeException: Error making 'createTable'
> RPC to Hive Metastore:
> at
> org.apache.impala.service.CatalogOpExecutor.createTable(CatalogOpExecutor.java:2107)
> at
> org.apache.impala.service.CatalogOpExecutor.createTable(CatalogOpExecutor.java:1936)
> at
> org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:330)
> at
> org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:167)
> Caused by: org.apache.thrift.TApplicationException: Invalid method name:
> 'create_table_req'
> at
> org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_create_table_req(ThriftHiveMetastore.java:1698)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.create_table_req(ThriftHiveMetastore.java:1685)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.create_table_with_environment_context(HiveMetaStoreClient.java:3359)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:990)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:975)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:208)
> at com.sun.proxy.$Proxy4.createTable(Unknown Source)
> at
> org.apache.impala.service.CatalogOpExecutor.createTable(CatalogOpExecutor.java:2086)
> ... 3 more
> I1218 11:10:13.230059 62868 status.cc:124] ImpalaRuntimeException: Error
> making 'createTable' RPC to Hive Metastore:
> CAUSED BY: TApplicationException: Invalid method name: 'create_table_req'
> @ 0xbc38a9
> @ 0x129139e
> @ 0xbb0317
> @ 0xb9f0e9
> @ 0xca58cc
> @ 0xca3e4c
> @ 0xb91059
> @ 0x1028a10
> @ 0x101e68e
> @ 0x101f521
> @ 0x131928f
> @ 0x1319e39
> @ 0x1a5f329
> @ 0x7fc4313d0e24
> @ 0x7fc42de2b34c
> E1218 11:10:13.230078 62868 catalog-server.cc:112] ImpalaRuntimeException:
> Error making 'createTable' RPC to Hive Metastore:
> CAUSED BY: TApplicationException: Invalid method name: 'create_table_req'
> I1218 11:10:13.230155 62868 catalog-server.cc:116] ExecDdl():
> response=TDdlExecResponse {
> 01: result (struct) = TCatalogUpdateResult {
> 01: catalog_service_id (struct) = TUniqueId {
> 01: hi (i64) = 0,
> 02: lo (i64) = 0,
> },
> 02: version (i64) = 0,
> 03: status (struct) = TStatus {
> 01: status_code (i32) = 2,
> 02: error_msgs (list) = list<string>[1] {
> [0] = "ImpalaRuntimeException: Error making 'createTable' RPC to
> Hive Metastore: \nCAUSED BY: TApplicationException: Invalid method name:
> 'create_table_req'",
> },
> },
> 04: is_invalidate (bool) = false,
> },
> }
>
>
> --
> Regards
> Akila Wajirasena
>
>
>
>