You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@impala.apache.org by Dinujaya Kaweesha <di...@gmail.com> on 2021/05/11 03:41:26 UTC
Impala table creation failure while using Amazon S3 as the data store
Hi all,
I've been working with kudu, impala, hive, namenode, datanode and several
other services in the Hadoop ecosystem for some time and I have tried
recently to create tables in impala and use Amazon S3 as the data store as
mentioned in this page(
https://impala.apache.org/docs/build/html/topics/impala_s3.html) and faced
below mentioned issue while creating table.
*[0.0.0.0:11432 <http://0.0.0.0:11432/>] default> create table
sample_table_on_s3 (id int, val int, ex_filed string,name string, assertion
boolean, city string, country string) stored as parquet location
's3a://impala-data-store-s3/impala-tbl';*
*Query: create table sample_table_on_s3 (id int, val int, ex_filed
string,name string, assertion boolean, city string, country string) stored
as parquet location 's3a://impala-data-store-s3/impala-tbl'*
*ERROR: ImpalaRuntimeException: Error making 'createTable' RPC to Hive
Metastore:*
*CAUSED BY: MetaException: java.lang.RuntimeException:
java.lang.ClassNotFoundException: Class
org.apache.hadoop.fs.s3a.S3AFileSystem not found*
*[0.0.0.0:11432 <http://0.0.0.0:11432/>] default>*
Has anyone else faced this issue?
Software versions:
- Impala – 3.4.0
- Hive - 3.1.2
- Hadoop-3.3.0
Thanks & Regards,
Dinujaya Kaweesha
Re: Impala table creation failure while using Amazon S3 as the data store
Posted by Quanlong Huang <hu...@gmail.com>.
>
> No hive tables are also not created.
You need to make sure Hive works with this statement first. Impala depends
on HMS to create the table directory on S3.
Do we need to add additional jars apart from default jars come with impala
> and hive build to make it work?
After compiling Impala, you can find the dependent jars under
${IMPALA_HOME}/fe/target/dependency. The S3AFileSystem is
in hadoop-aws-xxx.jar. You need to make sure they are included in the
classpath of your HMS.
On Tue, May 11, 2021 at 1:02 PM Dinujaya Kaweesha <di...@gmail.com>
wrote:
> Hi Quanlong,
>
> Thank you very much for your email. No hive tables are also not created.
> Do we need to add additional jars apart from default jars come with impala
> and hive build to make it work?
>
> Thanks & Regards,
> Dinujaya Kaweesha
>
> On Tue, 11 May 2021 at 10:00, Quanlong Huang <hu...@gmail.com>
> wrote:
>
>> Hi Dinujaya,
>>
>> Does it work if you create the table in Hive? It seems the s3 jars are
>> not placed in HMS's classpath, so HMS failed to create the table.
>>
>> Quanlong
>>
>> On Tue, May 11, 2021 at 11:41 AM Dinujaya Kaweesha <
>> dinukaweesha@gmail.com> wrote:
>>
>>> Hi all,
>>>
>>>
>>>
>>> I've been working with kudu, impala, hive, namenode, datanode and
>>> several other services in the Hadoop ecosystem for some time and I have
>>> tried recently to create tables in impala and use Amazon S3 as the data
>>> store as mentioned in this page(
>>> https://impala.apache.org/docs/build/html/topics/impala_s3.html) and
>>> faced below mentioned issue while creating table.
>>>
>>>
>>>
>>> *[0.0.0.0:11432 <http://0.0.0.0:11432/>] default> create table
>>> sample_table_on_s3 (id int, val int, ex_filed string,name string, assertion
>>> boolean, city string, country string) stored as parquet location
>>> 's3a://impala-data-store-s3/impala-tbl';*
>>>
>>> *Query: create table sample_table_on_s3 (id int, val int, ex_filed
>>> string,name string, assertion boolean, city string, country string) stored
>>> as parquet location 's3a://impala-data-store-s3/impala-tbl'*
>>>
>>> *ERROR: ImpalaRuntimeException: Error making 'createTable' RPC to Hive
>>> Metastore:*
>>>
>>> *CAUSED BY: MetaException: java.lang.RuntimeException:
>>> java.lang.ClassNotFoundException: Class
>>> org.apache.hadoop.fs.s3a.S3AFileSystem not found*
>>>
>>>
>>>
>>> *[0.0.0.0:11432 <http://0.0.0.0:11432/>] default>*
>>>
>>> Has anyone else faced this issue?
>>>
>>> Software versions:
>>>
>>> - Impala – 3.4.0
>>> - Hive - 3.1.2
>>> - Hadoop-3.3.0
>>>
>>>
>>> Thanks & Regards,
>>> Dinujaya Kaweesha
>>>
>>
Re: Impala table creation failure while using Amazon S3 as the data store
Posted by Dinujaya Kaweesha <di...@gmail.com>.
Hi Quanlong,
Thank you very much for your email. No hive tables are also not created. Do
we need to add additional jars apart from default jars come with impala and
hive build to make it work?
Thanks & Regards,
Dinujaya Kaweesha
On Tue, 11 May 2021 at 10:00, Quanlong Huang <hu...@gmail.com>
wrote:
> Hi Dinujaya,
>
> Does it work if you create the table in Hive? It seems the s3 jars are not
> placed in HMS's classpath, so HMS failed to create the table.
>
> Quanlong
>
> On Tue, May 11, 2021 at 11:41 AM Dinujaya Kaweesha <di...@gmail.com>
> wrote:
>
>> Hi all,
>>
>>
>>
>> I've been working with kudu, impala, hive, namenode, datanode and
>> several other services in the Hadoop ecosystem for some time and I have
>> tried recently to create tables in impala and use Amazon S3 as the data
>> store as mentioned in this page(
>> https://impala.apache.org/docs/build/html/topics/impala_s3.html) and
>> faced below mentioned issue while creating table.
>>
>>
>>
>> *[0.0.0.0:11432 <http://0.0.0.0:11432/>] default> create table
>> sample_table_on_s3 (id int, val int, ex_filed string,name string, assertion
>> boolean, city string, country string) stored as parquet location
>> 's3a://impala-data-store-s3/impala-tbl';*
>>
>> *Query: create table sample_table_on_s3 (id int, val int, ex_filed
>> string,name string, assertion boolean, city string, country string) stored
>> as parquet location 's3a://impala-data-store-s3/impala-tbl'*
>>
>> *ERROR: ImpalaRuntimeException: Error making 'createTable' RPC to Hive
>> Metastore:*
>>
>> *CAUSED BY: MetaException: java.lang.RuntimeException:
>> java.lang.ClassNotFoundException: Class
>> org.apache.hadoop.fs.s3a.S3AFileSystem not found*
>>
>>
>>
>> *[0.0.0.0:11432 <http://0.0.0.0:11432/>] default>*
>>
>> Has anyone else faced this issue?
>>
>> Software versions:
>>
>> - Impala – 3.4.0
>> - Hive - 3.1.2
>> - Hadoop-3.3.0
>>
>>
>> Thanks & Regards,
>> Dinujaya Kaweesha
>>
>
Re: Impala table creation failure while using Amazon S3 as the data store
Posted by Quanlong Huang <hu...@gmail.com>.
Hi Dinujaya,
Does it work if you create the table in Hive? It seems the s3 jars are not
placed in HMS's classpath, so HMS failed to create the table.
Quanlong
On Tue, May 11, 2021 at 11:41 AM Dinujaya Kaweesha <di...@gmail.com>
wrote:
> Hi all,
>
>
>
> I've been working with kudu, impala, hive, namenode, datanode and several
> other services in the Hadoop ecosystem for some time and I have tried
> recently to create tables in impala and use Amazon S3 as the data store as
> mentioned in this page(
> https://impala.apache.org/docs/build/html/topics/impala_s3.html) and
> faced below mentioned issue while creating table.
>
>
>
> *[0.0.0.0:11432 <http://0.0.0.0:11432/>] default> create table
> sample_table_on_s3 (id int, val int, ex_filed string,name string, assertion
> boolean, city string, country string) stored as parquet location
> 's3a://impala-data-store-s3/impala-tbl';*
>
> *Query: create table sample_table_on_s3 (id int, val int, ex_filed
> string,name string, assertion boolean, city string, country string) stored
> as parquet location 's3a://impala-data-store-s3/impala-tbl'*
>
> *ERROR: ImpalaRuntimeException: Error making 'createTable' RPC to Hive
> Metastore:*
>
> *CAUSED BY: MetaException: java.lang.RuntimeException:
> java.lang.ClassNotFoundException: Class
> org.apache.hadoop.fs.s3a.S3AFileSystem not found*
>
>
>
> *[0.0.0.0:11432 <http://0.0.0.0:11432/>] default>*
>
> Has anyone else faced this issue?
>
> Software versions:
>
> - Impala – 3.4.0
> - Hive - 3.1.2
> - Hadoop-3.3.0
>
>
> Thanks & Regards,
> Dinujaya Kaweesha
>