You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/02/26 09:08:19 UTC
[GitHub] [iceberg] allenzhg opened a new issue #2277: hive create external table for iceberg have error.
allenzhg opened a new issue #2277:
URL: https://github.com/apache/iceberg/issues/2277
env:
flink version: 1.11.3
hive version: 3.1.1
hadoop version: 3.1.2
The following is the hive table building statement ：
CREATE EXTERNAL TABLE sample2
STORED BY 'org.apache.iceberg.mr.hive.HiveIcebergStorageHandler'
LOCATION 'hdfs://xxx/usr/hive/warehouse/iceberg.db/sample1'
TBLPROPERTIES (
'iceberg.mr.catalog'='hadoop_catalog',
'iceberg.mr.catalog.hadoop.warehouse.location'='hdfs://xxx/usr/hive/warehouse')
;
error:
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: MetaException(message:org.apache.hadoop.hive.serde2.SerDeException Please provide an existing table or a valid schema)
help me!!!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] liudi1184 commented on issue #2277: hive create external table for iceberg have error.
Posted by GitBox <gi...@apache.org>.
liudi1184 commented on issue #2277:
URL: https://github.com/apache/iceberg/issues/2277#issuecomment-874444400
I also encountered this problem. Have you solved it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ubyyj commented on issue #2277: hive create external table for iceberg have error.
Posted by GitBox <gi...@apache.org>.
ubyyj commented on issue #2277:
URL: https://github.com/apache/iceberg/issues/2277#issuecomment-1022818167
same here. I already have an iceberg table store in s3, registered in a hive metastore. Now I want to access the table from another hive, so I wanted to create table on top of existing data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] allenzhg edited a comment on issue #2277: hive create external table for iceberg have error.
Posted by GitBox <gi...@apache.org>.
allenzhg edited a comment on issue #2277:
URL: https://github.com/apache/iceberg/issues/2277#issuecomment-787627901
> * case
1. hadoop_catalog does me created.
building statement is :
CREATE CATALOG hadoop_catalog WITH (
'type'='iceberg',
'catalog-type'='hadoop',
'warehouse'='hdfs://usr/hive/warehouse/',
'property-version'='1'
);
CREATE CATALOG hive_catalog WITH (
'type'='iceberg',
'catalog-type'='hive',
'uri'='thrift://172.24.xxx:xxx'
'clients'='20',
'property-version'='1',
'warehouse'='hdfs://usr/hive/warehouse/'
);
2. create same table for sample1 and have a same error.
create table statement in flink sql client:
USE CATALOG hive_catalog;
CREATE DATABASE iceberg_db;
CREATE TABLE sample1(
id BIGINT COMMENT 'unique id',
data STRING
);
thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on issue #2277: hive create external table for iceberg have error.
Posted by GitBox <gi...@apache.org>.
pvary commented on issue #2277:
URL: https://github.com/apache/iceberg/issues/2277#issuecomment-787907402
@allenzhg: Could you please explain what are you trying to archive here?
The table should be either in HiveCatalog, or in HadoopCatalog. The exact commands are described in https://iceberg.apache.org/hive/. If the documentation is not clear then please help us highlighting the issues, so we can fix them.
Thanks,
Peter
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on issue #2277: hive create external table for iceberg have error.
Posted by GitBox <gi...@apache.org>.
pvary commented on issue #2277:
URL: https://github.com/apache/iceberg/issues/2277#issuecomment-786639303
@allenzhg: What catalog do you try to do?
The valid values are:
- `hive` - HiveCatalog
- `hadoop` - HadoopCatalog
Where do you issue the table building statement? (Hive/Flink/Spark?)
Thanks,
Peter
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on issue #2277: hive create external table for iceberg have error.
Posted by GitBox <gi...@apache.org>.
openinx commented on issue #2277:
URL: https://github.com/apache/iceberg/issues/2277#issuecomment-788915005
@allenzhg , You mean you want to writing data by flink streaming job into apache iceberg table, then read it by using Hive sql query ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] allenzhg commented on issue #2277: hive create external table for iceberg have error.
Posted by GitBox <gi...@apache.org>.
allenzhg commented on issue #2277:
URL: https://github.com/apache/iceberg/issues/2277#issuecomment-787627901
> * case
1. hadoop_catalog does me created.
building statement is :
CREATE CATALOG hadoop_catalog WITH (
'type'='iceberg',
'catalog-type'='hadoop',
'warehouse'='hdfs://usr/hive/warehouse/',
'property-version'='1'
);
CREATE CATALOG hive_catalog WITH (
'type'='iceberg',
'catalog-type'='hive',
'uri'='thrift://172.24.xxx:xxx'
'clients'='20',
'property-version'='1',
'warehouse'='hdfs://usr/hive/warehouse/'
);
2. create same table for sample1 and have a same error.
create table statement in flink sql client:
USE CATALOG hive_catalog;
CREATE DATABASE iceberg_db;
CREATE TABLE sample1(
id BIGINT COMMENT 'unique id',
data STRING
);
thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] silvermissile removed a comment on issue #2277: hive create external table for iceberg have error.
Posted by GitBox <gi...@apache.org>.
silvermissile removed a comment on issue #2277:
URL: https://github.com/apache/iceberg/issues/2277#issuecomment-812299949
> @allenzhg , You mean you want to writing data by flink streaming job into apache iceberg table, then read it by using Hive sql query ?
yeah , I meet the same situation:
``` sql
CREATE EXTERNAL TABLE tmp.order_detail_sort
STORED BY 'org.apache.iceberg.mr.hive.HiveIcebergStorageHandler'
LOCATION 'hdfs:///data/new-ib/tmp/order_detail_sort' TBLPROPERTIES ( 'iceberg.mr.catalog'='hadoop',
'iceberg.mr.catalog.hadoop.warehouse.location'='hdfs:///data/new-ib') ;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on issue #2277: hive create external table for iceberg have error.
Posted by GitBox <gi...@apache.org>.
pvary commented on issue #2277:
URL: https://github.com/apache/iceberg/issues/2277#issuecomment-1022891138
@ubyyj: A few questions for future development:
- Is the 2nd HiveServer2/TezAM able to access the 1st HMS?
- Do you plan to change the tables from both Hive installations?
Thanks, Peter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] marton-bod commented on issue #2277: hive create external table for iceberg have error.
Posted by GitBox <gi...@apache.org>.
marton-bod commented on issue #2277:
URL: https://github.com/apache/iceberg/issues/2277#issuecomment-786749149
@allenzhg Did you create the Iceberg table from Flink and then try to create a Hive table on top of it?
You might want to reference this documentation: https://iceberg.apache.org/hive/, under the `Using Hadoop Catalog` section. A couple of things stand out:
- `'iceberg.mr.catalog'='hadoop_catalog'` is not a valid catalog value, as Peter pointed out. Can you retry with `'iceberg.mr.catalog'='hadoop'`?
- According to the docs, the table name you specify to the Hive DDL must match the table name of the Iceberg table. In this case, it seems that the Iceberg table is called `sample1`, whereas you are trying to create a Hive table called `sample2` on top of it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] silvermissile commented on issue #2277: hive create external table for iceberg have error.
Posted by GitBox <gi...@apache.org>.
silvermissile commented on issue #2277:
URL: https://github.com/apache/iceberg/issues/2277#issuecomment-812299949
> @allenzhg , You mean you want to writing data by flink streaming job into apache iceberg table, then read it by using Hive sql query ?
yeah , I meet the same situation:
``` sql
CREATE EXTERNAL TABLE tmp.order_detail_sort
STORED BY 'org.apache.iceberg.mr.hive.HiveIcebergStorageHandler'
LOCATION 'hdfs:///data/new-ib/tmp/order_detail_sort' TBLPROPERTIES ( 'iceberg.mr.catalog'='hadoop',
'iceberg.mr.catalog.hadoop.warehouse.location'='hdfs:///data/new-ib') ;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] allenzhg commented on issue #2277: hive create external table for iceberg have error.
Posted by GitBox <gi...@apache.org>.
allenzhg commented on issue #2277:
URL: https://github.com/apache/iceberg/issues/2277#issuecomment-787614495
> @allenzhg: What catalog do you try to do?
> The valid values are:
>
> * `hive` - HiveCatalog
> * `hadoop` - HadoopCatalog
>
> Where do you issue the table building statement? (Hive/Flink/Spark?)
>
> Thanks,
> Peter
the table building statement in hive. i wanted from hive read iceberg table content.
thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] liudi1184 commented on issue #2277: hive create external table for iceberg have error.
Posted by GitBox <gi...@apache.org>.
liudi1184 commented on issue #2277:
URL: https://github.com/apache/iceberg/issues/2277#issuecomment-874444400
I also encountered this problem. Have you solved it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org