You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/06/27 07:09:17 UTC
[GitHub] [doris] myfjdthink opened a new issue, #10452: [Feature] Doris support read iceberg table on google cloud storage
myfjdthink opened a new issue, #10452:
URL: https://github.com/apache/doris/issues/10452
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.
### Description
I have deployed a hive metastore service on dataproc, and it stores files on gcs with a storage path similar to `gs://iceberg-spark-warehouse/datasets.
`
I'm running into some problems
- doris does not have permission to read gcs files.
- doris does not recognize the gs protocol.
I see that trino(presto) [solves the problem](https://trino.io/docs/current/connector/hive-gcs-tutorial.html) like this
hope doris can add similar support
Thanks
### Use case
_No response_
### Related issues
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] myfjdthink commented on issue #10452: [Feature] Doris support read iceberg table on google cloud storage
Posted by "myfjdthink (via GitHub)" <gi...@apache.org>.
myfjdthink commented on issue #10452:
URL: https://github.com/apache/doris/issues/10452#issuecomment-1503032968
Update:
我在 1.2.2 版本上继续尝试，按如下方式配置 catalog
```sql
CREATE CATALOG iceberg PROPERTIES (
"type"="iceberg",
"warehouse" = "gs://gcs-bucket-xxx/hive-warehouse",
"iceberg.catalog.type"="hms",
"hive.metastore.uris" = "thrift://10.106.199.213:9083",
"AWS_ACCESS_KEY" = "xxx",
"AWS_SECRET_KEY" = "xxx",
"AWS_ENDPOINT" = "https://storage.googleapis.com/",
"AWS_REGION" = "us-east4",
"use_path_style" = "true"
);
```
配置后，可以访问 Hive metastore，尝试读取 Table List，一切正常
```sql
SWITCH iceberg
show databases;
```
正常
```sql
use public;
show tables;
```
正常
尝试读取 Table 里的数据，碰到问题
1. org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "gs"
添加 gcs-connector-hadoop3-latest.jar 依赖包后解决
2. Unknown file gs location for hms table
修改 fe/fe-core/src/main/java/org/apache/doris/planner/external/iceberg/IcebergScanProvider.java 里的判断逻辑，支持 GCS 可以解决
3. BE 无法识别 gs file
> errCode = 2, detailMessage = failed to init reader for file gs://gcs-bucket-xxx/hive-warehouse/public.db/table/data/xxx_bucket=6/20230411_075903_15214_h7ekb-1c62bdb7-6982-412d-8114-aeb0da5a6647.orc, err: s3 uri is invalid: gs://gcs-bucket-xxx/hive-warehouse/public.db/xxx-7efe259cb5274eeabfb6d6ba803f1843/data/xxx_bucket=6/20230411_075903_15214_h7ekb-1c62bdb7-6982-412d-8114-aeb0da5a6647.orc
搜索报错关键字，找不到报错的代码位置，Block 在这里了，
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] stalary commented on issue #10452: [Feature] Doris support read iceberg table on google cloud storage
Posted by GitBox <gi...@apache.org>.
stalary commented on issue #10452:
URL: https://github.com/apache/doris/issues/10452#issuecomment-1170050817
This feature is not developed, you can submit a pr~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] myfjdthink commented on issue #10452: [Feature] Doris support read iceberg table on google cloud storage
Posted by GitBox <gi...@apache.org>.
myfjdthink commented on issue #10452:
URL: https://github.com/apache/doris/issues/10452#issuecomment-1168365071
In addition to authorization issues, doirs are also unable to read data stored on gcs.
Take a look at this example
I wrote iceberg table on spark and the data is stored on gcs
and then read it in the hive environment, it works fine
`hive> select * from gs_table2;`
```
Query ID = nick_20220628041137_6fe75ec9-bae7-40b2-8d96-f6653fcdfb49
Total jobs = 1
Launching Job 1 out of 1
Status: Running (Executing on YARN cluster with App id application_1656302766976_0013)
----------------------------------------------------------------------------------------------
VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
----------------------------------------------------------------------------------------------
Map 1 .......... container SUCCEEDED 1 1 0 0 0 0
----------------------------------------------------------------------------------------------
VERTICES: 01/01 [==========================>>] 100% ELAPSED TIME: 8.34 s
----------------------------------------------------------------------------------------------
OK
1 a
2 b
3 c
4 a
1 a
2 b
3 c
4 a
Time taken: 9.569 seconds, Fetched: 8 row(s)
```
`hive> describe formatted gs_table2;`
```
OK
# col_name data_type comment
id int from deserializer
data string from deserializer
# Detailed Table Information
Database: gsdb
OwnerType: USER
Owner: nick
CreateTime: Tue Jun 28 02:34:36 UTC 2022
LastAccessTime: Sun Dec 14 22:38:21 UTC 1969
Retention: 2147483647
Location: gs://iceberg-spark/datasets/gsdb.db/gs_table2
Table Type: EXTERNAL_TABLE
Table Parameters:
EXTERNAL TRUE
metadata_location gs://iceberg-spark/datasets/gsdb.db/gs_table2/metadata/00002-833dafb8-cda5-4c7c-a2ea-04e2c84fa372.metadata.json
numFiles 8
numRows 8
owner nick
previous_metadata_location gs://iceberg-spark/datasets/gsdb.db/gs_table2/metadata/00001-0c51975f-0d0a-4c2e-a5d1-aad3251c0393.metadata.json
storage_handler org.apache.iceberg.mr.hive.HiveIcebergStorageHandler
table_type ICEBERG
totalSize 4952
transient_lastDdlTime 1656383676
uuid 86645efc-8a03-47f0-a397-1ec30877b1bb
# Storage Information
SerDe Library: org.apache.iceberg.mr.hive.HiveIcebergSerDe
InputFormat: org.apache.iceberg.mr.hive.HiveIcebergInputFormat
OutputFormat: org.apache.iceberg.mr.hive.HiveIcebergOutputFormat
Compressed: No
Num Buckets: 0
Bucket Columns: []
Sort Columns: []
Time taken: 0.206 seconds, Fetched: 34 row(s)
```
Then I try to create the iceberg table in doris
```sql
CREATE TABLE `gs_table2`
ENGINE = ICEBERG
PROPERTIES (
"iceberg.database" = "gsdb",
"iceberg.table" = "gs_table2",
"iceberg.hive.metastore.uris" = "thrift://10.201.0.104:9083",
"iceberg.catalog.type" = "HIVE_CATALOG"
);
```
sql execution timeout, table creation failed
Let's look at another example
I wrote the iceberg table on spark, the data is stored in hdfs
`hive> select * from test_table;`
```
Query ID = nick_20220628042917_7209ca76-1704-45e1-8678-88af4297b64a
Total jobs = 1
Launching Job 1 out of 1
Tez session was closed. Reopening...
Session re-established.
Session re-established.
Status: Running (Executing on YARN cluster with App id application_1656302766976_0014)
----------------------------------------------------------------------------------------------
VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
----------------------------------------------------------------------------------------------
Map 1 .......... container SUCCEEDED 1 1 0 0 0 0
----------------------------------------------------------------------------------------------
VERTICES: 01/01 [==========================>>] 100% ELAPSED TIME: 7.40 s
----------------------------------------------------------------------------------------------
OK
1 a
2 b
3 c
Time taken: 14.78 seconds, Fetched: 3 row(s)
```
`hive> describe formatted test_table;`
```
OK
# col_name data_type comment
id bigint from deserializer
data string from deserializer
# Detailed Table Information
Database: testdb
OwnerType: USER
Owner: nick
CreateTime: Thu Jun 23 11:11:56 UTC 2022
LastAccessTime: Wed Dec 10 07:15:41 UTC 1969
Retention: 2147483647
Location: hdfs://10.201.0.104:8020/user/hive/warehouse/testdb.db/test_table
Table Type: EXTERNAL_TABLE
Table Parameters:
EXTERNAL TRUE
metadata_location hdfs://10.201.0.104:8020/user/hive/warehouse/testdb.db/test_table/metadata/00001-ceb6024b-3d7d-4304-8fff-f2aad293d2cf.metadata.json
numFiles 3
numRows 3
owner nick
previous_metadata_location hdfs://10.201.0.104:8020/user/hive/warehouse/testdb.db/test_table/metadata/00000-f95f2c72-1e64-421b-a4b9-bccb77984d32.metadata.json
storage_handler org.apache.iceberg.mr.hive.HiveIcebergStorageHandler
table_type ICEBERG
totalSize 1929
transient_lastDdlTime 1655982716
uuid b3a2408e-bc56-4486-bd81-65b2be22b2f3
# Storage Information
SerDe Library: org.apache.iceberg.mr.hive.HiveIcebergSerDe
InputFormat: org.apache.iceberg.mr.hive.HiveIcebergInputFormat
OutputFormat: org.apache.iceberg.mr.hive.HiveIcebergOutputFormat
Compressed: No
Num Buckets: 0
Bucket Columns: []
Sort Columns: []
Time taken: 0.144 seconds, Fetched: 34 row(s)
```
and then try to create iceberg table in doris, created successfully and successfully read the data
```sql
CREATE TABLE `test_table`
ENGINE = ICEBERG
PROPERTIES (
"iceberg.database" = "testdb",
"iceberg.table" = "test_table",
"iceberg.hive.metastore.uris" = "thrift://10.201.0.104:9083",
"iceberg.catalog.type" = "HIVE_CATALOG"
);
select * from iceberg_db.test_table;
```
query result
data|id|
----+--+
a | 1|
b | 2|
c | 3|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] myfjdthink commented on issue #10452: [Feature] Doris support read iceberg table on google cloud storage
Posted by GitBox <gi...@apache.org>.
myfjdthink commented on issue #10452:
URL: https://github.com/apache/doris/issues/10452#issuecomment-1177395118
I try to fix this issue
this is the [changed files](https://github.com/apache/doris/compare/1.1.0-preview2...myfjdthink:feat-support-iceberg-on-gcs?expand=1)
Iceberg 0.12 doesn't support gcs yet, so I upgraded Iceberg to 0.13.2 to support gcs
With this, I was able to create tables
```sql
CREATE TABLE `gs_table`
ENGINE = ICEBERG
PROPERTIES (
"iceberg.database" = "gsdb",
"iceberg.table" = "gs_table2",
"iceberg.hive.metastore.uris" = "thrift://10.201.0.104:9083",
"iceberg.catalog.type" = "HIVE_CATALOG"
);
```
But I got an error when reading the table
```sql
select * from iceberg_db.gs_table;
```
Error message
```
errCode = 2, detailMessage = connect failed. gs://iceberg-spark-xxx
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] myfjdthink commented on issue #10452: [Feature] Doris support read iceberg table on google cloud storage
Posted by GitBox <gi...@apache.org>.
myfjdthink commented on issue #10452:
URL: https://github.com/apache/doris/issues/10452#issuecomment-1176093319
this is the error logs
```
2022-07-06 10:58:25,295 WARN (doris-mysql-nio-pool-3736|10326) [StmtExecutor.handleDdlStmt():1543] DDL statement(/* ApplicationName=DBeaver 22.1.1 - SQLEditor <Script-43.sql> */ CREATE TABLE `gs_table`
ENGINE = ICEBERG
PROPERTIES (
"iceberg.database" = "gsdb",
"iceberg.table" = "gs_table",
"iceberg.hive.metastore.uris" = "thrift://10.201.0.104:9083",
"iceberg.catalog.type" = "HIVE_CATALOG"
)) process failed.
org.apache.iceberg.exceptions.RuntimeIOException: Failed to get file system for path: gs://iceberg-spark-warehouse/datasets/gsdb.db/gs_table/metadata/00001-8ee13c7f-f1ad-4ca1-b88f-1d06a0f4899b.metadata.json
at org.apache.iceberg.hadoop.Util.getFs(Util.java:53) ~[iceberg-core-0.12.0.jar:?]
at org.apache.iceberg.hadoop.HadoopInputFile.fromLocation(HadoopInputFile.java:54) ~[iceberg-core-0.12.0.jar:?]
at org.apache.iceberg.hadoop.HadoopFileIO.newInputFile(HadoopFileIO.java:59) ~[iceberg-core-0.12.0.jar:?]
at org.apache.iceberg.TableMetadataParser.read(TableMetadataParser.java:252) ~[iceberg-core-0.12.0.jar:?]
at org.apache.iceberg.BaseMetastoreTableOperations.lambda$refreshFromMetadataLocation$0(BaseMetastoreTableOperations.java:179) ~[iceberg-core-0.12.0.jar:?]
at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:405) ~[iceberg-core-0.12.0.jar:?]
at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:214) ~[iceberg-core-0.12.0.jar:?]
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:198) ~[iceberg-core-0.12.0.jar:?]
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:190) ~[iceberg-core-0.12.0.jar:?]
at org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:178) ~[iceberg-core-0.12.0.jar:?]
at org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:160) ~[iceberg-core-0.12.0.jar:?]
at org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:200) ~[iceberg-hive-metastore-0.12.0.jar:?]
at org.apache.iceberg.BaseMetastoreTableOperations.refresh(BaseMetastoreTableOperations.java:94) ~[iceberg-core-0.12.0.jar:?]
at org.apache.iceberg.BaseMetastoreTableOperations.current(BaseMetastoreTableOperations.java:77) ~[iceberg-core-0.12.0.jar:?]
at org.apache.iceberg.BaseMetastoreCatalog.loadTable(BaseMetastoreCatalog.java:93) ~[iceberg-core-0.12.0.jar:?]
at org.apache.iceberg.catalog.Catalog.tableExists(Catalog.java:270) ~[iceberg-api-0.12.0.jar:?]
at org.apache.doris.external.iceberg.HiveCatalog.tableExists(HiveCatalog.java:57) ~[palo-fe.jar:0.15-SNAPSHOT]
at org.apache.doris.external.iceberg.IcebergCatalogMgr.getTableFromIceberg(IcebergCatalogMgr.java:151) ~[palo-fe.jar:0.15-SNAPSHOT]
at org.apache.doris.external.iceberg.IcebergCatalogMgr.createIcebergTable(IcebergCatalogMgr.java:201) ~[palo-fe.jar:0.15-SNAPSHOT]
at org.apache.doris.catalog.Catalog.createTable(Catalog.java:3059) ~[palo-fe.jar:0.15-SNAPSHOT]
at org.apache.doris.qe.DdlExecutor.execute(DdlExecutor.java:130) ~[palo-fe.jar:0.15-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.handleDdlStmt(StmtExecutor.java:1533) ~[palo-fe.jar:0.15-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:438) ~[palo-fe.jar:0.15-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:308) ~[palo-fe.jar:0.15-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:212) ~[palo-fe.jar:0.15-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.dispatch(ConnectProcessor.java:349) ~[palo-fe.jar:0.15-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.processOnce(ConnectProcessor.java:538) ~[palo-fe.jar:0.15-SNAPSHOT]
at org.apache.doris.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:50) ~[palo-fe.jar:0.15-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
Caused by: java.io.IOException: No FileSystem for scheme: gs
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2798) ~[hadoop-common-2.8.0.jar:?]
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2809) ~[hadoop-common-2.8.0.jar:?]
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100) ~[hadoop-common-2.8.0.jar:?]
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2848) ~[hadoop-common-2.8.0.jar:?]
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2830) ~[hadoop-common-2.8.0.jar:?]
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389) ~[hadoop-common-2.8.0.jar:?]
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:356) ~[hadoop-common-2.8.0.jar:?]
at org.apache.iceberg.hadoop.Util.getFs(Util.java:51) ~[iceberg-core-0.12.0.jar:?]
... 30 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] chenlinzhong closed issue #10452: [Feature] Doris support read iceberg table on google cloud storage
Posted by "chenlinzhong (via GitHub)" <gi...@apache.org>.
chenlinzhong closed issue #10452: [Feature] Doris support read iceberg table on google cloud storage
URL: https://github.com/apache/doris/issues/10452
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org