You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "okayhooni (via GitHub)" <gi...@apache.org> on 2024/03/14 00:57:56 UTC
[I] Is there any way to define Iceberg catalog and share it between DataStream API and Table/SQL API? [iceberg]
okayhooni opened a new issue, #9954:
URL: https://github.com/apache/iceberg/issues/9954
### Query engine
Flink
### Question
I tried to implement Flink application(source: Iceberg, sink: Iceberg). and use both Flink's datastream API and Table API for our convention for structure of Flink apps.
For this purpose, I want to define Iceberg catalog and share it between DataStream API and Table/SQL API.
But, I didn't find the way to share catalog among them. (I couldn't find the way except defining same catalog twice for each Flink API)
Is there any way to define Iceberg catalog and share it between DataStream API and Table/SQL API?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Is there any way to define Iceberg catalog and share it between DataStream API and Table/SQL API? [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #9954:
URL: https://github.com/apache/iceberg/issues/9954#issuecomment-2461060235
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Is there any way to define Iceberg catalog and share it between DataStream API and Table/SQL API? [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #9954:
URL: https://github.com/apache/iceberg/issues/9954#issuecomment-2430545073
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Is there any way to define Iceberg catalog and share it between DataStream API and Table/SQL API? [iceberg]
Posted by "dat-vikash (via GitHub)" <gi...@apache.org>.
dat-vikash commented on issue #9954:
URL: https://github.com/apache/iceberg/issues/9954#issuecomment-2077707511
hitting this as well. Would be great to have a way to register the catalog to the table env!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Is there any way to define Iceberg catalog and share it between DataStream API and Table/SQL API? [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on issue #9954:
URL: https://github.com/apache/iceberg/issues/9954#issuecomment-1996601635
@okayhooni: I was struggling with the same issue in my recent tests, and ended up creating my own test extension which encapsulated the catalog(sql) and catalog loader (java) creation internally based on the same parameters.
Seems like a useful idea to investigate, but I do not have time for it in the near future. If you want to try yourself, I would be happy to review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Is there any way to define Iceberg catalog and share it between DataStream API and Table/SQL API? [iceberg]
Posted by "dyzcs (via GitHub)" <gi...@apache.org>.
dyzcs commented on issue #9954:
URL: https://github.com/apache/iceberg/issues/9954#issuecomment-1996961367
@okayhooni I haven't used AWS Glue before, but as you said, this should be due to interface incompatibility(issue). The above code is available in Spark and Flink, and using Hive and HDFS catalog is acceptable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Is there any way to define Iceberg catalog and share it between DataStream API and Table/SQL API? [iceberg]
Posted by "okayhooni (via GitHub)" <gi...@apache.org>.
okayhooni commented on issue #9954:
URL: https://github.com/apache/iceberg/issues/9954#issuecomment-1997471383
@dyzcs Thank you for additional kind explanation..!
If catalog can be shared between DataStream API and Table API as your code, How can I use that catalog on SQL interface from `StreamTableEnvironment` ?
`registerCatalog` method of `TableEnvironment` has catalog parameter typed as `org.apache.flink.table.catalog`..!
But the catalog loaded from `CatalogLoader.custom()` is typed as `org.apache.iceberg.catalog.Catalog`..! so it cannot be the argument of `registerCatalog` method of `TableEnvironment`..!
```
val glueCatalogLoader = CatalogLoader.custom(
"iceberg_catalog",
Map(
"type" -> "iceberg",
"warehouse" -> "s3://thank-you-dyzcx",
"catalog-impl" -> "org.apache.iceberg.aws.glue.GlueCatalog",
"io-impl" -> "org.apache.iceberg.aws.s3.S3FileIO"
).asJava,
new Configuration,
"org.apache.iceberg.aws.glue.GlueCatalog"
)
val glueCatalog = glueCatalogLoader.loadCatalog // it is typed as `org.apache.iceberg.catalog.Catalog`
// tableEnv.registerCatalog() ~ NOT WORKING WITH `org.apache.iceberg.catalog.Catalog`, it needs `org.apache.flink.table.catalog`
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Is there any way to define Iceberg catalog and share it between DataStream API and Table/SQL API? [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #9954: Is there any way to define Iceberg catalog and share it between DataStream API and Table/SQL API?
URL: https://github.com/apache/iceberg/issues/9954
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Is there any way to define Iceberg catalog and share it between DataStream API and Table/SQL API? [iceberg]
Posted by "dyzcs (via GitHub)" <gi...@apache.org>.
dyzcs commented on issue #9954:
URL: https://github.com/apache/iceberg/issues/9954#issuecomment-1996566044
```
//1. create TableLoader configuration
Configuration conf = new Configuration();
//2. create catalog and set table store path
Catalog catalog = new HiveCatalog(conf,"path");
//3. load table
TableIdentifier name = TableIdentifier.of("", "");
Table table = catalog.loadTable(name);
```
According to my understanding, the above code can solve your problem, and this issue can be closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Is there any way to define Iceberg catalog and share it between DataStream API and Table/SQL API? [iceberg]
Posted by "okayhooni (via GitHub)" <gi...@apache.org>.
okayhooni commented on issue #9954:
URL: https://github.com/apache/iceberg/issues/9954#issuecomment-1996709246
@dyzcs Thank you for kind reply! I used Glue catalog and tried several approaches similar to your code example.. But it looks Catalog classes between Flink and Iceberg are not compatible with different interface.
@pvary Thank you for sharing experience! (I am newbie to Flink.. but I'll try to investigate it after being more accustomed with Flink!)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org