You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/02/20 10:41:21 UTC
[GitHub] [iceberg] zhangjun0x01 opened a new pull request #2259: Flink : change FlinkCatalog#loadIcebergTable method to public
zhangjun0x01 opened a new pull request #2259:
URL: https://github.com/apache/iceberg/pull/2259
In some cases, I need to get the iceberg table to do some work, such as `RewriteDataFiles`, or modify table schema. When we use the iceberg table of `HiveCatalog`, we may need to construct the `CatalogLoader` first, then construct the `TableLoader`, and then load the iceberg table. The code is relatively complicated, I personally prefer to use sql as much as possible, so I want to change the `org.apache.iceberg.flink.FlinkCatalog#loadIcebergTable` method to public so that we can easily get the iceberg table through the following code .
```
StreamTableEnvironment tenv = StreamTableEnvironment.create(env);
tenv.executeSql(
"CREATE CATALOG iceberg WITH (\n"
+ " 'type'='iceberg',\n"
+ " 'catalog-type'='hive',"
+ " 'warehouse'='hdfs://localhost/user/hive2/warehouse',\n"
+ " 'uri'='thrift://localhost:9083'"
+ ")");
FlinkCatalog flinkCatalog = (FlinkCatalog) tenv.getCatalog("iceberg").get();
Table icebergTable = flinkCatalog.loadIcebergTable(new ObjectPath("iceberg_db", "iceberg_table"));
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] zhangjun0x01 commented on pull request #2259: Flink: Make FlinkCatalog#loadIcebergTable public
Posted by GitBox <gi...@apache.org>.
zhangjun0x01 commented on pull request #2259:
URL: https://github.com/apache/iceberg/pull/2259#issuecomment-785511922
Great,I think it is a good idea to add a `catalog()` method to exposing iceberg catalog,I updated it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #2259: Flink: Make FlinkCatalog#loadIcebergTable public
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #2259:
URL: https://github.com/apache/iceberg/pull/2259#issuecomment-783787014
@zhangjun0x01, is it possible to expose a method on the table loaded by the catalog to access the Iceberg table instead? That's what we do in Spark. `SparkTable` has a `table` method to access the underlying Iceberg table. It isn't very exposed because you have to cast the Spark `Table` interface to Iceberg's `SparkTable` to use it. I'd prefer a solution like that to adding a new public method to a public API.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] zhangjun0x01 commented on pull request #2259: Flink: Make FlinkCatalog#loadIcebergTable public
Posted by GitBox <gi...@apache.org>.
zhangjun0x01 commented on pull request #2259:
URL: https://github.com/apache/iceberg/pull/2259#issuecomment-783804212
> @zhangjun0x01, is it possible to expose a method on the table loaded by the catalog to access the Iceberg table instead? That's what we do in Spark. `SparkTable` has a `table` method to access the underlying Iceberg table. It isn't very exposed because you have to cast the Spark `Table` interface to Iceberg's `SparkTable` to use it. I'd prefer a solution like that to adding a new public method to a public API.
@rdblue Thanks for your suggestion, I will think about how to implement it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] zhangjun0x01 commented on pull request #2259: Flink: Make FlinkCatalog#loadIcebergTable public
Posted by GitBox <gi...@apache.org>.
zhangjun0x01 commented on pull request #2259:
URL: https://github.com/apache/iceberg/pull/2259#issuecomment-784937086
@rdblue I check the code , found the [comment](https://github.com/apache/iceberg/blob/master/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java#L540) , `We can not create a IcebergCatalogTable extends CatalogTable` ,so in flink, we can not create a `SparkTable` implements `org.apache.spark.sql.connector.catalog.Table` like spark
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #2259: Flink: add a catalog() method for FlinkCatalog to exposing iceberg catalog
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #2259:
URL: https://github.com/apache/iceberg/pull/2259#issuecomment-791836795
Looks good now. Thanks @zhangjun0x01!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] zhangjun0x01 edited a comment on pull request #2259: Flink: Make FlinkCatalog#loadIcebergTable public
Posted by GitBox <gi...@apache.org>.
zhangjun0x01 edited a comment on pull request #2259:
URL: https://github.com/apache/iceberg/pull/2259#issuecomment-784937086
@rdblue I check the code , found the [comment](https://github.com/apache/iceberg/blob/master/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java#L540) , `We can not create a IcebergCatalogTable extends CatalogTable` ,so in flink, if we want to get the iceberg table in `FlinkCatalog` more conveniently , I think we can only change `loadIcebergTable` method to `public `
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #2259: Flink: Make FlinkCatalog#loadIcebergTable public
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #2259:
URL: https://github.com/apache/iceberg/pull/2259#issuecomment-785242961
Okay, sounds like we can't go that direction then. What about using the Iceberg catalogs directly? You should be able to create one easily and then use that directly rather than using a Flink catalog to do the loading. Or what about exposing the underlying Iceberg catalog from the Flink catalog? So you'd load the Flink catalog, call `catalog()` to get the Iceberg catalog and then be able to load tables from there? The advantage of that approach is that you'd add one method but be able to do any catalog operation directly rather than just load a table.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue merged pull request #2259: Flink: add a catalog() method for FlinkCatalog to exposing iceberg catalog
Posted by GitBox <gi...@apache.org>.
rdblue merged pull request #2259:
URL: https://github.com/apache/iceberg/pull/2259
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] zhangjun0x01 closed pull request #2259: Flink : change FlinkCatalog#loadIcebergTable method to public
Posted by GitBox <gi...@apache.org>.
zhangjun0x01 closed pull request #2259:
URL: https://github.com/apache/iceberg/pull/2259
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org