You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "swapna267 (via GitHub)" <gi...@apache.org> on 2025/01/27 21:41:32 UTC
[PR] support create table like in flink catalog and watermark in windows [iceberg]
swapna267 opened a new pull request, #12116:
URL: https://github.com/apache/iceberg/pull/12116
This PR addresses
1. Creation of dynamic Iceberg table in Flink Catalog using the underlying physical Iceberg table using LIKE clause.
2. Iceberg Source to support Source Watermark, so it can be used in Flink WINDOW functions.
```
CREATE TABLE table_wm (
eventTS AS CAST(t1 AS TIMESTAMP(3)),
WATERMARK FOR eventTS AS SOURCE_WATERMARK()
) WITH (
'connector'='iceberg',
'watermark-column'='t1',
) LIKE iceberg_catalog.db.table;
```
**Reference:**
https://github.com/apache/iceberg/issues/10219
https://github.com/apache/iceberg/pull/9346
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] support create table like in flink catalog and watermark in windows [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12116:
URL: https://github.com/apache/iceberg/pull/12116#discussion_r1936870349
##########
flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/source/IcebergTableSource.java:
##########
@@ -53,7 +54,8 @@ public class IcebergTableSource
implements ScanTableSource,
SupportsProjectionPushDown,
SupportsFilterPushDown,
- SupportsLimitPushDown {
+ SupportsLimitPushDown,
+ SupportsSourceWatermark {
Review Comment:
Please separate out the features to 2 PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] support create table like in flink catalog and watermark in windows [iceberg]
Posted by "swapna267 (via GitHub)" <gi...@apache.org>.
swapna267 commented on code in PR #12116:
URL: https://github.com/apache/iceberg/pull/12116#discussion_r1934453514
##########
flink/v1.20/flink/src/test/java/org/apache/iceberg/flink/TestIcebergConnector.java:
##########
@@ -256,43 +256,6 @@ public void testCatalogDatabaseConflictWithFlinkDatabase() {
.hasMessageStartingWith("Could not execute CreateTable in path");
}
- @TestTemplate
- public void testConnectorTableInIcebergCatalog() {
- // Create the catalog properties
- Map<String, String> catalogProps = Maps.newHashMap();
Review Comment:
This is testing the check mentioned in https://github.com/apache/iceberg/pull/12116#discussion_r1933510999
Fail creating a table in Iceberg Catalog if connector=iceberg is specified in the Option. As the check is been deleted, i removed this test case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] support create table like in flink catalog and watermark in windows [iceberg]
Posted by "swapna267 (via GitHub)" <gi...@apache.org>.
swapna267 commented on code in PR #12116:
URL: https://github.com/apache/iceberg/pull/12116#discussion_r1934430629
##########
flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -384,13 +395,6 @@ public void renameTable(ObjectPath tablePath, String newTableName, boolean ignor
@Override
public void createTable(ObjectPath tablePath, CatalogBaseTable table, boolean ignoreIfExists)
throws CatalogException, TableAlreadyExistException {
- if (Objects.equals(
- table.getOptions().get("connector"), FlinkDynamicTableFactory.FACTORY_IDENTIFIER)) {
- throw new IllegalArgumentException(
- "Cannot create the table with 'connector'='iceberg' table property in "
- + "an iceberg catalog, Please create table with 'connector'='iceberg' property in a non-iceberg catalog or "
- + "create table without 'connector'='iceberg' related properties in an iceberg table.");
- }
Review Comment:
Tables can be created using LIKE in
1. Flink Catalog - Not supported currently.
2. Another table in Iceberg catalog itself as detailed in [doc](https://iceberg.apache.org/docs/nightly/flink-ddl/#create-table-like)
This check basically fails, if we try to create table using LIKE in Iceberg catalog, basically case#2 if we have **connector=iceberg** in options . For example, DDL like below,
```
CREATE TABLE `hive_catalog`.`default`.`sample_like`
LIKE `hive_catalog`.`default`.`sample`
WITH ('connector'='iceberg')
```
In order to support Case#1 without user setting any extra Options using WITH clause, we need to add connector in getTable, https://github.com/apache/iceberg/blob/52bfbdc0b03bafb610efde8486a6633be2b9cc4f/flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java#L344
This check was added in very old PR,
https://github.com/apache/iceberg/pull/2666
https://github.com/apache/iceberg/pull/2666#discussion_r660569612 where Flink SQL didn't support CREATE TABLE A LIKE B , where A and B are in different Catalogs.
So, in this case by removing this check, we are ignoring connector option being passed, so following DDL can create table table_like in Flink catalog backed by `iceberg_catalog.db.table`. As we know source table is an Iceberg table, adding connector=iceberg would be redundant.
```
CREATE TABLE table_like (
eventTS AS CAST(t1 AS TIMESTAMP(3)),
) LIKE iceberg_catalog.db.table;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] support create table like in flink catalog and watermark in windows [iceberg]
Posted by "swapna267 (via GitHub)" <gi...@apache.org>.
swapna267 commented on code in PR #12116:
URL: https://github.com/apache/iceberg/pull/12116#discussion_r1947338028
##########
flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -384,13 +395,6 @@ public void renameTable(ObjectPath tablePath, String newTableName, boolean ignor
@Override
public void createTable(ObjectPath tablePath, CatalogBaseTable table, boolean ignoreIfExists)
throws CatalogException, TableAlreadyExistException {
- if (Objects.equals(
- table.getOptions().get("connector"), FlinkDynamicTableFactory.FACTORY_IDENTIFIER)) {
- throw new IllegalArgumentException(
- "Cannot create the table with 'connector'='iceberg' table property in "
- + "an iceberg catalog, Please create table with 'connector'='iceberg' property in a non-iceberg catalog or "
- + "create table without 'connector'='iceberg' related properties in an iceberg table.");
- }
Review Comment:
With current changes in PR, there is one side-effect. In Case 2 & 3 mentioned above, the extra properties added in getTable, https://github.com/apache/iceberg/blob/52bfbdc0b03bafb610efde8486a6633be2b9cc4f/flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java#L339-L344 will be added as Table properties in destination table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] support create table like in flink catalog and watermark in windows [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #12116: support create table like in flink catalog and watermark in windows
URL: https://github.com/apache/iceberg/pull/12116
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] support create table like in flink catalog and watermark in windows [iceberg]
Posted by "swapna267 (via GitHub)" <gi...@apache.org>.
swapna267 commented on code in PR #12116:
URL: https://github.com/apache/iceberg/pull/12116#discussion_r1947338028
##########
flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -384,13 +395,6 @@ public void renameTable(ObjectPath tablePath, String newTableName, boolean ignor
@Override
public void createTable(ObjectPath tablePath, CatalogBaseTable table, boolean ignoreIfExists)
throws CatalogException, TableAlreadyExistException {
- if (Objects.equals(
- table.getOptions().get("connector"), FlinkDynamicTableFactory.FACTORY_IDENTIFIER)) {
- throw new IllegalArgumentException(
- "Cannot create the table with 'connector'='iceberg' table property in "
- + "an iceberg catalog, Please create table with 'connector'='iceberg' property in a non-iceberg catalog or "
- + "create table without 'connector'='iceberg' related properties in an iceberg table.");
- }
Review Comment:
With current changes in PR, there is one side-effect. In Case 2 & 3 mentioned above, the extra properties added in getTable, https://github.com/apache/iceberg/blob/52bfbdc0b03bafb610efde8486a6633be2b9cc4f/flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java#L339-L344 will be added as Table properties in destination table. Looking into nicer ways to avoid that. An option can be to add an extraProperty like say , createtabletype -> like in getTable , which can be used to differentiate the way table is being created.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] support create table like in flink catalog and watermark in windows [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12116:
URL: https://github.com/apache/iceberg/pull/12116#discussion_r1949104342
##########
flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -384,13 +395,6 @@ public void renameTable(ObjectPath tablePath, String newTableName, boolean ignor
@Override
public void createTable(ObjectPath tablePath, CatalogBaseTable table, boolean ignoreIfExists)
throws CatalogException, TableAlreadyExistException {
- if (Objects.equals(
- table.getOptions().get("connector"), FlinkDynamicTableFactory.FACTORY_IDENTIFIER)) {
- throw new IllegalArgumentException(
- "Cannot create the table with 'connector'='iceberg' table property in "
- + "an iceberg catalog, Please create table with 'connector'='iceberg' property in a non-iceberg catalog or "
- + "create table without 'connector'='iceberg' related properties in an iceberg table.");
- }
Review Comment:
I don't get this:
> Currently the Catalog information comes when Catalog is created.
Let's talk offline
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] support create table like in flink catalog and watermark in windows [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12116:
URL: https://github.com/apache/iceberg/pull/12116#issuecomment-2719403005
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] support create table like in flink catalog and watermark in windows [iceberg]
Posted by "swapna267 (via GitHub)" <gi...@apache.org>.
swapna267 commented on code in PR #12116:
URL: https://github.com/apache/iceberg/pull/12116#discussion_r1947338028
##########
flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -384,13 +395,6 @@ public void renameTable(ObjectPath tablePath, String newTableName, boolean ignor
@Override
public void createTable(ObjectPath tablePath, CatalogBaseTable table, boolean ignoreIfExists)
throws CatalogException, TableAlreadyExistException {
- if (Objects.equals(
- table.getOptions().get("connector"), FlinkDynamicTableFactory.FACTORY_IDENTIFIER)) {
- throw new IllegalArgumentException(
- "Cannot create the table with 'connector'='iceberg' table property in "
- + "an iceberg catalog, Please create table with 'connector'='iceberg' property in a non-iceberg catalog or "
- + "create table without 'connector'='iceberg' related properties in an iceberg table.");
- }
Review Comment:
With current changes in PR, there is one side-effect. In Case 2 & 3 mentioned above, the extra properties added in getTable, https://github.com/apache/iceberg/blob/52bfbdc0b03bafb610efde8486a6633be2b9cc4f/flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java#L339-L344 will be added as Table properties in destination table. Looking into nicer ways to avoid that. An option can be to add an extraProperty like say , createtabletype -> like in getTable , which can be used to differentiate the way table is being created. And extra props can be dropped in case 2&3 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] support create table like in flink catalog and watermark in windows [iceberg]
Posted by "swapna267 (via GitHub)" <gi...@apache.org>.
swapna267 commented on code in PR #12116:
URL: https://github.com/apache/iceberg/pull/12116#discussion_r1934451707
##########
flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/source/IcebergTableSource.java:
##########
@@ -53,7 +54,8 @@ public class IcebergTableSource
implements ScanTableSource,
SupportsProjectionPushDown,
SupportsFilterPushDown,
- SupportsLimitPushDown {
+ SupportsLimitPushDown,
+ SupportsSourceWatermark {
Review Comment:
These features were driven by mainly a use case, where an iceberg table is needed to be used in Flink window functions. This needs an incoming table to have MILLISECOND precision timestamp column and also watermark to be defined on source table.
As iceberg only supports MICROSECOND timestamp columns, we need to have a table with computed columns and we can create these only in Flink Catalog. Iceberg catalog doesn't support creating tables with computed columns.
i am happy to split them into 2 separate PR's .
I have tests for CREATE TABLE LIKE.
As Watermark support is just making Source to implement interface and falling back to https://github.com/apache/iceberg/pull/9346 for core logic, i didn't have a test case. I can add a validation on if `watermark-column` is configured or not , so it can fail fast. And a test case around that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] support create table like in flink catalog and watermark in windows [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12116:
URL: https://github.com/apache/iceberg/pull/12116#discussion_r1936869792
##########
flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -384,13 +395,6 @@ public void renameTable(ObjectPath tablePath, String newTableName, boolean ignor
@Override
public void createTable(ObjectPath tablePath, CatalogBaseTable table, boolean ignoreIfExists)
throws CatalogException, TableAlreadyExistException {
- if (Objects.equals(
- table.getOptions().get("connector"), FlinkDynamicTableFactory.FACTORY_IDENTIFIER)) {
- throw new IllegalArgumentException(
- "Cannot create the table with 'connector'='iceberg' table property in "
- + "an iceberg catalog, Please create table with 'connector'='iceberg' property in a non-iceberg catalog or "
- + "create table without 'connector'='iceberg' related properties in an iceberg table.");
- }
Review Comment:
What happens when the source table is not an Iceberg table?
I'm trying to understand here, where we get the missing information in this case, and wether we have a way to check that we actually get the missing information. If we can create such a check, then we can still throw an exception when we don't get this information from any source
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] support create table like in flink catalog and watermark in windows [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12116:
URL: https://github.com/apache/iceberg/pull/12116#discussion_r1936872361
##########
flink/v1.20/flink/src/test/java/org/apache/iceberg/flink/TestIcebergConnector.java:
##########
@@ -256,43 +256,6 @@ public void testCatalogDatabaseConflictWithFlinkDatabase() {
.hasMessageStartingWith("Could not execute CreateTable in path");
}
- @TestTemplate
- public void testConnectorTableInIcebergCatalog() {
- // Create the catalog properties
- Map<String, String> catalogProps = Maps.newHashMap();
Review Comment:
I think that this is still a valid check in most cases. Only not valid when the table is created with "CREATE TABLE.. LIKE" and only if the source table is an iceberg table.
Do I miss something?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] support create table like in flink catalog and watermark in windows [iceberg]
Posted by "swapna267 (via GitHub)" <gi...@apache.org>.
swapna267 commented on code in PR #12116:
URL: https://github.com/apache/iceberg/pull/12116#discussion_r1934430629
##########
flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -384,13 +395,6 @@ public void renameTable(ObjectPath tablePath, String newTableName, boolean ignor
@Override
public void createTable(ObjectPath tablePath, CatalogBaseTable table, boolean ignoreIfExists)
throws CatalogException, TableAlreadyExistException {
- if (Objects.equals(
- table.getOptions().get("connector"), FlinkDynamicTableFactory.FACTORY_IDENTIFIER)) {
- throw new IllegalArgumentException(
- "Cannot create the table with 'connector'='iceberg' table property in "
- + "an iceberg catalog, Please create table with 'connector'='iceberg' property in a non-iceberg catalog or "
- + "create table without 'connector'='iceberg' related properties in an iceberg table.");
- }
Review Comment:
Tables can be created using LIKE
1. Flink Catalog - Not supported currently.
2. Another table in Iceberg catalog itself as detailed in [doc](https://iceberg.apache.org/docs/nightly/flink-ddl/#create-table-like)
This check basically fails, if we try to create table using LIKE in Iceberg catalog, basically case#2 if we have **connector=iceberg** in options.
In order to support Case#1 without user setting any extra Options using WITH clause, we need to add connector in getTable, https://github.com/apache/iceberg/blob/52bfbdc0b03bafb610efde8486a6633be2b9cc4f/flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java#L344
This check was added in very old PR,
https://github.com/apache/iceberg/pull/2666
https://github.com/apache/iceberg/pull/2666#discussion_r660569612 where Flink SQL didn't support CREATE TABLE A LIKE B , where A and B are in different Catalogs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] support create table like in flink catalog and watermark in windows [iceberg]
Posted by "swapna267 (via GitHub)" <gi...@apache.org>.
swapna267 commented on code in PR #12116:
URL: https://github.com/apache/iceberg/pull/12116#discussion_r1947295585
##########
flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -384,13 +395,6 @@ public void renameTable(ObjectPath tablePath, String newTableName, boolean ignor
@Override
public void createTable(ObjectPath tablePath, CatalogBaseTable table, boolean ignoreIfExists)
throws CatalogException, TableAlreadyExistException {
- if (Objects.equals(
- table.getOptions().get("connector"), FlinkDynamicTableFactory.FACTORY_IDENTIFIER)) {
- throw new IllegalArgumentException(
- "Cannot create the table with 'connector'='iceberg' table property in "
- + "an iceberg catalog, Please create table with 'connector'='iceberg' property in a non-iceberg catalog or "
- + "create table without 'connector'='iceberg' related properties in an iceberg table.");
- }
Review Comment:
createTable in this catalog is called only when Source table is an Iceberg Table. Currently the Catalog information comes when Catalog is created.
Following are the scenarios when getTable / createTable methods in this Catalog are being used.
1. Create Iceberg table in Iceberg catalog -> Only createTable is called, where Catalog instance has all catalog related info.
2. Create table in iceberg catalog catalog1 like table in iceberg catalog catalog1 -> getTable() sets schema/partitioning info , which is used to create the table with same schema/partitioning as source in Catalog1.
3. Create table in iceberg catalog catalog2 like table in iceberg catalog catalog1 -> getTable() sets schema/partitioning info , which is used to create the table with same schema/partitioning as source in Catalog2.
4. Create table in Flink catalog like table in iceberg catalog -> getTable() is only called to get the source table info and createTable called on Flink catalog, where connector/iceberg catalog properties are being used to instantiate FlinkDynamicTableFactory.
When createTable is invoked, currently there is no easy way to differentiate between Case 2) / Case 3) / Case 4) Or user is doing
```
CREATE TABLE `hive_catalog`.`default`.`sample_like`
WITH ('connector'='iceberg', 'catalog-name'='')
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] support create table like in flink catalog and watermark in windows [iceberg]
Posted by "swapna267 (via GitHub)" <gi...@apache.org>.
swapna267 commented on code in PR #12116:
URL: https://github.com/apache/iceberg/pull/12116#discussion_r1934430629
##########
flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -384,13 +395,6 @@ public void renameTable(ObjectPath tablePath, String newTableName, boolean ignor
@Override
public void createTable(ObjectPath tablePath, CatalogBaseTable table, boolean ignoreIfExists)
throws CatalogException, TableAlreadyExistException {
- if (Objects.equals(
- table.getOptions().get("connector"), FlinkDynamicTableFactory.FACTORY_IDENTIFIER)) {
- throw new IllegalArgumentException(
- "Cannot create the table with 'connector'='iceberg' table property in "
- + "an iceberg catalog, Please create table with 'connector'='iceberg' property in a non-iceberg catalog or "
- + "create table without 'connector'='iceberg' related properties in an iceberg table.");
- }
Review Comment:
Tables can be created using LIKE in
1. Flink Catalog - Not supported currently.
2. Another table in Iceberg catalog itself as detailed in [doc](https://iceberg.apache.org/docs/nightly/flink-ddl/#create-table-like)
This check basically fails, if we try to create table using LIKE in Iceberg catalog, basically case#2 if we have **connector=iceberg** in options.
In order to support Case#1 without user setting any extra Options using WITH clause, we need to add connector in getTable, https://github.com/apache/iceberg/blob/52bfbdc0b03bafb610efde8486a6633be2b9cc4f/flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java#L344
This check was added in very old PR,
https://github.com/apache/iceberg/pull/2666
https://github.com/apache/iceberg/pull/2666#discussion_r660569612 where Flink SQL didn't support CREATE TABLE A LIKE B , where A and B are in different Catalogs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] support create table like in flink catalog and watermark in windows [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #12116:
URL: https://github.com/apache/iceberg/pull/12116#issuecomment-2619253637
@swapna267: Started the test runs so we can see the status of the PR.
Could you please remove the Flink 1.18/1.9 version part of the changes?
It makes the review easier if we concentrate only a single Flink version, and do the backport later when the changes to the main version are merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] support create table like in flink catalog and watermark in windows [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12116:
URL: https://github.com/apache/iceberg/pull/12116#issuecomment-2741932274
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] support create table like in flink catalog and watermark in windows [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12116:
URL: https://github.com/apache/iceberg/pull/12116#discussion_r1933513558
##########
flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/source/IcebergTableSource.java:
##########
@@ -53,7 +54,8 @@ public class IcebergTableSource
implements ScanTableSource,
SupportsProjectionPushDown,
SupportsFilterPushDown,
- SupportsLimitPushDown {
+ SupportsLimitPushDown,
+ SupportsSourceWatermark {
Review Comment:
I think we have 2 feature in a single PR:
- CREATE TABLE LIKE
- Watermark support
Could we separate out these features to different PRs?
Could we write tests for both features?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] support create table like in flink catalog and watermark in windows [iceberg]
Posted by "swapna267 (via GitHub)" <gi...@apache.org>.
swapna267 commented on PR #12116:
URL: https://github.com/apache/iceberg/pull/12116#issuecomment-2619775785
@pvary reverted 1.18/1.19 changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] support create table like in flink catalog and watermark in windows [iceberg]
Posted by "swapna267 (via GitHub)" <gi...@apache.org>.
swapna267 commented on code in PR #12116:
URL: https://github.com/apache/iceberg/pull/12116#discussion_r1934451707
##########
flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/source/IcebergTableSource.java:
##########
@@ -53,7 +54,8 @@ public class IcebergTableSource
implements ScanTableSource,
SupportsProjectionPushDown,
SupportsFilterPushDown,
- SupportsLimitPushDown {
+ SupportsLimitPushDown,
+ SupportsSourceWatermark {
Review Comment:
These features were driven by mainly a use case, where an iceberg table is needed to be used in Flink window functions. This needs an incoming table to have MILLISECOND precision timestamp column and also watermark to be defined on source table.
As iceberg only supports MICROSECOND timestamp columns, we need to have a table with computed columns and we can create these only in Flink Catalog. Iceberg catalog doesn't support creating tables with computed columns.
That said, i am happy to split them into 2 separate PR's.
I have tests for CREATE TABLE LIKE.
As Watermark support is just making Source to implement interface and falling back to https://github.com/apache/iceberg/pull/9346 for core logic, i didn't have a test case. I can add a validation on if `watermark-column` is configured or not , so it can fail fast. And a test case around that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] support create table like in flink catalog and watermark in windows [iceberg]
Posted by "swapna267 (via GitHub)" <gi...@apache.org>.
swapna267 commented on PR #12116:
URL: https://github.com/apache/iceberg/pull/12116#issuecomment-2620523105
**Background:**
1. Creation of dynamic Iceberg table in Flink Catalog using the underlying physical Iceberg table using LIKE clause.
Currently (without the changes in PR), create table in flink catalog works by configuring flink connector as described in,
[flink-connector](https://iceberg.apache.org/docs/nightly/flink-connector/#flink-connector )
But that needs user to provide the schema for the table. A way to tackle that is to do create table LIKE using below DDL.
```
CREATE TABLE table_wm (
eventTS AS CAST(t1 AS TIMESTAMP(3)),
WATERMARK FOR eventTS AS SOURCE_WATERMARK()
) WITH (
'connector'='iceberg',
'catalog-name'='iceberg_catalog',
'catalog-database'='testdb',
'catalog-table'='t'
) LIKE iceberg_catalog.testdb.t;
```
Options like connector, catalog-name, catalog-database, catalog-table need to be duplicated as Iceberg FlinkCatalog doesn't return any catalog related properties during getTable. This PR addresses the issue by including these properties when getTable is called , which will be used by Flink when creating table in Flink Catalog.
2. Iceberg Source to support Source Watermark
As raised in https://github.com/apache/iceberg/issues/10219, Source watermark implemented as part of https://iceberg.apache.org/docs/nightly/flink-queries/#emitting-watermarks cannot be used in Flink window functions.
[SupportSourceWatermark.java](https://github.com/apache/flink/blob/release-1.20/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/abilities/SupportsSourceWatermark.java) interface to push down watermark to source.
Here we are falling back to read options implemented in https://github.com/apache/iceberg/pull/9346 , to configure the watermark column on Iceberg Source.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] support create table like in flink catalog and watermark in windows [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12116:
URL: https://github.com/apache/iceberg/pull/12116#discussion_r1933510999
##########
flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -384,13 +395,6 @@ public void renameTable(ObjectPath tablePath, String newTableName, boolean ignor
@Override
public void createTable(ObjectPath tablePath, CatalogBaseTable table, boolean ignoreIfExists)
throws CatalogException, TableAlreadyExistException {
- if (Objects.equals(
- table.getOptions().get("connector"), FlinkDynamicTableFactory.FACTORY_IDENTIFIER)) {
- throw new IllegalArgumentException(
- "Cannot create the table with 'connector'='iceberg' table property in "
- + "an iceberg catalog, Please create table with 'connector'='iceberg' property in a non-iceberg catalog or "
- + "create table without 'connector'='iceberg' related properties in an iceberg table.");
- }
Review Comment:
Why do we remove this check?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] support create table like in flink catalog and watermark in windows [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12116:
URL: https://github.com/apache/iceberg/pull/12116#discussion_r1933514597
##########
flink/v1.20/flink/src/test/java/org/apache/iceberg/flink/TestIcebergConnector.java:
##########
@@ -256,43 +256,6 @@ public void testCatalogDatabaseConflictWithFlinkDatabase() {
.hasMessageStartingWith("Could not execute CreateTable in path");
}
- @TestTemplate
- public void testConnectorTableInIcebergCatalog() {
- // Create the catalog properties
- Map<String, String> catalogProps = Maps.newHashMap();
Review Comment:
Why is this test removed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org