You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/04/14 11:46:10 UTC
[GitHub] [iceberg] wuwenchi opened a new pull request, #4561: Flink: support modify table properties with primary key.
wuwenchi opened a new pull request, #4561:
URL: https://github.com/apache/iceberg/pull/4561
Because we have iceberg table with primary key, and we want to alter table properties.
This PR supports 'ALTER TABLE ... SET ...' and 'ALTER TABLE ... RESET ...' to modify a table properties which has primary key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4561: Flink: support modify table properties with primary key.
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4561:
URL: https://github.com/apache/iceberg/pull/4561#discussion_r851365889
##########
flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -400,6 +400,30 @@ void createIcebergTable(ObjectPath tablePath, CatalogBaseTable table, boolean ig
}
}
+ private static void validateTableSchemaAndPartition(CatalogTable ct1, CatalogTable ct2) {
+ TableSchema ts1 = ct1.getSchema();
+ TableSchema ts2 = ct2.getSchema();
+ boolean equalsPrimary = false;
+ if (ts1.getPrimaryKey().isPresent() && ts2.getPrimaryKey().isPresent()) {
+ equalsPrimary = ts1.getPrimaryKey().get().getType().equals(ts2.getPrimaryKey().get().getType()) &&
+ ts1.getPrimaryKey().get().getColumns().equals(ts2.getPrimaryKey().get().getColumns());
+ } else if (!ts1.getPrimaryKey().isPresent() && !ts2.getPrimaryKey().isPresent()) {
+ equalsPrimary = true;
+ }
+
+ // For current Flink Catalog API, support for adding/removing/renaming columns cannot be done by comparing
+ // CatalogTable instances, unless the Flink schema contains Iceberg column IDs.
+ if (!equalsPrimary &&
+ ts1.getTableColumns().equals(ts2.getTableColumns()) &&
+ ts1.getWatermarkSpecs().equals(ts2.getWatermarkSpecs())) {
Review Comment:
Please fix indentation. This should be 2 indents, not 4.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #4561: Flink: support modify table properties with primary key.
Posted by GitBox <gi...@apache.org>.
rdblue commented on PR #4561:
URL: https://github.com/apache/iceberg/pull/4561#issuecomment-1101819065
@openinx, the test looks identical to the [test](https://github.com/apache/iceberg/pull/3614/files#diff-2c8ac6c96a4608e31ba079ccb7cd04dcd50cd9dc57c1c81eeeb5bc75c151c275R322) in #3614 from a different author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] wuwenchi commented on pull request #4561: Flink: support modify table properties with primary key.
Posted by GitBox <gi...@apache.org>.
wuwenchi commented on PR #4561:
URL: https://github.com/apache/iceberg/pull/4561#issuecomment-1101922521
@rdblue Sorry, I didn't notice this pr #3614 @Initial-neko before...
I want to add iceberg support for watermark and computed columns, and save these informations to table properties.
I found this problem when I alter a table with a primary key.
In fact, I write the test according to original test (
https://github.com/apache/iceberg/blob/master/flink/v1.14/flink/src/test/java/org/apache/iceberg/flink/TestFlinkCatalogTable.java#L296-L318 ) which written by @openinx , and I only add `primary key` in DDL sql.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] wuwenchi commented on a diff in pull request #4561: Flink: support modify table properties with primary key.
Posted by GitBox <gi...@apache.org>.
wuwenchi commented on code in PR #4561:
URL: https://github.com/apache/iceberg/pull/4561#discussion_r851619369
##########
flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -400,6 +400,30 @@ void createIcebergTable(ObjectPath tablePath, CatalogBaseTable table, boolean ig
}
}
+ private static void validateTableSchemaAndPartition(CatalogTable ct1, CatalogTable ct2) {
+ TableSchema ts1 = ct1.getSchema();
+ TableSchema ts2 = ct2.getSchema();
+ boolean equalsPrimary = false;
+ if (ts1.getPrimaryKey().isPresent() && ts2.getPrimaryKey().isPresent()) {
+ equalsPrimary = ts1.getPrimaryKey().get().getType().equals(ts2.getPrimaryKey().get().getType()) &&
+ ts1.getPrimaryKey().get().getColumns().equals(ts2.getPrimaryKey().get().getColumns());
+ } else if (!ts1.getPrimaryKey().isPresent() && !ts2.getPrimaryKey().isPresent()) {
+ equalsPrimary = true;
+ }
+
+ // For current Flink Catalog API, support for adding/removing/renaming columns cannot be done by comparing
+ // CatalogTable instances, unless the Flink schema contains Iceberg column IDs.
+ if (!equalsPrimary &&
+ ts1.getTableColumns().equals(ts2.getTableColumns()) &&
+ ts1.getWatermarkSpecs().equals(ts2.getWatermarkSpecs())) {
Review Comment:
@rdblue Thanks for your review, and I will modify it with 4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] yittg commented on a diff in pull request #4561: Flink: support modify table properties with primary key.
Posted by GitBox <gi...@apache.org>.
yittg commented on code in PR #4561:
URL: https://github.com/apache/iceberg/pull/4561#discussion_r851867733
##########
flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -400,6 +400,30 @@ void createIcebergTable(ObjectPath tablePath, CatalogBaseTable table, boolean ig
}
}
+ private static void validateTableSchemaAndPartition(CatalogTable ct1, CatalogTable ct2) {
+ TableSchema ts1 = ct1.getSchema();
+ TableSchema ts2 = ct2.getSchema();
+ boolean equalsPrimary = false;
+ if (ts1.getPrimaryKey().isPresent() && ts2.getPrimaryKey().isPresent()) {
+ equalsPrimary = ts1.getPrimaryKey().get().getType().equals(ts2.getPrimaryKey().get().getType()) &&
+ ts1.getPrimaryKey().get().getColumns().equals(ts2.getPrimaryKey().get().getColumns());
+ } else if (!ts1.getPrimaryKey().isPresent() && !ts2.getPrimaryKey().isPresent()) {
+ equalsPrimary = true;
+ }
Review Comment:
how about wrapping a function `primaryKeyEqualsIgnoreName`
##########
flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -400,6 +400,30 @@ void createIcebergTable(ObjectPath tablePath, CatalogBaseTable table, boolean ig
}
}
+ private static void validateTableSchemaAndPartition(CatalogTable ct1, CatalogTable ct2) {
+ TableSchema ts1 = ct1.getSchema();
+ TableSchema ts2 = ct2.getSchema();
+ boolean equalsPrimary = false;
+ if (ts1.getPrimaryKey().isPresent() && ts2.getPrimaryKey().isPresent()) {
+ equalsPrimary = ts1.getPrimaryKey().get().getType().equals(ts2.getPrimaryKey().get().getType()) &&
+ ts1.getPrimaryKey().get().getColumns().equals(ts2.getPrimaryKey().get().getColumns());
+ } else if (!ts1.getPrimaryKey().isPresent() && !ts2.getPrimaryKey().isPresent()) {
+ equalsPrimary = true;
+ }
+
+ // For current Flink Catalog API, support for adding/removing/renaming columns cannot be done by comparing
+ // CatalogTable instances, unless the Flink schema contains Iceberg column IDs.
+ if (!equalsPrimary &&
+ ts1.getTableColumns().equals(ts2.getTableColumns()) &&
+ ts1.getWatermarkSpecs().equals(ts2.getWatermarkSpecs())) {
+ throw new UnsupportedOperationException("Altering schema is not supported yet.");
Review Comment:
And would you mind validating it by `Objects.equals` as flink does originally?
##########
flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -400,6 +400,30 @@ void createIcebergTable(ObjectPath tablePath, CatalogBaseTable table, boolean ig
}
}
+ private static void validateTableSchemaAndPartition(CatalogTable ct1, CatalogTable ct2) {
+ TableSchema ts1 = ct1.getSchema();
+ TableSchema ts2 = ct2.getSchema();
+ boolean equalsPrimary = false;
+ if (ts1.getPrimaryKey().isPresent() && ts2.getPrimaryKey().isPresent()) {
+ equalsPrimary = ts1.getPrimaryKey().get().getType().equals(ts2.getPrimaryKey().get().getType()) &&
+ ts1.getPrimaryKey().get().getColumns().equals(ts2.getPrimaryKey().get().getColumns());
+ } else if (!ts1.getPrimaryKey().isPresent() && !ts2.getPrimaryKey().isPresent()) {
+ equalsPrimary = true;
+ }
+
+ // For current Flink Catalog API, support for adding/removing/renaming columns cannot be done by comparing
+ // CatalogTable instances, unless the Flink schema contains Iceberg column IDs.
+ if (!equalsPrimary &&
+ ts1.getTableColumns().equals(ts2.getTableColumns()) &&
+ ts1.getWatermarkSpecs().equals(ts2.getWatermarkSpecs())) {
+ throw new UnsupportedOperationException("Altering schema is not supported yet.");
Review Comment:
it should be `if (!(equalsPrimary && columns && watermarks)) {}`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] yittg commented on a diff in pull request #4561: Flink: support modify table properties with primary key.
Posted by GitBox <gi...@apache.org>.
yittg commented on code in PR #4561:
URL: https://github.com/apache/iceberg/pull/4561#discussion_r851868552
##########
flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -400,6 +400,30 @@ void createIcebergTable(ObjectPath tablePath, CatalogBaseTable table, boolean ig
}
}
+ private static void validateTableSchemaAndPartition(CatalogTable ct1, CatalogTable ct2) {
+ TableSchema ts1 = ct1.getSchema();
+ TableSchema ts2 = ct2.getSchema();
+ boolean equalsPrimary = false;
+ if (ts1.getPrimaryKey().isPresent() && ts2.getPrimaryKey().isPresent()) {
+ equalsPrimary = ts1.getPrimaryKey().get().getType().equals(ts2.getPrimaryKey().get().getType()) &&
+ ts1.getPrimaryKey().get().getColumns().equals(ts2.getPrimaryKey().get().getColumns());
+ } else if (!ts1.getPrimaryKey().isPresent() && !ts2.getPrimaryKey().isPresent()) {
+ equalsPrimary = true;
+ }
+
+ // For current Flink Catalog API, support for adding/removing/renaming columns cannot be done by comparing
+ // CatalogTable instances, unless the Flink schema contains Iceberg column IDs.
+ if (!equalsPrimary &&
+ ts1.getTableColumns().equals(ts2.getTableColumns()) &&
+ ts1.getWatermarkSpecs().equals(ts2.getWatermarkSpecs())) {
+ throw new UnsupportedOperationException("Altering schema is not supported yet.");
Review Comment:
And would you mind validating it by `Objects.equals` in the same order as flink does originally?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on a diff in pull request #4561: Flink: support modify table properties with primary key.
Posted by GitBox <gi...@apache.org>.
openinx commented on code in PR #4561:
URL: https://github.com/apache/iceberg/pull/4561#discussion_r851938969
##########
flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -400,6 +400,30 @@ void createIcebergTable(ObjectPath tablePath, CatalogBaseTable table, boolean ig
}
}
+ private static void validateTableSchemaAndPartition(CatalogTable ct1, CatalogTable ct2) {
+ TableSchema ts1 = ct1.getSchema();
+ TableSchema ts2 = ct2.getSchema();
+ boolean equalsPrimary = false;
+ if (ts1.getPrimaryKey().isPresent() && ts2.getPrimaryKey().isPresent()) {
+ equalsPrimary = ts1.getPrimaryKey().get().getType().equals(ts2.getPrimaryKey().get().getType()) &&
+ ts1.getPrimaryKey().get().getColumns().equals(ts2.getPrimaryKey().get().getColumns());
+ } else if (!ts1.getPrimaryKey().isPresent() && !ts2.getPrimaryKey().isPresent()) {
+ equalsPrimary = true;
+ }
+
+ // For current Flink Catalog API, support for adding/removing/renaming columns cannot be done by comparing
+ // CatalogTable instances, unless the Flink schema contains Iceberg column IDs.
Review Comment:
We've added the duplicated comments in two places, right ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #4561: Flink: support modify table properties with primary key.
Posted by GitBox <gi...@apache.org>.
rdblue commented on PR #4561:
URL: https://github.com/apache/iceberg/pull/4561#issuecomment-1100216684
@wuwenchi, can you please credit the author of the unit test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4561: Flink: support modify table properties with primary key.
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4561:
URL: https://github.com/apache/iceberg/pull/4561#discussion_r851798556
##########
flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -400,6 +400,30 @@ void createIcebergTable(ObjectPath tablePath, CatalogBaseTable table, boolean ig
}
}
+ private static void validateTableSchemaAndPartition(CatalogTable ct1, CatalogTable ct2) {
+ TableSchema ts1 = ct1.getSchema();
+ TableSchema ts2 = ct2.getSchema();
+ boolean equalsPrimary = false;
+ if (ts1.getPrimaryKey().isPresent() && ts2.getPrimaryKey().isPresent()) {
+ equalsPrimary = ts1.getPrimaryKey().get().getType().equals(ts2.getPrimaryKey().get().getType()) &&
+ ts1.getPrimaryKey().get().getColumns().equals(ts2.getPrimaryKey().get().getColumns());
+ } else if (!ts1.getPrimaryKey().isPresent() && !ts2.getPrimaryKey().isPresent()) {
+ equalsPrimary = true;
+ }
+
+ // For current Flink Catalog API, support for adding/removing/renaming columns cannot be done by comparing
+ // CatalogTable instances, unless the Flink schema contains Iceberg column IDs.
+ if (!equalsPrimary &&
+ ts1.getTableColumns().equals(ts2.getTableColumns()) &&
+ ts1.getWatermarkSpecs().equals(ts2.getWatermarkSpecs())) {
Review Comment:
@wuwenchi, could you please fix the other cases where lines are over-indented?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] wuwenchi commented on a diff in pull request #4561: Flink: support modify table properties with primary key.
Posted by GitBox <gi...@apache.org>.
wuwenchi commented on code in PR #4561:
URL: https://github.com/apache/iceberg/pull/4561#discussion_r851619369
##########
flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -400,6 +400,30 @@ void createIcebergTable(ObjectPath tablePath, CatalogBaseTable table, boolean ig
}
}
+ private static void validateTableSchemaAndPartition(CatalogTable ct1, CatalogTable ct2) {
+ TableSchema ts1 = ct1.getSchema();
+ TableSchema ts2 = ct2.getSchema();
+ boolean equalsPrimary = false;
+ if (ts1.getPrimaryKey().isPresent() && ts2.getPrimaryKey().isPresent()) {
+ equalsPrimary = ts1.getPrimaryKey().get().getType().equals(ts2.getPrimaryKey().get().getType()) &&
+ ts1.getPrimaryKey().get().getColumns().equals(ts2.getPrimaryKey().get().getColumns());
+ } else if (!ts1.getPrimaryKey().isPresent() && !ts2.getPrimaryKey().isPresent()) {
+ equalsPrimary = true;
+ }
+
+ // For current Flink Catalog API, support for adding/removing/renaming columns cannot be done by comparing
+ // CatalogTable instances, unless the Flink schema contains Iceberg column IDs.
+ if (!equalsPrimary &&
+ ts1.getTableColumns().equals(ts2.getTableColumns()) &&
+ ts1.getWatermarkSpecs().equals(ts2.getWatermarkSpecs())) {
Review Comment:
@rdblue Thanks for your review, and I will modify it with 2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on a diff in pull request #4561: Flink: support modify table properties with primary key.
Posted by GitBox <gi...@apache.org>.
openinx commented on code in PR #4561:
URL: https://github.com/apache/iceberg/pull/4561#discussion_r851939460
##########
flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -400,6 +400,30 @@ void createIcebergTable(ObjectPath tablePath, CatalogBaseTable table, boolean ig
}
}
+ private static void validateTableSchemaAndPartition(CatalogTable ct1, CatalogTable ct2) {
+ TableSchema ts1 = ct1.getSchema();
+ TableSchema ts2 = ct2.getSchema();
+ boolean equalsPrimary = false;
+ if (ts1.getPrimaryKey().isPresent() && ts2.getPrimaryKey().isPresent()) {
+ equalsPrimary = ts1.getPrimaryKey().get().getType().equals(ts2.getPrimaryKey().get().getType()) &&
+ ts1.getPrimaryKey().get().getColumns().equals(ts2.getPrimaryKey().get().getColumns());
+ } else if (!ts1.getPrimaryKey().isPresent() && !ts2.getPrimaryKey().isPresent()) {
+ equalsPrimary = true;
+ }
+
+ // For current Flink Catalog API, support for adding/removing/renaming columns cannot be done by comparing
+ // CatalogTable instances, unless the Flink schema contains Iceberg column IDs.
Review Comment:
Let's just remove the duplicate comments for line447~448.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #4561: Flink: support modify table properties with primary key.
Posted by GitBox <gi...@apache.org>.
rdblue commented on PR #4561:
URL: https://github.com/apache/iceberg/pull/4561#issuecomment-1100941694
@wuwenchi, please credit the original author of the test before we can merge this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #4561: Flink: support modify table properties with primary key.
Posted by GitBox <gi...@apache.org>.
rdblue commented on PR #4561:
URL: https://github.com/apache/iceberg/pull/4561#issuecomment-1102818720
@wuwenchi, thanks for clarifying how the test was written! Since this was mostly a copy of an existing test then I can see why it looks so similar to the one in the other PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] wuwenchi commented on pull request #4561: Flink: support modify table properties with primary key.
Posted by GitBox <gi...@apache.org>.
wuwenchi commented on PR #4561:
URL: https://github.com/apache/iceberg/pull/4561#issuecomment-1100647100
> @wuwenchi, can you please credit the author of the unit test?
@openinx Could you please review this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on pull request #4561: Flink: support modify table properties with primary key.
Posted by GitBox <gi...@apache.org>.
openinx commented on PR #4561:
URL: https://github.com/apache/iceberg/pull/4561#issuecomment-1101179019
> @wuwenchi, please credit the original author of the test before we can merge this.
@rdblue , this unit test was added by @wuwenchi for addressing the table properties changes for *primary key* tables, in theory the original author of it is @wuwenchi . So I think we don't need to credit others, right ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] wuwenchi commented on a diff in pull request #4561: Flink: support modify table properties with primary key.
Posted by GitBox <gi...@apache.org>.
wuwenchi commented on code in PR #4561:
URL: https://github.com/apache/iceberg/pull/4561#discussion_r851985358
##########
flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -400,6 +400,30 @@ void createIcebergTable(ObjectPath tablePath, CatalogBaseTable table, boolean ig
}
}
+ private static void validateTableSchemaAndPartition(CatalogTable ct1, CatalogTable ct2) {
+ TableSchema ts1 = ct1.getSchema();
+ TableSchema ts2 = ct2.getSchema();
+ boolean equalsPrimary = false;
+ if (ts1.getPrimaryKey().isPresent() && ts2.getPrimaryKey().isPresent()) {
+ equalsPrimary = ts1.getPrimaryKey().get().getType().equals(ts2.getPrimaryKey().get().getType()) &&
+ ts1.getPrimaryKey().get().getColumns().equals(ts2.getPrimaryKey().get().getColumns());
+ } else if (!ts1.getPrimaryKey().isPresent() && !ts2.getPrimaryKey().isPresent()) {
+ equalsPrimary = true;
+ }
Review Comment:
@openinx
Primarykey is a `UniqueConstraint` , so if use `boolean equalsPrimary = Objects.equals(ts1.getPrimaryKey(), ts2.getPrimaryKey());`, it will call `super.equals(o)`. Then
in `AbstractConstraint` 's equals function, it will compare `name` properity which generated by `TableSchema`'s `primaryKey`:
```
public Builder primaryKey(String... columns) {
return primaryKey(UUID.randomUUID().toString(), columns);
}
```
So, we should not compare the name attribute, and I only compare the type and columns from primarykeys.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on a diff in pull request #4561: Flink: support modify table properties with primary key.
Posted by GitBox <gi...@apache.org>.
openinx commented on code in PR #4561:
URL: https://github.com/apache/iceberg/pull/4561#discussion_r851995001
##########
flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -400,6 +400,30 @@ void createIcebergTable(ObjectPath tablePath, CatalogBaseTable table, boolean ig
}
}
+ private static void validateTableSchemaAndPartition(CatalogTable ct1, CatalogTable ct2) {
+ TableSchema ts1 = ct1.getSchema();
+ TableSchema ts2 = ct2.getSchema();
+ boolean equalsPrimary = false;
+ if (ts1.getPrimaryKey().isPresent() && ts2.getPrimaryKey().isPresent()) {
+ equalsPrimary = ts1.getPrimaryKey().get().getType().equals(ts2.getPrimaryKey().get().getType()) &&
+ ts1.getPrimaryKey().get().getColumns().equals(ts2.getPrimaryKey().get().getColumns());
+ } else if (!ts1.getPrimaryKey().isPresent() && !ts2.getPrimaryKey().isPresent()) {
+ equalsPrimary = true;
+ }
Review Comment:
Okay, that sounds reasonable.
Then I think we can use `Objects.equals` to check the equality for both `type` and `columns` fields.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on a diff in pull request #4561: Flink: support modify table properties with primary key.
Posted by GitBox <gi...@apache.org>.
openinx commented on code in PR #4561:
URL: https://github.com/apache/iceberg/pull/4561#discussion_r851938557
##########
flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -400,6 +400,30 @@ void createIcebergTable(ObjectPath tablePath, CatalogBaseTable table, boolean ig
}
}
+ private static void validateTableSchemaAndPartition(CatalogTable ct1, CatalogTable ct2) {
+ TableSchema ts1 = ct1.getSchema();
+ TableSchema ts2 = ct2.getSchema();
+ boolean equalsPrimary = false;
+ if (ts1.getPrimaryKey().isPresent() && ts2.getPrimaryKey().isPresent()) {
+ equalsPrimary = ts1.getPrimaryKey().get().getType().equals(ts2.getPrimaryKey().get().getType()) &&
+ ts1.getPrimaryKey().get().getColumns().equals(ts2.getPrimaryKey().get().getColumns());
+ } else if (!ts1.getPrimaryKey().isPresent() && !ts2.getPrimaryKey().isPresent()) {
+ equalsPrimary = true;
+ }
Review Comment:
Why not just use the one line sentence to check the equality ?
```
boolean equalsPrimary = Objects.equals(ts1.getPrimaryKey(), ts2.getPrimaryKey());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] wuwenchi commented on a diff in pull request #4561: Flink: support modify table properties with primary key.
Posted by GitBox <gi...@apache.org>.
wuwenchi commented on code in PR #4561:
URL: https://github.com/apache/iceberg/pull/4561#discussion_r851986421
##########
flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -400,6 +400,30 @@ void createIcebergTable(ObjectPath tablePath, CatalogBaseTable table, boolean ig
}
}
+ private static void validateTableSchemaAndPartition(CatalogTable ct1, CatalogTable ct2) {
+ TableSchema ts1 = ct1.getSchema();
+ TableSchema ts2 = ct2.getSchema();
+ boolean equalsPrimary = false;
+ if (ts1.getPrimaryKey().isPresent() && ts2.getPrimaryKey().isPresent()) {
+ equalsPrimary = ts1.getPrimaryKey().get().getType().equals(ts2.getPrimaryKey().get().getType()) &&
+ ts1.getPrimaryKey().get().getColumns().equals(ts2.getPrimaryKey().get().getColumns());
+ } else if (!ts1.getPrimaryKey().isPresent() && !ts2.getPrimaryKey().isPresent()) {
+ equalsPrimary = true;
+ }
+
+ // For current Flink Catalog API, support for adding/removing/renaming columns cannot be done by comparing
+ // CatalogTable instances, unless the Flink schema contains Iceberg column IDs.
+ if (!equalsPrimary &&
+ ts1.getTableColumns().equals(ts2.getTableColumns()) &&
+ ts1.getWatermarkSpecs().equals(ts2.getWatermarkSpecs())) {
+ throw new UnsupportedOperationException("Altering schema is not supported yet.");
Review Comment:
@yittg yes, I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] yittg commented on a diff in pull request #4561: Flink: support modify table properties with primary key.
Posted by GitBox <gi...@apache.org>.
yittg commented on code in PR #4561:
URL: https://github.com/apache/iceberg/pull/4561#discussion_r852531000
##########
flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -400,6 +400,30 @@ void createIcebergTable(ObjectPath tablePath, CatalogBaseTable table, boolean ig
}
}
+ private static void validateTableSchemaAndPartition(CatalogTable ct1, CatalogTable ct2) {
+ TableSchema ts1 = ct1.getSchema();
+ TableSchema ts2 = ct2.getSchema();
+ boolean equalsPrimary = false;
+ if (ts1.getPrimaryKey().isPresent() && ts2.getPrimaryKey().isPresent()) {
+ equalsPrimary = ts1.getPrimaryKey().get().getType().equals(ts2.getPrimaryKey().get().getType()) &&
+ ts1.getPrimaryKey().get().getColumns().equals(ts2.getPrimaryKey().get().getColumns());
+ } else if (!ts1.getPrimaryKey().isPresent() && !ts2.getPrimaryKey().isPresent()) {
+ equalsPrimary = true;
+ }
+
+ // For current Flink Catalog API, support for adding/removing/renaming columns cannot be done by comparing
+ // CatalogTable instances, unless the Flink schema contains Iceberg column IDs.
+ if (!equalsPrimary &&
+ ts1.getTableColumns().equals(ts2.getTableColumns()) &&
+ ts1.getWatermarkSpecs().equals(ts2.getWatermarkSpecs())) {
+ throw new UnsupportedOperationException("Altering schema is not supported yet.");
Review Comment:
@wuwenchi Would you mind adding some test cases for validating the expected exception?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on pull request #4561: Flink: support modify table properties with primary key.
Posted by GitBox <gi...@apache.org>.
openinx commented on PR #4561:
URL: https://github.com/apache/iceberg/pull/4561#issuecomment-1102578232
I think it's reasonable to add the new unit test based on the original unit tests. So I'm okay to skip the credit for the old PR. Thanks all for the reviewing !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx merged pull request #4561: Flink: support modify table properties with primary key.
Posted by GitBox <gi...@apache.org>.
openinx merged PR #4561:
URL: https://github.com/apache/iceberg/pull/4561
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org