You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2023/01/03 12:23:27 UTC
[GitHub] [flink] fsk119 opened a new pull request, #21592: [FLINK-30497][table-api] Introduce TableChange to represent DROP change
fsk119 opened a new pull request, #21592:
URL: https://github.com/apache/flink/pull/21592
<!--
*Thank you very much for contributing to Apache Flink - we are happy that you want to help us improve Flink. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
*Please understand that we do not do this to make contributions to Flink a hassle. In order to uphold a high standard of quality for code contributions, while at the same time managing a large number of contributions, we need contributors to prepare the contributions well, and give reviewers enough contextual information for the review. Please also understand that contributions that do not follow this guide will take longer to review and thus typically be picked up with lower priority by the community.*
## Contribution Checklist
- Make sure that the pull request corresponds to a [JIRA issue](https://issues.apache.org/jira/projects/FLINK/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no JIRA issue.
- Name the pull request in the form "[FLINK-XXXX] [component] Title of the pull request", where *FLINK-XXXX* should be replaced by the actual issue number. Skip *component* if you are unsure about which is the best component.
Typo fixes that have no associated JIRA issue should be named following this pattern: `[hotfix] [docs] Fix typo in event time introduction` or `[hotfix] [javadocs] Expand JavaDoc for PuncuatedWatermarkGenerator`.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes. You can set up Azure Pipelines CI to do that following [this guide](https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines#AzurePipelines-Tutorial:SettingupAzurePipelinesforaforkoftheFlinkrepository).
- Each pull request should address only one issue, not mix up code from multiple issues.
- Each commit in the pull request has a meaningful commit message (including the JIRA id)
- Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
**(The sections below can be removed for hotfixes of typos)**
-->
## What is the purpose of the change
*Introduce TableChange to represent DROP syntax.*
## Verifying this change
This change is already covered by existing tests, such as *SqlToOperationConverterTest*.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / **no**)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (**yes** / no)
- The serializers: (yes / **no** / don't know)
- The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know)
- The S3 file system connector: (yes / **no** / don't know)
## Documentation
- Does this pull request introduce a new feature? (**yes** / no)
- If yes, how is the feature documented? (not applicable / docs / **JavaDocs** / not documented)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fsk119 merged pull request #21592: [FLINK-30497][table-api] Introduce TableChange to represent DROP change
Posted by GitBox <gi...@apache.org>.
fsk119 merged PR #21592:
URL: https://github.com/apache/flink/pull/21592
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fsk119 commented on a diff in pull request #21592: [FLINK-30497][table-api] Introduce TableChange to represent DROP change
Posted by GitBox <gi...@apache.org>.
fsk119 commented on code in PR #21592:
URL: https://github.com/apache/flink/pull/21592#discussion_r1061102346
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/expressions/ColumnReferenceFinder.java:
##########
@@ -81,7 +82,11 @@ public static Set<String> findReferencedColumn(String columnName, ResolvedSchema
public static Set<String> findWatermarkReferencedColumn(ResolvedSchema schema) {
ColumnReferenceVisitor visitor = new ColumnReferenceVisitor(schema.getColumnNames());
return schema.getWatermarkSpecs().stream()
- .flatMap(spec -> visitor.visit(spec.getWatermarkExpression()).stream())
+ .flatMap(
+ spec ->
+ Stream.concat(
+ visitor.visit(spec.getWatermarkExpression()).stream(),
+ Stream.of(spec.getRowtimeAttribute())))
Review Comment:
For example, you can specify `WATERMARK FOR rowtime AS col - INTERVAL 5 SECOND`. I add a case in the `ColumnReferencesFinderTest`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] LadyForest commented on a diff in pull request #21592: [FLINK-30497][table-api] Introduce TableChange to represent DROP change
Posted by GitBox <gi...@apache.org>.
LadyForest commented on code in PR #21592:
URL: https://github.com/apache/flink/pull/21592#discussion_r1061167593
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/operations/SqlToOperationConverter.java:
##########
@@ -237,7 +236,7 @@ public class SqlToOperationConverter {
private final FlinkPlannerImpl flinkPlanner;
private final CatalogManager catalogManager;
private final SqlCreateTableConverter createTableConverter;
- private final AlterSchemaConverter alterSchemaConverter;
+ private final AlterSchemaConverter alterTableConverter;
Review Comment:
Nit: Align the variable name with the class name?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #21592: [FLINK-30497][table-api] Introduce TableChange to represent DROP change
Posted by GitBox <gi...@apache.org>.
flinkbot commented on PR #21592:
URL: https://github.com/apache/flink/pull/21592#issuecomment-1369711148
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0d6342200c04c0ab3db1a283ecb3410d05597df2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0d6342200c04c0ab3db1a283ecb3410d05597df2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0d6342200c04c0ab3db1a283ecb3410d05597df2 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] LadyForest commented on a diff in pull request #21592: [FLINK-30497][table-api] Introduce TableChange to represent DROP change
Posted by GitBox <gi...@apache.org>.
LadyForest commented on code in PR #21592:
URL: https://github.com/apache/flink/pull/21592#discussion_r1060581278
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/catalog/TableChange.java:
##########
@@ -229,6 +229,53 @@ static ModifyWatermark modify(WatermarkSpec newWatermarkSpec) {
return new ModifyWatermark(newWatermarkSpec);
}
+ /**
+ * A table change to drop column.
+ *
+ * <p>It is equal to the following statement:
+ *
+ * <pre>
+ * ALTER TABLE <table_name> DROP COLUMN <column_name>
+ * </pre>
+ *
+ * @param columnName the column to drop.
+ * @return a TableChange represents the modification.
+ */
+ static DropColumn dropColumn(String columnName) {
+ return new DropColumn(columnName);
+ }
+
+ /**
+ * A table change to drop watermark.
+ *
+ * <p>It is equal to the following statement:
+ *
+ * <pre>
+ * ALTER TABLE <table_name> DROP WATERMARK
+ * </pre>
+ *
+ * @return a TableChange represents the modification.
+ */
+ static DropWatermark dropWatermark() {
+ return DropWatermark.INSTANCE;
+ }
+
+ /**
+ * A table change to drop constraint.
+ *
+ * <p>It is equal to the following statement:
+ *
+ * <pre>
+ * ALTER TABLE <table_name> DROP CONSTRAINT <constraint_name>
+ * </pre>
+ *
+ * @param constraintName the constraint to drop.
+ * @return a TableChange represents the modification.
+ */
+ static DropUniqueConstraint dropConstraint(String constraintName) {
Review Comment:
AFAIK Flink does not support unique constraints, thus`UniqueConstraint` sounds slightly confusing. What about `DropConstraint`?
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/expressions/ColumnReferenceFinder.java:
##########
@@ -81,7 +82,11 @@ public static Set<String> findReferencedColumn(String columnName, ResolvedSchema
public static Set<String> findWatermarkReferencedColumn(ResolvedSchema schema) {
ColumnReferenceVisitor visitor = new ColumnReferenceVisitor(schema.getColumnNames());
return schema.getWatermarkSpecs().stream()
- .flatMap(spec -> visitor.visit(spec.getWatermarkExpression()).stream())
+ .flatMap(
+ spec ->
+ Stream.concat(
+ visitor.visit(spec.getWatermarkExpression()).stream(),
+ Stream.of(spec.getRowtimeAttribute())))
Review Comment:
I'm curious under what condition the watermark expression does not contain the rowtime field?
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/utils/OperationConverterUtils.java:
##########
@@ -133,6 +132,14 @@ public static Operation convertAddReplaceColumns(
newProperties,
catalogTable.getComment());
if (addReplaceColumns.isReplace()) {
+ // It's hard to determine how to decompose the ALTER TABLE REPLACE into multiple
+ // TableChanges. For example, with old schema <a INT, b INT, c INT> and the new schema
+ // <a INT, d INT>, there are multiple choices:
+ // plan 1: DROP COLUMN c, RENAME COLUMN b TO d;
+ // plan 2: DROP COLUMN b, RENAME COLUMN c TO d;
+ // So we don't translate with TableChanges here. One solution to solve this problem is
+ // the minimum edit distance, which tries to minimize the modification times, but it
+ // also can not give a unique answer.
Review Comment:
How about
```java
// It's hard to define how to convert the ALTER TABLE REPLACE into multiple
// TableChanges. For example, with old schema <a INT, b INT, c INT> and the new schema
// <a INT, d INT>, there are multiple alternatives:
// plan 1: DROP COLUMN c, RENAME COLUMN b TO d;
// plan 2: DROP COLUMN b, RENAME COLUMN c TO d;
// So we don't translate with TableChanges here. One workaround is
// to minimize the edit distance, i.e., minimize the modification times, but it
// still cannot provide a deterministic answer.
```
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/operations/AlterSchemaConverter.java:
##########
@@ -728,114 +724,166 @@ String getComment(SqlTableColumn column) {
}
}
- // --------------------------------------------------------------------------------------------
+ private static class ReferencesManager {
+
+ /** Available columns in the table. */
+ private final Set<String> columns;
+
+ /**
+ * Mappings about the column refers which columns, e.g. column `b` refers to the column `a`
+ * in the expression "b as a+1".
+ */
+ private final Map<String, Set<String>> columnToReferences;
+
+ /**
+ * Reverse mappings about the column refers which columns, e.g. column `a` has the
+ * dependency of column `b` in the expression "b as a+1".
+ */
+ private final Map<String, Set<String>> columnToDependencies;
+
+ /** Primary keys defined on the table. */
+ private final Set<String> primaryKeys;
+
+ /** The name of the column watermark expression depends on. */
+ private final Set<String> watermarkReferences;
+
+ /** The name of the column partition keys contains. */
+ private final Set<String> partitionKeys;
+
+ private ReferencesManager(
+ Set<String> columns,
+ Map<String, Set<String>> columnToReferences,
+ Map<String, Set<String>> columnToDependencies,
+ Set<String> primaryKeys,
+ Set<String> watermarkReferences,
+ Set<String> partitionKeys) {
+ this.columns = columns;
+ this.columnToReferences = columnToReferences;
+ this.columnToDependencies = columnToDependencies;
+ this.primaryKeys = primaryKeys;
+ this.watermarkReferences = watermarkReferences;
+ this.partitionKeys = partitionKeys;
+ }
+
+ static ReferencesManager create(ResolvedCatalogTable catalogTable) {
+ Map<String, Set<String>> columnToReferences = new HashMap<>();
+ Map<String, Set<String>> columnToDependencies = new HashMap<>();
+ catalogTable.getResolvedSchema().getColumns().stream()
+ .filter(column -> column instanceof Column.ComputedColumn)
+ .forEach(
+ column -> {
+ Set<String> referencedColumns =
+ ColumnReferenceFinder.findReferencedColumn(
+ column.getName(), catalogTable.getResolvedSchema());
+ for (String referencedColumn : referencedColumns) {
+ columnToReferences
+ .computeIfAbsent(
+ referencedColumn, key -> new HashSet<>())
+ .add(column.getName());
+ columnToDependencies
+ .computeIfAbsent(
+ column.getName(), key -> new HashSet<>())
+ .add(referencedColumn);
+ }
+ });
- private void validateColumnName(
- String oldColumnName,
- String newColumnName,
- ResolvedSchema oldSchema,
- List<String> partitionKeys) {
- validateColumnName(
- oldColumnName,
- oldSchema,
- partitionKeys,
- // fail the operation of renaming column, once the column derives a computed column
- (referencedColumn, computedColumn) -> referencedColumn.contains(oldColumnName));
- // validate new column
- if (oldSchema.getColumn(newColumnName).isPresent()) {
- throw new ValidationException(
- String.format(
- "%sThe column `%s` already existed in table schema.",
- EX_MSG_PREFIX, newColumnName));
+ return new ReferencesManager(
+ new HashSet<>(catalogTable.getResolvedSchema().getColumnNames()),
+ columnToReferences,
+ columnToDependencies,
+ catalogTable
+ .getResolvedSchema()
+ .getPrimaryKey()
+ .map(constraint -> new HashSet<>(constraint.getColumns()))
+ .orElse(new HashSet<>()),
+ ColumnReferenceFinder.findWatermarkReferencedColumn(
+ catalogTable.getResolvedSchema()),
+ new HashSet<>(catalogTable.getPartitionKeys()));
+ }
+
+ void dropColumn(String columnName) {
+ checkReferences(columnName);
+ if (primaryKeys.contains(columnName)) {
+ throw new ValidationException(
+ String.format(
+ "%sThe column %s is used as the primary key.",
+ EX_MSG_PREFIX, EncodingUtils.escapeIdentifier(columnName)));
+ }
+
+ columnToDependencies
+ .getOrDefault(columnName, Collections.emptySet())
+ .forEach(
+ referredColumn ->
+ columnToReferences.get(referredColumn).remove(columnName));
+ columnToDependencies.remove(columnName);
+ columns.remove(columnName);
}
- }
- private void validateColumnName(
- String columnToDrop,
- ResolvedSchema oldSchema,
- List<String> partitionKeys,
- Set<String> columnsToDrop) {
- validateColumnName(
- columnToDrop,
- oldSchema,
- partitionKeys,
- // fail the operation of dropping column, only if the column derives a computed
- // column, and the computed column is not being dropped along with the old column
- (referencedColumn, computedColumn) ->
- referencedColumn.contains(columnToDrop)
- && !columnsToDrop.contains(computedColumn.getName()));
- oldSchema
- .getPrimaryKey()
- .ifPresent(
- pk -> {
- if (pk.getColumns().contains(columnToDrop)) {
- throw new ValidationException(
- String.format(
- "%sThe column `%s` is used as the primary key.",
- EX_MSG_PREFIX, columnToDrop));
- }
- });
- }
+ void renameColumn(String columnName, String newName) {
+ checkReferences(columnName);
+ if (columns.contains(newName)) {
+ throw new ValidationException(
+ String.format(
+ "%sThe column `%s` already existed in table schema.",
+ EX_MSG_PREFIX, newName));
+ }
- private void validateColumnName(
- String columnToAlter,
- ResolvedSchema oldSchema,
- List<String> partitionKeys,
- BiFunction<Set<String>, Column.ComputedColumn, Boolean> computedColumnChecker) {
- // validate old column
- Set<String> tableColumns = new HashSet<>(oldSchema.getColumnNames());
- if (!tableColumns.contains(columnToAlter)) {
- throw new ValidationException(
- String.format(
- "%sThe column `%s` does not exist in the base table.",
- EX_MSG_PREFIX, columnToAlter));
+ columnToDependencies
+ .getOrDefault(columnName, Collections.emptySet())
+ .forEach(
+ referredColumn -> {
+ columnToReferences.get(referredColumn).remove(columnName);
+ columnToReferences.get(referredColumn).add(newName);
+ });
+ columnToDependencies.put(newName, columnToDependencies.remove(columnName));
+
+ columns.remove(columnName);
+ columns.add(newName);
+
+ primaryKeys.remove(columnName);
+ primaryKeys.add(newName);
}
- // validate old column name isn't referred by computed column case
- oldSchema.getColumns().stream()
- .filter(column -> column instanceof Column.ComputedColumn)
- .forEach(
- column -> {
- Column.ComputedColumn computedColumn = (Column.ComputedColumn) column;
- Set<String> referencedColumn =
- ColumnReferenceFinder.findReferencedColumn(
- computedColumn.getName(), oldSchema);
- if (computedColumnChecker.apply(referencedColumn, computedColumn)) {
- throw new ValidationException(
- String.format(
- "%sThe column `%s` is referenced by computed column %s.",
- EX_MSG_PREFIX,
- columnToAlter,
- computedColumn.asSummaryString()));
- }
- });
- // validate partition keys doesn't contain the old column
- if (partitionKeys.contains(columnToAlter)) {
- throw new ValidationException(
- String.format(
- "%sThe column `%s` is used as the partition keys.",
- EX_MSG_PREFIX, columnToAlter));
+ Set<String> lookupColumnDependencies(String columnName) {
+ return columnToDependencies.getOrDefault(columnName, Collections.emptySet());
}
Review Comment:
Nit: what about
```suggestion
int getColumnDependencyCount(String columnName) {
return columnToDependencies.getOrDefault(columnName, Collections.emptySet()).size();
}
```
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/operations/AlterSchemaConverter.java:
##########
@@ -728,114 +724,166 @@ String getComment(SqlTableColumn column) {
}
}
- // --------------------------------------------------------------------------------------------
+ private static class ReferencesManager {
+
+ /** Available columns in the table. */
+ private final Set<String> columns;
+
+ /**
+ * Mappings about the column refers which columns, e.g. column `b` refers to the column `a`
+ * in the expression "b as a+1".
+ */
+ private final Map<String, Set<String>> columnToReferences;
+
+ /**
+ * Reverse mappings about the column refers which columns, e.g. column `a` has the
+ * dependency of column `b` in the expression "b as a+1".
+ */
+ private final Map<String, Set<String>> columnToDependencies;
+
+ /** Primary keys defined on the table. */
+ private final Set<String> primaryKeys;
+
+ /** The name of the column watermark expression depends on. */
+ private final Set<String> watermarkReferences;
+
+ /** The name of the column partition keys contains. */
+ private final Set<String> partitionKeys;
+
+ private ReferencesManager(
+ Set<String> columns,
+ Map<String, Set<String>> columnToReferences,
+ Map<String, Set<String>> columnToDependencies,
+ Set<String> primaryKeys,
+ Set<String> watermarkReferences,
+ Set<String> partitionKeys) {
+ this.columns = columns;
+ this.columnToReferences = columnToReferences;
+ this.columnToDependencies = columnToDependencies;
+ this.primaryKeys = primaryKeys;
+ this.watermarkReferences = watermarkReferences;
+ this.partitionKeys = partitionKeys;
+ }
+
+ static ReferencesManager create(ResolvedCatalogTable catalogTable) {
+ Map<String, Set<String>> columnToReferences = new HashMap<>();
+ Map<String, Set<String>> columnToDependencies = new HashMap<>();
+ catalogTable.getResolvedSchema().getColumns().stream()
+ .filter(column -> column instanceof Column.ComputedColumn)
+ .forEach(
+ column -> {
+ Set<String> referencedColumns =
+ ColumnReferenceFinder.findReferencedColumn(
+ column.getName(), catalogTable.getResolvedSchema());
+ for (String referencedColumn : referencedColumns) {
+ columnToReferences
+ .computeIfAbsent(
+ referencedColumn, key -> new HashSet<>())
+ .add(column.getName());
+ columnToDependencies
+ .computeIfAbsent(
+ column.getName(), key -> new HashSet<>())
+ .add(referencedColumn);
+ }
+ });
- private void validateColumnName(
- String oldColumnName,
- String newColumnName,
- ResolvedSchema oldSchema,
- List<String> partitionKeys) {
- validateColumnName(
- oldColumnName,
- oldSchema,
- partitionKeys,
- // fail the operation of renaming column, once the column derives a computed column
- (referencedColumn, computedColumn) -> referencedColumn.contains(oldColumnName));
- // validate new column
- if (oldSchema.getColumn(newColumnName).isPresent()) {
- throw new ValidationException(
- String.format(
- "%sThe column `%s` already existed in table schema.",
- EX_MSG_PREFIX, newColumnName));
+ return new ReferencesManager(
+ new HashSet<>(catalogTable.getResolvedSchema().getColumnNames()),
+ columnToReferences,
+ columnToDependencies,
+ catalogTable
+ .getResolvedSchema()
+ .getPrimaryKey()
+ .map(constraint -> new HashSet<>(constraint.getColumns()))
+ .orElse(new HashSet<>()),
+ ColumnReferenceFinder.findWatermarkReferencedColumn(
+ catalogTable.getResolvedSchema()),
+ new HashSet<>(catalogTable.getPartitionKeys()));
+ }
+
+ void dropColumn(String columnName) {
+ checkReferences(columnName);
+ if (primaryKeys.contains(columnName)) {
+ throw new ValidationException(
+ String.format(
+ "%sThe column %s is used as the primary key.",
+ EX_MSG_PREFIX, EncodingUtils.escapeIdentifier(columnName)));
+ }
+
+ columnToDependencies
+ .getOrDefault(columnName, Collections.emptySet())
+ .forEach(
+ referredColumn ->
+ columnToReferences.get(referredColumn).remove(columnName));
+ columnToDependencies.remove(columnName);
+ columns.remove(columnName);
}
- }
- private void validateColumnName(
- String columnToDrop,
- ResolvedSchema oldSchema,
- List<String> partitionKeys,
- Set<String> columnsToDrop) {
- validateColumnName(
- columnToDrop,
- oldSchema,
- partitionKeys,
- // fail the operation of dropping column, only if the column derives a computed
- // column, and the computed column is not being dropped along with the old column
- (referencedColumn, computedColumn) ->
- referencedColumn.contains(columnToDrop)
- && !columnsToDrop.contains(computedColumn.getName()));
- oldSchema
- .getPrimaryKey()
- .ifPresent(
- pk -> {
- if (pk.getColumns().contains(columnToDrop)) {
- throw new ValidationException(
- String.format(
- "%sThe column `%s` is used as the primary key.",
- EX_MSG_PREFIX, columnToDrop));
- }
- });
- }
+ void renameColumn(String columnName, String newName) {
Review Comment:
Nit: `columnName` -> `oldColumnName`?
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/operations/SqlToOperationConverterTest.java:
##########
@@ -1507,19 +1507,15 @@ public void testFailedToAlterTableDropWatermark() throws Exception {
public void testAlterTableDropWatermark() throws Exception {
prepareNonManagedTable("tb1", true);
Operation operation = parse("alter table tb1 drop watermark");
- assertThat(operation).isInstanceOf(AlterTableSchemaOperation.class);
+ assertThat(operation).isInstanceOf(AlterTableChangeOperation.class);
assertThat(operation.asSummaryString())
- .isEqualTo(
- "ALTER TABLE cat1.db1.tb1 SET SCHEMA (\n"
- + " `a` INT NOT NULL,\n"
- + " `b` BIGINT NOT NULL,\n"
- + " `c` STRING NOT NULL COMMENT 'column comment',\n"
- + " `d` AS [a*(b+2 + a*b)],\n"
- + " `e` ROW<`f0` STRING, `f1` INT, `f2` ROW<`f0` DOUBLE, `f1` ARRAY<FLOAT>>>,\n"
- + " `f` AS [e.f1 + e.f2.f0],\n"
- + " `g` METADATA VIRTUAL,\n"
- + " `ts` TIMESTAMP(3) COMMENT 'just a comment'\n"
- + ")");
+ .isEqualTo("ALTER TABLE cat1.db1.tb1\n DROP WATERMARK");
+ assertThat(
+ ((AlterTableChangeOperation) operation)
+ .getNewTable()
+ .getUnresolvedSchema()
+ .getWatermarkSpecs())
+ .isEqualTo(Collections.emptyList());
Review Comment:
```suggestion
.isEmpty();
```
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/operations/AlterSchemaConverter.java:
##########
@@ -728,114 +724,166 @@ String getComment(SqlTableColumn column) {
}
}
- // --------------------------------------------------------------------------------------------
+ private static class ReferencesManager {
+
+ /** Available columns in the table. */
+ private final Set<String> columns;
+
+ /**
+ * Mappings about the column refers which columns, e.g. column `b` refers to the column `a`
+ * in the expression "b as a+1".
+ */
+ private final Map<String, Set<String>> columnToReferences;
+
+ /**
+ * Reverse mappings about the column refers which columns, e.g. column `a` has the
+ * dependency of column `b` in the expression "b as a+1".
+ */
+ private final Map<String, Set<String>> columnToDependencies;
+
+ /** Primary keys defined on the table. */
+ private final Set<String> primaryKeys;
+
+ /** The name of the column watermark expression depends on. */
+ private final Set<String> watermarkReferences;
+
+ /** The name of the column partition keys contains. */
+ private final Set<String> partitionKeys;
+
+ private ReferencesManager(
+ Set<String> columns,
+ Map<String, Set<String>> columnToReferences,
+ Map<String, Set<String>> columnToDependencies,
+ Set<String> primaryKeys,
+ Set<String> watermarkReferences,
+ Set<String> partitionKeys) {
+ this.columns = columns;
+ this.columnToReferences = columnToReferences;
+ this.columnToDependencies = columnToDependencies;
+ this.primaryKeys = primaryKeys;
+ this.watermarkReferences = watermarkReferences;
+ this.partitionKeys = partitionKeys;
+ }
+
+ static ReferencesManager create(ResolvedCatalogTable catalogTable) {
+ Map<String, Set<String>> columnToReferences = new HashMap<>();
+ Map<String, Set<String>> columnToDependencies = new HashMap<>();
+ catalogTable.getResolvedSchema().getColumns().stream()
+ .filter(column -> column instanceof Column.ComputedColumn)
+ .forEach(
+ column -> {
+ Set<String> referencedColumns =
+ ColumnReferenceFinder.findReferencedColumn(
+ column.getName(), catalogTable.getResolvedSchema());
+ for (String referencedColumn : referencedColumns) {
+ columnToReferences
+ .computeIfAbsent(
+ referencedColumn, key -> new HashSet<>())
+ .add(column.getName());
+ columnToDependencies
+ .computeIfAbsent(
+ column.getName(), key -> new HashSet<>())
+ .add(referencedColumn);
+ }
+ });
- private void validateColumnName(
- String oldColumnName,
- String newColumnName,
- ResolvedSchema oldSchema,
- List<String> partitionKeys) {
- validateColumnName(
- oldColumnName,
- oldSchema,
- partitionKeys,
- // fail the operation of renaming column, once the column derives a computed column
- (referencedColumn, computedColumn) -> referencedColumn.contains(oldColumnName));
- // validate new column
- if (oldSchema.getColumn(newColumnName).isPresent()) {
- throw new ValidationException(
- String.format(
- "%sThe column `%s` already existed in table schema.",
- EX_MSG_PREFIX, newColumnName));
+ return new ReferencesManager(
+ new HashSet<>(catalogTable.getResolvedSchema().getColumnNames()),
+ columnToReferences,
+ columnToDependencies,
+ catalogTable
+ .getResolvedSchema()
+ .getPrimaryKey()
+ .map(constraint -> new HashSet<>(constraint.getColumns()))
+ .orElse(new HashSet<>()),
+ ColumnReferenceFinder.findWatermarkReferencedColumn(
+ catalogTable.getResolvedSchema()),
+ new HashSet<>(catalogTable.getPartitionKeys()));
+ }
+
+ void dropColumn(String columnName) {
+ checkReferences(columnName);
+ if (primaryKeys.contains(columnName)) {
+ throw new ValidationException(
+ String.format(
+ "%sThe column %s is used as the primary key.",
+ EX_MSG_PREFIX, EncodingUtils.escapeIdentifier(columnName)));
+ }
+
+ columnToDependencies
+ .getOrDefault(columnName, Collections.emptySet())
+ .forEach(
+ referredColumn ->
+ columnToReferences.get(referredColumn).remove(columnName));
+ columnToDependencies.remove(columnName);
+ columns.remove(columnName);
}
- }
- private void validateColumnName(
- String columnToDrop,
- ResolvedSchema oldSchema,
- List<String> partitionKeys,
- Set<String> columnsToDrop) {
- validateColumnName(
- columnToDrop,
- oldSchema,
- partitionKeys,
- // fail the operation of dropping column, only if the column derives a computed
- // column, and the computed column is not being dropped along with the old column
- (referencedColumn, computedColumn) ->
- referencedColumn.contains(columnToDrop)
- && !columnsToDrop.contains(computedColumn.getName()));
- oldSchema
- .getPrimaryKey()
- .ifPresent(
- pk -> {
- if (pk.getColumns().contains(columnToDrop)) {
- throw new ValidationException(
- String.format(
- "%sThe column `%s` is used as the primary key.",
- EX_MSG_PREFIX, columnToDrop));
- }
- });
- }
+ void renameColumn(String columnName, String newName) {
+ checkReferences(columnName);
+ if (columns.contains(newName)) {
+ throw new ValidationException(
+ String.format(
+ "%sThe column `%s` already existed in table schema.",
+ EX_MSG_PREFIX, newName));
+ }
- private void validateColumnName(
- String columnToAlter,
- ResolvedSchema oldSchema,
- List<String> partitionKeys,
- BiFunction<Set<String>, Column.ComputedColumn, Boolean> computedColumnChecker) {
- // validate old column
- Set<String> tableColumns = new HashSet<>(oldSchema.getColumnNames());
- if (!tableColumns.contains(columnToAlter)) {
- throw new ValidationException(
- String.format(
- "%sThe column `%s` does not exist in the base table.",
- EX_MSG_PREFIX, columnToAlter));
+ columnToDependencies
+ .getOrDefault(columnName, Collections.emptySet())
+ .forEach(
+ referredColumn -> {
+ columnToReferences.get(referredColumn).remove(columnName);
+ columnToReferences.get(referredColumn).add(newName);
+ });
+ columnToDependencies.put(newName, columnToDependencies.remove(columnName));
+
+ columns.remove(columnName);
+ columns.add(newName);
+
+ primaryKeys.remove(columnName);
Review Comment:
I don't think we can directly add `newColumnName` to `primaryKeys`.
```suggestion
if (primaryKeys.remove(columnName)) {
primaryKeys.add(newName);
}
```
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/operations/SqlToOperationConverterTest.java:
##########
@@ -1474,25 +1470,29 @@ public void testFailedToAlterTableDropConstraint() throws Exception {
@Test
public void testAlterTableDropConstraint() throws Exception {
prepareNonManagedTable(true);
- String expectedSummaryString =
- "ALTER TABLE cat1.db1.tb1 SET SCHEMA (\n"
- + " `a` INT NOT NULL,\n"
- + " `b` BIGINT NOT NULL,\n"
- + " `c` STRING NOT NULL COMMENT 'column comment',\n"
- + " `d` AS [a*(b+2 + a*b)],\n"
- + " `e` ROW<`f0` STRING, `f1` INT, `f2` ROW<`f0` DOUBLE, `f1` ARRAY<FLOAT>>>,\n"
- + " `f` AS [e.f1 + e.f2.f0],\n"
- + " `g` METADATA VIRTUAL,\n"
- + " `ts` TIMESTAMP(3) COMMENT 'just a comment'\n"
- + ")";
+ String expectedSummaryString = "ALTER TABLE cat1.db1.tb1\n DROP CONSTRAINT ct1";
Operation operation = parse("alter table tb1 drop constraint ct1");
- assertThat(operation).isInstanceOf(AlterTableSchemaOperation.class);
+ assertThat(operation).isInstanceOf(AlterTableChangeOperation.class);
assertThat(operation.asSummaryString()).isEqualTo(expectedSummaryString);
+ assertThat(
+ ((AlterTableChangeOperation) operation)
+ .getNewTable()
+ .getUnresolvedSchema()
+ .getPrimaryKey()
+ .isPresent())
Review Comment:
```suggestion
isNotPresent()
```
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/operations/SqlToOperationConverterTest.java:
##########
@@ -1474,25 +1470,29 @@ public void testFailedToAlterTableDropConstraint() throws Exception {
@Test
public void testAlterTableDropConstraint() throws Exception {
prepareNonManagedTable(true);
- String expectedSummaryString =
- "ALTER TABLE cat1.db1.tb1 SET SCHEMA (\n"
- + " `a` INT NOT NULL,\n"
- + " `b` BIGINT NOT NULL,\n"
- + " `c` STRING NOT NULL COMMENT 'column comment',\n"
- + " `d` AS [a*(b+2 + a*b)],\n"
- + " `e` ROW<`f0` STRING, `f1` INT, `f2` ROW<`f0` DOUBLE, `f1` ARRAY<FLOAT>>>,\n"
- + " `f` AS [e.f1 + e.f2.f0],\n"
- + " `g` METADATA VIRTUAL,\n"
- + " `ts` TIMESTAMP(3) COMMENT 'just a comment'\n"
- + ")";
+ String expectedSummaryString = "ALTER TABLE cat1.db1.tb1\n DROP CONSTRAINT ct1";
Operation operation = parse("alter table tb1 drop constraint ct1");
- assertThat(operation).isInstanceOf(AlterTableSchemaOperation.class);
+ assertThat(operation).isInstanceOf(AlterTableChangeOperation.class);
assertThat(operation.asSummaryString()).isEqualTo(expectedSummaryString);
+ assertThat(
+ ((AlterTableChangeOperation) operation)
+ .getNewTable()
+ .getUnresolvedSchema()
+ .getPrimaryKey()
+ .isPresent())
+ .isEqualTo(false);
operation = parse("alter table tb1 drop primary key");
- assertThat(operation).isInstanceOf(AlterTableSchemaOperation.class);
+ assertThat(operation).isInstanceOf(AlterTableChangeOperation.class);
assertThat(operation.asSummaryString()).isEqualTo(expectedSummaryString);
+ assertThat(
+ ((AlterTableChangeOperation) operation)
+ .getNewTable()
+ .getUnresolvedSchema()
+ .getPrimaryKey()
+ .isPresent())
+ .isEqualTo(false);
Review Comment:
Nit
```suggestion
assertThat(
((AlterTableChangeOperation) operation)
.getNewTable()
.getUnresolvedSchema()
.getPrimaryKey()
)
.isNotPresent();
```
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/operations/AlterSchemaConverter.java:
##########
@@ -728,114 +724,166 @@ String getComment(SqlTableColumn column) {
}
}
- // --------------------------------------------------------------------------------------------
+ private static class ReferencesManager {
+
+ /** Available columns in the table. */
+ private final Set<String> columns;
+
+ /**
+ * Mappings about the column refers which columns, e.g. column `b` refers to the column `a`
+ * in the expression "b as a+1".
+ */
+ private final Map<String, Set<String>> columnToReferences;
+
+ /**
+ * Reverse mappings about the column refers which columns, e.g. column `a` has the
+ * dependency of column `b` in the expression "b as a+1".
+ */
+ private final Map<String, Set<String>> columnToDependencies;
+
+ /** Primary keys defined on the table. */
+ private final Set<String> primaryKeys;
+
+ /** The name of the column watermark expression depends on. */
+ private final Set<String> watermarkReferences;
+
+ /** The name of the column partition keys contains. */
+ private final Set<String> partitionKeys;
+
+ private ReferencesManager(
+ Set<String> columns,
+ Map<String, Set<String>> columnToReferences,
+ Map<String, Set<String>> columnToDependencies,
+ Set<String> primaryKeys,
+ Set<String> watermarkReferences,
+ Set<String> partitionKeys) {
+ this.columns = columns;
+ this.columnToReferences = columnToReferences;
+ this.columnToDependencies = columnToDependencies;
+ this.primaryKeys = primaryKeys;
+ this.watermarkReferences = watermarkReferences;
+ this.partitionKeys = partitionKeys;
+ }
+
+ static ReferencesManager create(ResolvedCatalogTable catalogTable) {
+ Map<String, Set<String>> columnToReferences = new HashMap<>();
+ Map<String, Set<String>> columnToDependencies = new HashMap<>();
+ catalogTable.getResolvedSchema().getColumns().stream()
+ .filter(column -> column instanceof Column.ComputedColumn)
+ .forEach(
+ column -> {
+ Set<String> referencedColumns =
+ ColumnReferenceFinder.findReferencedColumn(
+ column.getName(), catalogTable.getResolvedSchema());
+ for (String referencedColumn : referencedColumns) {
+ columnToReferences
+ .computeIfAbsent(
+ referencedColumn, key -> new HashSet<>())
+ .add(column.getName());
+ columnToDependencies
+ .computeIfAbsent(
+ column.getName(), key -> new HashSet<>())
+ .add(referencedColumn);
+ }
+ });
- private void validateColumnName(
- String oldColumnName,
- String newColumnName,
- ResolvedSchema oldSchema,
- List<String> partitionKeys) {
- validateColumnName(
- oldColumnName,
- oldSchema,
- partitionKeys,
- // fail the operation of renaming column, once the column derives a computed column
- (referencedColumn, computedColumn) -> referencedColumn.contains(oldColumnName));
- // validate new column
- if (oldSchema.getColumn(newColumnName).isPresent()) {
- throw new ValidationException(
- String.format(
- "%sThe column `%s` already existed in table schema.",
- EX_MSG_PREFIX, newColumnName));
+ return new ReferencesManager(
+ new HashSet<>(catalogTable.getResolvedSchema().getColumnNames()),
+ columnToReferences,
+ columnToDependencies,
+ catalogTable
+ .getResolvedSchema()
+ .getPrimaryKey()
+ .map(constraint -> new HashSet<>(constraint.getColumns()))
+ .orElse(new HashSet<>()),
+ ColumnReferenceFinder.findWatermarkReferencedColumn(
+ catalogTable.getResolvedSchema()),
+ new HashSet<>(catalogTable.getPartitionKeys()));
+ }
+
+ void dropColumn(String columnName) {
+ checkReferences(columnName);
+ if (primaryKeys.contains(columnName)) {
+ throw new ValidationException(
+ String.format(
+ "%sThe column %s is used as the primary key.",
+ EX_MSG_PREFIX, EncodingUtils.escapeIdentifier(columnName)));
+ }
+
+ columnToDependencies
+ .getOrDefault(columnName, Collections.emptySet())
+ .forEach(
+ referredColumn ->
+ columnToReferences.get(referredColumn).remove(columnName));
+ columnToDependencies.remove(columnName);
+ columns.remove(columnName);
}
- }
- private void validateColumnName(
- String columnToDrop,
- ResolvedSchema oldSchema,
- List<String> partitionKeys,
- Set<String> columnsToDrop) {
- validateColumnName(
- columnToDrop,
- oldSchema,
- partitionKeys,
- // fail the operation of dropping column, only if the column derives a computed
- // column, and the computed column is not being dropped along with the old column
- (referencedColumn, computedColumn) ->
- referencedColumn.contains(columnToDrop)
- && !columnsToDrop.contains(computedColumn.getName()));
- oldSchema
- .getPrimaryKey()
- .ifPresent(
- pk -> {
- if (pk.getColumns().contains(columnToDrop)) {
- throw new ValidationException(
- String.format(
- "%sThe column `%s` is used as the primary key.",
- EX_MSG_PREFIX, columnToDrop));
- }
- });
- }
+ void renameColumn(String columnName, String newName) {
+ checkReferences(columnName);
+ if (columns.contains(newName)) {
+ throw new ValidationException(
+ String.format(
+ "%sThe column `%s` already existed in table schema.",
+ EX_MSG_PREFIX, newName));
+ }
- private void validateColumnName(
- String columnToAlter,
- ResolvedSchema oldSchema,
- List<String> partitionKeys,
- BiFunction<Set<String>, Column.ComputedColumn, Boolean> computedColumnChecker) {
- // validate old column
- Set<String> tableColumns = new HashSet<>(oldSchema.getColumnNames());
- if (!tableColumns.contains(columnToAlter)) {
- throw new ValidationException(
- String.format(
- "%sThe column `%s` does not exist in the base table.",
- EX_MSG_PREFIX, columnToAlter));
+ columnToDependencies
+ .getOrDefault(columnName, Collections.emptySet())
+ .forEach(
+ referredColumn -> {
+ columnToReferences.get(referredColumn).remove(columnName);
+ columnToReferences.get(referredColumn).add(newName);
+ });
+ columnToDependencies.put(newName, columnToDependencies.remove(columnName));
+
+ columns.remove(columnName);
+ columns.add(newName);
+
+ primaryKeys.remove(columnName);
Review Comment:
Btw, I don't think we need to update `primaryKeys` here because `buildUpdatedPrimaryKey` will compute the primary key name.
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/operations/AlterSchemaConverter.java:
##########
@@ -728,114 +724,166 @@ String getComment(SqlTableColumn column) {
}
}
- // --------------------------------------------------------------------------------------------
+ private static class ReferencesManager {
+
+ /** Available columns in the table. */
+ private final Set<String> columns;
+
+ /**
+ * Mappings about the column refers which columns, e.g. column `b` refers to the column `a`
+ * in the expression "b as a+1".
+ */
+ private final Map<String, Set<String>> columnToReferences;
+
+ /**
+ * Reverse mappings about the column refers which columns, e.g. column `a` has the
+ * dependency of column `b` in the expression "b as a+1".
+ */
+ private final Map<String, Set<String>> columnToDependencies;
+
+ /** Primary keys defined on the table. */
+ private final Set<String> primaryKeys;
+
+ /** The name of the column watermark expression depends on. */
+ private final Set<String> watermarkReferences;
+
+ /** The name of the column partition keys contains. */
+ private final Set<String> partitionKeys;
+
+ private ReferencesManager(
+ Set<String> columns,
+ Map<String, Set<String>> columnToReferences,
+ Map<String, Set<String>> columnToDependencies,
+ Set<String> primaryKeys,
+ Set<String> watermarkReferences,
+ Set<String> partitionKeys) {
+ this.columns = columns;
+ this.columnToReferences = columnToReferences;
+ this.columnToDependencies = columnToDependencies;
+ this.primaryKeys = primaryKeys;
+ this.watermarkReferences = watermarkReferences;
+ this.partitionKeys = partitionKeys;
+ }
+
+ static ReferencesManager create(ResolvedCatalogTable catalogTable) {
+ Map<String, Set<String>> columnToReferences = new HashMap<>();
+ Map<String, Set<String>> columnToDependencies = new HashMap<>();
+ catalogTable.getResolvedSchema().getColumns().stream()
+ .filter(column -> column instanceof Column.ComputedColumn)
+ .forEach(
+ column -> {
+ Set<String> referencedColumns =
+ ColumnReferenceFinder.findReferencedColumn(
+ column.getName(), catalogTable.getResolvedSchema());
+ for (String referencedColumn : referencedColumns) {
+ columnToReferences
+ .computeIfAbsent(
+ referencedColumn, key -> new HashSet<>())
+ .add(column.getName());
+ columnToDependencies
+ .computeIfAbsent(
+ column.getName(), key -> new HashSet<>())
+ .add(referencedColumn);
+ }
+ });
- private void validateColumnName(
- String oldColumnName,
- String newColumnName,
- ResolvedSchema oldSchema,
- List<String> partitionKeys) {
- validateColumnName(
- oldColumnName,
- oldSchema,
- partitionKeys,
- // fail the operation of renaming column, once the column derives a computed column
- (referencedColumn, computedColumn) -> referencedColumn.contains(oldColumnName));
- // validate new column
- if (oldSchema.getColumn(newColumnName).isPresent()) {
- throw new ValidationException(
- String.format(
- "%sThe column `%s` already existed in table schema.",
- EX_MSG_PREFIX, newColumnName));
+ return new ReferencesManager(
+ new HashSet<>(catalogTable.getResolvedSchema().getColumnNames()),
+ columnToReferences,
+ columnToDependencies,
+ catalogTable
+ .getResolvedSchema()
+ .getPrimaryKey()
+ .map(constraint -> new HashSet<>(constraint.getColumns()))
+ .orElse(new HashSet<>()),
+ ColumnReferenceFinder.findWatermarkReferencedColumn(
+ catalogTable.getResolvedSchema()),
+ new HashSet<>(catalogTable.getPartitionKeys()));
+ }
+
+ void dropColumn(String columnName) {
+ checkReferences(columnName);
+ if (primaryKeys.contains(columnName)) {
+ throw new ValidationException(
+ String.format(
+ "%sThe column %s is used as the primary key.",
+ EX_MSG_PREFIX, EncodingUtils.escapeIdentifier(columnName)));
+ }
+
+ columnToDependencies
+ .getOrDefault(columnName, Collections.emptySet())
+ .forEach(
+ referredColumn ->
+ columnToReferences.get(referredColumn).remove(columnName));
+ columnToDependencies.remove(columnName);
+ columns.remove(columnName);
}
- }
- private void validateColumnName(
- String columnToDrop,
- ResolvedSchema oldSchema,
- List<String> partitionKeys,
- Set<String> columnsToDrop) {
- validateColumnName(
- columnToDrop,
- oldSchema,
- partitionKeys,
- // fail the operation of dropping column, only if the column derives a computed
- // column, and the computed column is not being dropped along with the old column
- (referencedColumn, computedColumn) ->
- referencedColumn.contains(columnToDrop)
- && !columnsToDrop.contains(computedColumn.getName()));
- oldSchema
- .getPrimaryKey()
- .ifPresent(
- pk -> {
- if (pk.getColumns().contains(columnToDrop)) {
- throw new ValidationException(
- String.format(
- "%sThe column `%s` is used as the primary key.",
- EX_MSG_PREFIX, columnToDrop));
- }
- });
- }
+ void renameColumn(String columnName, String newName) {
+ checkReferences(columnName);
+ if (columns.contains(newName)) {
+ throw new ValidationException(
+ String.format(
+ "%sThe column `%s` already existed in table schema.",
+ EX_MSG_PREFIX, newName));
+ }
- private void validateColumnName(
- String columnToAlter,
- ResolvedSchema oldSchema,
- List<String> partitionKeys,
- BiFunction<Set<String>, Column.ComputedColumn, Boolean> computedColumnChecker) {
- // validate old column
- Set<String> tableColumns = new HashSet<>(oldSchema.getColumnNames());
- if (!tableColumns.contains(columnToAlter)) {
- throw new ValidationException(
- String.format(
- "%sThe column `%s` does not exist in the base table.",
- EX_MSG_PREFIX, columnToAlter));
+ columnToDependencies
+ .getOrDefault(columnName, Collections.emptySet())
+ .forEach(
+ referredColumn -> {
+ columnToReferences.get(referredColumn).remove(columnName);
+ columnToReferences.get(referredColumn).add(newName);
+ });
+ columnToDependencies.put(newName, columnToDependencies.remove(columnName));
+
+ columns.remove(columnName);
+ columns.add(newName);
Review Comment:
Do we need to update `columnToDependencies` and `columnToReferences`?
`ALTER TABLE RENAME` is not a bulk operation; there won't exist any further calls to get value from them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] fsk119 commented on a diff in pull request #21592: [FLINK-30497][table-api] Introduce TableChange to represent DROP change
Posted by GitBox <gi...@apache.org>.
fsk119 commented on code in PR #21592:
URL: https://github.com/apache/flink/pull/21592#discussion_r1061088996
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/operations/AlterSchemaConverter.java:
##########
@@ -728,114 +724,166 @@ String getComment(SqlTableColumn column) {
}
}
- // --------------------------------------------------------------------------------------------
+ private static class ReferencesManager {
+
+ /** Available columns in the table. */
+ private final Set<String> columns;
+
+ /**
+ * Mappings about the column refers which columns, e.g. column `b` refers to the column `a`
+ * in the expression "b as a+1".
+ */
+ private final Map<String, Set<String>> columnToReferences;
+
+ /**
+ * Reverse mappings about the column refers which columns, e.g. column `a` has the
+ * dependency of column `b` in the expression "b as a+1".
+ */
+ private final Map<String, Set<String>> columnToDependencies;
+
+ /** Primary keys defined on the table. */
+ private final Set<String> primaryKeys;
+
+ /** The name of the column watermark expression depends on. */
+ private final Set<String> watermarkReferences;
+
+ /** The name of the column partition keys contains. */
+ private final Set<String> partitionKeys;
+
+ private ReferencesManager(
+ Set<String> columns,
+ Map<String, Set<String>> columnToReferences,
+ Map<String, Set<String>> columnToDependencies,
+ Set<String> primaryKeys,
+ Set<String> watermarkReferences,
+ Set<String> partitionKeys) {
+ this.columns = columns;
+ this.columnToReferences = columnToReferences;
+ this.columnToDependencies = columnToDependencies;
+ this.primaryKeys = primaryKeys;
+ this.watermarkReferences = watermarkReferences;
+ this.partitionKeys = partitionKeys;
+ }
+
+ static ReferencesManager create(ResolvedCatalogTable catalogTable) {
+ Map<String, Set<String>> columnToReferences = new HashMap<>();
+ Map<String, Set<String>> columnToDependencies = new HashMap<>();
+ catalogTable.getResolvedSchema().getColumns().stream()
+ .filter(column -> column instanceof Column.ComputedColumn)
+ .forEach(
+ column -> {
+ Set<String> referencedColumns =
+ ColumnReferenceFinder.findReferencedColumn(
+ column.getName(), catalogTable.getResolvedSchema());
+ for (String referencedColumn : referencedColumns) {
+ columnToReferences
+ .computeIfAbsent(
+ referencedColumn, key -> new HashSet<>())
+ .add(column.getName());
+ columnToDependencies
+ .computeIfAbsent(
+ column.getName(), key -> new HashSet<>())
+ .add(referencedColumn);
+ }
+ });
- private void validateColumnName(
- String oldColumnName,
- String newColumnName,
- ResolvedSchema oldSchema,
- List<String> partitionKeys) {
- validateColumnName(
- oldColumnName,
- oldSchema,
- partitionKeys,
- // fail the operation of renaming column, once the column derives a computed column
- (referencedColumn, computedColumn) -> referencedColumn.contains(oldColumnName));
- // validate new column
- if (oldSchema.getColumn(newColumnName).isPresent()) {
- throw new ValidationException(
- String.format(
- "%sThe column `%s` already existed in table schema.",
- EX_MSG_PREFIX, newColumnName));
+ return new ReferencesManager(
+ new HashSet<>(catalogTable.getResolvedSchema().getColumnNames()),
+ columnToReferences,
+ columnToDependencies,
+ catalogTable
+ .getResolvedSchema()
+ .getPrimaryKey()
+ .map(constraint -> new HashSet<>(constraint.getColumns()))
+ .orElse(new HashSet<>()),
+ ColumnReferenceFinder.findWatermarkReferencedColumn(
+ catalogTable.getResolvedSchema()),
+ new HashSet<>(catalogTable.getPartitionKeys()));
+ }
+
+ void dropColumn(String columnName) {
+ checkReferences(columnName);
+ if (primaryKeys.contains(columnName)) {
+ throw new ValidationException(
+ String.format(
+ "%sThe column %s is used as the primary key.",
+ EX_MSG_PREFIX, EncodingUtils.escapeIdentifier(columnName)));
+ }
+
+ columnToDependencies
+ .getOrDefault(columnName, Collections.emptySet())
+ .forEach(
+ referredColumn ->
+ columnToReferences.get(referredColumn).remove(columnName));
+ columnToDependencies.remove(columnName);
+ columns.remove(columnName);
}
- }
- private void validateColumnName(
- String columnToDrop,
- ResolvedSchema oldSchema,
- List<String> partitionKeys,
- Set<String> columnsToDrop) {
- validateColumnName(
- columnToDrop,
- oldSchema,
- partitionKeys,
- // fail the operation of dropping column, only if the column derives a computed
- // column, and the computed column is not being dropped along with the old column
- (referencedColumn, computedColumn) ->
- referencedColumn.contains(columnToDrop)
- && !columnsToDrop.contains(computedColumn.getName()));
- oldSchema
- .getPrimaryKey()
- .ifPresent(
- pk -> {
- if (pk.getColumns().contains(columnToDrop)) {
- throw new ValidationException(
- String.format(
- "%sThe column `%s` is used as the primary key.",
- EX_MSG_PREFIX, columnToDrop));
- }
- });
- }
+ void renameColumn(String columnName, String newName) {
+ checkReferences(columnName);
+ if (columns.contains(newName)) {
+ throw new ValidationException(
+ String.format(
+ "%sThe column `%s` already existed in table schema.",
+ EX_MSG_PREFIX, newName));
+ }
- private void validateColumnName(
- String columnToAlter,
- ResolvedSchema oldSchema,
- List<String> partitionKeys,
- BiFunction<Set<String>, Column.ComputedColumn, Boolean> computedColumnChecker) {
- // validate old column
- Set<String> tableColumns = new HashSet<>(oldSchema.getColumnNames());
- if (!tableColumns.contains(columnToAlter)) {
- throw new ValidationException(
- String.format(
- "%sThe column `%s` does not exist in the base table.",
- EX_MSG_PREFIX, columnToAlter));
+ columnToDependencies
+ .getOrDefault(columnName, Collections.emptySet())
+ .forEach(
+ referredColumn -> {
+ columnToReferences.get(referredColumn).remove(columnName);
+ columnToReferences.get(referredColumn).add(newName);
+ });
+ columnToDependencies.put(newName, columnToDependencies.remove(columnName));
+
+ columns.remove(columnName);
+ columns.add(newName);
+
+ primaryKeys.remove(columnName);
Review Comment:
OK. I remove this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org