You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Timm0 (via GitHub)" <gi...@apache.org> on 2026/08/03 09:55:42 UTC
[PR] [FLINK-40271][table] Fix SQL serialization of Table API group window properties [flink]
Timm0 opened a new pull request, #28886:
URL: https://github.com/apache/flink/pull/28886
<!--
*Thank you very much for contributing to Apache Flink - we are happy that you want to help us improve Flink. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
*Please understand that we do not do this to make contributions to Flink a hassle. In order to uphold a high standard of quality for code contributions, while at the same time managing a large number of contributions, we need contributors to prepare the contributions well, and give reviewers enough contextual information for the review. Please also understand that contributions that do not follow this guide will take longer to review and thus typically be picked up with lower priority by the community.*
## Contribution Checklist
- Make sure that the pull request corresponds to a [JIRA issue](https://issues.apache.org/jira/projects/FLINK/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no JIRA issue.
- Name the pull request in the form "[FLINK-XXXX] [component] Title of the pull request", where *FLINK-XXXX* should be replaced by the actual issue number. Skip *component* if you are unsure about which is the best component.
Typo fixes that have no associated JIRA issue should be named following this pattern: `[hotfix] [docs] Fix typo in event time introduction` or `[hotfix] [javadocs] Expand JavaDoc for PuncuatedWatermarkGenerator`.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes. You can set up Azure Pipelines CI to do that following [this guide](https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines#AzurePipelines-Tutorial:SettingupAzurePipelinesforaforkoftheFlinkrepository).
- Each pull request should address only one issue, not mix up code from multiple issues.
- Each commit in the pull request has a meaningful commit message (including the JIRA id)
- Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
**(The sections below can be removed for hotfixes of typos)**
-->
## What is the purpose of the change
Table API group-window properties — `$("w").start()`, `$("w").end()`, `$("w").rowtime()` and `$("w").proctime()` — were not translated to the corresponding windowing-TVF output columns when a `WindowAggregateQueryOperation` is serialized back to SQL. The group window alias was emitted as if it were a column reference, so the generated statement failed to validate with `Column 'w' not found in any table`. This change maps each property onto the TVF's `window_start` / `window_end` / `window_time` columns and adds `window_time` to the `GROUP BY` when it is projected, so the serialized SQL parses and produces the same results as the Table API program it came from.
## Brief change log
- Serialize window properties to the windowing-TVF output columns `window_start`, `window_end` and `window_time` instead of emitting the group window alias, which produced unparseable SQL
- Add `window_time` to the `GROUP BY` when a `rowtime()`/`proctime()` property is projected
- Reject a `proctime()` property on an event-time group window with a `TableException`
- Add `TableTestProgram#getRunTableApiTestStep`
- Add rowtime and proctime serialization test programs, plus semantic coverage of the generated SQL for the rowtime case
## Verifying this change
- New tests in `QueryOperationSqlSerializationTest` and `QueryOperationSqlSemanticTest`
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? n/a
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR, please
change the checkbox below to `[X]` and replace the placeholder in the "Generated-by"
line with the tool name and version. Otherwise remove the "Generated-by" line.
See the ASF Generative Tooling Guidance for details:
https://www.apache.org/legal/generative-tooling.html
You are responsible for the quality and correctness of every change in this PR
regardless of the tooling used. Low-effort AI-generated PRs will be closed. See
AGENTS.md for the full guidance.
-->
- [x] Yes
Generated-by: Claude Code Opus 5 (1M context)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40271][table] Fix SQL serialization of Table API group window properties [flink]
Posted by "Timm0 (via GitHub)" <gi...@apache.org>.
Timm0 commented on code in PR #28886:
URL: https://github.com/apache/flink/pull/28886#discussion_r3735820762
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/api/QueryOperationSqlSerializationTest.java:
##########
@@ -138,6 +124,46 @@ void testSqlAsJobNameForQueryOperation(TableTestProgram program) {
assertThat(streamGraph.getJobName()).isEqualTo(sqlStep.sql);
}
+ @Test
+ void testProctimeWindowGeneratedSqlPlans() {
Review Comment:
Tried this, but a `proctime` window can't be covered by `QueryOperationSqlSemanticTest`. A proctime window only fires from a processing-time timer, and since `CommonSemanticTestBase` doesn't set terminating=false on the source (unlike `RestoreTestBase` for example) the bounded source ends the job before the timer fires and the sink stays empty.
I kept the test in `QueryOperationSqlSerializationTest` and added a comment to the test program.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40271][table] Fix SQL serialization of Table API group window properties [flink]
Posted by "Timm0 (via GitHub)" <gi...@apache.org>.
Timm0 commented on code in PR #28886:
URL: https://github.com/apache/flink/pull/28886#discussion_r3735820762
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/api/QueryOperationSqlSerializationTest.java:
##########
@@ -138,6 +124,46 @@ void testSqlAsJobNameForQueryOperation(TableTestProgram program) {
assertThat(streamGraph.getJobName()).isEqualTo(sqlStep.sql);
}
+ @Test
+ void testProctimeWindowGeneratedSqlPlans() {
Review Comment:
Tried this, but a `proctime` window can't be covered by `QueryOperationSqlSemanticTest`. A proctime window only fires from a processing-time timer, and since `CommonSemanticTestBase` doesn't set terminating=false on the source (unlike `RestoreTestBase` for example) the bounded source ends the job before the timer fires and the sink stays empty.
I kept the test in `QueryOperationSqlSerializationTest` for now and added a comment to the test program. Let me know if you want me to do this differently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40271][table] Fix SQL serialization of Table API group window properties [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #28886:
URL: https://github.com/apache/flink/pull/28886#issuecomment-5164990469
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "91115b26b584d540eac5ac7d928543ac033ff652",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91115b26b584d540eac5ac7d928543ac033ff652",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91115b26b584d540eac5ac7d928543ac033ff652 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40271][table] Fix SQL serialization of Table API group window properties [flink]
Posted by "Timm0 (via GitHub)" <gi...@apache.org>.
Timm0 commented on code in PR #28886:
URL: https://github.com/apache/flink/pull/28886#discussion_r3735795860
##########
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/operations/WindowAggregateQueryOperation.java:
##########
@@ -94,35 +107,112 @@ public String asSummaryString() {
@Override
public String asSerializableString(SqlFactory sqlFactory) {
+ final List<WindowColumn> windowColumns = resolveWindowColumns();
return String.format(
"SELECT %s FROM TABLE(%s\n) %s GROUP BY %s",
- Stream.of(
- groupingExpressions.stream(),
- aggregateExpressions.stream(),
- windowPropertiesExpressions.stream())
- .flatMap(Function.identity())
- .map(
- expr ->
- OperationExpressionsUtils.scopeReferencesWithAlias(
- INPUT_ALIAS, expr))
- .map(
- resolvedExpression ->
- resolvedExpression.asSerializableString(sqlFactory))
- .collect(Collectors.joining(", ")),
+ serializeSelectList(windowColumns, sqlFactory),
OperationUtils.indent(
groupWindow.asSerializableString(
child.asSerializableString(sqlFactory), sqlFactory)),
INPUT_ALIAS,
- Stream.concat(
- Stream.of("window_start", "window_end"),
- groupingExpressions.stream()
- .map(
- expr ->
- OperationExpressionsUtils
- .scopeReferencesWithAlias(
- INPUT_ALIAS, expr))
- .map(expr -> expr.asSerializableString(sqlFactory)))
- .collect(Collectors.joining(", ")));
+ serializeGroupBy(windowColumns, sqlFactory));
+ }
+
+ private List<WindowColumn> resolveWindowColumns() {
+ return windowPropertiesExpressions.stream()
+ .map(property -> new WindowColumn(aliasOf(property), windowColumnOf(property)))
+ .collect(Collectors.toList());
+ }
+
+ private static String aliasOf(ResolvedExpression aliasedProperty) {
+ return OperationExpressionsUtils.extractName(aliasedProperty)
+ .orElseThrow(
+ () ->
+ new TableException(
+ "Expected a named alias over a window property. Got: "
+ + aliasedProperty));
+ }
+
+ /** The windowing TVF output column that the given window property denotes. */
+ private String windowColumnOf(ResolvedExpression aliasedProperty) {
+ final FunctionDefinition property = windowPropertyOf(aliasedProperty);
+ if (BuiltInFunctionDefinitions.WINDOW_START == property) {
+ return WINDOW_START_COLUMN;
+ }
+ if (BuiltInFunctionDefinitions.WINDOW_END == property) {
+ return WINDOW_END_COLUMN;
+ }
+ if (BuiltInFunctionDefinitions.ROWTIME == property) {
+ return WINDOW_TIME_COLUMN;
+ }
+ if (BuiltInFunctionDefinitions.PROCTIME == property) {
+ checkWindowIsProcessingTime();
+ return WINDOW_TIME_COLUMN;
+ }
+ throw new TableException("Unsupported window property: " + property);
+ }
+
+ private static FunctionDefinition windowPropertyOf(ResolvedExpression aliasedProperty) {
+ final List<ResolvedExpression> children = aliasedProperty.getResolvedChildren();
+ if (!children.isEmpty() && children.get(0) instanceof CallExpression) {
+ final FunctionDefinition property =
+ ((CallExpression) children.get(0)).getFunctionDefinition();
+ if (BuiltInFunctionDefinitions.WINDOW_PROPERTIES.contains(property)) {
Review Comment:
Moved them to `OperationExpressionsUtils` and added a doc string for each of those.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40271][table] Fix SQL serialization of Table API group window properties [flink]
Posted by "Timm0 (via GitHub)" <gi...@apache.org>.
Timm0 commented on code in PR #28886:
URL: https://github.com/apache/flink/pull/28886#discussion_r3735791790
##########
flink-table/flink-table-api-java/src/test/java/org/apache/flink/table/operations/QueryOperationTest.java:
##########
@@ -134,4 +140,52 @@ void testIndentation() {
+ " secondLevel1\n"
+ " thirdLevel1");
}
+
+ @Test
+ void testWindowPropertiesSharingAnAliasAreAllSerialized() {
Review Comment:
Removed this test from `QueryOperationTest`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40271][table] Fix SQL serialization of Table API group window properties [flink]
Posted by "twalthr (via GitHub)" <gi...@apache.org>.
twalthr commented on code in PR #28886:
URL: https://github.com/apache/flink/pull/28886#discussion_r3734332008
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/api/QueryOperationSqlSerializationTest.java:
##########
@@ -138,6 +124,46 @@ void testSqlAsJobNameForQueryOperation(TableTestProgram program) {
assertThat(streamGraph.getJobName()).isEqualTo(sqlStep.sql);
}
+ @Test
+ void testProctimeWindowGeneratedSqlPlans() {
Review Comment:
Could also be replaced with a QueryOperationSqlSemanticTest?
##########
flink-table/flink-table-api-java/src/test/java/org/apache/flink/table/operations/QueryOperationTest.java:
##########
@@ -134,4 +140,52 @@ void testIndentation() {
+ " secondLevel1\n"
+ " thirdLevel1");
}
+
+ @Test
+ void testWindowPropertiesSharingAnAliasAreAllSerialized() {
Review Comment:
I'm wondering whether we need this test at all. The code is hard to read and to maintain. In the end QueryOperationSqlSemanticTest is what counts.
##########
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/operations/WindowAggregateQueryOperation.java:
##########
@@ -94,35 +107,112 @@ public String asSummaryString() {
@Override
public String asSerializableString(SqlFactory sqlFactory) {
+ final List<WindowColumn> windowColumns = resolveWindowColumns();
return String.format(
"SELECT %s FROM TABLE(%s\n) %s GROUP BY %s",
- Stream.of(
- groupingExpressions.stream(),
- aggregateExpressions.stream(),
- windowPropertiesExpressions.stream())
- .flatMap(Function.identity())
- .map(
- expr ->
- OperationExpressionsUtils.scopeReferencesWithAlias(
- INPUT_ALIAS, expr))
- .map(
- resolvedExpression ->
- resolvedExpression.asSerializableString(sqlFactory))
- .collect(Collectors.joining(", ")),
+ serializeSelectList(windowColumns, sqlFactory),
OperationUtils.indent(
groupWindow.asSerializableString(
child.asSerializableString(sqlFactory), sqlFactory)),
INPUT_ALIAS,
- Stream.concat(
- Stream.of("window_start", "window_end"),
- groupingExpressions.stream()
- .map(
- expr ->
- OperationExpressionsUtils
- .scopeReferencesWithAlias(
- INPUT_ALIAS, expr))
- .map(expr -> expr.asSerializableString(sqlFactory)))
- .collect(Collectors.joining(", ")));
+ serializeGroupBy(windowColumns, sqlFactory));
+ }
+
+ private List<WindowColumn> resolveWindowColumns() {
+ return windowPropertiesExpressions.stream()
+ .map(property -> new WindowColumn(aliasOf(property), windowColumnOf(property)))
+ .collect(Collectors.toList());
+ }
+
+ private static String aliasOf(ResolvedExpression aliasedProperty) {
+ return OperationExpressionsUtils.extractName(aliasedProperty)
+ .orElseThrow(
+ () ->
+ new TableException(
+ "Expected a named alias over a window property. Got: "
+ + aliasedProperty));
+ }
+
+ /** The windowing TVF output column that the given window property denotes. */
+ private String windowColumnOf(ResolvedExpression aliasedProperty) {
+ final FunctionDefinition property = windowPropertyOf(aliasedProperty);
+ if (BuiltInFunctionDefinitions.WINDOW_START == property) {
+ return WINDOW_START_COLUMN;
+ }
+ if (BuiltInFunctionDefinitions.WINDOW_END == property) {
+ return WINDOW_END_COLUMN;
+ }
+ if (BuiltInFunctionDefinitions.ROWTIME == property) {
+ return WINDOW_TIME_COLUMN;
+ }
+ if (BuiltInFunctionDefinitions.PROCTIME == property) {
+ checkWindowIsProcessingTime();
+ return WINDOW_TIME_COLUMN;
+ }
+ throw new TableException("Unsupported window property: " + property);
+ }
+
+ private static FunctionDefinition windowPropertyOf(ResolvedExpression aliasedProperty) {
+ final List<ResolvedExpression> children = aliasedProperty.getResolvedChildren();
+ if (!children.isEmpty() && children.get(0) instanceof CallExpression) {
+ final FunctionDefinition property =
+ ((CallExpression) children.get(0)).getFunctionDefinition();
+ if (BuiltInFunctionDefinitions.WINDOW_PROPERTIES.contains(property)) {
Review Comment:
I was confused to see `WINDOW_PROPERTIES` in `BuiltInFunctionDefinitions`. Could we move these Table API specific lists (ORDERING, TIME_ATTRIBUTES etc.) into OperationExpressionsUtils or some other Table API specific class?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40271][table] Fix SQL serialization of Table API group window properties [flink]
Posted by "autophagy (via GitHub)" <gi...@apache.org>.
autophagy commented on code in PR #28886:
URL: https://github.com/apache/flink/pull/28886#discussion_r3729945063
##########
flink-table/flink-table-api-java/src/test/java/org/apache/flink/table/operations/QueryOperationTest.java:
##########
@@ -134,4 +140,52 @@ void testIndentation() {
+ " secondLevel1\n"
+ " thirdLevel1");
}
+
+ @Test
+ void testWindowPropertiesSharingAnAliasAreAllSerialized() {
Review Comment:
Is the scenario in this test actually reachable, can 2 window properties actually share an alias? iirc the aliases are autogenerated and unique, but I might be missing something
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40271][table] Fix SQL serialization of Table API group window properties [flink]
Posted by "twalthr (via GitHub)" <gi...@apache.org>.
twalthr merged PR #28886:
URL: https://github.com/apache/flink/pull/28886
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org