You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by tw...@apache.org on 2026/04/17 15:10:27 UTC
(flink) branch master updated: [FLINK-39256][table] Support ORDER BY clause in Process Table Functions
This is an automated email from the ASF dual-hosted git repository.
twalthr pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/master by this push:
new 3afaa8cf645 [FLINK-39256][table] Support ORDER BY clause in Process Table Functions
3afaa8cf645 is described below
commit 3afaa8cf645e39559b5e01d8ef2ae143846b161a
Author: Timo Walther <tw...@apache.org>
AuthorDate: Fri Apr 17 17:10:20 2026 +0200
[FLINK-39256][table] Support ORDER BY clause in Process Table Functions
This closes #27929.
---
docs/content.zh/docs/dev/table/functions/ptfs.md | 178 ++++++-
docs/content/docs/dev/table/functions/ptfs.md | 183 +++++++-
.../apache/flink/table/api/PartitionedTable.java | 33 +-
.../apache/flink/table/api/internal/TableImpl.java | 27 +-
.../resolver/rules/ResolveCallByArgumentsRule.java | 77 ++-
.../table/operations/PartitionQueryOperation.java | 93 +++-
.../operations/utils/OperationTreeBuilder.java | 8 +
.../table/functions/ProcessTableFunction.java | 104 ++++-
.../flink/table/functions/TableSemantics.java | 47 ++
.../table/types/inference/SystemTypeInference.java | 91 +++-
.../types/inference/utils/TableSemanticsMock.java | 10 +
.../planner/calcite/FlinkConvertletTable.java | 22 +-
.../table/planner/calcite/RexTableArgCall.java | 201 +++++++-
.../inference/CallBindingCallContext.java | 37 +-
.../inference/OperatorBindingCallContext.java | 23 +-
.../planner/plan/QueryOperationConverter.java | 18 +-
.../nodes/exec/serde/RexNodeJsonDeserializer.java | 15 +-
.../nodes/exec/serde/RexNodeJsonSerializer.java | 8 +
.../stream/StreamExecProcessTableFunction.java | 75 ++-
.../stream/StreamPhysicalProcessTableFunction.java | 31 ++
.../ProjectWindowTableFunctionTransposeRule.java | 4 +-
.../table/api/QueryOperationSqlSemanticTest.java | 5 +-
.../api/QueryOperationSqlSerializationTest.java | 5 +-
.../table/api/QueryOperationTestPrograms.java | 50 +-
.../nodes/exec/serde/RexNodeJsonSerdeTest.java | 4 +-
.../stream/ProcessTableFunctionRestoreTests.java | 4 +-
.../stream/ProcessTableFunctionSemanticTests.java | 5 +-
.../stream/ProcessTableFunctionTestPrograms.java | 284 +++++++++++
.../exec/stream/ProcessTableFunctionTestUtils.java | 83 ++++
.../plan/nodes/exec/testutils/RestoreTestBase.java | 14 +
.../plan/stream/sql/ProcessTableFunctionTest.java | 51 +-
.../plan/stream/sql/ProcessTableFunctionTest.xml | 23 +
.../plan/process-multi-input-order-by-restore.json | 519 +++++++++++++++++++++
.../savepoint/_metadata | Bin 0 -> 28159 bytes
.../plan/process-order-by-restore.json | 320 +++++++++++++
.../process-order-by-restore/savepoint/_metadata | Bin 0 -> 21613 bytes
.../plan/metadata/FlinkRelMdHandlerTestBase.scala | 2 +-
.../runtime/generated/ProcessTableRunner.java | 13 +-
.../process/AbstractProcessTableOperator.java | 11 +-
.../operators/process/ExternalTimeContext.java | 9 +
.../runtime/operators/process/InputSortBuffer.java | 255 ++++++++++
.../process/MailboxPartialWatermarkProcessor.java | 153 ++++++
.../operators/process/ProcessRowTableOperator.java | 6 +-
.../operators/process/ProcessSetTableOperator.java | 162 ++++++-
.../process/ProcessTableOperatorFactory.java | 19 +-
.../process/ReadableInternalTimeContext.java | 12 +-
.../operators/process/RuntimeTableSemantics.java | 13 +-
.../operators/process/InputSortBufferTest.java | 461 ++++++++++++++++++
.../MailboxPartialWatermarkProcessorTest.java | 351 ++++++++++++++
tools/maven/suppressions.xml | 2 +-
50 files changed, 3989 insertions(+), 132 deletions(-)
diff --git a/docs/content.zh/docs/dev/table/functions/ptfs.md b/docs/content.zh/docs/dev/table/functions/ptfs.md
index ac41b8a1147..9a077a6473c 100644
--- a/docs/content.zh/docs/dev/table/functions/ptfs.md
+++ b/docs/content.zh/docs/dev/table/functions/ptfs.md
@@ -932,22 +932,37 @@ table.
The timestamp of the firing timer currently being processed within the `onTimer()` method.
+#### Table Watermark
+
+`TimeContext#tableWatermark()` returns the event-time watermark of the input table currently being processed.
+
+Watermarks are generated in sources and sent through the topology for advancing the logical clock. The current watermark
+of an input table is the minimum watermark of all upstream Flink subtasks producing the table.
+
+In multi-input scenarios, each input table can have its own independent watermark. This method returns the watermark
+specific to the input table that is currently being processed in the `eval()` method, rather than the global minimum
+watermark across all input tables (which is returned by `currentWatermark()`).
+
+This is particularly useful for late event detection on a per-input basis.
+
+It returns the current watermark of the input table being processed. `null` if called within the `onTimer()` method or a
+watermark has not yet been received from all upstream Flink subtasks producing the table.
+
#### Current Watermark
-`TimeContext#currentWatermark()` returns the current event-time watermark.
+`TimeContext#currentWatermark()` returns the current event-time watermark at this PTF instance.
-Watermarks are generated in sources and sent through the topology for advancing the logical clock in each Flink subtask.
-The current watermark of a Flink subtask is the global minimum watermark of all inputs (i.e. across all parallel inputs
-and table partitions).
+Watermarks are generated in sources and sent through the topology for advancing the logical clock. The current watermark
+of a PTF instance is the global minimum watermark of all input tables (i.e., across all upstream Flink subtasks and
+table partitions).
This method returns the current watermark of the Flink subtask that evaluates the PTF. Thus, the returned timestamp
-represents the entire Flink subtask, independent of the currently processed partition. This behavior is similar to a call
-to `SELECT CURRENT_WATERMARK(...)` in SQL.
+represents the entire Flink subtask, independent of the currently processed input table and partition. This behavior is
+similar to a call to `SELECT CURRENT_WATERMARK(...)` in SQL.
-If a watermark was not received from all inputs, the method returns `null`.
-
-In case this method is called within the `onTimer()` method, the returned watermark is the triggering watermark
-that currently fires the timer.
+It returns the current watermark at the PTF instance across all upstream Flink subtasks and table partitions. A `null`
+value is returned if no minimum logical time could be calculated across all inputs; this happens during startup
+or recovery when one or more active (i.e. not idle) inputs haven't sent a watermark yet.
### Timers
@@ -1015,6 +1030,149 @@ not needed anymore via `Context#clearAllTimers()` or `TimeContext#clearTimer(Str
{{< top >}}
+Ordering
+--------
+
+A PTF that takes a table with set semantics can optionally specify an ORDER BY clause in the
+function call to define the order in which rows are processed within each partition. The ORDER BY
+clause guarantees that rows are delivered to the eval() method in the specified order.
+
+The ORDER BY clause requires that the first column is a time attribute column (i.e., a
+TIMESTAMP or TIMESTAMP_LTZ column with a watermark declaration). The first ORDER BY column must
+be specified in ascending order. This ensures that rows are processed in event-time order.
+Additional columns can be specified as secondary sort keys to define the ordering of
+rows with the same timestamp.
+
+{{< tabs "2137eeed-order-by-syntax" >}}
+{{< tab "SQL" >}}
+```sql
+SELECT * FROM my_ptf(
+ input_table => TABLE source_table
+ PARTITION BY user_id
+ ORDER BY (event_time ASC, priority DESC NULLS FIRST)
+)
+```
+{{< /tab >}}
+{{< tab "Java" >}}
+```java
+env.from("source_table")
+ .partitionBy($("user_id"))
+ .orderBy($("event_time").asc(), $("priority").desc())
+ .process(MyPTF.class, descriptor("event_time").asArgument("on_time"))
+```
+{{< /tab >}}
+{{< /tabs >}}
+
+### Difference Between ORDER BY and on_time Argument
+
+While both ORDER BY and the `on_time` argument relate to time attributes, they serve
+different purposes:
+
+- **on_time**: Declares which time attribute column powers the time context (`TimeContext#time()`) and
+ output timestamp. It does NOT affect the processing order of rows.
+- **ORDER BY**: Physically buffers and sorts rows within each partition to guarantee ordered delivery
+ to the eval() method. If both ORDER BY and `on_time` are specified for the same table argument, they
+ must reference the same time attribute column.
+
+### Ordering Guarantees and Late Events
+
+When ORDER BY is specified on a time attribute column, the framework maintains a sort buffer
+per partition and input table to reorder out-of-order events. The sort buffer is flushed when the
+watermark for the given input table advances, at which point all buffered rows with timestamps
+less than or equal to the watermark are delivered to the eval() method in sorted order. Late
+events (arriving after the watermark) are dropped to maintain the ordering guarantee.
+
+The following example demonstrates ordered processing with secondary sorting. First, the function implementation:
+
+```java
+// Function that processes events in order and captures the ordering
+public static class OrderedProcessor extends ProcessTableFunction<List<Event>> {
+ public record Event(Integer score, Instant ts) {}
+
+ public static class BufferState {
+ // Stores all input events that enter the eval() after sorting
+ public List<Event> events = new ArrayList<>();
+ }
+
+ public void eval(
+ Context ctx,
+ @StateHint BufferState state,
+ @ArgumentHint(SET_SEMANTIC_TABLE) Row input
+ ) {
+ // Optional: Access ordering information at runtime
+ TableSemantics semantics = ctx.tableSemanticsFor("input");
+ int[] orderColumns = semantics.orderByColumns();
+ SortDirection[] directions = semantics.orderByDirections();
+
+ // Buffer the incoming row
+ state.events.add(new Event(input.getFieldAs("score"), input.getFieldAs("ts")));
+
+ // Emit current buffer state
+ collect(state.events);
+ }
+}
+```
+
+The function can be called using either SQL or Table API:
+
+{{< tabs "2237eeed-order-by-example" >}}
+{{< tab "SQL" >}}
+```sql
+-- Create a watermarked table
+CREATE TABLE Events (
+ name STRING,
+ score INT,
+ ts TIMESTAMP_LTZ(3),
+ WATERMARK FOR ts AS ts - INTERVAL '1' SECOND
+) WITH (
+ 'connector' = 'datagen'
+);
+
+-- Register the function
+CREATE FUNCTION OrderedProcessor AS 'org.example.OrderedProcessor';
+
+-- Use ORDER BY with primary and secondary sort columns
+SELECT * FROM OrderedProcessor(
+ input => TABLE Events
+ PARTITION BY name
+ ORDER BY (ts ASC, score DESC)
+);
+```
+{{< /tab >}}
+{{< tab "Java" >}}
+```java
+TableEnvironment env = TableEnvironment.create(EnvironmentSettings.inStreamingMode());
+
+// Create a watermarked table
+env.executeSql(
+ "CREATE TABLE Events (" +
+ "name STRING, " +
+ "score INT, " +
+ "ts TIMESTAMP_LTZ(3), " +
+ "WATERMARK FOR ts AS ts - INTERVAL '1' SECOND" +
+ ") WITH ('connector' = 'datagen')"
+);
+
+// Use orderBy() with primary and secondary sort columns
+env.from("Events")
+ .partitionBy($("name"))
+ .orderBy($("ts").asc(), $("score").desc())
+ .process(OrderedProcessor.class)
+ .execute()
+ .print();
+```
+{{< /tab >}}
+{{< /tabs >}}
+
+In this example:
+- Events are first sorted by `ts` (ascending), ensuring event-time order
+- Events with the same timestamp are then sorted by `score`
+- Late events (with timestamp less than the watermark) are automatically dropped
+- The `TableSemantics` API provides runtime access to the ordering configuration
+- The output is an ever-growing list of sorted input events
+
+{{< top >}}
+
Multiple Tables
---------------
diff --git a/docs/content/docs/dev/table/functions/ptfs.md b/docs/content/docs/dev/table/functions/ptfs.md
index ac41b8a1147..1b919dbc0d3 100644
--- a/docs/content/docs/dev/table/functions/ptfs.md
+++ b/docs/content/docs/dev/table/functions/ptfs.md
@@ -838,7 +838,8 @@ class can be passed as an argument to `timeContext()`.
Every PTF takes an optional `on_time` argument. The `on_time` argument in the function call declares the time attribute
column for which a watermark has been declared. When processing a table's row, this timestamp can be accessed via
-`TimeContext#time()` and the watermark via `TimeContext#currentWatermark()` respectively.
+`TimeContext#time()` and the watermark via `TimeContext#currentWatermark()`/`TimeContext#tableWatermark()`
+respectively.
Specifying an `on_time` argument in the function call instructs the framework to return a `rowtime` column in the
function's output for subsequent time-based operations.
@@ -932,22 +933,37 @@ table.
The timestamp of the firing timer currently being processed within the `onTimer()` method.
+#### Table Watermark
+
+`TimeContext#tableWatermark()` returns the event-time watermark of the input table currently being processed.
+
+Watermarks are generated in sources and sent through the topology for advancing the logical clock. The current watermark
+of an input table is the minimum watermark of all upstream Flink subtasks producing the table.
+
+In multi-input scenarios, each input table can have its own independent watermark. This method returns the watermark
+specific to the input table that is currently being processed in the `eval()` method, rather than the global minimum
+watermark across all input tables (which is returned by `currentWatermark()`).
+
+This is particularly useful for late event detection on a per-input basis.
+
+It returns the current watermark of the input table being processed. `null` if called within the `onTimer()` method or a
+watermark has not yet been received from all upstream Flink subtasks producing the table.
+
#### Current Watermark
-`TimeContext#currentWatermark()` returns the current event-time watermark.
+`TimeContext#currentWatermark()` returns the current event-time watermark at this PTF instance.
-Watermarks are generated in sources and sent through the topology for advancing the logical clock in each Flink subtask.
-The current watermark of a Flink subtask is the global minimum watermark of all inputs (i.e. across all parallel inputs
-and table partitions).
+Watermarks are generated in sources and sent through the topology for advancing the logical clock. The current watermark
+of a PTF instance is the global minimum watermark of all input tables (i.e., across all upstream Flink subtasks and
+table partitions).
This method returns the current watermark of the Flink subtask that evaluates the PTF. Thus, the returned timestamp
-represents the entire Flink subtask, independent of the currently processed partition. This behavior is similar to a call
-to `SELECT CURRENT_WATERMARK(...)` in SQL.
+represents the entire Flink subtask, independent of the currently processed input table and partition. This behavior is
+similar to a call to `SELECT CURRENT_WATERMARK(...)` in SQL.
-If a watermark was not received from all inputs, the method returns `null`.
-
-In case this method is called within the `onTimer()` method, the returned watermark is the triggering watermark
-that currently fires the timer.
+It returns the current watermark at the PTF instance across all upstream Flink subtasks and table partitions. A `null`
+value is returned if no minimum logical time could be calculated across all inputs; this happens during startup
+or recovery when one or more active (i.e. not idle) inputs haven't sent a watermark yet.
### Timers
@@ -1015,6 +1031,149 @@ not needed anymore via `Context#clearAllTimers()` or `TimeContext#clearTimer(Str
{{< top >}}
+Ordering
+--------
+
+A PTF that takes a table with set semantics can optionally specify an ORDER BY clause in the
+function call to define the order in which rows are processed within each partition. The ORDER BY
+clause guarantees that rows are delivered to the eval() method in the specified order.
+
+The ORDER BY clause requires that the first column is a time attribute column (i.e., a
+TIMESTAMP or TIMESTAMP_LTZ column with a watermark declaration). The first ORDER BY column must
+be specified in ascending order. This ensures that rows are processed in event-time order.
+Additional columns can be specified as secondary sort keys to define the ordering of
+rows with the same timestamp.
+
+{{< tabs "2137eeed-order-by-syntax" >}}
+{{< tab "SQL" >}}
+```sql
+SELECT * FROM my_ptf(
+ input_table => TABLE source_table
+ PARTITION BY user_id
+ ORDER BY (event_time ASC, priority DESC NULLS FIRST)
+)
+```
+{{< /tab >}}
+{{< tab "Java" >}}
+```java
+env.from("source_table")
+ .partitionBy($("user_id"))
+ .orderBy($("event_time").asc(), $("priority").desc())
+ .process(MyPTF.class, descriptor("event_time").asArgument("on_time"))
+```
+{{< /tab >}}
+{{< /tabs >}}
+
+### Difference Between ORDER BY and on_time Argument
+
+While both ORDER BY and the `on_time` argument relate to time attributes, they serve
+different purposes:
+
+- **on_time**: Declares which time attribute column powers the time context (`TimeContext#time()`) and
+ output timestamp. It does NOT affect the processing order of rows.
+- **ORDER BY**: Physically buffers and sorts rows within each partition to guarantee ordered delivery
+ to the eval() method. If both ORDER BY and `on_time` are specified for the same table argument, they
+ must reference the same time attribute column.
+
+### Ordering Guarantees and Late Events
+
+When ORDER BY is specified on a time attribute column, the framework maintains a sort buffer
+per partition and input table to reorder out-of-order events. The sort buffer is flushed when the
+watermark for the given input table advances, at which point all buffered rows with timestamps
+less than or equal to the watermark are delivered to the eval() method in sorted order. Late
+events (arriving after the watermark) are dropped to maintain the ordering guarantee.
+
+The following example demonstrates ordered processing with secondary sorting. First, the function implementation:
+
+```java
+// Function that processes events in order and captures the ordering
+public static class OrderedProcessor extends ProcessTableFunction<List<Event>> {
+ public record Event(Integer score, Instant ts) {}
+
+ public static class BufferState {
+ // Stores all input events that enter the eval() after sorting
+ public List<Event> events = new ArrayList<>();
+ }
+
+ public void eval(
+ Context ctx,
+ @StateHint BufferState state,
+ @ArgumentHint(SET_SEMANTIC_TABLE) Row input
+ ) {
+ // Optional: Access ordering information at runtime
+ TableSemantics semantics = ctx.tableSemanticsFor("input");
+ int[] orderColumns = semantics.orderByColumns();
+ SortDirection[] directions = semantics.orderByDirections();
+
+ // Buffer the incoming row
+ state.events.add(new Event(input.getFieldAs("score"), input.getFieldAs("ts")));
+
+ // Emit current buffer state
+ collect(state.events);
+ }
+}
+```
+
+The function can be called using either SQL or Table API:
+
+{{< tabs "2237eeed-order-by-example" >}}
+{{< tab "SQL" >}}
+```sql
+-- Create a watermarked table
+CREATE TABLE Events (
+ name STRING,
+ score INT,
+ ts TIMESTAMP_LTZ(3),
+ WATERMARK FOR ts AS ts - INTERVAL '1' SECOND
+) WITH (
+ 'connector' = 'datagen'
+);
+
+-- Register the function
+CREATE FUNCTION OrderedProcessor AS 'org.example.OrderedProcessor';
+
+-- Use ORDER BY with primary and secondary sort columns
+SELECT * FROM OrderedProcessor(
+ input => TABLE Events
+ PARTITION BY name
+ ORDER BY (ts ASC, score DESC)
+);
+```
+{{< /tab >}}
+{{< tab "Java" >}}
+```java
+TableEnvironment env = TableEnvironment.create(EnvironmentSettings.inStreamingMode());
+
+// Create a watermarked table
+env.executeSql(
+ "CREATE TABLE Events (" +
+ "name STRING, " +
+ "score INT, " +
+ "ts TIMESTAMP_LTZ(3), " +
+ "WATERMARK FOR ts AS ts - INTERVAL '1' SECOND" +
+ ") WITH ('connector' = 'datagen')"
+);
+
+// Use orderBy() with primary and secondary sort columns
+env.from("Events")
+ .partitionBy($("name"))
+ .orderBy($("ts").asc(), $("score").desc())
+ .process(OrderedProcessor.class)
+ .execute()
+ .print();
+```
+{{< /tab >}}
+{{< /tabs >}}
+
+In this example:
+- Events are first sorted by `ts` (ascending), ensuring event-time order
+- Events with the same timestamp are then sorted by `score`
+- Late events (with timestamp less than the watermark) are automatically dropped
+- The `TableSemantics` API provides runtime access to the ordering configuration
+- The output is an ever-growing list of sorted input events
+
+{{< top >}}
+
Multiple Tables
---------------
@@ -1165,7 +1324,7 @@ END;
Pass-Through Columns
--------------------
-Depending on the table semantics and whether an `on_time` argument has been defined, the system adds addition columns for
+Depending on the table semantics and whether an `on_time` argument has been defined, the system adds additional columns for
every function output.
For table arguments with set semantics, the output is prefixed with the `PARTITION BY` columns.
diff --git a/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/PartitionedTable.java b/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/PartitionedTable.java
index 32180e51fb9..a83b29fecad 100644
--- a/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/PartitionedTable.java
+++ b/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/PartitionedTable.java
@@ -20,6 +20,7 @@ package org.apache.flink.table.api;
import org.apache.flink.annotation.PublicEvolving;
import org.apache.flink.table.annotation.ArgumentTrait;
+import org.apache.flink.table.expressions.Expression;
import org.apache.flink.table.functions.ProcessTableFunction;
import org.apache.flink.table.functions.UserDefinedFunction;
@@ -35,6 +36,36 @@ import org.apache.flink.table.functions.UserDefinedFunction;
@PublicEvolving
public interface PartitionedTable {
+ /**
+ * Sorts the partitioned table within each partition.
+ *
+ * <p>This is useful for process table functions that need ordered input within each set scoped
+ * by the key.
+ *
+ * <p>Note: The first sorting column must be a time attribute column in ascending order for
+ * which a watermark has been declared. The watermarked column must be forwarded without any
+ * modification to ensure event-time sorting. Secondary ordering columns can differ and are used
+ * for deterministic ordering within the same timestamp.
+ *
+ * <p>Example:
+ *
+ * <pre>{@code
+ * // Primary ordering by watermarked timestamp, secondary by score
+ * env.fromCall(
+ * "MyPTF",
+ * table
+ * .partitionBy($("key"))
+ * .orderBy($("ts").asc(), $("score").desc())
+ * .asArgument("input_table")
+ * )
+ * }</pre>
+ *
+ * @param fields expressions for ordering (e.g., {@code $("ts").asc(), $("score").desc()})
+ * @return a partitioned and ordered table
+ * @see ProcessTableFunction
+ */
+ PartitionedTable orderBy(Expression... fields);
+
/**
* Converts this table object into a named argument.
*
@@ -50,8 +81,8 @@ public interface PartitionedTable {
* )
* }</pre>
*
- * @see ProcessTableFunction
* @return an expression that can be passed into {@link TableEnvironment#fromCall}.
+ * @see ProcessTableFunction
*/
ApiExpression asArgument(String name);
diff --git a/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/internal/TableImpl.java b/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/internal/TableImpl.java
index 73d70a83dfd..f63d4032067 100644
--- a/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/internal/TableImpl.java
+++ b/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/internal/TableImpl.java
@@ -878,16 +878,27 @@ public class TableImpl implements Table {
private final TableImpl table;
private final List<Expression> partitionKeys;
+ private final List<Expression> orderKeys;
private PartitionedTableImpl(TableImpl table, List<Expression> partitionKeys) {
+ this(table, partitionKeys, Collections.emptyList());
+ }
+
+ private PartitionedTableImpl(
+ TableImpl table, List<Expression> partitionKeys, List<Expression> orderKeys) {
this.table = table;
this.partitionKeys = partitionKeys;
+ this.orderKeys = orderKeys;
+ }
+
+ @Override
+ public PartitionedTable orderBy(Expression... fields) {
+ return new PartitionedTableImpl(table, partitionKeys, Arrays.asList(fields));
}
@Override
public ApiExpression asArgument(String name) {
- return createArgumentExpression(
- createPartitionQueryOperation(), table.tableEnvironment, name);
+ return createArgumentExpression(createQueryOperation(), table.tableEnvironment, name);
}
@Override
@@ -895,7 +906,7 @@ public class TableImpl implements Table {
return table.tableEnvironment.fromCall(
path,
unionTableAndArguments(
- createPartitionQueryOperation(), table.tableEnvironment, arguments));
+ createQueryOperation(), table.tableEnvironment, arguments));
}
@Override
@@ -903,11 +914,15 @@ public class TableImpl implements Table {
return table.tableEnvironment.fromCall(
function,
unionTableAndArguments(
- createPartitionQueryOperation(), table.tableEnvironment, arguments));
+ createQueryOperation(), table.tableEnvironment, arguments));
}
- private QueryOperation createPartitionQueryOperation() {
- return table.operationTreeBuilder.partition(partitionKeys, table.operationTree);
+ private QueryOperation createQueryOperation() {
+ if (orderKeys.isEmpty()) {
+ return table.operationTreeBuilder.partition(partitionKeys, table.operationTree);
+ }
+ return table.operationTreeBuilder.partition(
+ partitionKeys, orderKeys, table.operationTree);
}
}
diff --git a/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/expressions/resolver/rules/ResolveCallByArgumentsRule.java b/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/expressions/resolver/rules/ResolveCallByArgumentsRule.java
index 2af2a97399d..be8e578deea 100644
--- a/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/expressions/resolver/rules/ResolveCallByArgumentsRule.java
+++ b/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/expressions/resolver/rules/ResolveCallByArgumentsRule.java
@@ -28,6 +28,7 @@ import org.apache.flink.table.connector.ChangelogMode;
import org.apache.flink.table.expressions.CallExpression;
import org.apache.flink.table.expressions.Expression;
import org.apache.flink.table.expressions.ExpressionUtils;
+import org.apache.flink.table.expressions.FieldReferenceExpression;
import org.apache.flink.table.expressions.ModelReferenceExpression;
import org.apache.flink.table.expressions.ResolvedExpression;
import org.apache.flink.table.expressions.TableReferenceExpression;
@@ -745,16 +746,29 @@ final class ResolveCallByArgumentsRule implements ResolverRule {
@Override
public int[] partitionByColumns() {
- if (!(operation instanceof PartitionQueryOperation)) {
+ final PartitionQueryOperation partitionOperation = findPartitionOperation(operation);
+ if (partitionOperation == null) {
return new int[0];
}
- final PartitionQueryOperation partitionOperation = (PartitionQueryOperation) operation;
return partitionOperation.getPartitionKeys();
}
@Override
public int[] orderByColumns() {
- return new int[0];
+ final PartitionQueryOperation partitionOperation = findPartitionOperation(operation);
+ if (partitionOperation == null) {
+ return new int[0];
+ }
+ return extractOrderByColumns(partitionOperation);
+ }
+
+ @Override
+ public SortDirection[] orderByDirections() {
+ final PartitionQueryOperation partitionOperation = findPartitionOperation(operation);
+ if (partitionOperation == null) {
+ return new SortDirection[0];
+ }
+ return extractOrderByDirections(partitionOperation);
}
@Override
@@ -766,6 +780,63 @@ final class ResolveCallByArgumentsRule implements ResolverRule {
public Optional<ChangelogMode> changelogMode() {
return Optional.empty();
}
+
+ private PartitionQueryOperation findPartitionOperation(QueryOperation op) {
+ if (op instanceof PartitionQueryOperation) {
+ return (PartitionQueryOperation) op;
+ }
+ return null;
+ }
+
+ private int[] extractOrderByColumns(PartitionQueryOperation partitionOperation) {
+ return partitionOperation.getOrderExpressions().stream()
+ .mapToInt(this::extractFieldIndex)
+ .toArray();
+ }
+
+ private SortDirection[] extractOrderByDirections(
+ PartitionQueryOperation partitionOperation) {
+ return partitionOperation.getOrderExpressions().stream()
+ .map(this::extractSortDirection)
+ .toArray(SortDirection[]::new);
+ }
+
+ private int extractFieldIndex(ResolvedExpression orderExpr) {
+ // Order expressions are typically CallExpressions wrapping ORDER_ASC or ORDER_DESC
+ if (orderExpr instanceof CallExpression) {
+ final CallExpression call = (CallExpression) orderExpr;
+ if (call.getChildren().size() == 1) {
+ final Expression child = call.getChildren().get(0);
+ if (child instanceof FieldReferenceExpression) {
+ return ((FieldReferenceExpression) child).getFieldIndex();
+ }
+ }
+ }
+ // Fallback: if it's directly a field reference
+ if (orderExpr instanceof FieldReferenceExpression) {
+ return ((FieldReferenceExpression) orderExpr).getFieldIndex();
+ }
+ throw new TableException(
+ "Unable to extract field index from order expression: " + orderExpr);
+ }
+
+ private SortDirection extractSortDirection(ResolvedExpression orderExpr) {
+ // Check if wrapped in ORDER_ASC or ORDER_DESC
+ if (orderExpr instanceof CallExpression) {
+ final CallExpression call = (CallExpression) orderExpr;
+ final FunctionDefinition functionDef = call.getFunctionDefinition();
+
+ if (functionDef == BuiltInFunctionDefinitions.ORDER_DESC) {
+ // DESC defaults to NULLS FIRST in SQL
+ return SortDirection.DESC_NULLS_FIRST;
+ } else if (functionDef == BuiltInFunctionDefinitions.ORDER_ASC) {
+ // ASC defaults to NULLS LAST in SQL
+ return SortDirection.ASC_NULLS_LAST;
+ }
+ }
+ // Default is ascending with nulls last
+ return SortDirection.ASC_NULLS_LAST;
+ }
}
private static class TableApiModelSemantics implements ModelSemantics {
diff --git a/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/operations/PartitionQueryOperation.java b/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/operations/PartitionQueryOperation.java
index 0ba38fabab5..a248ab81ea2 100644
--- a/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/operations/PartitionQueryOperation.java
+++ b/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/operations/PartitionQueryOperation.java
@@ -19,10 +19,16 @@
package org.apache.flink.table.operations;
import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableException;
import org.apache.flink.table.catalog.ResolvedSchema;
+import org.apache.flink.table.expressions.CallExpression;
+import org.apache.flink.table.expressions.Expression;
import org.apache.flink.table.expressions.FieldReferenceExpression;
import org.apache.flink.table.expressions.ResolvedExpression;
import org.apache.flink.table.expressions.SqlFactory;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.FunctionDefinition;
+import org.apache.flink.table.functions.TableSemantics.SortDirection;
import java.util.LinkedHashMap;
import java.util.List;
@@ -34,11 +40,20 @@ import java.util.stream.Collectors;
public class PartitionQueryOperation implements QueryOperation {
private final List<ResolvedExpression> partitionExpressions;
+ private final List<ResolvedExpression> orderExpressions;
private final QueryOperation child;
public PartitionQueryOperation(
List<ResolvedExpression> partitionExpressions, QueryOperation child) {
+ this(partitionExpressions, List.of(), child);
+ }
+
+ public PartitionQueryOperation(
+ List<ResolvedExpression> partitionExpressions,
+ List<ResolvedExpression> orderExpressions,
+ QueryOperation child) {
this.partitionExpressions = partitionExpressions;
+ this.orderExpressions = orderExpressions;
this.child = child;
}
@@ -50,14 +65,80 @@ public class PartitionQueryOperation implements QueryOperation {
.toArray();
}
+ public List<ResolvedExpression> getOrderExpressions() {
+ return orderExpressions;
+ }
+
+ public int[] getOrderKeys() {
+ return orderExpressions.stream().mapToInt(this::extractFieldIndex).toArray();
+ }
+
+ public SortDirection[] getOrderDirections() {
+ return orderExpressions.stream()
+ .map(this::extractSortDirection)
+ .toArray(SortDirection[]::new);
+ }
+
+ private int extractFieldIndex(ResolvedExpression orderExpr) {
+ // Order expressions are typically CallExpressions wrapping ORDER_ASC or ORDER_DESC
+ if (orderExpr instanceof CallExpression) {
+ final CallExpression call = (CallExpression) orderExpr;
+ if (call.getChildren().size() == 1) {
+ final Expression child = call.getChildren().get(0);
+ if (child instanceof FieldReferenceExpression) {
+ return ((FieldReferenceExpression) child).getFieldIndex();
+ }
+ }
+ }
+ // Fallback: if it's directly a field reference
+ if (orderExpr instanceof FieldReferenceExpression) {
+ return ((FieldReferenceExpression) orderExpr).getFieldIndex();
+ }
+ throw new TableException(
+ "Unable to extract field index from order expression: " + orderExpr);
+ }
+
+ private SortDirection extractSortDirection(ResolvedExpression orderExpr) {
+ // Check if wrapped in ORDER_ASC or ORDER_DESC
+ if (orderExpr instanceof CallExpression) {
+ final CallExpression call = (CallExpression) orderExpr;
+ final FunctionDefinition functionDef = call.getFunctionDefinition();
+
+ if (functionDef == BuiltInFunctionDefinitions.ORDER_DESC) {
+ return SortDirection.DESC_NULLS_FIRST;
+ } else if (functionDef == BuiltInFunctionDefinitions.ORDER_ASC) {
+ return SortDirection.ASC_NULLS_LAST;
+ }
+ }
+ // Default to ASC if no explicit direction
+ return SortDirection.ASC_NULLS_LAST;
+ }
+
@Override
public String asSerializableString(SqlFactory sqlFactory) {
- return String.format(
- "(%s\n) PARTITION BY (%s)",
- OperationUtils.indent(child.asSerializableString(sqlFactory)),
- partitionExpressions.stream()
- .map(expr -> expr.asSerializableString(sqlFactory))
- .collect(Collectors.joining(", ")));
+ String result =
+ String.format(
+ "(%s\n)", OperationUtils.indent(child.asSerializableString(sqlFactory)));
+
+ if (!partitionExpressions.isEmpty()) {
+ result +=
+ " PARTITION BY ("
+ + partitionExpressions.stream()
+ .map(expr -> expr.asSerializableString(sqlFactory))
+ .collect(Collectors.joining(", "))
+ + ")";
+ }
+
+ if (!orderExpressions.isEmpty()) {
+ result +=
+ " ORDER BY ("
+ + orderExpressions.stream()
+ .map(expr -> expr.asSerializableString(sqlFactory))
+ .collect(Collectors.joining(", "))
+ + ")";
+ }
+
+ return result;
}
@Override
diff --git a/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/operations/utils/OperationTreeBuilder.java b/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/operations/utils/OperationTreeBuilder.java
index f54c9c51f32..2cf8691f165 100644
--- a/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/operations/utils/OperationTreeBuilder.java
+++ b/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/operations/utils/OperationTreeBuilder.java
@@ -616,6 +616,14 @@ public final class OperationTreeBuilder {
return new PartitionQueryOperation(resolvedPartitionKeys, child);
}
+ public QueryOperation partition(
+ List<Expression> partitionKeys, List<Expression> orderKeys, QueryOperation child) {
+ final ExpressionResolver resolver = getResolverBuilder(child).build();
+ final List<ResolvedExpression> resolvedPartitionKeys = resolver.resolve(partitionKeys);
+ final List<ResolvedExpression> resolvedOrderKeys = resolver.resolve(orderKeys);
+ return new PartitionQueryOperation(resolvedPartitionKeys, resolvedOrderKeys, child);
+ }
+
private QueryOperation valuesInternal(
@Nullable ResolvedSchema valuesSchema, Expression... expressions) {
if (expressions.length == 0) {
diff --git a/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/ProcessTableFunction.java b/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/ProcessTableFunction.java
index 19079315579..bec4bcc8593 100644
--- a/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/ProcessTableFunction.java
+++ b/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/ProcessTableFunction.java
@@ -351,7 +351,8 @@ import java.time.LocalDateTime;
* <p>Every PTF takes an optional {@code on_time} argument. The {@code on_time} argument in the
* function call declares the time attribute column for which a watermark has been declared. When
* processing a table's row, this timestamp can be accessed via {@link TimeContext#time()} and the
- * watermark via {@link TimeContext#currentWatermark()} respectively.
+ * watermark via {@link TimeContext#currentWatermark()}/{@link TimeContext#tableWatermark()}
+ * respectively.
*
* <p>Specifying an {@code on_time} argument in the function call instructs the framework to return
* a {@code rowtime} column in the function's output for subsequent time-based operations.
@@ -410,8 +411,57 @@ import java.time.LocalDateTime;
* not needed anymore via {@link Context#clearAllTimers()} or {@link
* TimeContext#clearTimer(String)}.
*
- * @param <T> The type of the output row. Either an explicit composite type or an atomic type that
- * is implicitly wrapped into a row consisting of one field.
+ * <h1>Ordering</h1>
+ *
+ * <p>A PTF that takes a table with set semantics can optionally specify an ORDER BY clause in the
+ * function call to define the order in which rows are processed within each partition. The ORDER BY
+ * clause guarantees that rows are delivered to the eval() method in the specified order.
+ *
+ * <p>The ORDER BY clause requires that the first column is a time attribute column (i.e., a
+ * TIMESTAMP or TIMESTAMP_LTZ column with a watermark declaration). The first ORDER BY column must
+ * be specified in ascending order. This ensures that rows are processed in event-time order.
+ * Additional columns can be specified as secondary sort keys to define the ordering of rows with
+ * the same timestamp.
+ *
+ * <p>ORDER BY provides strict time-based ordering at the cost of holding back records until a
+ * watermark advances the logical clock. Consequently, any late data that violates this ordering is
+ * dropped. If you need to process late data or access intermediate results, it is highly
+ * recommended to bypass ORDER BY and implement custom handling within the PTF using state and
+ * timers.
+ *
+ * <p>Example SQL syntax:
+ *
+ * <pre>{@code
+ * SELECT * FROM my_ptf(
+ * input_table => TABLE source_table
+ * PARTITION BY user_id
+ * ORDER BY (event_time ASC, priority DESC NULLS FIRST)
+ * )
+ * }</pre>
+ *
+ * <h2>Difference Between ORDER BY and on_time Argument</h2>
+ *
+ * <p>While both ORDER BY and the {@code on_time} argument relate to time attributes, they serve
+ * different purposes:
+ *
+ * <ul>
+ * <li><b>on_time</b>: Declares which time attribute column powers the time context ({@link
+ * TimeContext#time()}) and output timestamp. It does NOT affect the processing order of rows.
+ * <li><b>ORDER BY</b>: Physically buffers and sorts rows within each partition to guarantee
+ * ordered delivery to the eval() method. If both ORDER BY and {@code on_time} are specified
+ * for the same table argument, they must reference the same time attribute column.
+ * </ul>
+ *
+ * <h2>Ordering Guarantees and Late Events</h2>
+ *
+ * <p>When ORDER BY is specified on a time attribute column, the framework maintains a sort buffer
+ * per partition and input table to reorder out-of-order events. The sort buffer is flushed when the
+ * watermark for the given input table advances, at which point all buffered rows with timestamps
+ * less than or equal to the watermark are delivered to the eval() method in sorted order. Late
+ * events (arriving after the watermark) are dropped to maintain the ordering guarantee.
+ *
+ * @param <T> The type of the output row. Either an explicit composite type or an atomic type
+ * implicitly wrapped into a row consisting of one field.
*/
@PublicEvolving
public abstract class ProcessTableFunction<T> extends UserDefinedFunction {
@@ -558,25 +608,47 @@ public abstract class ProcessTableFunction<T> extends UserDefinedFunction {
TimeType time();
/**
- * Returns the current event-time watermark.
+ * Returns the event-time watermark of the input table currently being processed.
*
* <p>Watermarks are generated in sources and sent through the topology for advancing the
- * logical clock in each Flink subtask. The current watermark of a Flink subtask is the
- * global minimum watermark of all inputs (i.e. across all parallel inputs and table
- * partitions).
+ * logical clock. The current watermark of an input table is the minimum watermark of all
+ * upstream Flink subtasks producing the table.
*
- * <p>This method returns the current watermark of the Flink subtask that evaluates the PTF.
- * Thus, the returned timestamp represents the entire Flink subtask, independent of the
- * currently processed partition. This behavior is similar to a call to {@code SELECT
- * CURRENT_WATERMARK(...)} in SQL.
+ * <p>In multi-input scenarios, each input table can have its own independent watermark.
+ * This method returns the watermark specific to the input table that is currently being
+ * processed in the {@code eval()} method, rather than the global minimum watermark across
+ * all input tables (which is returned by {@link #currentWatermark()}).
+ *
+ * <p>This is particularly useful for late event detection on a per-input basis.
+ *
+ * <p>If a watermark has not yet been received from all upstream Flink subtasks producing
+ * the table, the method returns {@code null}.
+ *
+ * @return The current watermark of the input table being processed. {@code null} if called
+ * within the {@code onTimer()} method or a watermark has not yet been received from all
+ * upstream Flink subtasks producing the table.
+ */
+ TimeType tableWatermark();
+
+ /**
+ * Returns the current event-time watermark at this PTF instance.
*
- * <p>If a watermark was not received from all inputs, the method returns {@code null}.
+ * <p>Watermarks are generated in sources and sent through the topology for advancing the
+ * logical clock. The current watermark of a PTF instance is the global minimum watermark of
+ * all input tables (i.e., across all upstream Flink subtasks and table partitions).
+ *
+ * <p>This method returns the current watermark of the Flink subtask that evaluates the PTF.
+ * Thus, the returned timestamp represents the watermark of the entire Flink subtask,
+ * independent of the currently processed input table and partition. This behavior is
+ * similar to a call to {@code SELECT CURRENT_WATERMARK(...)} in SQL.
*
- * <p>In case this method is called within the {@code onTimer()} method, the returned
- * watermark is the triggering watermark that currently fires the timer.
+ * <p>If a watermark has not yet been received from all input tables, the method returns
+ * {@code null}.
*
- * @return the current watermark of the Flink subtask, or {@code null} if no common logical
- * time could be determined from the inputs
+ * @return The current watermark at the PTF instance across all upstream Flink subtasks and
+ * table partitions. A null value is returned if no minimum logical time could be
+ * calculated across all inputs; this happens during startup or recovery when one or
+ * more active (i.e. not idle) inputs haven't sent a watermark yet.
*/
TimeType currentWatermark();
diff --git a/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableSemantics.java b/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableSemantics.java
index 53732d4f691..f63566befce 100644
--- a/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableSemantics.java
+++ b/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableSemantics.java
@@ -94,6 +94,17 @@ public interface TableSemantics {
*/
int[] orderByColumns();
+ /**
+ * Returns information about the sort direction for each ORDER BY column. Applies only to table
+ * arguments with set semantics.
+ *
+ * <p>The returned array has the same length as {@link #orderByColumns()} and each element
+ * corresponds to the sort direction of the column at the same index.
+ *
+ * @return An array of {@link SortDirection} values corresponding to the ORDER BY columns.
+ */
+ SortDirection[] orderByDirections();
+
/**
* Returns information about the time attribute of the passed table. The time attribute column
* powers the concept of rowtime and timers. Applies to both table arguments with row and set
@@ -116,4 +127,40 @@ public interface TableSemantics {
* type inference phase as the changelog mode is still unknown.
*/
Optional<ChangelogMode> changelogMode();
+
+ /** The sort direction for ORDER BY columns in table arguments with set semantics. */
+ @PublicEvolving
+ enum SortDirection {
+ /** Ascending order with nulls first. */
+ ASC_NULLS_FIRST(false, true),
+ /** Ascending order with nulls last. */
+ ASC_NULLS_LAST(false, false),
+ /** Descending order with nulls first. */
+ DESC_NULLS_FIRST(true, true),
+ /** Descending order with nulls last. */
+ DESC_NULLS_LAST(true, false);
+
+ private final boolean descending;
+ private final boolean nullsFirst;
+
+ SortDirection(boolean descending, boolean nullsFirst) {
+ this.descending = descending;
+ this.nullsFirst = nullsFirst;
+ }
+
+ /** Returns true if this is a descending sort direction. */
+ public boolean isDescending() {
+ return descending;
+ }
+
+ /** Returns true if nulls should be sorted first. */
+ public boolean isNullsFirst() {
+ return nullsFirst;
+ }
+
+ /** Returns true if nulls should be sorted last. */
+ public boolean isNullsLast() {
+ return !nullsFirst;
+ }
+ }
}
diff --git a/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/SystemTypeInference.java b/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/SystemTypeInference.java
index 27dd4b9b981..92fd79f407b 100644
--- a/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/SystemTypeInference.java
+++ b/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/SystemTypeInference.java
@@ -27,6 +27,7 @@ import org.apache.flink.table.functions.FunctionDefinition;
import org.apache.flink.table.functions.FunctionKind;
import org.apache.flink.table.functions.ProcessTableFunction;
import org.apache.flink.table.functions.TableSemantics;
+import org.apache.flink.table.functions.TableSemantics.SortDirection;
import org.apache.flink.table.types.DataType;
import org.apache.flink.table.types.logical.LocalZonedTimestampType;
import org.apache.flink.table.types.logical.LogicalType;
@@ -374,14 +375,20 @@ public class SystemTypeInference {
if (!staticArg.is(StaticArgumentTrait.TABLE)) {
return;
}
+ final String argName = staticArg.getName();
final RowType rowType =
LogicalTypeUtils.toRowType(args.get(pos).getLogicalType());
final int onTimeColumn =
- findUniqueOnTimeColumn(
- staticArg.getName(), rowType, onTimeFields);
+ findUniqueOnTimeColumn(argName, rowType, onTimeFields);
if (onTimeColumn >= 0) {
usedOnTimeFields.add(rowType.getFieldNames().get(onTimeColumn));
onTimeColumns.add(rowType.getTypeAt(onTimeColumn));
+ // If ORDER BY is defined, it must match the on_time column
+ final TableSemantics semantics =
+ callContext
+ .getTableSemantics(pos)
+ .orElseThrow(IllegalStateException::new);
+ checkOrderByMatchesOnTime(argName, semantics, onTimeColumn);
return;
}
if (staticArg.is(StaticArgumentTrait.REQUIRE_ON_TIME)) {
@@ -390,9 +397,9 @@ public class SystemTypeInference {
"Table argument '%s' requires a time attribute. "
+ "Please provide one using the implicit `on_time` argument. "
+ "For example: myFunction(..., on_time => DESCRIPTOR(`my_timestamp`)",
- staticArg.getName()));
+ argName));
} else {
- missingOnTimeColumns.add(staticArg.getName());
+ missingOnTimeColumns.add(argName);
}
});
@@ -500,9 +507,21 @@ public class SystemTypeInference {
}
}
- private static boolean isUnsupportedOnTimeColumn(LogicalType type) {
- return !LogicalTypeChecks.canBeTimeAttributeType(type)
- || LogicalTypeChecks.getPrecision(type) > 3;
+ private static void checkOrderByMatchesOnTime(
+ String argName, TableSemantics semantics, int onTimeColumn) {
+ final int[] orderByColumns = semantics.orderByColumns();
+ if (orderByColumns.length == 0) {
+ return;
+ }
+ final int firstOrderByColumn = orderByColumns[0];
+ if (onTimeColumn != firstOrderByColumn) {
+ throw new ValidationException(
+ String.format(
+ "Invalid ORDER BY clause for table argument '%s'. "
+ + "When both ORDER BY and the 'on_time' argument are defined, "
+ + "the referenced timestamp columns must match.",
+ argName));
+ }
}
}
@@ -667,6 +686,64 @@ public class SystemTypeInference {
"Table argument '%s' requires a PARTITION BY clause for parallel processing.",
staticArg.getName()));
}
+
+ checkOrderBy(staticArg, semantics);
+ }
+
+ private static void checkOrderByDuplicates(
+ StaticArgument staticArg, List<String> fieldNames, int[] orderByColumns) {
+ final Set<Integer> seenColumns = new HashSet<>();
+ for (int columnIndex : orderByColumns) {
+ if (!seenColumns.add(columnIndex)) {
+ throw new ValidationException(
+ String.format(
+ "Invalid ORDER BY clause for table argument '%s'. "
+ + "Column '%s' appears more than once in ORDER BY.",
+ staticArg.getName(), fieldNames.get(columnIndex)));
+ }
+ }
+ }
+
+ private static void checkOrderBy(StaticArgument staticArg, TableSemantics semantics) {
+ final int[] orderByColumns = semantics.orderByColumns();
+ if (orderByColumns.length == 0) {
+ return;
+ }
+
+ final LogicalType tableType = semantics.dataType().getLogicalType();
+ final List<String> fieldNames = LogicalTypeChecks.getFieldNames(tableType);
+ final List<LogicalType> fieldTypes = LogicalTypeChecks.getFieldTypes(tableType);
+
+ checkOrderByDuplicates(staticArg, fieldNames, orderByColumns);
+
+ // The first ORDER BY column must be a supported time attribute
+ final int firstOrderByColumn = orderByColumns[0];
+ final LogicalType firstOrderByType = fieldTypes.get(firstOrderByColumn);
+ if (isUnsupportedOnTimeColumn(firstOrderByType)) {
+ throw new ValidationException(
+ String.format(
+ "Invalid ORDER BY clause for table argument '%s'. "
+ + "The first ORDER BY column must be a TIMESTAMP or TIMESTAMP_LTZ column (up to precision 3). "
+ + "However, column '%s' has data type '%s'.",
+ staticArg.getName(),
+ fieldNames.get(firstOrderByColumn),
+ firstOrderByType.asSummaryString()));
+ }
+
+ // The first ORDER BY column must be ascending
+ final SortDirection[] orderByDirections = semantics.orderByDirections();
+ if (orderByDirections[0] != SortDirection.ASC_NULLS_LAST) {
+ throw new ValidationException(
+ String.format(
+ "Invalid ORDER BY clause for table argument '%s'. "
+ + "The first ORDER BY column on a timestamp column must be in ascending order.",
+ staticArg.getName()));
+ }
}
}
+
+ private static boolean isUnsupportedOnTimeColumn(LogicalType type) {
+ return !LogicalTypeChecks.canBeTimeAttributeType(type)
+ || LogicalTypeChecks.getPrecision(type) > 3;
+ }
}
diff --git a/flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/inference/utils/TableSemanticsMock.java b/flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/inference/utils/TableSemanticsMock.java
index 1a3a1e06ab1..fe881f8fd1f 100644
--- a/flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/inference/utils/TableSemanticsMock.java
+++ b/flink-table/flink-table-common/src/test/java/org/apache/flink/table/types/inference/utils/TableSemanticsMock.java
@@ -32,6 +32,7 @@ public class TableSemanticsMock implements TableSemantics {
private final DataType dataType;
private final int[] partitionByColumns;
private final int[] orderByColumns;
+ private final SortDirection[] orderByDirections;
private final int timeColumn;
private final ChangelogMode changelogMode;
@@ -48,6 +49,10 @@ public class TableSemanticsMock implements TableSemantics {
this.dataType = dataType;
this.partitionByColumns = partitionByColumns;
this.orderByColumns = orderByColumns;
+ this.orderByDirections = new SortDirection[orderByColumns.length];
+ for (int i = 0; i < orderByColumns.length; i++) {
+ this.orderByDirections[i] = SortDirection.ASC_NULLS_LAST;
+ }
this.timeColumn = timeColumn;
this.changelogMode = changelogMode;
}
@@ -67,6 +72,11 @@ public class TableSemanticsMock implements TableSemantics {
return orderByColumns;
}
+ @Override
+ public SortDirection[] orderByDirections() {
+ return orderByDirections;
+ }
+
@Override
public int timeColumn() {
return timeColumn;
diff --git a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/calcite/FlinkConvertletTable.java b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/calcite/FlinkConvertletTable.java
index 8af320cf673..546d83b480a 100644
--- a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/calcite/FlinkConvertletTable.java
+++ b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/calcite/FlinkConvertletTable.java
@@ -19,6 +19,8 @@
package org.apache.flink.table.planner.calcite;
import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.planner.calcite.RexTableArgCall.OrderByInfo;
+import org.apache.flink.table.planner.calcite.RexTableArgCall.SortOrder;
import org.apache.flink.table.planner.functions.sql.FlinkSqlOperatorTable;
import org.apache.calcite.rel.type.RelDataType;
@@ -175,16 +177,26 @@ public class FlinkConvertletTable implements SqlRexConvertletTable {
final SqlBasicCall setSemanticsCall = (SqlBasicCall) operand;
final SqlNodeList partitionKeys = setSemanticsCall.operand(1);
final SqlNodeList orderKeys = setSemanticsCall.operand(2);
- checkArgument(
- orderKeys.isEmpty(), "Table functions do not support order keys yet.");
- final int[] keys = getPartitionKeyIndices(tableType, partitionKeys);
+ final int[] partitionKeyIndices = getPartitionKeyIndices(tableType, partitionKeys);
+ final OrderByInfo orderByInfo =
+ RexTableArgCall.extractOrderByInfo(tableType.getFieldNames(), orderKeys);
rewrittenOperands.add(
- new RexTableArgCall(tableType, tableInputCount++, keys, new int[0]));
+ new RexTableArgCall(
+ tableType,
+ tableInputCount++,
+ partitionKeyIndices,
+ orderByInfo.columns,
+ orderByInfo.order));
} else if (operand.isA(SqlKind.QUERY)) {
final RelDataType tableType = cx.getValidator().getValidatedNodeType(operand);
rewrittenOperands.add(
- new RexTableArgCall(tableType, tableInputCount++, new int[0], new int[0]));
+ new RexTableArgCall(
+ tableType,
+ tableInputCount++,
+ new int[0],
+ new int[0],
+ new SortOrder[0]));
} else {
rewrittenOperands.add(cx.convertExpression(operand));
}
diff --git a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/calcite/RexTableArgCall.java b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/calcite/RexTableArgCall.java
index 7cf58c36ca7..daa508577f4 100644
--- a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/calcite/RexTableArgCall.java
+++ b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/calcite/RexTableArgCall.java
@@ -18,6 +18,9 @@
package org.apache.flink.table.planner.calcite;
+import org.apache.flink.table.api.CompiledPlan;
+import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.table.functions.TableSemantics.SortDirection;
import org.apache.flink.table.planner.functions.sql.SqlTableArgOperator;
import org.apache.flink.table.types.inference.StaticArgument;
import org.apache.flink.table.types.inference.StaticArgumentTrait;
@@ -25,12 +28,24 @@ import org.apache.flink.table.types.inference.StaticArgumentTrait;
import org.apache.calcite.rel.type.RelDataType;
import org.apache.calcite.rex.RexCall;
import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlNodeList;
+
+import javax.annotation.Nullable;
import java.util.Arrays;
import java.util.List;
import java.util.Objects;
import java.util.stream.Collectors;
+import static org.apache.flink.table.functions.TableSemantics.SortDirection.ASC_NULLS_FIRST;
+import static org.apache.flink.table.functions.TableSemantics.SortDirection.ASC_NULLS_LAST;
+import static org.apache.flink.table.functions.TableSemantics.SortDirection.DESC_NULLS_FIRST;
+import static org.apache.flink.table.functions.TableSemantics.SortDirection.DESC_NULLS_LAST;
+
/**
* A special {@link RexCall} that represents a table argument in a signature of {@link
* StaticArgument}s. The table arguments describe a {@link StaticArgumentTrait#SET_SEMANTIC_TABLE}
@@ -40,15 +55,50 @@ import java.util.stream.Collectors;
*/
public class RexTableArgCall extends RexCall {
+ /**
+ * Sort order for ORDER BY columns.
+ *
+ * <p>Note: Be careful when changing this class. It ends up in {@link CompiledPlan}. This is
+ * also the reason why it has been separated from API-facing {@link SortDirection}.
+ */
+ public enum SortOrder {
+ ASC_NULLS_FIRST,
+ ASC_NULLS_LAST,
+ DESC_NULLS_FIRST,
+ DESC_NULLS_LAST;
+
+ public static SortOrder fromSortDirection(SortDirection direction) {
+ switch (direction) {
+ case ASC_NULLS_FIRST:
+ return ASC_NULLS_FIRST;
+ case ASC_NULLS_LAST:
+ return ASC_NULLS_LAST;
+ case DESC_NULLS_FIRST:
+ return DESC_NULLS_FIRST;
+ case DESC_NULLS_LAST:
+ return DESC_NULLS_LAST;
+ default:
+ throw new IllegalArgumentException("Unknown sort direction: " + direction);
+ }
+ }
+ }
+
private final int inputIndex;
private final int[] partitionKeys;
private final int[] orderKeys;
+ private final SortOrder[] sortOrder;
- public RexTableArgCall(RelDataType type, int inputIndex, int[] partitionKeys, int[] orderKeys) {
+ public RexTableArgCall(
+ RelDataType type,
+ int inputIndex,
+ int[] partitionKeys,
+ int[] orderKeys,
+ SortOrder[] order) {
super(type, SqlTableArgOperator.INSTANCE, List.of());
this.inputIndex = inputIndex;
this.partitionKeys = partitionKeys;
this.orderKeys = orderKeys;
+ this.sortOrder = order;
}
public int getInputIndex() {
@@ -63,6 +113,10 @@ public class RexTableArgCall extends RexCall {
return orderKeys;
}
+ public SortOrder[] getSortOrder() {
+ return sortOrder;
+ }
+
@Override
protected String computeDigest(boolean withType) {
final StringBuilder sb = new StringBuilder(op.getName());
@@ -74,28 +128,46 @@ public class RexTableArgCall extends RexCall {
sb.append(":");
sb.append(type.getFullTypeString());
}
- formatKeys(sb, partitionKeys, " PARTITION BY");
- formatKeys(sb, orderKeys, " ORDER BY");
+ formatPartitionKeys(sb, partitionKeys);
+ formatOrderKeys(sb);
return sb.toString();
}
- private void formatKeys(StringBuilder sb, int[] keys, String prefix) {
+ private void formatPartitionKeys(StringBuilder sb, int[] keys) {
if (keys.length == 0) {
return;
}
sb.append(
Arrays.stream(keys)
.mapToObj(key -> "$" + key)
- .collect(Collectors.joining(", ", prefix + "(", ")")));
+ .collect(Collectors.joining(", ", " PARTITION BY" + "(", ")")));
+ }
+
+ private void formatOrderKeys(StringBuilder sb) {
+ if (orderKeys.length == 0) {
+ return;
+ }
+ sb.append(" ORDER BY(");
+ for (int i = 0; i < orderKeys.length; i++) {
+ if (i > 0) {
+ sb.append(", ");
+ }
+ sb.append("$").append(orderKeys[i]);
+ if (i < sortOrder.length) {
+ sb.append(" ").append(sortOrder[i]);
+ }
+ }
+ sb.append(")");
}
@Override
public RexCall clone(RelDataType type, List<RexNode> operands) {
- return new RexTableArgCall(type, inputIndex, partitionKeys, orderKeys);
+ return new RexTableArgCall(type, inputIndex, partitionKeys, orderKeys, sortOrder);
}
- public RexTableArgCall copy(RelDataType type, int[] partitionKeys, int[] orderKeys) {
- return new RexTableArgCall(type, inputIndex, partitionKeys, orderKeys);
+ public RexTableArgCall copy(
+ RelDataType type, int[] partitionKeys, int[] orderKeys, SortOrder[] order) {
+ return new RexTableArgCall(type, inputIndex, partitionKeys, orderKeys, order);
}
@Override
@@ -112,7 +184,8 @@ public class RexTableArgCall extends RexCall {
final RexTableArgCall that = (RexTableArgCall) o;
return inputIndex == that.inputIndex
&& Arrays.equals(partitionKeys, that.partitionKeys)
- && Arrays.equals(orderKeys, that.orderKeys);
+ && Arrays.equals(orderKeys, that.orderKeys)
+ && Arrays.equals(sortOrder, that.sortOrder);
}
@Override
@@ -120,6 +193,116 @@ public class RexTableArgCall extends RexCall {
int result = Objects.hash(super.hashCode(), inputIndex);
result = 31 * result + Arrays.hashCode(partitionKeys);
result = 31 * result + Arrays.hashCode(orderKeys);
+ result = 31 * result + Arrays.hashCode(sortOrder);
return result;
}
+
+ // --------------------------------------------------------------------------------------------
+ // Utility methods for ORDER BY extraction
+ // --------------------------------------------------------------------------------------------
+
+ /** Result holder for ORDER BY information. */
+ public static class OrderByInfo {
+ public final int[] columns;
+ public final SortOrder[] order;
+
+ public OrderByInfo(int[] columns, SortOrder[] order) {
+ this.columns = columns;
+ this.order = order;
+ }
+ }
+
+ /**
+ * Extracts ORDER BY information from a SQL node list.
+ *
+ * @param fieldNames the list of field names from the table type
+ * @param orderKeys the SQL node list containing ORDER BY clauses (may be null)
+ * @return OrderByInfo containing column indices and sort directions
+ */
+ public static OrderByInfo extractOrderByInfo(
+ List<String> fieldNames, @Nullable SqlNodeList orderKeys) {
+ if (orderKeys == null || orderKeys.isEmpty()) {
+ return new OrderByInfo(new int[0], new SortOrder[0]);
+ }
+
+ final int[] columns = new int[orderKeys.size()];
+ final SortOrder[] order = new SortOrder[orderKeys.size()];
+
+ for (int i = 0; i < orderKeys.size(); i++) {
+ SqlNode orderItem = orderKeys.get(i);
+ boolean descending = false;
+ Boolean nullsFirst = null;
+ SqlNode columnNode = orderItem;
+
+ // Unwrap NULLS FIRST/LAST and DESCENDING operators
+ while (columnNode instanceof SqlCall) {
+ final SqlCall call = (SqlCall) columnNode;
+ final SqlKind kind = call.getKind();
+ if (kind == SqlKind.DESCENDING) {
+ descending = true;
+ columnNode = call.operand(0);
+ } else if (kind == SqlKind.NULLS_FIRST) {
+ nullsFirst = true;
+ columnNode = call.operand(0);
+ } else if (kind == SqlKind.NULLS_LAST) {
+ nullsFirst = false;
+ columnNode = call.operand(0);
+ } else {
+ break;
+ }
+ }
+
+ // Determine the sort direction with explicit null handling
+ final SortOrder direction = getSortOrder(descending, nullsFirst);
+
+ if (!(columnNode instanceof SqlIdentifier)) {
+ throw new ValidationException("Column reference expected for ORDER BY clause.");
+ }
+ final String columnName = ((SqlIdentifier) columnNode).getSimple();
+ final int pos = fieldNames.indexOf(columnName);
+ if (pos < 0) {
+ throw new ValidationException(
+ String.format(
+ "Invalid column '%s' for ORDER BY clause. "
+ + "Available columns are: %s",
+ columnName, fieldNames));
+ }
+ columns[i] = pos;
+ order[i] = direction;
+ }
+
+ return new OrderByInfo(columns, order);
+ }
+
+ private static SortOrder getSortOrder(boolean descending, Boolean nullsFirst) {
+ final SortOrder direction;
+ if (descending) {
+ // Default for DESC is NULLS FIRST
+ direction =
+ (nullsFirst == null || nullsFirst)
+ ? SortOrder.DESC_NULLS_FIRST
+ : SortOrder.DESC_NULLS_LAST;
+ } else {
+ // Default for ASC is NULLS LAST
+ direction =
+ (nullsFirst == null || !nullsFirst)
+ ? SortOrder.ASC_NULLS_LAST
+ : SortOrder.ASC_NULLS_FIRST;
+ }
+ return direction;
+ }
+
+ /**
+ * Converts an array of {@link SortOrder} to an array of {@link SortDirection}.
+ *
+ * @param orders the array of Order values to convert
+ * @return the corresponding array of SortDirection values
+ */
+ public static SortDirection[] toSortDirections(SortOrder[] orders) {
+ final SortDirection[] directions = new SortDirection[orders.length];
+ for (int i = 0; i < orders.length; i++) {
+ directions[i] = SortDirection.valueOf(orders[i].name());
+ }
+ return directions;
+ }
}
diff --git a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/inference/CallBindingCallContext.java b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/inference/CallBindingCallContext.java
index 01c569a19ef..065a3033545 100644
--- a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/inference/CallBindingCallContext.java
+++ b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/inference/CallBindingCallContext.java
@@ -26,6 +26,7 @@ import org.apache.flink.table.functions.FunctionDefinition;
import org.apache.flink.table.functions.ModelSemantics;
import org.apache.flink.table.functions.TableSemantics;
import org.apache.flink.table.planner.calcite.FlinkTypeFactory;
+import org.apache.flink.table.planner.calcite.RexTableArgCall;
import org.apache.flink.table.types.DataType;
import org.apache.flink.table.types.inference.CallContext;
import org.apache.flink.table.types.inference.StaticArgument;
@@ -212,23 +213,20 @@ public final class CallBindingCallContext extends AbstractSqlCallContext {
private final DataType dataType;
private final int[] partitionByColumns;
+ private final int[] orderByColumns;
+ private final SortDirection[] orderByDirections;
public static CallBindingTableSemantics create(
DataType tableDataType, StaticArgument staticArg, SqlNode sqlNode) {
- checkNoOrderBy(sqlNode);
+ final SqlNodeList orderByList = getSemanticsComponent(sqlNode, 2);
+ final RexTableArgCall.OrderByInfo orderByInfo =
+ RexTableArgCall.extractOrderByInfo(
+ DataType.getFieldNames(tableDataType), orderByList);
return new CallBindingTableSemantics(
createDataType(tableDataType, staticArg),
- createPartitionByColumns(tableDataType, sqlNode));
- }
-
- private static void checkNoOrderBy(SqlNode sqlNode) {
- final SqlNodeList orderByList = getSemanticsComponent(sqlNode, 2);
- if (orderByList == null) {
- return;
- }
- if (!orderByList.isEmpty()) {
- throw new ValidationException("ORDER BY clause is currently not supported.");
- }
+ createPartitionByColumns(tableDataType, sqlNode),
+ orderByInfo.columns,
+ RexTableArgCall.toSortDirections(orderByInfo.order));
}
private static @Nullable SqlNodeList getSemanticsComponent(SqlNode sqlNode, int pos) {
@@ -274,9 +272,15 @@ public final class CallBindingCallContext extends AbstractSqlCallContext {
.toArray();
}
- private CallBindingTableSemantics(DataType dataType, int[] partitionByColumns) {
+ private CallBindingTableSemantics(
+ DataType dataType,
+ int[] partitionByColumns,
+ int[] orderByColumns,
+ SortDirection[] orderByDirections) {
this.dataType = dataType;
this.partitionByColumns = partitionByColumns;
+ this.orderByColumns = orderByColumns;
+ this.orderByDirections = orderByDirections;
}
@Override
@@ -291,7 +295,12 @@ public final class CallBindingCallContext extends AbstractSqlCallContext {
@Override
public int[] orderByColumns() {
- return new int[0];
+ return orderByColumns;
+ }
+
+ @Override
+ public SortDirection[] orderByDirections() {
+ return orderByDirections;
}
@Override
diff --git a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/inference/OperatorBindingCallContext.java b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/inference/OperatorBindingCallContext.java
index f1afecd4ace..f31406dad19 100644
--- a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/inference/OperatorBindingCallContext.java
+++ b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/inference/OperatorBindingCallContext.java
@@ -19,7 +19,6 @@
package org.apache.flink.table.planner.functions.inference;
import org.apache.flink.annotation.Internal;
-import org.apache.flink.table.api.ValidationException;
import org.apache.flink.table.catalog.DataTypeFactory;
import org.apache.flink.table.connector.ChangelogMode;
import org.apache.flink.table.functions.FunctionDefinition;
@@ -280,6 +279,8 @@ public final class OperatorBindingCallContext extends AbstractSqlCallContext {
private final DataType dataType;
private final int[] partitionByColumns;
+ private final int[] orderByColumns;
+ private final SortDirection[] orderByDirections;
private final int timeColumn;
private final @Nullable ChangelogMode changelogMode;
@@ -289,10 +290,11 @@ public final class OperatorBindingCallContext extends AbstractSqlCallContext {
RexTableArgCall tableArgCall,
int timeColumn,
@Nullable ChangelogMode changelogMode) {
- checkNoOrderBy(tableArgCall);
return new OperatorBindingTableSemantics(
createDataType(tableDataType, staticArg),
tableArgCall.getPartitionKeys(),
+ tableArgCall.getOrderKeys(),
+ RexTableArgCall.toSortDirections(tableArgCall.getSortOrder()),
timeColumn,
changelogMode);
}
@@ -300,20 +302,18 @@ public final class OperatorBindingCallContext extends AbstractSqlCallContext {
private OperatorBindingTableSemantics(
DataType dataType,
int[] partitionByColumns,
+ int[] orderByColumns,
+ SortDirection[] orderByDirections,
int timeColumn,
@Nullable ChangelogMode changelogMode) {
this.dataType = dataType;
this.partitionByColumns = partitionByColumns;
+ this.orderByColumns = orderByColumns;
+ this.orderByDirections = orderByDirections;
this.timeColumn = timeColumn;
this.changelogMode = changelogMode;
}
- private static void checkNoOrderBy(RexTableArgCall tableArgCall) {
- if (tableArgCall.getOrderKeys().length > 0) {
- throw new ValidationException("ORDER BY clause is currently not supported.");
- }
- }
-
private static DataType createDataType(DataType tableDataType, StaticArgument staticArg) {
final DataType dataType = staticArg.getDataType().orElse(null);
if (dataType != null) {
@@ -336,7 +336,12 @@ public final class OperatorBindingCallContext extends AbstractSqlCallContext {
@Override
public int[] orderByColumns() {
- return new int[0];
+ return orderByColumns;
+ }
+
+ @Override
+ public SortDirection[] orderByDirections() {
+ return orderByDirections;
}
@Override
diff --git a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/QueryOperationConverter.java b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/QueryOperationConverter.java
index 3120e23359a..2fb2b5b71fb 100644
--- a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/QueryOperationConverter.java
+++ b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/QueryOperationConverter.java
@@ -42,6 +42,7 @@ import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
import org.apache.flink.table.functions.FunctionDefinition;
import org.apache.flink.table.functions.TableFunction;
import org.apache.flink.table.functions.TableFunctionDefinition;
+import org.apache.flink.table.functions.TableSemantics.SortDirection;
import org.apache.flink.table.legacy.sources.LookupableTableSource;
import org.apache.flink.table.legacy.sources.TableSource;
import org.apache.flink.table.operations.AggregateQueryOperation;
@@ -69,6 +70,7 @@ import org.apache.flink.table.planner.calcite.FlinkContext;
import org.apache.flink.table.planner.calcite.FlinkRelBuilder;
import org.apache.flink.table.planner.calcite.FlinkTypeFactory;
import org.apache.flink.table.planner.calcite.RexTableArgCall;
+import org.apache.flink.table.planner.calcite.RexTableArgCall.SortOrder;
import org.apache.flink.table.planner.connectors.DynamicSourceUtils;
import org.apache.flink.table.planner.expressions.RexNodeExpression;
import org.apache.flink.table.planner.expressions.SqlAggFunctionVisitor;
@@ -316,6 +318,8 @@ public class QueryOperationConverter extends QueryOperationDefaultVisitor<RelNod
typeFactory.buildRelNodeRowType(
(RowType) tableArgType);
final int[] partitionKeys;
+ final int[] orderKeys;
+ final SortOrder[] sortOrders;
if (tableRef.getQueryOperation()
instanceof PartitionQueryOperation) {
final PartitionQueryOperation partitionOperation =
@@ -323,15 +327,27 @@ public class QueryOperationConverter extends QueryOperationDefaultVisitor<RelNod
tableRef.getQueryOperation();
partitionKeys =
partitionOperation.getPartitionKeys();
+ orderKeys = partitionOperation.getOrderKeys();
+ final SortDirection[] directions =
+ partitionOperation.getOrderDirections();
+ sortOrders = new SortOrder[directions.length];
+ for (int i = 0; i < directions.length; i++) {
+ sortOrders[i] =
+ SortOrder.fromSortDirection(
+ directions[i]);
+ }
} else {
partitionKeys = new int[0];
+ orderKeys = new int[0];
+ sortOrders = new SortOrder[0];
}
final RexTableArgCall tableArgCall =
new RexTableArgCall(
rowType,
inputStack.size(),
partitionKeys,
- new int[0]);
+ orderKeys,
+ sortOrders);
inputStack.add(relBuilder.build());
return tableArgCall;
}
diff --git a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/serde/RexNodeJsonDeserializer.java b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/serde/RexNodeJsonDeserializer.java
index 269a7991135..49720d39cc0 100644
--- a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/serde/RexNodeJsonDeserializer.java
+++ b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/serde/RexNodeJsonDeserializer.java
@@ -28,6 +28,7 @@ import org.apache.flink.table.functions.FunctionIdentifier;
import org.apache.flink.table.functions.UserDefinedFunction;
import org.apache.flink.table.functions.UserDefinedFunctionHelper;
import org.apache.flink.table.planner.calcite.RexTableArgCall;
+import org.apache.flink.table.planner.calcite.RexTableArgCall.SortOrder;
import org.apache.flink.table.planner.functions.bridging.BridgingSqlAggFunction;
import org.apache.flink.table.planner.functions.bridging.BridgingSqlFunction;
import org.apache.flink.table.planner.functions.sql.BuiltInSqlOperator;
@@ -94,6 +95,7 @@ import static org.apache.flink.table.planner.plan.nodes.exec.serde.RexNodeJsonSe
import static org.apache.flink.table.planner.plan.nodes.exec.serde.RexNodeJsonSerializer.FIELD_NAME_NAME;
import static org.apache.flink.table.planner.plan.nodes.exec.serde.RexNodeJsonSerializer.FIELD_NAME_NULL_AS;
import static org.apache.flink.table.planner.plan.nodes.exec.serde.RexNodeJsonSerializer.FIELD_NAME_OPERANDS;
+import static org.apache.flink.table.planner.plan.nodes.exec.serde.RexNodeJsonSerializer.FIELD_NAME_ORDER_DIRECTIONS;
import static org.apache.flink.table.planner.plan.nodes.exec.serde.RexNodeJsonSerializer.FIELD_NAME_ORDER_KEYS;
import static org.apache.flink.table.planner.plan.nodes.exec.serde.RexNodeJsonSerializer.FIELD_NAME_PARTITION_KEYS;
import static org.apache.flink.table.planner.plan.nodes.exec.serde.RexNodeJsonSerializer.FIELD_NAME_RANGES;
@@ -338,7 +340,18 @@ final class RexNodeJsonDeserializer extends StdDeserializer<RexNode> {
orderKeys[i] = orderKeysNode.get(i).asInt();
}
- return new RexTableArgCall(callType, inputIndex, partitionKeys, orderKeys);
+ final JsonNode orderDirectionsNode = jsonNode.get(FIELD_NAME_ORDER_DIRECTIONS);
+ final SortOrder[] order;
+ if (orderDirectionsNode != null && !orderDirectionsNode.isEmpty()) {
+ order = new SortOrder[orderDirectionsNode.size()];
+ for (int i = 0; i < orderDirectionsNode.size(); ++i) {
+ order[i] = SortOrder.valueOf(orderDirectionsNode.get(i).asText());
+ }
+ } else {
+ order = new SortOrder[0];
+ }
+
+ return new RexTableArgCall(callType, inputIndex, partitionKeys, orderKeys, order);
}
private static RexNode deserializeCall(JsonNode jsonNode, SerdeContext serdeContext)
diff --git a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/serde/RexNodeJsonSerializer.java b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/serde/RexNodeJsonSerializer.java
index aa4fad24f86..a342dc66913 100644
--- a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/serde/RexNodeJsonSerializer.java
+++ b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/serde/RexNodeJsonSerializer.java
@@ -34,6 +34,7 @@ import org.apache.flink.table.functions.TableAggregateFunctionDefinition;
import org.apache.flink.table.functions.TableFunctionDefinition;
import org.apache.flink.table.functions.UserDefinedFunction;
import org.apache.flink.table.planner.calcite.RexTableArgCall;
+import org.apache.flink.table.planner.calcite.RexTableArgCall.SortOrder;
import org.apache.flink.table.planner.functions.bridging.BridgingSqlAggFunction;
import org.apache.flink.table.planner.functions.bridging.BridgingSqlFunction;
import org.apache.flink.table.planner.functions.sql.BuiltInSqlOperator;
@@ -127,6 +128,7 @@ final class RexNodeJsonSerializer extends StdSerializer<RexNode> {
static final String KIND_TABLE_ARG_CALL = "TABLE_ARG_CALL";
static final String FIELD_NAME_PARTITION_KEYS = "partitionKeys";
static final String FIELD_NAME_ORDER_KEYS = "orderKeys";
+ static final String FIELD_NAME_ORDER_DIRECTIONS = "orderDirections";
RexNodeJsonSerializer() {
super(RexNode.class);
@@ -342,6 +344,12 @@ final class RexNodeJsonSerializer extends StdSerializer<RexNode> {
gen.writeArray(tableArgCall.getPartitionKeys(), 0, tableArgCall.getPartitionKeys().length);
gen.writeFieldName(FIELD_NAME_ORDER_KEYS);
gen.writeArray(tableArgCall.getOrderKeys(), 0, tableArgCall.getOrderKeys().length);
+ gen.writeFieldName(FIELD_NAME_ORDER_DIRECTIONS);
+ gen.writeStartArray();
+ for (SortOrder order : tableArgCall.getSortOrder()) {
+ gen.writeString(order.name());
+ }
+ gen.writeEndArray();
serializerProvider.defaultSerializeField(FIELD_NAME_TYPE, tableArgCall.getType(), gen);
gen.writeEndObject();
}
diff --git a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/stream/StreamExecProcessTableFunction.java b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/stream/StreamExecProcessTableFunction.java
index d04ee8757c8..12a20833068 100644
--- a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/stream/StreamExecProcessTableFunction.java
+++ b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/stream/StreamExecProcessTableFunction.java
@@ -19,6 +19,7 @@
package org.apache.flink.table.planner.plan.nodes.exec.stream;
import org.apache.flink.FlinkVersion;
+import org.apache.flink.api.common.state.StateTtlConfig;
import org.apache.flink.api.dag.Transformation;
import org.apache.flink.api.java.functions.KeySelector;
import org.apache.flink.configuration.ReadableConfig;
@@ -28,12 +29,14 @@ import org.apache.flink.table.api.DataTypes;
import org.apache.flink.table.connector.ChangelogMode;
import org.apache.flink.table.data.RowData;
import org.apache.flink.table.functions.ProcessTableFunction;
+import org.apache.flink.table.functions.TableSemantics;
import org.apache.flink.table.planner.calcite.FlinkTypeFactory;
import org.apache.flink.table.planner.calcite.RexTableArgCall;
import org.apache.flink.table.planner.codegen.CodeGeneratorContext;
import org.apache.flink.table.planner.codegen.EqualiserCodeGenerator;
import org.apache.flink.table.planner.codegen.HashCodeGenerator;
import org.apache.flink.table.planner.codegen.ProcessTableRunnerGenerator;
+import org.apache.flink.table.planner.codegen.sort.SortCodeGenerator;
import org.apache.flink.table.planner.delegation.PlannerBase;
import org.apache.flink.table.planner.plan.nodes.exec.ExecNodeBase;
import org.apache.flink.table.planner.plan.nodes.exec.ExecNodeConfig;
@@ -41,12 +44,15 @@ import org.apache.flink.table.planner.plan.nodes.exec.ExecNodeContext;
import org.apache.flink.table.planner.plan.nodes.exec.ExecNodeMetadata;
import org.apache.flink.table.planner.plan.nodes.exec.InputProperty;
import org.apache.flink.table.planner.plan.nodes.exec.SingleTransformationTranslator;
+import org.apache.flink.table.planner.plan.nodes.exec.StateMetadata;
+import org.apache.flink.table.planner.plan.nodes.exec.spec.SortSpec;
import org.apache.flink.table.planner.plan.nodes.exec.utils.ExecNodeUtil;
import org.apache.flink.table.planner.plan.nodes.exec.utils.TransformationMetadata;
import org.apache.flink.table.planner.plan.nodes.physical.stream.StreamPhysicalProcessTableFunction;
import org.apache.flink.table.planner.plan.utils.KeySelectorUtil;
import org.apache.flink.table.runtime.generated.GeneratedHashFunction;
import org.apache.flink.table.runtime.generated.GeneratedProcessTableRunner;
+import org.apache.flink.table.runtime.generated.GeneratedRecordComparator;
import org.apache.flink.table.runtime.generated.GeneratedRecordEqualiser;
import org.apache.flink.table.runtime.keyselector.RowDataKeySelector;
import org.apache.flink.table.runtime.operators.process.ProcessTableOperatorFactory;
@@ -54,6 +60,7 @@ import org.apache.flink.table.runtime.operators.process.RuntimeChangelogMode;
import org.apache.flink.table.runtime.operators.process.RuntimeStateInfo;
import org.apache.flink.table.runtime.operators.process.RuntimeTableSemantics;
import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.table.runtime.util.StateConfigUtil;
import org.apache.flink.table.types.DataType;
import org.apache.flink.table.types.inference.StaticArgument;
import org.apache.flink.table.types.inference.StaticArgumentTrait;
@@ -224,14 +231,36 @@ public class StreamExecProcessTableFunction extends ExecNodeBase<RowData>
final RuntimeChangelogMode producedChangelogMode =
RuntimeChangelogMode.serialize(outputChangelogMode);
+ // Generate comparators for ORDER BY columns
+ final GeneratedRecordComparator[] orderByComparators =
+ generateOrderByComparators(
+ planner.getFlinkContext().getClassLoader(),
+ ctx,
+ config,
+ providedInputArgs,
+ operands);
+
+ // Create TTL config for input sort buffers (one per input)
+ // Use empty state metadata list as input sort buffers don't have user-defined state
+ final List<Long> inputBufferTtlMillis =
+ StateMetadata.getStateTtlForMultiInputOperator(
+ config, runtimeTableSemantics.size(), List.of());
+ // Convert to StateTtlConfig list (one per input)
+ final List<StateTtlConfig> inputBufferTtlConfigs =
+ inputBufferTtlMillis.stream()
+ .map(StateConfigUtil::createTtlConfig)
+ .collect(Collectors.toList());
+
final ProcessTableOperatorFactory operatorFactory =
new ProcessTableOperatorFactory(
runtimeTableSemantics,
runtimeStateInfos,
+ orderByComparators,
generatedRunner,
stateHashCode,
stateEquals,
- producedChangelogMode);
+ producedChangelogMode,
+ inputBufferTtlConfigs);
final String effectiveUid =
uid != null ? uid : createTransformationUid(PROCESS_TRANSFORMATION, config);
@@ -282,6 +311,8 @@ public class StreamExecProcessTableFunction extends ExecNodeBase<RowData>
tableArgCall.getInputIndex(),
dataType,
tableArgCall.getPartitionKeys(),
+ tableArgCall.getOrderKeys(),
+ RexTableArgCall.toSortDirections(tableArgCall.getSortOrder()),
consumedChangelogMode,
tableArg.is(StaticArgumentTrait.PASS_COLUMNS_THROUGH),
tableArg.is(StaticArgumentTrait.SET_SEMANTIC_TABLE),
@@ -364,4 +395,46 @@ public class StreamExecProcessTableFunction extends ExecNodeBase<RowData>
}
return globalRetentionTime;
}
+
+ private GeneratedRecordComparator[] generateOrderByComparators(
+ ClassLoader classLoader,
+ CodeGeneratorContext ctx,
+ ExecNodeConfig config,
+ List<Ord<StaticArgument>> providedInputArgs,
+ List<RexNode> operands) {
+ final GeneratedRecordComparator[] comparators =
+ new GeneratedRecordComparator[providedInputArgs.size()];
+
+ for (int i = 0; i < providedInputArgs.size(); i++) {
+ final Ord<StaticArgument> providedInputArg = providedInputArgs.get(i);
+ final RexTableArgCall tableArgCall = (RexTableArgCall) operands.get(providedInputArg.i);
+ final int[] orderKeys = tableArgCall.getOrderKeys();
+
+ if (orderKeys.length > 0) {
+ final RowType inputType = FlinkTypeFactory.toLogicalRowType(tableArgCall.getType());
+ final TableSemantics.SortDirection[] sortDirections =
+ RexTableArgCall.toSortDirections(tableArgCall.getSortOrder());
+
+ // Convert to SortSpec
+ final SortSpec.SortSpecBuilder builder = SortSpec.builder();
+ for (int j = 0; j < orderKeys.length; j++) {
+ final TableSemantics.SortDirection direction = sortDirections[j];
+ builder.addField(
+ orderKeys[j], !direction.isDescending(), direction.isNullsLast());
+ }
+ final SortSpec sortSpec = builder.build();
+
+ final SortCodeGenerator sortCodeGen =
+ new SortCodeGenerator(config, classLoader, inputType, sortSpec, ctx);
+
+ comparators[i] =
+ sortCodeGen.generateRecordComparator(
+ "ProcessTableFunctionOrderByComparator" + i);
+ } else {
+ comparators[i] = null;
+ }
+ }
+
+ return comparators;
+ }
}
diff --git a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/physical/stream/StreamPhysicalProcessTableFunction.java b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/physical/stream/StreamPhysicalProcessTableFunction.java
index 0a88b6f9102..754fe1b3284 100644
--- a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/physical/stream/StreamPhysicalProcessTableFunction.java
+++ b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/physical/stream/StreamPhysicalProcessTableFunction.java
@@ -267,6 +267,7 @@ public class StreamPhysicalProcessTableFunction extends AbstractRelNode
inputs.stream()
.map(RelNode::getRowType)
.forEach(rowType -> verifyTimeAttribute(rowType, onTimeFields));
+ verifyOrderByTimeAttributes(inputs, call);
}
private static void verifyTimeAttribute(RelDataType rowType, Set<String> onTimeFields) {
@@ -304,6 +305,36 @@ public class StreamPhysicalProcessTableFunction extends AbstractRelNode
}
}
+ private static void verifyOrderByTimeAttributes(List<RelNode> inputs, RexCall call) {
+ final List<Ord<StaticArgument>> providedInputArgs = getProvidedInputArgs(call);
+ final List<RexNode> operands = call.getOperands();
+
+ for (Ord<StaticArgument> providedInputArg : providedInputArgs) {
+ final RexTableArgCall tableArgCall = (RexTableArgCall) operands.get(providedInputArg.i);
+ final int[] orderKeys = tableArgCall.getOrderKeys();
+
+ // Skip if no ORDER BY
+ if (orderKeys.length == 0) {
+ continue;
+ }
+
+ final int firstOrderByColumn = orderKeys[0];
+ final RelDataType inputType = inputs.get(tableArgCall.getInputIndex()).getRowType();
+ final RelDataTypeField firstOrderByField =
+ inputType.getFieldList().get(firstOrderByColumn);
+
+ // Verify that the first ORDER BY column is a valid time attribute
+ if (!FlinkTypeFactory.isTimeIndicatorType(firstOrderByField.getType())) {
+ throw new ValidationException(
+ String.format(
+ "Invalid ORDER BY clause for table argument '%s'. The first ORDER BY column '%s' is not a valid time attribute. "
+ + "Only columns with a watermark declaration qualify for ORDER BY. "
+ + "Also, make sure that the watermarked column is forwarded without any modification.",
+ providedInputArg.e.getName(), firstOrderByField.getName()));
+ }
+ }
+ }
+
private static void verifyPassThroughColumnsForUpdates(
List<Ord<StaticArgument>> providedInputArgs, ChangelogMode requiredChangelogMode) {
if (!requiredChangelogMode.containsOnly(RowKind.INSERT)
diff --git a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/logical/ProjectWindowTableFunctionTransposeRule.java b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/logical/ProjectWindowTableFunctionTransposeRule.java
index 2bec8a826fa..57c4e6c78af 100644
--- a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/logical/ProjectWindowTableFunctionTransposeRule.java
+++ b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/logical/ProjectWindowTableFunctionTransposeRule.java
@@ -20,6 +20,7 @@ package org.apache.flink.table.planner.plan.rules.logical;
import org.apache.flink.table.planner.calcite.FlinkTypeFactory;
import org.apache.flink.table.planner.calcite.RexTableArgCall;
+import org.apache.flink.table.planner.calcite.RexTableArgCall.SortOrder;
import org.apache.flink.table.planner.functions.sql.SqlWindowTableFunction;
import org.apache.flink.table.planner.plan.logical.SessionWindowSpec;
import org.apache.flink.table.planner.plan.logical.TimeAttributeWindowingStrategy;
@@ -213,8 +214,9 @@ public class ProjectWindowTableFunctionTransposeRule extends RelOptRule {
Arrays.stream(tableArgCall.getPartitionKeys()).map(mapping::get).toArray();
final int[] newOrderKeys =
Arrays.stream(tableArgCall.getOrderKeys()).map(mapping::get).toArray();
+ final SortOrder[] newOrder = tableArgCall.getSortOrder();
final RexTableArgCall projectedTableArgCall =
- tableArgCall.copy(newTableType, newPartitionKeys, newOrderKeys);
+ tableArgCall.copy(newTableType, newPartitionKeys, newOrderKeys, newOrder);
newOperands.set(0, projectedTableArgCall);
diff --git a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/api/QueryOperationSqlSemanticTest.java b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/api/QueryOperationSqlSemanticTest.java
index aeb8c7028a8..8483fdff25f 100644
--- a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/api/QueryOperationSqlSemanticTest.java
+++ b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/api/QueryOperationSqlSemanticTest.java
@@ -57,8 +57,9 @@ public class QueryOperationSqlSemanticTest extends SemanticTestBase {
QueryOperationTestPrograms.OVER_WINDOW_ROWS_UNBOUNDED_NO_PARTITION,
QueryOperationTestPrograms.OVER_WINDOW_LAG,
QueryOperationTestPrograms.ACCESSING_NESTED_COLUMN,
- QueryOperationTestPrograms.ROW_SEMANTIC_TABLE_PTF,
- QueryOperationTestPrograms.SET_SEMANTIC_TABLE_PTF,
+ QueryOperationTestPrograms.PTF_ROW_SEMANTIC_TABLE,
+ QueryOperationTestPrograms.PTF_SET_SEMANTIC_TABLE,
+ QueryOperationTestPrograms.PTF_ORDER_BY,
QueryOperationTestPrograms.ML_PREDICT_MODEL_API,
QueryOperationTestPrograms.ASYNC_ML_PREDICT_TABLE_API_MAP_EXPRESSION_CONFIG,
QueryOperationTestPrograms.ASYNC_ML_PREDICT_MODEL_API,
diff --git a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/api/QueryOperationSqlSerializationTest.java b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/api/QueryOperationSqlSerializationTest.java
index 639a3eb4e5a..96e53484d20 100644
--- a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/api/QueryOperationSqlSerializationTest.java
+++ b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/api/QueryOperationSqlSerializationTest.java
@@ -68,8 +68,9 @@ public class QueryOperationSqlSerializationTest implements TableTestProgramRunne
QueryOperationTestPrograms.OVER_WINDOW_ROWS_UNBOUNDED_NO_PARTITION,
QueryOperationTestPrograms.OVER_WINDOW_LAG,
QueryOperationTestPrograms.ACCESSING_NESTED_COLUMN,
- QueryOperationTestPrograms.ROW_SEMANTIC_TABLE_PTF,
- QueryOperationTestPrograms.SET_SEMANTIC_TABLE_PTF,
+ QueryOperationTestPrograms.PTF_ROW_SEMANTIC_TABLE,
+ QueryOperationTestPrograms.PTF_SET_SEMANTIC_TABLE,
+ QueryOperationTestPrograms.PTF_ORDER_BY,
QueryOperationTestPrograms.INLINE_FUNCTION_SERIALIZATION,
QueryOperationTestPrograms.ML_PREDICT_MODEL_API,
QueryOperationTestPrograms.ASYNC_ML_PREDICT_MODEL_API,
diff --git a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/api/QueryOperationTestPrograms.java b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/api/QueryOperationTestPrograms.java
index 677d957275a..71e049a84ff 100644
--- a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/api/QueryOperationTestPrograms.java
+++ b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/api/QueryOperationTestPrograms.java
@@ -28,6 +28,7 @@ import org.apache.flink.table.planner.factories.TestValuesModelFactory;
import org.apache.flink.table.planner.plan.nodes.exec.stream.ProcessTableFunctionTestUtils.ChainedReceivingFunction;
import org.apache.flink.table.planner.plan.nodes.exec.stream.ProcessTableFunctionTestUtils.ChainedSendingFunction;
import org.apache.flink.table.planner.plan.nodes.exec.stream.ProcessTableFunctionTestUtils.RowSemanticTableFunction;
+import org.apache.flink.table.planner.plan.nodes.exec.stream.ProcessTableFunctionTestUtils.SetSemanticTableFunction;
import org.apache.flink.table.planner.plan.utils.JavaUserDefinedAggFunctions;
import org.apache.flink.table.planner.runtime.utils.JavaUserDefinedTableFunctions;
import org.apache.flink.table.test.program.SinkTestStep;
@@ -60,6 +61,7 @@ import static org.apache.flink.table.planner.plan.nodes.exec.stream.MLPredictTes
import static org.apache.flink.table.planner.plan.nodes.exec.stream.MLPredictTestPrograms.SYNC_MODEL;
import static org.apache.flink.table.planner.plan.nodes.exec.stream.ProcessTableFunctionTestUtils.BASE_SINK_SCHEMA;
import static org.apache.flink.table.planner.plan.nodes.exec.stream.ProcessTableFunctionTestUtils.BASIC_VALUES;
+import static org.apache.flink.table.planner.plan.nodes.exec.stream.ProcessTableFunctionTestUtils.KEYED_BASE_SINK_SCHEMA;
import static org.apache.flink.table.planner.plan.nodes.exec.stream.ProcessTableFunctionTestUtils.KEYED_TIMED_BASE_SINK_SCHEMA;
import static org.apache.flink.table.planner.plan.nodes.exec.stream.ProcessTableFunctionTestUtils.TIMED_SOURCE;
@@ -1034,8 +1036,8 @@ public class QueryOperationTestPrograms {
"sink")
.build();
- public static final TableTestProgram ROW_SEMANTIC_TABLE_PTF =
- TableTestProgram.of("process-row-table-api", "table with row semantics")
+ public static final TableTestProgram PTF_ROW_SEMANTIC_TABLE =
+ TableTestProgram.of("ptf-row-semantic-table", "table with row semantics")
// TODO [FLINK-38233]: Remove this config when PTF support in
// StreamNonDeterministicUpdatePlanVisitor is added.
.setupConfig(
@@ -1064,8 +1066,8 @@ public class QueryOperationTestPrograms {
"sink")
.build();
- static final TableTestProgram SET_SEMANTIC_TABLE_PTF =
- TableTestProgram.of("partitioned-ptf", "verifies SQL serialization")
+ static final TableTestProgram PTF_SET_SEMANTIC_TABLE =
+ TableTestProgram.of("ptf-set-semantic-table", "verifies SQL serialization")
// TODO [FLINK-38233]: Remove this config when PTF support in
// StreamNonDeterministicUpdatePlanVisitor is added.
.setupConfig(
@@ -1269,4 +1271,44 @@ public class QueryOperationTestPrograms {
$("b")),
"sink")
.build();
+
+ static final TableTestProgram PTF_ORDER_BY =
+ TableTestProgram.of("ptf-order-by", "verifies SQL serialization with ORDER BY clause")
+ // TODO [FLINK-38233]: Remove this config when PTF support in
+ // StreamNonDeterministicUpdatePlanVisitor is added.
+ .setupConfig(
+ OptimizerConfigOptions.TABLE_OPTIMIZER_NONDETERMINISTIC_UPDATE_STRATEGY,
+ OptimizerConfigOptions.NonDeterministicUpdateStrategy.IGNORE)
+ .setupTemporarySystemFunction("f", SetSemanticTableFunction.class)
+ .setupTableSource(TIMED_SOURCE)
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink")
+ .addSchema(KEYED_BASE_SINK_SCHEMA)
+ .consumedValues(
+ "+I[Bob, {+I[Bob, 1, 1970-01-01T00:00:00Z], 1}]",
+ "+I[Alice, {+I[Alice, 1, 1970-01-01T00:00:00.001Z], 1}]",
+ "+I[Bob, {+I[Bob, 2, 1970-01-01T00:00:00.002Z], 1}]",
+ "+I[Bob, {+I[Bob, 3, 1970-01-01T00:00:00.003Z], 1}]",
+ "+I[Bob, {+I[Bob, 4, 1970-01-01T00:00:00.004Z], 1}]",
+ "+I[Bob, {+I[Bob, 5, 1970-01-01T00:00:00.005Z], 1}]",
+ "+I[Bob, {+I[Bob, 6, 1970-01-01T00:00:00.006Z], 1}]")
+ .build())
+ .runSql(
+ "SELECT `$$T_FUNC`.`name`, `$$T_FUNC`.`out` FROM TABLE(\n"
+ + " `f`(\n"
+ + " (\n"
+ + " SELECT `$$T_SOURCE`.`name`, `$$T_SOURCE`.`score`, `$$T_SOURCE`.`ts` FROM `default_catalog`.`default_database`.`t` $$T_SOURCE\n"
+ + " ) PARTITION BY (`name`) ORDER BY (`ts` ASC, `score` DESC), 1, DEFAULT, 'f')\n"
+ + ") $$T_FUNC")
+ .runTableApi(
+ env ->
+ env.fromCall(
+ "f",
+ env.from("t")
+ .partitionBy($("name"))
+ .orderBy($("ts").asc(), $("score").desc())
+ .asArgument("r"),
+ lit(1).asArgument("i")),
+ "sink")
+ .build();
}
diff --git a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/serde/RexNodeJsonSerdeTest.java b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/serde/RexNodeJsonSerdeTest.java
index ad8583a23a5..d5fdb0709df 100644
--- a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/serde/RexNodeJsonSerdeTest.java
+++ b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/serde/RexNodeJsonSerdeTest.java
@@ -42,6 +42,7 @@ import org.apache.flink.table.module.Module;
import org.apache.flink.table.planner.calcite.FlinkTypeFactory;
import org.apache.flink.table.planner.calcite.FlinkTypeSystem;
import org.apache.flink.table.planner.calcite.RexTableArgCall;
+import org.apache.flink.table.planner.calcite.RexTableArgCall.SortOrder;
import org.apache.flink.table.planner.functions.bridging.BridgingSqlFunction;
import org.apache.flink.table.planner.functions.sql.FlinkSqlOperatorTable;
import org.apache.flink.table.planner.functions.utils.UserDefinedFunctionUtils;
@@ -839,7 +840,8 @@ public class RexNodeJsonSerdeTest {
Arrays.asList("f1", "f2")),
0,
new int[] {1},
- new int[] {0}));
+ new int[] {0},
+ new SortOrder[] {SortOrder.ASC_NULLS_LAST}));
}
// --------------------------------------------------------------------------------------------
diff --git a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ProcessTableFunctionRestoreTests.java b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ProcessTableFunctionRestoreTests.java
index baa60c612c0..d95c4a7dd73 100644
--- a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ProcessTableFunctionRestoreTests.java
+++ b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ProcessTableFunctionRestoreTests.java
@@ -38,6 +38,8 @@ public class ProcessTableFunctionRestoreTests extends RestoreTestBase {
ProcessTableFunctionTestPrograms.PROCESS_NAMED_TIMERS_RESTORE,
ProcessTableFunctionTestPrograms.PROCESS_MULTI_INPUT_RESTORE,
ProcessTableFunctionTestPrograms.PROCESS_MAP_STATE_RESTORE,
- ProcessTableFunctionTestPrograms.PROCESS_UPDATING_OUTPUT_UPSERT_RESTORE);
+ ProcessTableFunctionTestPrograms.PROCESS_UPDATING_OUTPUT_UPSERT_RESTORE,
+ ProcessTableFunctionTestPrograms.PROCESS_ORDER_BY_RESTORE,
+ ProcessTableFunctionTestPrograms.PROCESS_MULTI_INPUT_ORDER_BY_RESTORE);
}
}
diff --git a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ProcessTableFunctionSemanticTests.java b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ProcessTableFunctionSemanticTests.java
index cace4ad26c5..7903b93bc5d 100644
--- a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ProcessTableFunctionSemanticTests.java
+++ b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ProcessTableFunctionSemanticTests.java
@@ -100,6 +100,9 @@ public class ProcessTableFunctionSemanticTests extends SemanticTestBase {
ProcessTableFunctionTestPrograms.PROCESS_MULTI_INPUT,
ProcessTableFunctionTestPrograms.PROCESS_MULTI_INPUT_WITH_SCALAR_ARGS,
ProcessTableFunctionTestPrograms.PROCESS_STATEFUL_MULTI_INPUT_WITH_TIMEOUT,
- ProcessTableFunctionTestPrograms.PROCESS_UPDATING_MULTI_INPUT);
+ ProcessTableFunctionTestPrograms.PROCESS_UPDATING_MULTI_INPUT,
+ ProcessTableFunctionTestPrograms.PROCESS_ORDER_BY,
+ ProcessTableFunctionTestPrograms.PROCESS_MULTI_INPUT_ORDER_BY,
+ ProcessTableFunctionTestPrograms.PROCESS_ORDER_BY_TABLE_API);
}
}
diff --git a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ProcessTableFunctionTestPrograms.java b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ProcessTableFunctionTestPrograms.java
index e65dd8319f1..7248ff1755a 100644
--- a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ProcessTableFunctionTestPrograms.java
+++ b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ProcessTableFunctionTestPrograms.java
@@ -37,12 +37,14 @@ import org.apache.flink.table.planner.plan.nodes.exec.stream.ProcessTableFunctio
import org.apache.flink.table.planner.plan.nodes.exec.stream.ProcessTableFunctionTestUtils.ListStateFunction;
import org.apache.flink.table.planner.plan.nodes.exec.stream.ProcessTableFunctionTestUtils.MapStateFunction;
import org.apache.flink.table.planner.plan.nodes.exec.stream.ProcessTableFunctionTestUtils.MultiInputFunction;
+import org.apache.flink.table.planner.plan.nodes.exec.stream.ProcessTableFunctionTestUtils.MultiInputOrderByFunction;
import org.apache.flink.table.planner.plan.nodes.exec.stream.ProcessTableFunctionTestUtils.MultiInputWithScalarArgsFunction;
import org.apache.flink.table.planner.plan.nodes.exec.stream.ProcessTableFunctionTestUtils.MultiStateFunction;
import org.apache.flink.table.planner.plan.nodes.exec.stream.ProcessTableFunctionTestUtils.NamedTimersFunction;
import org.apache.flink.table.planner.plan.nodes.exec.stream.ProcessTableFunctionTestUtils.NonNullMapStateFunction;
import org.apache.flink.table.planner.plan.nodes.exec.stream.ProcessTableFunctionTestUtils.OptionalOnTimeFunction;
import org.apache.flink.table.planner.plan.nodes.exec.stream.ProcessTableFunctionTestUtils.OptionalPartitionOnTimeFunction;
+import org.apache.flink.table.planner.plan.nodes.exec.stream.ProcessTableFunctionTestUtils.OrderByFunction;
import org.apache.flink.table.planner.plan.nodes.exec.stream.ProcessTableFunctionTestUtils.PojoArgsFunction;
import org.apache.flink.table.planner.plan.nodes.exec.stream.ProcessTableFunctionTestUtils.PojoCreatingFunction;
import org.apache.flink.table.planner.plan.nodes.exec.stream.ProcessTableFunctionTestUtils.PojoStateFunction;
@@ -95,6 +97,7 @@ import static org.apache.flink.table.planner.plan.nodes.exec.stream.ProcessTable
import static org.apache.flink.table.planner.plan.nodes.exec.stream.ProcessTableFunctionTestUtils.TIMED_MULTI_BASE_SINK_SCHEMA;
import static org.apache.flink.table.planner.plan.nodes.exec.stream.ProcessTableFunctionTestUtils.TIMED_SOURCE;
import static org.apache.flink.table.planner.plan.nodes.exec.stream.ProcessTableFunctionTestUtils.TIMED_SOURCE_LATE_EVENTS;
+import static org.apache.flink.table.planner.plan.nodes.exec.stream.ProcessTableFunctionTestUtils.TIMED_SOURCE_LATE_EVENTS_LARGE;
import static org.apache.flink.table.planner.plan.nodes.exec.stream.ProcessTableFunctionTestUtils.TIMED_SOURCE_SCHEMA;
import static org.apache.flink.table.planner.plan.nodes.exec.stream.ProcessTableFunctionTestUtils.UPDATING_VALUES;
@@ -1556,4 +1559,285 @@ public class ProcessTableFunctionTestPrograms {
+ "scoreTable => TABLE scores PARTITION BY name, "
+ "cityTable => TABLE city PARTITION BY name)")
.build();
+
+ public static final TableTestProgram PROCESS_ORDER_BY =
+ TableTestProgram.of(
+ "process-order-by",
+ "test the ORDER BY clause on watermarked column and secondary ordering, late events are dropped")
+ .setupTemporarySystemFunction("f", OrderByFunction.class)
+ .setupTableSource(TIMED_SOURCE_LATE_EVENTS_LARGE)
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink")
+ .addSchema(KEYED_TIMED_BASE_SINK_SCHEMA)
+ .consumedValues(
+ // Schema:
+ // (key, current watermark, table watermark, current
+ // timestamp, captured order, output timestamp)
+ //
+ // (Alice, 2) triggers flushing the sort buffer
+ // Watermark transitions from 1969-12-31T23:59:59.995Z
+ // to 1970-01-01T00:00:00.009Z
+ // (Bob, 1) arrives
+ "+I[Bob, {1969-12-31T23:59:59.995Z, 1969-12-31T23:59:59.995Z, 1970-01-01T00:00:00Z, [+I[Bob, 1, 1970-01-01T00:00:00Z]]}, 1970-01-01T00:00:00Z]",
+ // (Alice, 1) arrives
+ "+I[Alice, {1969-12-31T23:59:59.995Z, 1969-12-31T23:59:59.995Z, 1970-01-01T00:00:00.001Z, [+I[Alice, 1, 1970-01-01T00:00:00.001Z]]}, 1970-01-01T00:00:00.001Z]",
+ // (Bob, 3) arrives, reordered by secondary sorting,
+ // not (Bob, 2)
+ "+I[Bob, {1969-12-31T23:59:59.995Z, 1969-12-31T23:59:59.995Z, 1970-01-01T00:00:00.003Z, [+I[Bob, 1, 1970-01-01T00:00:00Z], +I[Bob, 3, 1970-01-01T00:00:00.003Z]]}, 1970-01-01T00:00:00.003Z]",
+ // (Bob, 2) arrives
+ "+I[Bob, {1969-12-31T23:59:59.995Z, 1969-12-31T23:59:59.995Z, 1970-01-01T00:00:00.003Z, [+I[Bob, 1, 1970-01-01T00:00:00Z], +I[Bob, 3, 1970-01-01T00:00:00.003Z], +I[Bob, 2, 1970-01-01T00:00:00.003Z]]}, 1970-01-01T00:00:00.003Z]",
+ // (Bob, 4) arrives
+ "+I[Bob, {1969-12-31T23:59:59.995Z, 1969-12-31T23:59:59.995Z, 1970-01-01T00:00:00.005Z, [+I[Bob, 1, 1970-01-01T00:00:00Z], +I[Bob, 3, 1970-01-01T00:00:00.003Z], +I[Bob, 2, 1970-01-01T00:00:00.003Z], +I[Bob, 4, 1970-01-01T00:00:00.005Z]]}, 1970-01-01T00:00:00.005Z]",
+ // (Bob, 7) triggers flushing the sort buffer
+ // Watermark transitions from 1970-01-01T00:00:00.009Z
+ // to 1970-01-01T00:00:00.012Z
+ // (Bob, 5) arrives
+ "+I[Bob, {1970-01-01T00:00:00.009Z, 1970-01-01T00:00:00.009Z, 1970-01-01T00:00:00.010Z, [+I[Bob, 1, 1970-01-01T00:00:00Z], +I[Bob, 3, 1970-01-01T00:00:00.003Z], +I[Bob, 2, 1970-01-01T00:00:00.003Z], +I[Bob, 4, 1970-01-01T00:00:00.005Z], +I[Bob, 5, 1970-01-01T00:00:00.010Z]]}, 1970-01-01T00:00:00.010Z]",
+ // (Bob, 6) arrives
+ "+I[Bob, {1970-01-01T00:00:00.009Z, 1970-01-01T00:00:00.009Z, 1970-01-01T00:00:00.012Z, [+I[Bob, 1, 1970-01-01T00:00:00Z], +I[Bob, 3, 1970-01-01T00:00:00.003Z], +I[Bob, 2, 1970-01-01T00:00:00.003Z], +I[Bob, 4, 1970-01-01T00:00:00.005Z], +I[Bob, 5, 1970-01-01T00:00:00.010Z], +I[Bob, 6, 1970-01-01T00:00:00.012Z]]}, 1970-01-01T00:00:00.012Z]",
+ // Job is terminating, watermark transitions from
+ // 1970-01-01T00:00:00.012Z to MAX (end of input)
+ // (Alice, 2) arrives
+ "+I[Alice, {1970-01-01T00:00:00.012Z, 1970-01-01T00:00:00.012Z, 1970-01-01T00:00:00.019Z, [+I[Alice, 1, 1970-01-01T00:00:00.001Z], +I[Alice, 2, 1970-01-01T00:00:00.019Z]]}, 1970-01-01T00:00:00.019Z]",
+ // (Alice, 3), (Bob, 5), and (Bob, 6) are dropped
+ // because they are marked as late
+ // (Bob, 7) arrives
+ "+I[Bob, {1970-01-01T00:00:00.012Z, 1970-01-01T00:00:00.012Z, 1970-01-01T00:00:00.022Z, [+I[Bob, 1, 1970-01-01T00:00:00Z], +I[Bob, 3, 1970-01-01T00:00:00.003Z], +I[Bob, 2, 1970-01-01T00:00:00.003Z], +I[Bob, 4, 1970-01-01T00:00:00.005Z], +I[Bob, 5, 1970-01-01T00:00:00.010Z], +I[Bob, 6, 1970-01-01T00:00:00.012Z], +I[Bob, 7, 1970-01-01T00:00:00.022Z]]}, 1970-01-01T00:00:00.022Z]")
+ .build())
+ .runSql(
+ "INSERT INTO sink SELECT * FROM f("
+ + "r => TABLE t_large PARTITION BY name ORDER BY (ts ASC, score DESC), "
+ + "on_time => DESCRIPTOR(ts))")
+ .build();
+
+ public static final TableTestProgram PROCESS_MULTI_INPUT_ORDER_BY =
+ TableTestProgram.of(
+ "process-multi-input-order-by",
+ "test the ORDER BY clause on watermarked column and secondary ordering per input, late events are dropped")
+ .setupTemporarySystemFunction("f", MultiInputOrderByFunction.class)
+ .setupTableSource(TIMED_SOURCE_LATE_EVENTS)
+ .setupTableSource(TIMED_SOURCE_LATE_EVENTS_LARGE)
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink")
+ .addSchema(TIMED_MULTI_BASE_SINK_SCHEMA)
+ .consumedValues(
+ // Schema:
+ // (key1, key2, input, table watermark, current
+ // timestamp,
+ // captured order, output timestamp)
+ //
+ // Input 1: (Alice, 1) triggers flushing the sort buffer
+ // Table watermark 1 transitions from
+ // 1969-12-31T23:59:59.999Z
+ // to 1970-01-01T00:00:00.000Z
+ // (Bob, 1) arrives
+ "+I[Bob, Bob, {in1, 1969-12-31T23:59:59.999Z, 1970-01-01T00:00:00Z, [+I[Bob, 1, 1970-01-01T00:00:00Z]]}, 1970-01-01T00:00:00Z]",
+ // Input 1: (Bob, 1) triggers flushing the sort buffer
+ // Table watermark 1 transitions from
+ // 1970-01-01T00:00:00.000Z
+ // to 1970-01-01T00:01:39.998Z
+ // (Alice, 1) arrives
+ "+I[Alice, Alice, {in1, 1970-01-01T00:00:00Z, 1970-01-01T00:00:00.001Z, [+I[Alice, 1, 1970-01-01T00:00:00.001Z]]}, 1970-01-01T00:00:00.001Z]",
+ // Input 1: (Bob, 3) and (Bob, 4) are dropped because
+ // they are marked as late
+ // Table watermark 1 transitions from
+ // 1970-01-01T00:01:39.998Z
+ // to MAX (end of input)
+ // (Bob, 2) arrives
+ "+I[Bob, Bob, {in1, 1970-01-01T00:01:39.998Z, 1970-01-01T00:01:39.999Z, [+I[Bob, 1, 1970-01-01T00:00:00Z], +I[Bob, 2, 1970-01-01T00:01:39.999Z]]}, 1970-01-01T00:01:39.999Z]",
+ // Input 2: (Alice, 2) triggers flushing the sort buffer
+ // Table watermark 2 transitions from
+ // 1969-12-31T23:59:59.995Z
+ // to 1970-01-01T00:00:00.009Z
+ // (Bob, 1) to (Bob, 4) and (Alice, 1) arrives
+ "+I[Bob, Bob, {in2, 1969-12-31T23:59:59.995Z, 1970-01-01T00:00:00Z, [+I[Bob, 1, 1970-01-01T00:00:00Z]]}, 1970-01-01T00:00:00Z]",
+ "+I[Bob, Bob, {in2, 1969-12-31T23:59:59.995Z, 1970-01-01T00:00:00.003Z, [+I[Bob, 1, 1970-01-01T00:00:00Z], +I[Bob, 3, 1970-01-01T00:00:00.003Z]]}, 1970-01-01T00:00:00.003Z]",
+ "+I[Bob, Bob, {in2, 1969-12-31T23:59:59.995Z, 1970-01-01T00:00:00.003Z, [+I[Bob, 1, 1970-01-01T00:00:00Z], +I[Bob, 3, 1970-01-01T00:00:00.003Z], +I[Bob, 2, 1970-01-01T00:00:00.003Z]]}, 1970-01-01T00:00:00.003Z]",
+ "+I[Bob, Bob, {in2, 1969-12-31T23:59:59.995Z, 1970-01-01T00:00:00.005Z, [+I[Bob, 1, 1970-01-01T00:00:00Z], +I[Bob, 3, 1970-01-01T00:00:00.003Z], +I[Bob, 2, 1970-01-01T00:00:00.003Z], +I[Bob, 4, 1970-01-01T00:00:00.005Z]]}, 1970-01-01T00:00:00.005Z]",
+ "+I[Alice, Alice, {in2, 1969-12-31T23:59:59.995Z, 1970-01-01T00:00:00.001Z, [+I[Alice, 1, 1970-01-01T00:00:00.001Z]]}, 1970-01-01T00:00:00.001Z]",
+ // Input 2: (Bob, 7) triggers flushing the sort buffer
+ // Table watermark 2 transitions from
+ // 1970-01-01T00:00:00.009Z
+ // to 1970-01-01T00:00:00.012Z
+ // (Bob, 5) and (Bob, 6) arrives
+ "+I[Bob, Bob, {in2, 1970-01-01T00:00:00.009Z, 1970-01-01T00:00:00.010Z, [+I[Bob, 1, 1970-01-01T00:00:00Z], +I[Bob, 3, 1970-01-01T00:00:00.003Z], +I[Bob, 2, 1970-01-01T00:00:00.003Z], +I[Bob, 4, 1970-01-01T00:00:00.005Z], +I[Bob, 5, 1970-01-01T00:00:00.010Z]]}, 1970-01-01T00:00:00.010Z]",
+ "+I[Bob, Bob, {in2, 1970-01-01T00:00:00.009Z, 1970-01-01T00:00:00.012Z, [+I[Bob, 1, 1970-01-01T00:00:00Z], +I[Bob, 3, 1970-01-01T00:00:00.003Z], +I[Bob, 2, 1970-01-01T00:00:00.003Z], +I[Bob, 4, 1970-01-01T00:00:00.005Z], +I[Bob, 5, 1970-01-01T00:00:00.010Z], +I[Bob, 6, 1970-01-01T00:00:00.012Z]]}, 1970-01-01T00:00:00.012Z]",
+ // Table watermark 2 transitions from
+ // 1970-01-01T00:00:00.012Z
+ // to MAX (end of input)
+ // (Bob, 7) and (Alice, 2) arrives
+ "+I[Bob, Bob, {in2, 1970-01-01T00:00:00.012Z, 1970-01-01T00:00:00.022Z, [+I[Bob, 1, 1970-01-01T00:00:00Z], +I[Bob, 3, 1970-01-01T00:00:00.003Z], +I[Bob, 2, 1970-01-01T00:00:00.003Z], +I[Bob, 4, 1970-01-01T00:00:00.005Z], +I[Bob, 5, 1970-01-01T00:00:00.010Z], +I[Bob, 6, 1970-01-01T00:00:00.012Z], +I[Bob, 7, 1970-01-01T00:00:00.022Z]]}, 1970-01-01T00:00:00.022Z]",
+ "+I[Alice, Alice, {in2, 1970-01-01T00:00:00.012Z, 1970-01-01T00:00:00.019Z, [+I[Alice, 1, 1970-01-01T00:00:00.001Z], +I[Alice, 2, 1970-01-01T00:00:00.019Z]]}, 1970-01-01T00:00:00.019Z]")
+ .build())
+ .runSql(
+ "INSERT INTO sink SELECT * FROM f("
+ + "in1 => TABLE t PARTITION BY name ORDER BY (ts ASC, score DESC), "
+ + "in2 => TABLE t_large PARTITION BY name ORDER BY (ts ASC, score DESC), "
+ + "on_time => DESCRIPTOR(ts))")
+ .build();
+
+ public static final TableTestProgram PROCESS_ORDER_BY_TABLE_API =
+ TableTestProgram.of(
+ "process-order-by-table-api",
+ "test the ORDER BY clause via Table API on watermarked column and secondary ordering, late events are dropped")
+ .setupTemporarySystemFunction("f", OrderByFunction.class)
+ .setupTableSource(TIMED_SOURCE_LATE_EVENTS_LARGE)
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink")
+ .addSchema(KEYED_TIMED_BASE_SINK_SCHEMA)
+ .consumedValues(
+ // Schema:
+ // (key, current watermark, table watermark, current
+ // timestamp, captured order, output timestamp)
+ // Same expected output as PROCESS_ORDER_BY
+ "+I[Bob, {1969-12-31T23:59:59.995Z, 1969-12-31T23:59:59.995Z, 1970-01-01T00:00:00Z, [+I[Bob, 1, 1970-01-01T00:00:00Z]]}, 1970-01-01T00:00:00Z]",
+ "+I[Alice, {1969-12-31T23:59:59.995Z, 1969-12-31T23:59:59.995Z, 1970-01-01T00:00:00.001Z, [+I[Alice, 1, 1970-01-01T00:00:00.001Z]]}, 1970-01-01T00:00:00.001Z]",
+ "+I[Bob, {1969-12-31T23:59:59.995Z, 1969-12-31T23:59:59.995Z, 1970-01-01T00:00:00.003Z, [+I[Bob, 1, 1970-01-01T00:00:00Z], +I[Bob, 3, 1970-01-01T00:00:00.003Z]]}, 1970-01-01T00:00:00.003Z]",
+ "+I[Bob, {1969-12-31T23:59:59.995Z, 1969-12-31T23:59:59.995Z, 1970-01-01T00:00:00.003Z, [+I[Bob, 1, 1970-01-01T00:00:00Z], +I[Bob, 3, 1970-01-01T00:00:00.003Z], +I[Bob, 2, 1970-01-01T00:00:00.003Z]]}, 1970-01-01T00:00:00.003Z]",
+ "+I[Bob, {1969-12-31T23:59:59.995Z, 1969-12-31T23:59:59.995Z, 1970-01-01T00:00:00.005Z, [+I[Bob, 1, 1970-01-01T00:00:00Z], +I[Bob, 3, 1970-01-01T00:00:00.003Z], +I[Bob, 2, 1970-01-01T00:00:00.003Z], +I[Bob, 4, 1970-01-01T00:00:00.005Z]]}, 1970-01-01T00:00:00.005Z]",
+ "+I[Bob, {1970-01-01T00:00:00.009Z, 1970-01-01T00:00:00.009Z, 1970-01-01T00:00:00.010Z, [+I[Bob, 1, 1970-01-01T00:00:00Z], +I[Bob, 3, 1970-01-01T00:00:00.003Z], +I[Bob, 2, 1970-01-01T00:00:00.003Z], +I[Bob, 4, 1970-01-01T00:00:00.005Z], +I[Bob, 5, 1970-01-01T00:00:00.010Z]]}, 1970-01-01T00:00:00.010Z]",
+ "+I[Bob, {1970-01-01T00:00:00.009Z, 1970-01-01T00:00:00.009Z, 1970-01-01T00:00:00.012Z, [+I[Bob, 1, 1970-01-01T00:00:00Z], +I[Bob, 3, 1970-01-01T00:00:00.003Z], +I[Bob, 2, 1970-01-01T00:00:00.003Z], +I[Bob, 4, 1970-01-01T00:00:00.005Z], +I[Bob, 5, 1970-01-01T00:00:00.010Z], +I[Bob, 6, 1970-01-01T00:00:00.012Z]]}, 1970-01-01T00:00:00.012Z]",
+ "+I[Alice, {1970-01-01T00:00:00.012Z, 1970-01-01T00:00:00.012Z, 1970-01-01T00:00:00.019Z, [+I[Alice, 1, 1970-01-01T00:00:00.001Z], +I[Alice, 2, 1970-01-01T00:00:00.019Z]]}, 1970-01-01T00:00:00.019Z]",
+ "+I[Bob, {1970-01-01T00:00:00.012Z, 1970-01-01T00:00:00.012Z, 1970-01-01T00:00:00.022Z, [+I[Bob, 1, 1970-01-01T00:00:00Z], +I[Bob, 3, 1970-01-01T00:00:00.003Z], +I[Bob, 2, 1970-01-01T00:00:00.003Z], +I[Bob, 4, 1970-01-01T00:00:00.005Z], +I[Bob, 5, 1970-01-01T00:00:00.010Z], +I[Bob, 6, 1970-01-01T00:00:00.012Z], +I[Bob, 7, 1970-01-01T00:00:00.022Z]]}, 1970-01-01T00:00:00.022Z]")
+ .build())
+ .runTableApi(
+ env ->
+ env.fromCall(
+ OrderByFunction.class,
+ env.from("t_large")
+ .partitionBy($("name"))
+ .orderBy($("ts").asc(), $("score").desc())
+ .asArgument("r"),
+ descriptor("ts").asArgument("on_time")),
+ "sink")
+ .build();
+
+ public static final TableTestProgram PROCESS_ORDER_BY_RESTORE =
+ TableTestProgram.of(
+ "process-order-by-restore",
+ "test the ORDER BY clause on watermarked column for restore tests")
+ .setupTemporarySystemFunction("f", OrderByFunction.class)
+ .setupTableSource(
+ SourceTestStep.newBuilder("t_large")
+ .addSchema(
+ "name STRING",
+ "score INT",
+ "ts TIMESTAMP_LTZ(3)",
+ "WATERMARK FOR ts AS ts - INTERVAL '0.010' SECOND")
+ .producedBeforeRestore(
+ Row.of("Bob", 1, Instant.ofEpochMilli(0)),
+ Row.of("Alice", 1, Instant.ofEpochMilli(1)),
+ Row.of("Bob", 2, Instant.ofEpochMilli(3)),
+ Row.of("Bob", 3, Instant.ofEpochMilli(3)),
+ Row.of("Bob", 4, Instant.ofEpochMilli(5)),
+ Row.of("Alice", 2, Instant.ofEpochMilli(19)),
+ Row.of("Alice", 3, Instant.ofEpochMilli(1)))
+ .producedAfterRestore(
+ Row.of("Bob", 5, Instant.ofEpochMilli(10)),
+ Row.of("Bob", 6, Instant.ofEpochMilli(12)),
+ Row.of("Bob", 7, Instant.ofEpochMilli(22)))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink")
+ .addSchema(KEYED_TIMED_BASE_SINK_SCHEMA)
+ .consumedBeforeRestore(
+ // Schema:
+ // (key, current watermark, table watermark, current
+ // timestamp, captured order, output timestamp)
+ "+I[Bob, {1969-12-31T23:59:59.995Z, 1969-12-31T23:59:59.995Z, 1970-01-01T00:00:00Z, [+I[Bob, 1, 1970-01-01T00:00:00Z]]}, 1970-01-01T00:00:00Z]",
+ "+I[Alice, {1969-12-31T23:59:59.995Z, 1969-12-31T23:59:59.995Z, 1970-01-01T00:00:00.001Z, [+I[Alice, 1, 1970-01-01T00:00:00.001Z]]}, 1970-01-01T00:00:00.001Z]",
+ "+I[Bob, {1969-12-31T23:59:59.995Z, 1969-12-31T23:59:59.995Z, 1970-01-01T00:00:00.003Z, [+I[Bob, 1, 1970-01-01T00:00:00Z], +I[Bob, 3, 1970-01-01T00:00:00.003Z]]}, 1970-01-01T00:00:00.003Z]",
+ "+I[Bob, {1969-12-31T23:59:59.995Z, 1969-12-31T23:59:59.995Z, 1970-01-01T00:00:00.003Z, [+I[Bob, 1, 1970-01-01T00:00:00Z], +I[Bob, 3, 1970-01-01T00:00:00.003Z], +I[Bob, 2, 1970-01-01T00:00:00.003Z]]}, 1970-01-01T00:00:00.003Z]",
+ "+I[Bob, {1969-12-31T23:59:59.995Z, 1969-12-31T23:59:59.995Z, 1970-01-01T00:00:00.005Z, [+I[Bob, 1, 1970-01-01T00:00:00Z], +I[Bob, 3, 1970-01-01T00:00:00.003Z], +I[Bob, 2, 1970-01-01T00:00:00.003Z], +I[Bob, 4, 1970-01-01T00:00:00.005Z]]}, 1970-01-01T00:00:00.005Z]")
+ .consumedAfterRestore(
+ // Currently, we don't checkpoint watermarks. Thus, the
+ // logical clock will go backwards after restore. If the
+ // watermark strategy is not chosen well, it can mess up
+ // the initial ordering right after restore. This
+ // is the intended behavior for PTFs for now and should
+ // be addressed upstream (most likely in source splits).
+ "+I[Bob, {1970-01-01T00:00:00.002Z, 1970-01-01T00:00:00.002Z, 1970-01-01T00:00:00.010Z, [+I[Bob, 1, 1970-01-01T00:00:00Z], +I[Bob, 3, 1970-01-01T00:00:00.003Z], +I[Bob, 2, 1970-01-01T00:00:00.003Z], +I[Bob, 4, 1970-01-01T00:00:00.005Z], +I[Bob, 5, 1970-01-01T00:00:00.010Z]]}, 1970-01-01T00:00:00.010Z]",
+ "+I[Bob, {1970-01-01T00:00:00.002Z, 1970-01-01T00:00:00.002Z, 1970-01-01T00:00:00.012Z, [+I[Bob, 1, 1970-01-01T00:00:00Z], +I[Bob, 3, 1970-01-01T00:00:00.003Z], +I[Bob, 2, 1970-01-01T00:00:00.003Z], +I[Bob, 4, 1970-01-01T00:00:00.005Z], +I[Bob, 5, 1970-01-01T00:00:00.010Z], +I[Bob, 6, 1970-01-01T00:00:00.012Z]]}, 1970-01-01T00:00:00.012Z]",
+ "+I[Alice, {1970-01-01T00:00:00.012Z, 1970-01-01T00:00:00.012Z, 1970-01-01T00:00:00.019Z, [+I[Alice, 1, 1970-01-01T00:00:00.001Z], +I[Alice, 2, 1970-01-01T00:00:00.019Z]]}, 1970-01-01T00:00:00.019Z]",
+ "+I[Bob, {1970-01-01T00:00:00.012Z, 1970-01-01T00:00:00.012Z, 1970-01-01T00:00:00.022Z, [+I[Bob, 1, 1970-01-01T00:00:00Z], +I[Bob, 3, 1970-01-01T00:00:00.003Z], +I[Bob, 2, 1970-01-01T00:00:00.003Z], +I[Bob, 4, 1970-01-01T00:00:00.005Z], +I[Bob, 5, 1970-01-01T00:00:00.010Z], +I[Bob, 6, 1970-01-01T00:00:00.012Z], +I[Bob, 7, 1970-01-01T00:00:00.022Z]]}, 1970-01-01T00:00:00.022Z]")
+ .build())
+ .runSql(
+ "INSERT INTO sink SELECT * FROM f("
+ + "r => TABLE t_large PARTITION BY name ORDER BY (ts ASC, score DESC), "
+ + "on_time => DESCRIPTOR(ts))")
+ .build();
+
+ public static final TableTestProgram PROCESS_MULTI_INPUT_ORDER_BY_RESTORE =
+ TableTestProgram.of(
+ "process-multi-input-order-by-restore",
+ "test the multi-input ORDER BY clause on watermarked column for restore tests")
+ .setupTemporarySystemFunction("f", MultiInputOrderByFunction.class)
+ .setupTableSource(
+ SourceTestStep.newBuilder("t")
+ .addSchema(
+ "name STRING",
+ "score INT",
+ "ts TIMESTAMP_LTZ(3)",
+ "WATERMARK FOR ts AS ts - INTERVAL '0.001' SECOND")
+ .producedBeforeRestore(
+ Row.of("Bob", 1, Instant.ofEpochMilli(0)),
+ Row.of("Alice", 1, Instant.ofEpochMilli(1)),
+ Row.of("Bob", 2, Instant.ofEpochMilli(99999)),
+ Row.of("Bob", 3, Instant.ofEpochMilli(3)),
+ Row.of("Bob", 4, Instant.ofEpochMilli(4)))
+ .producedAfterRestore()
+ .build())
+ .setupTableSource(
+ SourceTestStep.newBuilder("t_large")
+ .addSchema(
+ "name STRING",
+ "score INT",
+ "ts TIMESTAMP_LTZ(3)",
+ "WATERMARK FOR ts AS ts - INTERVAL '0.010' SECOND")
+ .producedBeforeRestore(
+ Row.of("Bob", 1, Instant.ofEpochMilli(0)),
+ Row.of("Alice", 1, Instant.ofEpochMilli(1)),
+ Row.of("Bob", 2, Instant.ofEpochMilli(3)),
+ Row.of("Bob", 3, Instant.ofEpochMilli(3)),
+ Row.of("Bob", 4, Instant.ofEpochMilli(5)),
+ Row.of("Alice", 2, Instant.ofEpochMilli(19)),
+ Row.of("Alice", 3, Instant.ofEpochMilli(1)))
+ .producedAfterRestore(
+ Row.of("Bob", 5, Instant.ofEpochMilli(10)),
+ Row.of("Bob", 6, Instant.ofEpochMilli(12)),
+ Row.of("Bob", 7, Instant.ofEpochMilli(22)))
+ .build())
+ .setupTableSink(
+ SinkTestStep.newBuilder("sink")
+ .addSchema(TIMED_MULTI_BASE_SINK_SCHEMA)
+ .consumedBeforeRestore(
+ // Schema:
+ // (key1, key2, input, table watermark, current
+ // timestamp, captured order, output timestamp)
+ "+I[Bob, Bob, {in1, 1969-12-31T23:59:59.999Z, 1970-01-01T00:00:00Z, [+I[Bob, 1, 1970-01-01T00:00:00Z]]}, 1970-01-01T00:00:00Z]",
+ "+I[Alice, Alice, {in1, 1970-01-01T00:00:00Z, 1970-01-01T00:00:00.001Z, [+I[Alice, 1, 1970-01-01T00:00:00.001Z]]}, 1970-01-01T00:00:00.001Z]",
+ "+I[Bob, Bob, {in2, 1969-12-31T23:59:59.995Z, 1970-01-01T00:00:00Z, [+I[Bob, 1, 1970-01-01T00:00:00Z]]}, 1970-01-01T00:00:00Z]",
+ "+I[Bob, Bob, {in2, 1969-12-31T23:59:59.995Z, 1970-01-01T00:00:00.003Z, [+I[Bob, 1, 1970-01-01T00:00:00Z], +I[Bob, 3, 1970-01-01T00:00:00.003Z]]}, 1970-01-01T00:00:00.003Z]",
+ "+I[Bob, Bob, {in2, 1969-12-31T23:59:59.995Z, 1970-01-01T00:00:00.003Z, [+I[Bob, 1, 1970-01-01T00:00:00Z], +I[Bob, 3, 1970-01-01T00:00:00.003Z], +I[Bob, 2, 1970-01-01T00:00:00.003Z]]}, 1970-01-01T00:00:00.003Z]",
+ "+I[Bob, Bob, {in2, 1969-12-31T23:59:59.995Z, 1970-01-01T00:00:00.005Z, [+I[Bob, 1, 1970-01-01T00:00:00Z], +I[Bob, 3, 1970-01-01T00:00:00.003Z], +I[Bob, 2, 1970-01-01T00:00:00.003Z], +I[Bob, 4, 1970-01-01T00:00:00.005Z]]}, 1970-01-01T00:00:00.005Z]",
+ "+I[Alice, Alice, {in2, 1969-12-31T23:59:59.995Z, 1970-01-01T00:00:00.001Z, [+I[Alice, 1, 1970-01-01T00:00:00.001Z]]}, 1970-01-01T00:00:00.001Z]")
+ .consumedAfterRestore(
+ // Watermark resets after restore but captured order is
+ // still accurate.
+ // (Bob, 2) is flushed because the source for input 1
+ // shuts down
+ "+I[Bob, Bob, {in1, null, 1970-01-01T00:01:39.999Z, [+I[Bob, 1, 1970-01-01T00:00:00Z], +I[Bob, 2, 1970-01-01T00:01:39.999Z]]}, 1970-01-01T00:01:39.999Z]",
+ // (Bob, 7) triggers watermark up to (Bob, 6)
+ "+I[Bob, Bob, {in2, 1970-01-01T00:00:00.002Z, 1970-01-01T00:00:00.010Z, [+I[Bob, 1, 1970-01-01T00:00:00Z], +I[Bob, 3, 1970-01-01T00:00:00.003Z], +I[Bob, 2, 1970-01-01T00:00:00.003Z], +I[Bob, 4, 1970-01-01T00:00:00.005Z], +I[Bob, 5, 1970-01-01T00:00:00.010Z]]}, 1970-01-01T00:00:00.010Z]",
+ "+I[Bob, Bob, {in2, 1970-01-01T00:00:00.002Z, 1970-01-01T00:00:00.012Z, [+I[Bob, 1, 1970-01-01T00:00:00Z], +I[Bob, 3, 1970-01-01T00:00:00.003Z], +I[Bob, 2, 1970-01-01T00:00:00.003Z], +I[Bob, 4, 1970-01-01T00:00:00.005Z], +I[Bob, 5, 1970-01-01T00:00:00.010Z], +I[Bob, 6, 1970-01-01T00:00:00.012Z]]}, 1970-01-01T00:00:00.012Z]",
+ // Source for input 2 shuts down
+ "+I[Bob, Bob, {in2, 1970-01-01T00:00:00.012Z, 1970-01-01T00:00:00.022Z, [+I[Bob, 1, 1970-01-01T00:00:00Z], +I[Bob, 3, 1970-01-01T00:00:00.003Z], +I[Bob, 2, 1970-01-01T00:00:00.003Z], +I[Bob, 4, 1970-01-01T00:00:00.005Z], +I[Bob, 5, 1970-01-01T00:00:00.010Z], +I[Bob, 6, 1970-01-01T00:00:00.012Z], +I[Bob, 7, 1970-01-01T00:00:00.022Z]]}, 1970-01-01T00:00:00.022Z]",
+ "+I[Alice, Alice, {in2, 1970-01-01T00:00:00.012Z, 1970-01-01T00:00:00.019Z, [+I[Alice, 1, 1970-01-01T00:00:00.001Z], +I[Alice, 2, 1970-01-01T00:00:00.019Z]]}, 1970-01-01T00:00:00.019Z]")
+ .build())
+ .runSql(
+ "INSERT INTO sink SELECT * FROM f("
+ + "in1 => TABLE t PARTITION BY name ORDER BY (ts ASC, score DESC), "
+ + "in2 => TABLE t_large PARTITION BY name ORDER BY (ts ASC, score DESC), "
+ + "on_time => DESCRIPTOR(ts))")
+ .build();
}
diff --git a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ProcessTableFunctionTestUtils.java b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ProcessTableFunctionTestUtils.java
index cab52688050..13ca08280b6 100644
--- a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ProcessTableFunctionTestUtils.java
+++ b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/ProcessTableFunctionTestUtils.java
@@ -46,6 +46,7 @@ import java.time.Duration;
import java.time.Instant;
import java.time.LocalDateTime;
import java.time.Period;
+import java.util.ArrayList;
import java.util.Arrays;
import java.util.EnumSet;
import java.util.HashMap;
@@ -63,6 +64,7 @@ import static org.apache.flink.table.annotation.ArgumentTrait.REQUIRE_UPDATE_BEF
import static org.apache.flink.table.annotation.ArgumentTrait.ROW_SEMANTIC_TABLE;
import static org.apache.flink.table.annotation.ArgumentTrait.SET_SEMANTIC_TABLE;
import static org.apache.flink.table.annotation.ArgumentTrait.SUPPORT_UPDATES;
+import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
/** Testing functions for {@link ProcessTableFunction}. */
@@ -138,6 +140,27 @@ public class ProcessTableFunctionTestUtils {
Row.of("Bob", 4, Instant.ofEpochMilli(4)))
.build();
+ public static final SourceTestStep TIMED_SOURCE_LATE_EVENTS_LARGE =
+ SourceTestStep.newBuilder("t_large")
+ .addSchema(
+ "name STRING",
+ "score INT",
+ "ts TIMESTAMP_LTZ(3)",
+ "WATERMARK FOR ts AS ts - INTERVAL '0.010' SECOND")
+ .producedValues(
+ Row.of("Bob", 1, Instant.ofEpochMilli(0)),
+ Row.of("Alice", 1, Instant.ofEpochMilli(1)),
+ Row.of("Bob", 2, Instant.ofEpochMilli(3)),
+ Row.of("Bob", 3, Instant.ofEpochMilli(3)),
+ Row.of("Bob", 4, Instant.ofEpochMilli(5)),
+ Row.of("Alice", 2, Instant.ofEpochMilli(19)),
+ // Late event
+ Row.of("Alice", 3, Instant.ofEpochMilli(1)),
+ Row.of("Bob", 5, Instant.ofEpochMilli(10)),
+ Row.of("Bob", 6, Instant.ofEpochMilli(12)),
+ Row.of("Bob", 7, Instant.ofEpochMilli(22)))
+ .build();
+
/** Corresponds to {@link AppendProcessTableFunctionBase}. */
public static final List<String> BASE_SINK_SCHEMA = List.of("`out` STRING");
@@ -546,6 +569,10 @@ public class ProcessTableFunctionTestUtils {
public static class NamedTimersFunction extends AppendProcessTableFunctionBase {
public void eval(Context ctx, @ArgumentHint({SET_SEMANTIC_TABLE, REQUIRE_ON_TIME}) Row r) {
final TimeContext<Long> timeCtx = ctx.timeContext(Long.class);
+
+ // For single inputs, the table watermark and current watermark always match
+ assertThat(timeCtx.tableWatermark()).isEqualTo(timeCtx.currentWatermark());
+
collectEvalEvent(timeCtx, r);
if (timeCtx.time() == 0) {
@@ -560,6 +587,8 @@ public class ProcessTableFunctionTestUtils {
public void onTimer(OnTimerContext ctx) {
final TimeContext<Long> timeCtx = ctx.timeContext(Long.class);
+ // Firing timers don't reference a table
+ assertThat(timeCtx.tableWatermark()).isNull();
// read
collectOnTimerEvent(ctx);
if (ctx.currentTimer().equals("timeout1")) {
@@ -579,6 +608,10 @@ public class ProcessTableFunctionTestUtils {
public static class UnnamedTimersFunction extends AppendProcessTableFunctionBase {
public void eval(Context ctx, @ArgumentHint({SET_SEMANTIC_TABLE, REQUIRE_ON_TIME}) Row r) {
final TimeContext<Long> timeCtx = ctx.timeContext(Long.class);
+
+ // For single inputs, the table watermark and current watermark always match
+ assertThat(timeCtx.tableWatermark()).isEqualTo(timeCtx.currentWatermark());
+
collectEvalEvent(timeCtx, r);
if (timeCtx.time() == 0) {
@@ -591,6 +624,8 @@ public class ProcessTableFunctionTestUtils {
public void onTimer(OnTimerContext ctx) {
final TimeContext<Long> timeCtx = ctx.timeContext(Long.class);
+ // Firing timers don't reference a table
+ assertThat(timeCtx.tableWatermark()).isNull();
// read
collectOnTimerEvent(ctx);
if (timeCtx.time() == 1) {
@@ -1042,6 +1077,54 @@ public class ProcessTableFunctionTestUtils {
}
}
+ /** Captures the order and time how rows enter the function. */
+ public static class OrderByFunction extends AppendProcessTableFunctionBase {
+
+ public static class CapturedOrder {
+ public List<String> list = new ArrayList<>();
+ }
+
+ public void eval(
+ Context ctx,
+ @StateHint CapturedOrder capturedOrder,
+ @ArgumentHint({SET_SEMANTIC_TABLE, OPTIONAL_PARTITION_BY}) Row r) {
+ capturedOrder.list.add(r.toString());
+ final TimeContext<Instant> timeCtx = ctx.timeContext(Instant.class);
+ collectObjects(
+ timeCtx.currentWatermark(),
+ timeCtx.tableWatermark(),
+ timeCtx.time(),
+ capturedOrder.list);
+ }
+ }
+
+ /** Captures the order and time how rows enter the function from multiple inputs. */
+ public static class MultiInputOrderByFunction extends AppendProcessTableFunctionBase {
+
+ public static class CapturedOrder {
+ public List<String> list1 = new ArrayList<>();
+ public List<String> list2 = new ArrayList<>();
+ }
+
+ public void eval(
+ Context ctx,
+ @StateHint CapturedOrder capturedOrder,
+ @ArgumentHint(SET_SEMANTIC_TABLE) Row in1,
+ @ArgumentHint(SET_SEMANTIC_TABLE) Row in2) {
+ final TimeContext<Instant> timeCtx = ctx.timeContext(Instant.class);
+ if (in1 != null) {
+ capturedOrder.list1.add(in1.toString());
+ collectObjects(
+ "in1", timeCtx.tableWatermark(), timeCtx.time(), capturedOrder.list1);
+ }
+ if (in2 != null) {
+ capturedOrder.list2.add(in2.toString());
+ collectObjects(
+ "in2", timeCtx.tableWatermark(), timeCtx.time(), capturedOrder.list2);
+ }
+ }
+ }
+
// --------------------------------------------------------------------------------------------
// Helpers
// --------------------------------------------------------------------------------------------
diff --git a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/testutils/RestoreTestBase.java b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/testutils/RestoreTestBase.java
index f6ec45fb5a0..d64ac4f858e 100644
--- a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/testutils/RestoreTestBase.java
+++ b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/testutils/RestoreTestBase.java
@@ -301,6 +301,7 @@ public abstract class RestoreTestBase implements TableTestProgramRunner {
options.put("data-id", id);
options.put("terminating", "false");
options.put("runtime-source", "NewSource");
+ enablePerRecordWatermarks(sourceTestStep, options);
sourceTestStep.apply(tEnv, options);
}
@@ -393,6 +394,7 @@ public abstract class RestoreTestBase implements TableTestProgramRunner {
options.put("connector", "values");
options.put("data-id", id);
options.put("runtime-source", "NewSource");
+ enablePerRecordWatermarks(sourceTestStep, options);
if (afterRestoreSource == AfterRestoreSource.INFINITE) {
options.put("terminating", "false");
}
@@ -461,4 +463,16 @@ public abstract class RestoreTestBase implements TableTestProgramRunner {
return TestValuesTableFactory.getResultsAsStrings(tableName);
}
}
+
+ private static void enablePerRecordWatermarks(
+ SourceTestStep sourceTestStep, Map<String, String> options) {
+ // This is in sync with SemanticTestBase and is a better default for time-based testing
+ // because processing-time watermark emission would otherwise lead to flaky tests.
+ // Restore tests that fail with this setting can override it in the test program.
+ if (sourceTestStep.schemaComponents.stream().anyMatch(c -> c.startsWith("WATERMARK FOR"))) {
+ options.put("disable-lookup", "true");
+ options.put("enable-watermark-push-down", "true");
+ options.put("scan.watermark.emit.strategy", "on-event");
+ }
+ }
}
diff --git a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/stream/sql/ProcessTableFunctionTest.java b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/stream/sql/ProcessTableFunctionTest.java
index 659ee47cb97..c00f76fbb66 100644
--- a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/stream/sql/ProcessTableFunctionTest.java
+++ b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/stream/sql/ProcessTableFunctionTest.java
@@ -192,6 +192,16 @@ public class ProcessTableFunctionTest extends TableTestBase {
"SELECT `out`, `rowtime` FROM f(r => TABLE t_watermarked, on_time => DESCRIPTOR(ts))");
}
+ @Test
+ void testOrderBy() {
+ util.addTemporarySystemFunction("f", SetSemanticTableFunction.class);
+ util.verifyRelPlan(
+ "SELECT `out`, `rowtime` FROM f("
+ + "r => TABLE t_watermarked PARTITION BY name ORDER BY (ts ASC, score DESC NULLS FIRST), "
+ + "i => 42, "
+ + "on_time => DESCRIPTOR(ts))");
+ }
+
@Test
void testMissingUid() {
// Function name contains special characters and can thus not be used as UID
@@ -351,10 +361,45 @@ public class ProcessTableFunctionTest extends TableTestBase {
"SELECT * FROM f(r => TABLE t PARTITION BY invalid, i => 1)",
"Invalid column 'invalid' for PARTITION BY clause. Available columns are: [name, score]"),
ErrorSpec.ofSelect(
- "unsupported order by",
+ "order by on row semantic table",
+ RowSemanticTableFunction.class,
+ "SELECT * FROM f(r => TABLE t_watermarked ORDER BY ts, i => 1)",
+ "Only tables with set semantics may be ordered."),
+ ErrorSpec.ofSelect(
+ "order by on non-timestamp column",
+ SetSemanticTableFunction.class,
+ "SELECT * FROM f(r => TABLE t_watermarked PARTITION BY name ORDER BY score, i => 1)",
+ "Invalid ORDER BY clause for table argument 'r'. "
+ + "The first ORDER BY column must be a TIMESTAMP or TIMESTAMP_LTZ column (up to precision 3). "
+ + "However, column 'score' has data type 'INT'."),
+ ErrorSpec.ofSelect(
+ "order by descending timestamp",
+ SetSemanticTableFunction.class,
+ "SELECT * FROM f(r => TABLE t_watermarked PARTITION BY name ORDER BY ts DESC, i => 1)",
+ "Invalid ORDER BY clause for table argument 'r'. "
+ + "The first ORDER BY column on a timestamp column must be in ascending order."),
+ ErrorSpec.ofSelect(
+ "order by mismatch with on_time",
+ SetSemanticTableFunction.class,
+ "WITH dual_ts AS (SELECT name, score, ts, TO_TIMESTAMP_LTZ(10000, 3) AS ts2 FROM t_watermarked) "
+ + "SELECT * FROM f(r => TABLE dual_ts PARTITION BY name ORDER BY ts2, i => 1, on_time => DESCRIPTOR(ts))",
+ "Invalid ORDER BY clause for table argument 'r'. When both ORDER BY and the 'on_time' "
+ + "argument are defined, the referenced timestamp columns must match."),
+ ErrorSpec.ofSelect(
+ "order by duplicate column",
+ SetSemanticTableFunction.class,
+ "SELECT * FROM f(r => TABLE t_watermarked PARTITION BY name ORDER BY (ts, score, ts), i => 1)",
+ "Invalid ORDER BY clause for table argument 'r'. "
+ + "Column 'ts' appears more than once in ORDER BY."),
+ ErrorSpec.ofSelect(
+ "order by on non-time attribute",
SetSemanticTableFunction.class,
- "SELECT * FROM f(r => TABLE t PARTITION BY name ORDER BY score, i => 1)",
- "ORDER BY clause is currently not supported."),
+ "WITH dual_ts AS (SELECT name, score, TO_TIMESTAMP_LTZ(10000, 3) AS ts FROM t_watermarked) "
+ + "SELECT * FROM f(r => TABLE dual_ts PARTITION BY name ORDER BY ts, i => 1)",
+ "Invalid ORDER BY clause for table argument 'r'. The first ORDER BY "
+ + "column 'ts' is not a valid time attribute. Only columns with "
+ + "a watermark declaration qualify for ORDER BY. Also, make sure "
+ + "that the watermarked column is forwarded without any modification."),
ErrorSpec.ofSelect(
"updates into insert-only table arg",
SetSemanticTableFunction.class,
diff --git a/flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/stream/sql/ProcessTableFunctionTest.xml b/flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/stream/sql/ProcessTableFunctionTest.xml
index 55fa321f1eb..d89f766b84a 100644
--- a/flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/stream/sql/ProcessTableFunctionTest.xml
+++ b/flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/stream/sql/ProcessTableFunctionTest.xml
@@ -123,6 +123,29 @@ LogicalProject(out=[$0], rowtime=[$1])
ProcessTableFunction(invocation=[f(TABLE(#0), DESCRIPTOR(_UTF-16LE'ts'), DEFAULT())], uid=[null], select=[out,rowtime], rowType=[RecordType(VARCHAR(2147483647) out, TIMESTAMP_LTZ(3) *ROWTIME* rowtime)])
+- WatermarkAssigner(rowtime=[ts], watermark=[ts])
+- TableSourceScan(table=[[default_catalog, default_database, t_watermarked]], fields=[name, score, ts])
+]]>
+ </Resource>
+ </TestCase>
+ <TestCase name="testOrderBy">
+ <Resource name="sql">
+ <![CDATA[SELECT `out`, `rowtime` FROM f(r => TABLE t_watermarked PARTITION BY name ORDER BY (ts ASC, score DESC NULLS FIRST), i => 42, on_time => DESCRIPTOR(ts))]]>
+ </Resource>
+ <Resource name="ast">
+ <![CDATA[
+LogicalProject(out=[$1], rowtime=[$2])
++- LogicalTableFunctionScan(invocation=[f(TABLE(#0) PARTITION BY($0) ORDER BY($2 ASC_NULLS_LAST, $1 DESC_NULLS_FIRST), 42, DESCRIPTOR(_UTF-16LE'ts'), DEFAULT())], rowType=[RecordType(VARCHAR(2147483647) name, VARCHAR(2147483647) out, TIMESTAMP_LTZ(3) *ROWTIME* rowtime)])
+ +- LogicalProject(name=[$0], score=[$1], ts=[$2])
+ +- LogicalWatermarkAssigner(rowtime=[ts], watermark=[$2])
+ +- LogicalTableScan(table=[[default_catalog, default_database, t_watermarked]])
+]]>
+ </Resource>
+ <Resource name="optimized rel plan">
+ <![CDATA[
+Calc(select=[out, rowtime])
++- ProcessTableFunction(invocation=[f(TABLE(#0) PARTITION BY($0) ORDER BY($2 ASC_NULLS_LAST, $1 DESC_NULLS_FIRST), 42, DESCRIPTOR(_UTF-16LE'ts'), DEFAULT())], uid=[f], select=[name,out,rowtime], rowType=[RecordType(VARCHAR(2147483647) name, VARCHAR(2147483647) out, TIMESTAMP_LTZ(3) *ROWTIME* rowtime)])
+ +- Exchange(distribution=[hash[name]])
+ +- WatermarkAssigner(rowtime=[ts], watermark=[ts])
+ +- TableSourceScan(table=[[default_catalog, default_database, t_watermarked]], fields=[name, score, ts])
]]>
</Resource>
</TestCase>
diff --git a/flink-table/flink-table-planner/src/test/resources/restore-tests/stream-exec-process-table-function_1/process-multi-input-order-by-restore/plan/process-multi-input-order-by-restore.json b/flink-table/flink-table-planner/src/test/resources/restore-tests/stream-exec-process-table-function_1/process-multi-input-order-by-restore/plan/process-multi-input-order-by-restore.json
new file mode 100644
index 00000000000..8196c964041
--- /dev/null
+++ b/flink-table/flink-table-planner/src/test/resources/restore-tests/stream-exec-process-table-function_1/process-multi-input-order-by-restore/plan/process-multi-input-order-by-restore.json
@@ -0,0 +1,519 @@
+{
+ "flinkVersion" : "2.3",
+ "nodes" : [ {
+ "id" : 5,
+ "type" : "stream-exec-table-source-scan_2",
+ "scanTableSource" : {
+ "table" : {
+ "identifier" : "`default_catalog`.`default_database`.`t`",
+ "resolvedTable" : {
+ "schema" : {
+ "columns" : [ {
+ "name" : "name",
+ "dataType" : "VARCHAR(2147483647)"
+ }, {
+ "name" : "score",
+ "dataType" : "INT"
+ }, {
+ "name" : "ts",
+ "dataType" : {
+ "type" : "TIMESTAMP_WITH_LOCAL_TIME_ZONE",
+ "precision" : 3,
+ "kind" : "ROWTIME"
+ }
+ } ],
+ "watermarkSpecs" : [ {
+ "rowtimeAttribute" : "ts",
+ "expression" : {
+ "rexNode" : {
+ "kind" : "CALL",
+ "syntax" : "BINARY",
+ "internalName" : "$-$1",
+ "operands" : [ {
+ "kind" : "INPUT_REF",
+ "inputIndex" : 2,
+ "type" : "TIMESTAMP(3) WITH LOCAL TIME ZONE"
+ }, {
+ "kind" : "LITERAL",
+ "value" : "1",
+ "type" : "INTERVAL SECOND(6) NOT NULL"
+ } ],
+ "type" : "TIMESTAMP(3) WITH LOCAL TIME ZONE"
+ },
+ "serializableString" : "`ts` - INTERVAL '0.001' SECOND"
+ }
+ } ]
+ }
+ }
+ },
+ "abilities" : [ {
+ "type" : "WatermarkPushDown",
+ "watermarkExpr" : {
+ "kind" : "CALL",
+ "syntax" : "BINARY",
+ "internalName" : "$-$1",
+ "operands" : [ {
+ "kind" : "INPUT_REF",
+ "inputIndex" : 2,
+ "type" : "TIMESTAMP(3) WITH LOCAL TIME ZONE"
+ }, {
+ "kind" : "LITERAL",
+ "value" : "1",
+ "type" : "INTERVAL SECOND(6) NOT NULL"
+ } ],
+ "type" : "TIMESTAMP(3) WITH LOCAL TIME ZONE"
+ },
+ "rowtimeExpr" : {
+ "kind" : "INPUT_REF",
+ "inputIndex" : 2,
+ "type" : "TIMESTAMP(3) WITH LOCAL TIME ZONE"
+ },
+ "idleTimeoutMillis" : -1,
+ "producedType" : {
+ "type" : "ROW",
+ "nullable" : false,
+ "fields" : [ {
+ "name" : "name",
+ "fieldType" : "VARCHAR(2147483647)"
+ }, {
+ "name" : "score",
+ "fieldType" : "INT"
+ }, {
+ "name" : "ts",
+ "fieldType" : {
+ "type" : "TIMESTAMP_WITH_LOCAL_TIME_ZONE",
+ "precision" : 3,
+ "kind" : "ROWTIME"
+ }
+ } ]
+ },
+ "watermarkParams" : {
+ "emitStrategy" : "ON_EVENT",
+ "alignGroupName" : null,
+ "alignMaxDrift" : "PT0S",
+ "alignUpdateInterval" : "PT1S",
+ "sourceIdleTimeout" : -1
+ }
+ } ]
+ },
+ "outputType" : {
+ "type" : "ROW",
+ "fields" : [ {
+ "name" : "name",
+ "fieldType" : "VARCHAR(2147483647)"
+ }, {
+ "name" : "score",
+ "fieldType" : "INT"
+ }, {
+ "name" : "ts",
+ "fieldType" : {
+ "type" : "TIMESTAMP_WITH_LOCAL_TIME_ZONE",
+ "precision" : 3,
+ "kind" : "ROWTIME"
+ }
+ } ]
+ },
+ "description" : "TableSourceScan(table=[[default_catalog, default_database, t, watermark=[-(ts, 1:INTERVAL SECOND)], watermarkEmitStrategy=[on-event]]], fields=[name, score, ts])"
+ }, {
+ "id" : 6,
+ "type" : "stream-exec-exchange_1",
+ "inputProperties" : [ {
+ "requiredDistribution" : {
+ "type" : "HASH",
+ "keys" : [ 0 ]
+ },
+ "damBehavior" : "PIPELINED",
+ "priority" : 0
+ } ],
+ "outputType" : {
+ "type" : "ROW",
+ "fields" : [ {
+ "name" : "name",
+ "fieldType" : "VARCHAR(2147483647)"
+ }, {
+ "name" : "score",
+ "fieldType" : "INT"
+ }, {
+ "name" : "ts",
+ "fieldType" : {
+ "type" : "TIMESTAMP_WITH_LOCAL_TIME_ZONE",
+ "precision" : 3,
+ "kind" : "ROWTIME"
+ }
+ } ]
+ },
+ "description" : "Exchange(distribution=[hash[name]])"
+ }, {
+ "id" : 7,
+ "type" : "stream-exec-table-source-scan_2",
+ "scanTableSource" : {
+ "table" : {
+ "identifier" : "`default_catalog`.`default_database`.`t_large`",
+ "resolvedTable" : {
+ "schema" : {
+ "columns" : [ {
+ "name" : "name",
+ "dataType" : "VARCHAR(2147483647)"
+ }, {
+ "name" : "score",
+ "dataType" : "INT"
+ }, {
+ "name" : "ts",
+ "dataType" : {
+ "type" : "TIMESTAMP_WITH_LOCAL_TIME_ZONE",
+ "precision" : 3,
+ "kind" : "ROWTIME"
+ }
+ } ],
+ "watermarkSpecs" : [ {
+ "rowtimeAttribute" : "ts",
+ "expression" : {
+ "rexNode" : {
+ "kind" : "CALL",
+ "syntax" : "BINARY",
+ "internalName" : "$-$1",
+ "operands" : [ {
+ "kind" : "INPUT_REF",
+ "inputIndex" : 2,
+ "type" : "TIMESTAMP(3) WITH LOCAL TIME ZONE"
+ }, {
+ "kind" : "LITERAL",
+ "value" : "10",
+ "type" : "INTERVAL SECOND(6) NOT NULL"
+ } ],
+ "type" : "TIMESTAMP(3) WITH LOCAL TIME ZONE"
+ },
+ "serializableString" : "`ts` - INTERVAL '0.010' SECOND"
+ }
+ } ]
+ }
+ }
+ },
+ "abilities" : [ {
+ "type" : "WatermarkPushDown",
+ "watermarkExpr" : {
+ "kind" : "CALL",
+ "syntax" : "BINARY",
+ "internalName" : "$-$1",
+ "operands" : [ {
+ "kind" : "INPUT_REF",
+ "inputIndex" : 2,
+ "type" : "TIMESTAMP(3) WITH LOCAL TIME ZONE"
+ }, {
+ "kind" : "LITERAL",
+ "value" : "10",
+ "type" : "INTERVAL SECOND(6) NOT NULL"
+ } ],
+ "type" : "TIMESTAMP(3) WITH LOCAL TIME ZONE"
+ },
+ "rowtimeExpr" : {
+ "kind" : "INPUT_REF",
+ "inputIndex" : 2,
+ "type" : "TIMESTAMP(3) WITH LOCAL TIME ZONE"
+ },
+ "idleTimeoutMillis" : -1,
+ "producedType" : {
+ "type" : "ROW",
+ "nullable" : false,
+ "fields" : [ {
+ "name" : "name",
+ "fieldType" : "VARCHAR(2147483647)"
+ }, {
+ "name" : "score",
+ "fieldType" : "INT"
+ }, {
+ "name" : "ts",
+ "fieldType" : {
+ "type" : "TIMESTAMP_WITH_LOCAL_TIME_ZONE",
+ "precision" : 3,
+ "kind" : "ROWTIME"
+ }
+ } ]
+ },
+ "watermarkParams" : {
+ "emitStrategy" : "ON_EVENT",
+ "alignGroupName" : null,
+ "alignMaxDrift" : "PT0S",
+ "alignUpdateInterval" : "PT1S",
+ "sourceIdleTimeout" : -1
+ }
+ } ]
+ },
+ "outputType" : {
+ "type" : "ROW",
+ "fields" : [ {
+ "name" : "name",
+ "fieldType" : "VARCHAR(2147483647)"
+ }, {
+ "name" : "score",
+ "fieldType" : "INT"
+ }, {
+ "name" : "ts",
+ "fieldType" : {
+ "type" : "TIMESTAMP_WITH_LOCAL_TIME_ZONE",
+ "precision" : 3,
+ "kind" : "ROWTIME"
+ }
+ } ]
+ },
+ "description" : "TableSourceScan(table=[[default_catalog, default_database, t_large, watermark=[-(ts, 10:INTERVAL SECOND)], watermarkEmitStrategy=[on-event]]], fields=[name, score, ts])"
+ }, {
+ "id" : 8,
+ "type" : "stream-exec-exchange_1",
+ "inputProperties" : [ {
+ "requiredDistribution" : {
+ "type" : "HASH",
+ "keys" : [ 0 ]
+ },
+ "damBehavior" : "PIPELINED",
+ "priority" : 0
+ } ],
+ "outputType" : {
+ "type" : "ROW",
+ "fields" : [ {
+ "name" : "name",
+ "fieldType" : "VARCHAR(2147483647)"
+ }, {
+ "name" : "score",
+ "fieldType" : "INT"
+ }, {
+ "name" : "ts",
+ "fieldType" : {
+ "type" : "TIMESTAMP_WITH_LOCAL_TIME_ZONE",
+ "precision" : 3,
+ "kind" : "ROWTIME"
+ }
+ } ]
+ },
+ "description" : "Exchange(distribution=[hash[name]])"
+ }, {
+ "id" : 9,
+ "type" : "stream-exec-process-table-function_1",
+ "inputProperties" : [ {
+ "requiredDistribution" : {
+ "type" : "UNKNOWN"
+ },
+ "damBehavior" : "PIPELINED",
+ "priority" : 0
+ }, {
+ "requiredDistribution" : {
+ "type" : "UNKNOWN"
+ },
+ "damBehavior" : "PIPELINED",
+ "priority" : 0
+ } ],
+ "outputType" : {
+ "type" : "ROW",
+ "fields" : [ {
+ "name" : "name",
+ "fieldType" : "VARCHAR(2147483647)"
+ }, {
+ "name" : "name0",
+ "fieldType" : "VARCHAR(2147483647)"
+ }, {
+ "name" : "out",
+ "fieldType" : "VARCHAR(2147483647)"
+ }, {
+ "name" : "rowtime",
+ "fieldType" : {
+ "type" : "TIMESTAMP_WITH_LOCAL_TIME_ZONE",
+ "nullable" : false,
+ "precision" : 3,
+ "kind" : "ROWTIME"
+ }
+ } ]
+ },
+ "description" : "ProcessTableFunction(invocation=[f(TABLE(#0) PARTITION BY($0) ORDER BY($2 ASC_NULLS_LAST, $1 DESC_NULLS_FIRST), TABLE(#1) PARTITION BY($0) ORDER BY($2 ASC_NULLS_LAST, $1 DESC_NULLS_FIRST), DESCRIPTOR(_UTF-16LE'ts'), DEFAULT())], uid=[f], select=[name,name0,out,rowtime], rowType=[RecordType(VARCHAR(2147483647) name, VARCHAR(2147483647) name0, VARCHAR(2147483647) out, TIMESTAMP_LTZ(3) *ROWTIME* rowtime)])",
+ "uid" : "f",
+ "functionCall" : {
+ "kind" : "CALL",
+ "systemName" : "f",
+ "operands" : [ {
+ "kind" : "TABLE_ARG_CALL",
+ "inputIndex" : 0,
+ "partitionKeys" : [ 0 ],
+ "orderKeys" : [ 2, 1 ],
+ "orderDirections" : [ "ASC_NULLS_LAST", "DESC_NULLS_FIRST" ],
+ "type" : {
+ "type" : "ROW",
+ "nullable" : false,
+ "fields" : [ {
+ "name" : "name",
+ "fieldType" : "VARCHAR(2147483647)"
+ }, {
+ "name" : "score",
+ "fieldType" : "INT"
+ }, {
+ "name" : "ts",
+ "fieldType" : {
+ "type" : "TIMESTAMP_WITH_LOCAL_TIME_ZONE",
+ "precision" : 3,
+ "kind" : "ROWTIME"
+ }
+ } ]
+ }
+ }, {
+ "kind" : "TABLE_ARG_CALL",
+ "inputIndex" : 1,
+ "partitionKeys" : [ 0 ],
+ "orderKeys" : [ 2, 1 ],
+ "orderDirections" : [ "ASC_NULLS_LAST", "DESC_NULLS_FIRST" ],
+ "type" : {
+ "type" : "ROW",
+ "nullable" : false,
+ "fields" : [ {
+ "name" : "name",
+ "fieldType" : "VARCHAR(2147483647)"
+ }, {
+ "name" : "score",
+ "fieldType" : "INT"
+ }, {
+ "name" : "ts",
+ "fieldType" : {
+ "type" : "TIMESTAMP_WITH_LOCAL_TIME_ZONE",
+ "precision" : 3,
+ "kind" : "ROWTIME"
+ }
+ } ]
+ }
+ }, {
+ "kind" : "CALL",
+ "internalName" : "$DESCRIPTOR$1",
+ "operands" : [ {
+ "kind" : "LITERAL",
+ "value" : "ts",
+ "type" : "CHAR(2) NOT NULL"
+ } ],
+ "type" : "DESCRIPTOR NOT NULL"
+ }, {
+ "kind" : "CALL",
+ "syntax" : "SPECIAL",
+ "internalName" : "$DEFAULT$1",
+ "type" : "VARCHAR(2147483647)"
+ } ],
+ "type" : {
+ "type" : "ROW",
+ "nullable" : false,
+ "fields" : [ {
+ "name" : "name",
+ "fieldType" : "VARCHAR(2147483647)"
+ }, {
+ "name" : "name0",
+ "fieldType" : "VARCHAR(2147483647)"
+ }, {
+ "name" : "out",
+ "fieldType" : "VARCHAR(2147483647)"
+ }, {
+ "name" : "rowtime",
+ "fieldType" : {
+ "type" : "TIMESTAMP_WITH_LOCAL_TIME_ZONE",
+ "nullable" : false,
+ "precision" : 3,
+ "kind" : "ROWTIME"
+ }
+ } ]
+ }
+ },
+ "inputChangelogModes" : [ [ "INSERT" ], [ "INSERT" ] ],
+ "outputChangelogMode" : [ "INSERT" ]
+ }, {
+ "id" : 10,
+ "type" : "stream-exec-sink_2",
+ "configuration" : {
+ "table.exec.sink.keyed-shuffle" : "AUTO",
+ "table.exec.sink.not-null-enforcer" : "ERROR",
+ "table.exec.sink.rowtime-inserter" : "ENABLED",
+ "table.exec.sink.type-length-enforcer" : "IGNORE",
+ "table.exec.sink.upsert-materialize" : "AUTO"
+ },
+ "dynamicTableSink" : {
+ "table" : {
+ "identifier" : "`default_catalog`.`default_database`.`sink`",
+ "resolvedTable" : {
+ "schema" : {
+ "columns" : [ {
+ "name" : "name",
+ "dataType" : "VARCHAR(2147483647)"
+ }, {
+ "name" : "name0",
+ "dataType" : "VARCHAR(2147483647)"
+ }, {
+ "name" : "out",
+ "dataType" : "VARCHAR(2147483647)"
+ }, {
+ "name" : "rowtime",
+ "dataType" : "TIMESTAMP(3) WITH LOCAL TIME ZONE"
+ } ]
+ }
+ }
+ }
+ },
+ "inputChangelogMode" : [ "INSERT" ],
+ "upsertMaterializeStrategy" : "MAP",
+ "inputProperties" : [ {
+ "requiredDistribution" : {
+ "type" : "UNKNOWN"
+ },
+ "damBehavior" : "PIPELINED",
+ "priority" : 0
+ } ],
+ "outputType" : {
+ "type" : "ROW",
+ "fields" : [ {
+ "name" : "name",
+ "fieldType" : "VARCHAR(2147483647)"
+ }, {
+ "name" : "name0",
+ "fieldType" : "VARCHAR(2147483647)"
+ }, {
+ "name" : "out",
+ "fieldType" : "VARCHAR(2147483647)"
+ }, {
+ "name" : "rowtime",
+ "fieldType" : {
+ "type" : "TIMESTAMP_WITH_LOCAL_TIME_ZONE",
+ "nullable" : false,
+ "precision" : 3,
+ "kind" : "ROWTIME"
+ }
+ } ]
+ },
+ "description" : "Sink(table=[default_catalog.default_database.sink], fields=[name, name0, out, rowtime])"
+ } ],
+ "edges" : [ {
+ "source" : 5,
+ "target" : 6,
+ "shuffle" : {
+ "type" : "FORWARD"
+ },
+ "shuffleMode" : "PIPELINED"
+ }, {
+ "source" : 7,
+ "target" : 8,
+ "shuffle" : {
+ "type" : "FORWARD"
+ },
+ "shuffleMode" : "PIPELINED"
+ }, {
+ "source" : 6,
+ "target" : 9,
+ "shuffle" : {
+ "type" : "FORWARD"
+ },
+ "shuffleMode" : "PIPELINED"
+ }, {
+ "source" : 8,
+ "target" : 9,
+ "shuffle" : {
+ "type" : "FORWARD"
+ },
+ "shuffleMode" : "PIPELINED"
+ }, {
+ "source" : 9,
+ "target" : 10,
+ "shuffle" : {
+ "type" : "FORWARD"
+ },
+ "shuffleMode" : "PIPELINED"
+ } ]
+}
\ No newline at end of file
diff --git a/flink-table/flink-table-planner/src/test/resources/restore-tests/stream-exec-process-table-function_1/process-multi-input-order-by-restore/savepoint/_metadata b/flink-table/flink-table-planner/src/test/resources/restore-tests/stream-exec-process-table-function_1/process-multi-input-order-by-restore/savepoint/_metadata
new file mode 100644
index 00000000000..d85030a2dc8
Binary files /dev/null and b/flink-table/flink-table-planner/src/test/resources/restore-tests/stream-exec-process-table-function_1/process-multi-input-order-by-restore/savepoint/_metadata differ
diff --git a/flink-table/flink-table-planner/src/test/resources/restore-tests/stream-exec-process-table-function_1/process-order-by-restore/plan/process-order-by-restore.json b/flink-table/flink-table-planner/src/test/resources/restore-tests/stream-exec-process-table-function_1/process-order-by-restore/plan/process-order-by-restore.json
new file mode 100644
index 00000000000..134a6625c42
--- /dev/null
+++ b/flink-table/flink-table-planner/src/test/resources/restore-tests/stream-exec-process-table-function_1/process-order-by-restore/plan/process-order-by-restore.json
@@ -0,0 +1,320 @@
+{
+ "flinkVersion" : "2.3",
+ "nodes" : [ {
+ "id" : 1,
+ "type" : "stream-exec-table-source-scan_2",
+ "scanTableSource" : {
+ "table" : {
+ "identifier" : "`default_catalog`.`default_database`.`t_large`",
+ "resolvedTable" : {
+ "schema" : {
+ "columns" : [ {
+ "name" : "name",
+ "dataType" : "VARCHAR(2147483647)"
+ }, {
+ "name" : "score",
+ "dataType" : "INT"
+ }, {
+ "name" : "ts",
+ "dataType" : {
+ "type" : "TIMESTAMP_WITH_LOCAL_TIME_ZONE",
+ "precision" : 3,
+ "kind" : "ROWTIME"
+ }
+ } ],
+ "watermarkSpecs" : [ {
+ "rowtimeAttribute" : "ts",
+ "expression" : {
+ "rexNode" : {
+ "kind" : "CALL",
+ "syntax" : "BINARY",
+ "internalName" : "$-$1",
+ "operands" : [ {
+ "kind" : "INPUT_REF",
+ "inputIndex" : 2,
+ "type" : "TIMESTAMP(3) WITH LOCAL TIME ZONE"
+ }, {
+ "kind" : "LITERAL",
+ "value" : "10",
+ "type" : "INTERVAL SECOND(6) NOT NULL"
+ } ],
+ "type" : "TIMESTAMP(3) WITH LOCAL TIME ZONE"
+ },
+ "serializableString" : "`ts` - INTERVAL '0.010' SECOND"
+ }
+ } ]
+ }
+ }
+ },
+ "abilities" : [ {
+ "type" : "WatermarkPushDown",
+ "watermarkExpr" : {
+ "kind" : "CALL",
+ "syntax" : "BINARY",
+ "internalName" : "$-$1",
+ "operands" : [ {
+ "kind" : "INPUT_REF",
+ "inputIndex" : 2,
+ "type" : "TIMESTAMP(3) WITH LOCAL TIME ZONE"
+ }, {
+ "kind" : "LITERAL",
+ "value" : "10",
+ "type" : "INTERVAL SECOND(6) NOT NULL"
+ } ],
+ "type" : "TIMESTAMP(3) WITH LOCAL TIME ZONE"
+ },
+ "rowtimeExpr" : {
+ "kind" : "INPUT_REF",
+ "inputIndex" : 2,
+ "type" : "TIMESTAMP(3) WITH LOCAL TIME ZONE"
+ },
+ "idleTimeoutMillis" : -1,
+ "producedType" : {
+ "type" : "ROW",
+ "nullable" : false,
+ "fields" : [ {
+ "name" : "name",
+ "fieldType" : "VARCHAR(2147483647)"
+ }, {
+ "name" : "score",
+ "fieldType" : "INT"
+ }, {
+ "name" : "ts",
+ "fieldType" : {
+ "type" : "TIMESTAMP_WITH_LOCAL_TIME_ZONE",
+ "precision" : 3,
+ "kind" : "ROWTIME"
+ }
+ } ]
+ },
+ "watermarkParams" : {
+ "emitStrategy" : "ON_EVENT",
+ "alignGroupName" : null,
+ "alignMaxDrift" : "PT0S",
+ "alignUpdateInterval" : "PT1S",
+ "sourceIdleTimeout" : -1
+ }
+ } ]
+ },
+ "outputType" : {
+ "type" : "ROW",
+ "fields" : [ {
+ "name" : "name",
+ "fieldType" : "VARCHAR(2147483647)"
+ }, {
+ "name" : "score",
+ "fieldType" : "INT"
+ }, {
+ "name" : "ts",
+ "fieldType" : {
+ "type" : "TIMESTAMP_WITH_LOCAL_TIME_ZONE",
+ "precision" : 3,
+ "kind" : "ROWTIME"
+ }
+ } ]
+ },
+ "description" : "TableSourceScan(table=[[default_catalog, default_database, t_large, watermark=[-(ts, 10:INTERVAL SECOND)], watermarkEmitStrategy=[on-event]]], fields=[name, score, ts])"
+ }, {
+ "id" : 2,
+ "type" : "stream-exec-exchange_1",
+ "inputProperties" : [ {
+ "requiredDistribution" : {
+ "type" : "HASH",
+ "keys" : [ 0 ]
+ },
+ "damBehavior" : "PIPELINED",
+ "priority" : 0
+ } ],
+ "outputType" : {
+ "type" : "ROW",
+ "fields" : [ {
+ "name" : "name",
+ "fieldType" : "VARCHAR(2147483647)"
+ }, {
+ "name" : "score",
+ "fieldType" : "INT"
+ }, {
+ "name" : "ts",
+ "fieldType" : {
+ "type" : "TIMESTAMP_WITH_LOCAL_TIME_ZONE",
+ "precision" : 3,
+ "kind" : "ROWTIME"
+ }
+ } ]
+ },
+ "description" : "Exchange(distribution=[hash[name]])"
+ }, {
+ "id" : 3,
+ "type" : "stream-exec-process-table-function_1",
+ "inputProperties" : [ {
+ "requiredDistribution" : {
+ "type" : "UNKNOWN"
+ },
+ "damBehavior" : "PIPELINED",
+ "priority" : 0
+ } ],
+ "outputType" : {
+ "type" : "ROW",
+ "fields" : [ {
+ "name" : "name",
+ "fieldType" : "VARCHAR(2147483647)"
+ }, {
+ "name" : "out",
+ "fieldType" : "VARCHAR(2147483647)"
+ }, {
+ "name" : "rowtime",
+ "fieldType" : {
+ "type" : "TIMESTAMP_WITH_LOCAL_TIME_ZONE",
+ "nullable" : false,
+ "precision" : 3,
+ "kind" : "ROWTIME"
+ }
+ } ]
+ },
+ "description" : "ProcessTableFunction(invocation=[f(TABLE(#0) PARTITION BY($0) ORDER BY($2 ASC_NULLS_LAST, $1 DESC_NULLS_FIRST), DESCRIPTOR(_UTF-16LE'ts'), DEFAULT())], uid=[f], select=[name,out,rowtime], rowType=[RecordType(VARCHAR(2147483647) name, VARCHAR(2147483647) out, TIMESTAMP_LTZ(3) *ROWTIME* rowtime)])",
+ "uid" : "f",
+ "functionCall" : {
+ "kind" : "CALL",
+ "systemName" : "f",
+ "operands" : [ {
+ "kind" : "TABLE_ARG_CALL",
+ "inputIndex" : 0,
+ "partitionKeys" : [ 0 ],
+ "orderKeys" : [ 2, 1 ],
+ "orderDirections" : [ "ASC_NULLS_LAST", "DESC_NULLS_FIRST" ],
+ "type" : {
+ "type" : "ROW",
+ "nullable" : false,
+ "fields" : [ {
+ "name" : "name",
+ "fieldType" : "VARCHAR(2147483647)"
+ }, {
+ "name" : "score",
+ "fieldType" : "INT"
+ }, {
+ "name" : "ts",
+ "fieldType" : {
+ "type" : "TIMESTAMP_WITH_LOCAL_TIME_ZONE",
+ "precision" : 3,
+ "kind" : "ROWTIME"
+ }
+ } ]
+ }
+ }, {
+ "kind" : "CALL",
+ "internalName" : "$DESCRIPTOR$1",
+ "operands" : [ {
+ "kind" : "LITERAL",
+ "value" : "ts",
+ "type" : "CHAR(2) NOT NULL"
+ } ],
+ "type" : "DESCRIPTOR NOT NULL"
+ }, {
+ "kind" : "CALL",
+ "syntax" : "SPECIAL",
+ "internalName" : "$DEFAULT$1",
+ "type" : "VARCHAR(2147483647)"
+ } ],
+ "type" : {
+ "type" : "ROW",
+ "nullable" : false,
+ "fields" : [ {
+ "name" : "name",
+ "fieldType" : "VARCHAR(2147483647)"
+ }, {
+ "name" : "out",
+ "fieldType" : "VARCHAR(2147483647)"
+ }, {
+ "name" : "rowtime",
+ "fieldType" : {
+ "type" : "TIMESTAMP_WITH_LOCAL_TIME_ZONE",
+ "nullable" : false,
+ "precision" : 3,
+ "kind" : "ROWTIME"
+ }
+ } ]
+ }
+ },
+ "inputChangelogModes" : [ [ "INSERT" ] ],
+ "outputChangelogMode" : [ "INSERT" ]
+ }, {
+ "id" : 4,
+ "type" : "stream-exec-sink_2",
+ "configuration" : {
+ "table.exec.sink.keyed-shuffle" : "AUTO",
+ "table.exec.sink.not-null-enforcer" : "ERROR",
+ "table.exec.sink.rowtime-inserter" : "ENABLED",
+ "table.exec.sink.type-length-enforcer" : "IGNORE",
+ "table.exec.sink.upsert-materialize" : "AUTO"
+ },
+ "dynamicTableSink" : {
+ "table" : {
+ "identifier" : "`default_catalog`.`default_database`.`sink`",
+ "resolvedTable" : {
+ "schema" : {
+ "columns" : [ {
+ "name" : "name",
+ "dataType" : "VARCHAR(2147483647)"
+ }, {
+ "name" : "out",
+ "dataType" : "VARCHAR(2147483647)"
+ }, {
+ "name" : "rowtime",
+ "dataType" : "TIMESTAMP(3) WITH LOCAL TIME ZONE"
+ } ]
+ }
+ }
+ }
+ },
+ "inputChangelogMode" : [ "INSERT" ],
+ "upsertMaterializeStrategy" : "MAP",
+ "inputProperties" : [ {
+ "requiredDistribution" : {
+ "type" : "UNKNOWN"
+ },
+ "damBehavior" : "PIPELINED",
+ "priority" : 0
+ } ],
+ "outputType" : {
+ "type" : "ROW",
+ "fields" : [ {
+ "name" : "name",
+ "fieldType" : "VARCHAR(2147483647)"
+ }, {
+ "name" : "out",
+ "fieldType" : "VARCHAR(2147483647)"
+ }, {
+ "name" : "rowtime",
+ "fieldType" : {
+ "type" : "TIMESTAMP_WITH_LOCAL_TIME_ZONE",
+ "nullable" : false,
+ "precision" : 3,
+ "kind" : "ROWTIME"
+ }
+ } ]
+ },
+ "description" : "Sink(table=[default_catalog.default_database.sink], fields=[name, out, rowtime])"
+ } ],
+ "edges" : [ {
+ "source" : 1,
+ "target" : 2,
+ "shuffle" : {
+ "type" : "FORWARD"
+ },
+ "shuffleMode" : "PIPELINED"
+ }, {
+ "source" : 2,
+ "target" : 3,
+ "shuffle" : {
+ "type" : "FORWARD"
+ },
+ "shuffleMode" : "PIPELINED"
+ }, {
+ "source" : 3,
+ "target" : 4,
+ "shuffle" : {
+ "type" : "FORWARD"
+ },
+ "shuffleMode" : "PIPELINED"
+ } ]
+}
\ No newline at end of file
diff --git a/flink-table/flink-table-planner/src/test/resources/restore-tests/stream-exec-process-table-function_1/process-order-by-restore/savepoint/_metadata b/flink-table/flink-table-planner/src/test/resources/restore-tests/stream-exec-process-table-function_1/process-order-by-restore/savepoint/_metadata
new file mode 100644
index 00000000000..5b17e1978ad
Binary files /dev/null and b/flink-table/flink-table-planner/src/test/resources/restore-tests/stream-exec-process-table-function_1/process-order-by-restore/savepoint/_metadata differ
diff --git a/flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/plan/metadata/FlinkRelMdHandlerTestBase.scala b/flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/plan/metadata/FlinkRelMdHandlerTestBase.scala
index 88dd7214062..eaca9fa23c9 100644
--- a/flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/plan/metadata/FlinkRelMdHandlerTestBase.scala
+++ b/flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/plan/metadata/FlinkRelMdHandlerTestBase.scala
@@ -3305,7 +3305,7 @@ class FlinkRelMdHandlerTestBase {
ImmutableList.of(calcOnStudentScan),
relBuilder.call(
FlinkSqlOperatorTable.TUMBLE,
- new RexTableArgCall(outputRowType, 0, Array(), Array()),
+ new RexTableArgCall(outputRowType, 0, Array(), Array(), Array()),
relBuilder.call(FlinkSqlOperatorTable.DESCRIPTOR, relBuilder.literal("ptime")),
rexBuilder.makeIntervalLiteral(
bd(600000L),
diff --git a/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/generated/ProcessTableRunner.java b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/generated/ProcessTableRunner.java
index eda3a6c2442..318dd4ae0d3 100644
--- a/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/generated/ProcessTableRunner.java
+++ b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/generated/ProcessTableRunner.java
@@ -64,6 +64,7 @@ public abstract class ProcessTableRunner extends AbstractRichFunction {
protected RowData inputRow;
// Current time
+ private long tableWatermark = Long.MIN_VALUE;
private long currentWatermark = Long.MIN_VALUE;
private @Nullable Long rowtime;
private @Nullable StringData timerName;
@@ -104,7 +105,7 @@ public abstract class ProcessTableRunner extends AbstractRichFunction {
this.valueStateFromFunction = new RowData[stateHandles.length];
}
- public void ingestTableEvent(int pos, RowData row, int timeColumn) {
+ public void ingestTableEvent(int pos, RowData row, int timeColumn, long watermark) {
evalCollector.setPrefix(pos, row);
if (timeColumn == -1) {
rowtime = null;
@@ -117,6 +118,7 @@ public abstract class ProcessTableRunner extends AbstractRichFunction {
}
inputIndex = pos;
inputRow = row;
+ tableWatermark = watermark;
}
public void ingestTimerEvent(RowData key, @Nullable StringData name, long timerTime) {
@@ -126,10 +128,11 @@ public abstract class ProcessTableRunner extends AbstractRichFunction {
}
rowtime = timerTime;
timerName = name;
+ tableWatermark = Long.MIN_VALUE;
}
- public void ingestWatermarkEvent(long watermarkTime) {
- currentWatermark = watermarkTime;
+ public void ingestCurrentWatermarkEvent(long watermark) {
+ currentWatermark = watermark;
}
public void clearAllState() {
@@ -144,6 +147,10 @@ public abstract class ProcessTableRunner extends AbstractRichFunction {
return currentWatermark;
}
+ public long getTableWatermark() {
+ return tableWatermark;
+ }
+
public @Nullable Long getTime() {
return rowtime;
}
diff --git a/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/AbstractProcessTableOperator.java b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/AbstractProcessTableOperator.java
index 5432a826cc2..93eee5e8e4f 100644
--- a/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/AbstractProcessTableOperator.java
+++ b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/AbstractProcessTableOperator.java
@@ -148,7 +148,8 @@ public abstract class AbstractProcessTableOperator extends AbstractStreamOperato
@Override
public void processWatermark(Watermark mark) throws Exception {
super.processWatermark(mark);
- processTableRunner.ingestWatermarkEvent(mark.getTimestamp());
+ // TODO this line has issues with interruptible timers, see FLINK-39437
+ processTableRunner.ingestCurrentWatermarkEvent(mark.getTimestamp());
}
@Override
@@ -164,6 +165,10 @@ public abstract class AbstractProcessTableOperator extends AbstractStreamOperato
@Override
public void onProcessingTime(InternalTimer<RowData, Object> timer) throws Exception {}
+ // --------------------------------------------------------------------------------------------
+ // Context implementation
+ // --------------------------------------------------------------------------------------------
+
/** Implementation of {@link ProcessTableFunction.Context}. */
@Internal
public class RunnerContext implements ProcessTableFunction.Context {
@@ -207,7 +212,9 @@ public abstract class AbstractProcessTableOperator extends AbstractStreamOperato
}
internalTimeContext.setTime(
- processTableRunner.getCurrentWatermark(), processTableRunner.getTime());
+ processTableRunner.getTableWatermark(),
+ processTableRunner.getCurrentWatermark(),
+ processTableRunner.getTime());
return (TimeContext<TimeType>)
new ExternalTimeContext<>(internalTimeContext, timeConverter);
diff --git a/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/ExternalTimeContext.java b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/ExternalTimeContext.java
index 96ec7cbec23..b849e38465f 100644
--- a/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/ExternalTimeContext.java
+++ b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/ExternalTimeContext.java
@@ -50,6 +50,15 @@ class ExternalTimeContext<TimeConversion>
return timeConverter.toExternal(internal);
}
+ @Override
+ public TimeConversion tableWatermark() {
+ final Long internal = internalTimeContext.tableWatermark();
+ if (internal == null) {
+ return null;
+ }
+ return timeConverter.toExternal(internal);
+ }
+
@Override
public TimeConversion currentWatermark() {
final Long internal = internalTimeContext.currentWatermark();
diff --git a/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/InputSortBuffer.java b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/InputSortBuffer.java
new file mode 100644
index 00000000000..ff6ff5676e1
--- /dev/null
+++ b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/InputSortBuffer.java
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.process;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.KeyedStateStore;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.StateTtlConfig;
+import org.apache.flink.api.common.typeutils.base.ListSerializer;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.functions.ProcessTableFunction;
+import org.apache.flink.table.runtime.generated.RecordComparator;
+import org.apache.flink.table.runtime.operators.sink.SortedLongSerializer;
+import org.apache.flink.table.runtime.typeutils.InternalSerializers;
+import org.apache.flink.table.types.logical.LogicalType;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Timer-based sorting buffer for handling ORDER BY in {@link ProcessTableFunction} inputs.
+ *
+ * <p>An instance buffers rows by timestamp and emits them in sorted order when watermark advances.
+ * It implements {@link Triggerable} to handle timer events, registering a single timer per key at
+ * {@code lastWatermark + 1}. This reduces the timer overhead from O(K × T) to O(K) where K = keys
+ * and T = unique timestamps per key.
+ *
+ * <p>When a timer fires, all buffered data with timestamp ≤ current watermark is processed in
+ * sorted order according to the ORDER BY specification.
+ */
+@Internal
+class InputSortBuffer implements Triggerable<RowData, VoidNamespace> {
+
+ private static final Set<String> ORDERED_STATE_BACKENDS = Set.of("rocksdb", "forst");
+
+ /** Functional interface for consuming sorted rows. */
+ interface SortedRowConsumer {
+ void accept(RowData row) throws Exception;
+ }
+
+ private final int timeColumn;
+ private final RecordComparator comparator;
+ private final SortedRowConsumer rowConsumer;
+ private final boolean isOrderedStateBackend;
+ private final MapState<Long, List<RowData>> dataState;
+
+ private InternalTimerService<VoidNamespace> timerService;
+
+ private long triggeringWatermark = Long.MIN_VALUE;
+
+ InputSortBuffer(
+ int inputIdx,
+ LogicalType inputRowType,
+ int timeColumn,
+ RecordComparator comparator,
+ KeyedStateStore keyedStateStore,
+ SortedRowConsumer rowConsumer,
+ StateTtlConfig ttlConfig,
+ KeyedStateBackend<RowData> keyedStateBackend) {
+ this.timeColumn = timeColumn;
+ this.comparator = comparator;
+ this.rowConsumer = rowConsumer;
+ this.isOrderedStateBackend = isOrderedStateBackend(keyedStateBackend);
+
+ final MapStateDescriptor<Long, List<RowData>> mapStateDescriptor =
+ new MapStateDescriptor<>(
+ "input-sort-buffer-data-" + inputIdx,
+ SortedLongSerializer.INSTANCE,
+ new ListSerializer<>(InternalSerializers.create(inputRowType)));
+
+ if (ttlConfig.isEnabled()) {
+ mapStateDescriptor.enableTimeToLive(ttlConfig);
+ }
+
+ this.dataState = keyedStateStore.getMapState(mapStateDescriptor);
+ }
+
+ /** Sets the timer service after construction. Must be called before processing any elements. */
+ void setTimerService(InternalTimerService<VoidNamespace> timerService) {
+ this.timerService = timerService;
+ }
+
+ /** Returns the timer service for this sorting service. */
+ InternalTimerService<VoidNamespace> getTimerService() {
+ return timerService;
+ }
+
+ /** Processes an input element, buffering it for later sorted emission. */
+ void processElement(RowData input) throws Exception {
+ final long rowTime = input.getLong(timeColumn);
+
+ // Drop late events
+ if (rowTime <= triggeringWatermark) {
+ return;
+ }
+
+ // Add row to the buffer for this timestamp
+ List<RowData> rows = dataState.get(rowTime);
+ final boolean isNewTimestamp = rows == null;
+ if (isNewTimestamp) {
+ rows = new ArrayList<>(1);
+ }
+ rows.add(input);
+ dataState.put(rowTime, rows);
+
+ // Register timer at triggeringWatermark + 1 (idempotent - only one timer per key)
+ // This allows batch processing of all timestamps when watermark advances
+ timerService.registerEventTimeTimer(VoidNamespace.INSTANCE, triggeringWatermark + 1);
+ }
+
+ /** Updates the watermark that triggers the current sorting. */
+ public void updateWatermark(long watermark) {
+ triggeringWatermark = watermark;
+ }
+
+ @Override
+ public void onEventTime(InternalTimer<RowData, VoidNamespace> timer) throws Exception {
+ // Timer fired for this key (key context set by Flink timer service)
+ // Process ALL buffered data with rowTime <= current watermark
+ // Must process timestamps in sorted order to maintain ORDER BY semantics
+
+ final boolean hasRemainingData =
+ processSortedTimestamps(
+ dataState,
+ triggeringWatermark,
+ comparator,
+ rowConsumer,
+ isOrderedStateBackend);
+
+ // Re-register timer if any data remains (timestamps > watermark)
+ if (hasRemainingData) {
+ timerService.registerEventTimeTimer(VoidNamespace.INSTANCE, triggeringWatermark + 1);
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<RowData, VoidNamespace> timer) {
+ // Not used - only event-time timers are registered
+ }
+
+ /**
+ * Detects if the given state backend maintains keys in sorted order.
+ *
+ * @param backend the keyed state backend
+ * @return true if the backend is RocksDB or ForSt, false otherwise
+ */
+ private static boolean isOrderedStateBackend(KeyedStateBackend<?> backend) {
+ return ORDERED_STATE_BACKENDS.contains(backend.getBackendTypeIdentifier());
+ }
+
+ /**
+ * Processes buffered timestamps in sorted order, ensuring ORDER BY correctness.
+ *
+ * <p>For ordered state backends (RocksDB/ForSt), entries are already sorted by the backend, so
+ * we can directly iterate and process them. For unordered backends, we collect all timestamps,
+ * sort them manually, and then process each timestamp.
+ *
+ * @param dataState the MapState containing buffered rows by timestamp
+ * @param watermark the current watermark timestamp
+ * @param comparator optional comparator for secondary sorting within a timestamp
+ * @param rowConsumer consumer to emit sorted rows
+ * @param isOrderedBackend whether the state backend maintains sorted key order
+ * @return true if there are remaining timestamps > watermark, false otherwise
+ * @throws Exception if processing fails
+ */
+ private static boolean processSortedTimestamps(
+ MapState<Long, List<RowData>> dataState,
+ long watermark,
+ RecordComparator comparator,
+ SortedRowConsumer rowConsumer,
+ boolean isOrderedBackend)
+ throws Exception {
+
+ boolean hasRemainingData = false;
+
+ if (isOrderedBackend) {
+ // Ordered backend (RocksDB/ForSt): entries already sorted, iterate directly
+ final Iterator<Map.Entry<Long, List<RowData>>> iterator =
+ dataState.entries().iterator();
+ while (iterator.hasNext()) {
+ final Map.Entry<Long, List<RowData>> entry = iterator.next();
+ final long timestamp = entry.getKey();
+
+ if (timestamp <= watermark) {
+ final List<RowData> rows = entry.getValue();
+ if (comparator != null) {
+ rows.sort(comparator);
+ }
+ for (RowData row : rows) {
+ rowConsumer.accept(row);
+ }
+ iterator.remove();
+ } else {
+ // Early exit: all remaining entries have timestamp > watermark
+ hasRemainingData = true;
+ break;
+ }
+ }
+ } else {
+ // Unordered backend: collect, sort, then process
+ final List<Long> readyTimestamps = new ArrayList<>();
+
+ for (Long ts : dataState.keys()) {
+ if (ts <= watermark) {
+ readyTimestamps.add(ts);
+ } else {
+ hasRemainingData = true;
+ }
+ }
+
+ Collections.sort(readyTimestamps);
+
+ for (Long timestamp : readyTimestamps) {
+ final List<RowData> rows = dataState.get(timestamp);
+ if (rows != null) {
+ if (comparator != null) {
+ rows.sort(comparator);
+ }
+ for (RowData row : rows) {
+ rowConsumer.accept(row);
+ }
+ dataState.remove(timestamp);
+ }
+ }
+ }
+
+ return hasRemainingData;
+ }
+}
diff --git a/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/MailboxPartialWatermarkProcessor.java b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/MailboxPartialWatermarkProcessor.java
new file mode 100644
index 00000000000..7996a1d7625
--- /dev/null
+++ b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/MailboxPartialWatermarkProcessor.java
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.process;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.operators.MailboxExecutor;
+import org.apache.flink.api.common.operators.MailboxExecutor.MailOptions;
+import org.apache.flink.streaming.api.operators.InternalTimeServiceManager;
+import org.apache.flink.streaming.api.operators.InternalTimerServiceImpl;
+import org.apache.flink.streaming.api.operators.MailboxWatermarkProcessor;
+import org.apache.flink.streaming.api.watermark.Watermark;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.List;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A specialized {@link MailboxWatermarkProcessor} for {@link AbstractProcessTableOperator} that
+ * manages a list of {@link InternalTimerServiceImpl} instances directly instead of delegating to
+ * {@link InternalTimeServiceManager}.
+ *
+ * <p>This processor advances watermarks across multiple timer services in a controlled manner,
+ * allowing for interruptible timer firing via mailbox execution. When any mail is present in the
+ * mailbox, the process of firing timers is interrupted and a continuation is scheduled to finish it
+ * later.
+ *
+ * <p>When the watermark has been fully advanced across all timer services, the derived watermark is
+ * made available via a consumer callback. This watermark represents the progress of the managed
+ * timer services only, not a global watermark.
+ *
+ * <p>Note that interrupting firing timers can change the order of some invocations. It is possible
+ * that between firing timers, some records might be processed.
+ */
+@Internal
+public class MailboxPartialWatermarkProcessor {
+ protected static final Logger LOG =
+ LoggerFactory.getLogger(MailboxPartialWatermarkProcessor.class);
+
+ private final String description;
+ private final MailboxExecutor mailboxExecutor;
+ private final List<InternalTimerServiceImpl<?, ?>> timerServices;
+ private final WatermarkConsumer watermarkConsumer;
+
+ /**
+ * Flag to indicate whether a progress watermark is scheduled in the mailbox. This is used to
+ * avoid duplicate scheduling in case we have multiple watermarks to process.
+ */
+ private boolean progressWatermarkScheduled = false;
+
+ private Watermark maxInputWatermark = Watermark.UNINITIALIZED;
+
+ public MailboxPartialWatermarkProcessor(
+ String description,
+ MailboxExecutor mailboxExecutor,
+ List<InternalTimerServiceImpl<?, ?>> timerServices,
+ WatermarkConsumer watermarkConsumer) {
+ this.description = checkNotNull(description);
+ this.mailboxExecutor = checkNotNull(mailboxExecutor);
+ this.timerServices = checkNotNull(timerServices);
+ this.watermarkConsumer = checkNotNull(watermarkConsumer);
+ }
+
+ public void emitWatermarkInsideMailbox(Watermark mark) throws Exception {
+ maxInputWatermark =
+ new Watermark(Math.max(maxInputWatermark.getTimestamp(), mark.getTimestamp()));
+ emitWatermarkInsideMailbox();
+ }
+
+ private void emitWatermarkInsideMailbox() throws Exception {
+ // Try to progress min watermark as far as we can.
+ if (tryAdvanceWatermark(maxInputWatermark, mailboxExecutor::shouldInterrupt)) {
+ // In case output watermark has fully progressed, notify the consumer.
+ watermarkConsumer.accept(maxInputWatermark);
+ } else if (!progressWatermarkScheduled) {
+ progressWatermarkScheduled = true;
+ // We still have work to do, but we need to let other mails to be processed first.
+ mailboxExecutor.execute(
+ MailOptions.deferrable(),
+ () -> {
+ progressWatermarkScheduled = false;
+ emitWatermarkInsideMailbox();
+ },
+ description);
+ } else {
+ // We're not guaranteed that MailboxProcessor is going to process all mails before
+ // processing additional input, so the advanceWatermark could be called before the
+ // previous watermark is fully processed.
+ LOG.debug("{} is already scheduled, skipping.", description);
+ }
+ }
+
+ /**
+ * Attempts to advance the watermark across all timer services. If the {@code
+ * shouldStopAdvancingFn} returns true during processing, the advancement is interrupted and can
+ * be resumed later.
+ *
+ * @param watermark the target watermark to advance to
+ * @param shouldStopAdvancingFn function that indicates whether to interrupt the advancement
+ * @return true if the watermark has been fully processed across all timer services, false
+ * otherwise
+ */
+ private boolean tryAdvanceWatermark(
+ Watermark watermark, ShouldStopAdvancingFn shouldStopAdvancingFn) throws Exception {
+ final long targetWatermark = watermark.getTimestamp();
+
+ // Advance watermark across all timer services
+ for (InternalTimerServiceImpl<?, ?> timerService : timerServices) {
+ if (!timerService.tryAdvanceWatermark(targetWatermark, shouldStopAdvancingFn::test)) {
+ // Interrupted while processing this timer service
+ return false;
+ }
+ }
+
+ // All timer services have been fully processed
+ return true;
+ }
+
+ /** Signals whether the watermark should continue advancing. */
+ @FunctionalInterface
+ public interface ShouldStopAdvancingFn {
+
+ /**
+ * @return {@code true} if firing timers should be interrupted.
+ */
+ boolean test();
+ }
+
+ /** Consumer when a watermark has been fully processed by the mailbox. */
+ @FunctionalInterface
+ public interface WatermarkConsumer {
+
+ void accept(Watermark watermark) throws Exception;
+ }
+}
diff --git a/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/ProcessRowTableOperator.java b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/ProcessRowTableOperator.java
index 075f589e926..1964b2d2376 100644
--- a/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/ProcessRowTableOperator.java
+++ b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/ProcessRowTableOperator.java
@@ -78,7 +78,11 @@ public class ProcessRowTableOperator extends AbstractProcessTableOperator
@Override
public void processElement(StreamRecord<RowData> element) throws Exception {
if (inputSemantics != null) {
- processTableRunner.ingestTableEvent(0, element.getValue(), inputSemantics.timeColumn());
+ processTableRunner.ingestTableEvent(
+ 0,
+ element.getValue(),
+ inputSemantics.timeColumn(),
+ processTableRunner.getCurrentWatermark());
}
processTableRunner.processEval();
}
diff --git a/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/ProcessSetTableOperator.java b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/ProcessSetTableOperator.java
index 948464d5f3f..a371465cadd 100644
--- a/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/ProcessSetTableOperator.java
+++ b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/ProcessSetTableOperator.java
@@ -18,18 +18,31 @@
package org.apache.flink.table.runtime.operators.process;
+import org.apache.flink.api.common.operators.MailboxExecutor;
+import org.apache.flink.api.common.state.KeyedStateStore;
+import org.apache.flink.api.common.state.StateTtlConfig;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
import org.apache.flink.streaming.api.operators.AbstractInput;
import org.apache.flink.streaming.api.operators.Input;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.InternalTimerServiceImpl;
import org.apache.flink.streaming.api.operators.MultipleInputStreamOperator;
import org.apache.flink.streaming.api.operators.StreamOperatorParameters;
+import org.apache.flink.streaming.api.watermark.Watermark;
import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
import org.apache.flink.table.data.RowData;
import org.apache.flink.table.functions.ProcessTableFunction;
import org.apache.flink.table.functions.TableSemantics;
import org.apache.flink.table.runtime.generated.HashFunction;
import org.apache.flink.table.runtime.generated.ProcessTableRunner;
+import org.apache.flink.table.runtime.generated.RecordComparator;
import org.apache.flink.table.runtime.generated.RecordEqualiser;
+import org.apache.flink.table.runtime.operators.process.InputSortBuffer.SortedRowConsumer;
+import org.apache.flink.table.types.logical.LogicalType;
+import java.util.ArrayList;
+import java.util.Arrays;
import java.util.List;
import java.util.stream.Collectors;
import java.util.stream.IntStream;
@@ -41,14 +54,24 @@ import java.util.stream.IntStream;
public class ProcessSetTableOperator extends AbstractProcessTableOperator
implements MultipleInputStreamOperator<RowData> {
+ private final RecordComparator[] orderByComparators;
+ private final List<StateTtlConfig> inputBufferTtlConfigs;
+ private final MailboxExecutor mailboxExecutor;
+
+ private transient Watermark[] inputWatermarks;
+ private transient InputSortBuffer[] inputSortBuffers;
+ private transient MailboxPartialWatermarkProcessor[] inputWatermarkProcessors;
+
public ProcessSetTableOperator(
StreamOperatorParameters<RowData> parameters,
List<RuntimeTableSemantics> tableSemantics,
List<RuntimeStateInfo> stateInfos,
+ RecordComparator[] orderByComparators,
ProcessTableRunner processTableRunner,
HashFunction[] stateHashCode,
RecordEqualiser[] stateEquals,
- RuntimeChangelogMode producedChangelogMode) {
+ RuntimeChangelogMode producedChangelogMode,
+ List<StateTtlConfig> inputBufferTtlConfigs) {
super(
parameters,
tableSemantics,
@@ -57,6 +80,18 @@ public class ProcessSetTableOperator extends AbstractProcessTableOperator
stateHashCode,
stateEquals,
producedChangelogMode);
+ this.orderByComparators = orderByComparators;
+ this.inputBufferTtlConfigs = inputBufferTtlConfigs;
+ this.mailboxExecutor = parameters.getMailboxExecutor();
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ setInputWatermarks();
+ setInputSortBuffers();
+ setInputWatermarkProcessors();
}
@Override
@@ -71,12 +106,131 @@ public class ProcessSetTableOperator extends AbstractProcessTableOperator
@Override
public void processElement(StreamRecord<RowData> element)
throws Exception {
- processTableRunner.ingestTableEvent(
- inputIdx, element.getValue(), timeColumn);
- processTableRunner.processEval();
+ final InputSortBuffer sortBuffer = inputSortBuffers[inputIdx];
+ if (sortBuffer != null) {
+ sortBuffer.processElement(element.getValue());
+ } else {
+ processTableEvent(inputIdx, element.getValue(), timeColumn);
+ }
+ }
+
+ @Override
+ public void processWatermark(Watermark mark) throws Exception {
+ // Immediately update the sort buffer to avoid
+ // adding late events while sorting is still in progress
+ if (inputSortBuffers[inputIdx] != null) {
+ inputSortBuffers[inputIdx].updateWatermark(
+ mark.getTimestamp());
+ }
+ // Handle watermarks per-input but in an interruptable way.
+ // Advances the combined watermark eventually.
+ inputWatermarkProcessors[inputIdx].emitWatermarkInsideMailbox(
+ mark);
}
};
})
.collect(Collectors.toList());
}
+
+ private void processTableEvent(int inputIdx, RowData row, int timeColumn) throws Exception {
+ processTableRunner.ingestTableEvent(
+ inputIdx, row, timeColumn, inputWatermarks[inputIdx].getTimestamp());
+ processTableRunner.processEval();
+ }
+
+ // --------------------------------------------------------------------------------------------
+ // Watermark handling
+ // --------------------------------------------------------------------------------------------
+
+ private void setInputWatermarks() {
+ // Initialize per-input watermark tracking
+ inputWatermarks = new Watermark[tableSemantics.size()];
+ Arrays.fill(inputWatermarks, Watermark.UNINITIALIZED);
+ }
+
+ private void setInputSortBuffers() {
+ inputSortBuffers = new InputSortBuffer[tableSemantics.size()];
+
+ IntStream.range(0, tableSemantics.size())
+ .forEach(
+ inputIdx -> {
+ final RuntimeTableSemantics semantics = tableSemantics.get(inputIdx);
+ if (semantics.orderByColumns().length == 0) {
+ return; // No ORDER BY for this input
+ }
+ assert orderByComparators[inputIdx] != null;
+ final RecordComparator comparator = orderByComparators[inputIdx];
+ final int timeColumn = semantics.orderByColumns()[0];
+ final LogicalType inputType = semantics.dataType().getLogicalType();
+ final KeyedStateStore stateStore =
+ getKeyedStateStore().orElseThrow(IllegalStateException::new);
+
+ final int idx = inputIdx;
+ final SortedRowConsumer rowConsumer =
+ row -> processTableEvent(idx, row, timeColumn);
+
+ // Get TTL config for this specific input
+ final StateTtlConfig ttlConfig = inputBufferTtlConfigs.get(inputIdx);
+
+ // Create a sort buffer (it implements Triggerable)
+ inputSortBuffers[inputIdx] =
+ new InputSortBuffer(
+ inputIdx,
+ inputType,
+ timeColumn,
+ comparator,
+ stateStore,
+ rowConsumer,
+ ttlConfig,
+ getKeyedStateBackend());
+
+ // Create a dedicated timer service
+ InternalTimerService<VoidNamespace> timerService =
+ getInternalTimerService(
+ "input-sort-buffer-" + inputIdx,
+ VoidNamespaceSerializer.INSTANCE,
+ inputSortBuffers[inputIdx]);
+
+ // Inject timer service
+ inputSortBuffers[inputIdx].setTimerService(timerService);
+ });
+ }
+
+ private void setInputWatermarkProcessors() {
+ // Multi-inputs always use interruptible timers per input
+ inputWatermarkProcessors = new MailboxPartialWatermarkProcessor[tableSemantics.size()];
+
+ IntStream.range(0, tableSemantics.size())
+ .forEach(
+ pos -> {
+ final int inputIdx = pos;
+
+ // Collect timer services for this input
+ List<InternalTimerServiceImpl<?, ?>> timerServices = new ArrayList<>();
+
+ if (inputSortBuffers[inputIdx] != null) {
+ InternalTimerService<?> timerService =
+ inputSortBuffers[inputIdx].getTimerService();
+ timerServices.add((InternalTimerServiceImpl<?, ?>) timerService);
+ }
+
+ // Create a watermark processor for this input
+ inputWatermarkProcessors[inputIdx] =
+ new MailboxPartialWatermarkProcessor(
+ "input-" + inputIdx + "-watermark",
+ mailboxExecutor,
+ timerServices,
+ derivedWatermark -> {
+ // When timer processing completes:
+ // - Update per-input watermark
+ // - Trigger global watermark
+ advanceInputWatermark(inputIdx, derivedWatermark);
+ reportWatermark(derivedWatermark, inputIdx + 1);
+ });
+ });
+ }
+
+ private void advanceInputWatermark(int inputIdx, Watermark watermark) {
+ inputWatermarks[inputIdx] = watermark;
+ }
}
diff --git a/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/ProcessTableOperatorFactory.java b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/ProcessTableOperatorFactory.java
index d0eed417f11..eb2632955a6 100644
--- a/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/ProcessTableOperatorFactory.java
+++ b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/ProcessTableOperatorFactory.java
@@ -18,15 +18,18 @@
package org.apache.flink.table.runtime.operators.process;
+import org.apache.flink.api.common.state.StateTtlConfig;
import org.apache.flink.streaming.api.operators.AbstractStreamOperatorFactory;
import org.apache.flink.streaming.api.operators.StreamOperator;
import org.apache.flink.streaming.api.operators.StreamOperatorParameters;
import org.apache.flink.table.data.RowData;
import org.apache.flink.table.runtime.generated.GeneratedHashFunction;
import org.apache.flink.table.runtime.generated.GeneratedProcessTableRunner;
+import org.apache.flink.table.runtime.generated.GeneratedRecordComparator;
import org.apache.flink.table.runtime.generated.GeneratedRecordEqualiser;
import org.apache.flink.table.runtime.generated.HashFunction;
import org.apache.flink.table.runtime.generated.ProcessTableRunner;
+import org.apache.flink.table.runtime.generated.RecordComparator;
import org.apache.flink.table.runtime.generated.RecordEqualiser;
import java.util.Arrays;
@@ -39,30 +42,40 @@ public class ProcessTableOperatorFactory extends AbstractStreamOperatorFactory<R
private final List<RuntimeTableSemantics> tableSemantics;
private final List<RuntimeStateInfo> stateInfos;
+ private final GeneratedRecordComparator[] generatedOrderByComparators;
private final GeneratedProcessTableRunner generatedProcessTableRunner;
private final GeneratedHashFunction[] generatedStateHashCode;
private final GeneratedRecordEqualiser[] generatedStateEquals;
private final RuntimeChangelogMode producedChangelogMode;
+ private final List<StateTtlConfig> inputBufferTtlConfigs;
public ProcessTableOperatorFactory(
List<RuntimeTableSemantics> tableSemantics,
List<RuntimeStateInfo> stateInfos,
+ GeneratedRecordComparator[] generatedOrderByComparators,
GeneratedProcessTableRunner generatedProcessTableRunner,
GeneratedHashFunction[] generatedStateHashCode,
GeneratedRecordEqualiser[] generatedStateEquals,
- RuntimeChangelogMode producedChangelogMode) {
+ RuntimeChangelogMode producedChangelogMode,
+ List<StateTtlConfig> inputBufferTtlConfigs) {
this.tableSemantics = tableSemantics;
this.stateInfos = stateInfos;
+ this.generatedOrderByComparators = generatedOrderByComparators;
this.generatedProcessTableRunner = generatedProcessTableRunner;
this.generatedStateHashCode = generatedStateHashCode;
this.generatedStateEquals = generatedStateEquals;
this.producedChangelogMode = producedChangelogMode;
+ this.inputBufferTtlConfigs = inputBufferTtlConfigs;
}
@Override
@SuppressWarnings({"rawtypes", "unchecked"})
public StreamOperator createStreamOperator(StreamOperatorParameters parameters) {
final ClassLoader classLoader = parameters.getContainingTask().getUserCodeClassLoader();
+ final RecordComparator[] orderByComparators =
+ Arrays.stream(generatedOrderByComparators)
+ .map(g -> g == null ? null : g.newInstance(classLoader))
+ .toArray(RecordComparator[]::new);
final ProcessTableRunner runner = generatedProcessTableRunner.newInstance(classLoader);
final HashFunction[] stateHashCode =
Arrays.stream(generatedStateHashCode)
@@ -77,10 +90,12 @@ public class ProcessTableOperatorFactory extends AbstractStreamOperatorFactory<R
parameters,
tableSemantics,
stateInfos,
+ orderByComparators,
runner,
stateHashCode,
stateEquals,
- producedChangelogMode);
+ producedChangelogMode,
+ inputBufferTtlConfigs);
} else {
return new ProcessRowTableOperator(
parameters,
diff --git a/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/ReadableInternalTimeContext.java b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/ReadableInternalTimeContext.java
index 2e0f631fb3d..cbdadac3eb3 100644
--- a/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/ReadableInternalTimeContext.java
+++ b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/ReadableInternalTimeContext.java
@@ -34,10 +34,12 @@ import static org.apache.flink.table.functions.ProcessTableFunction.TimeContext;
@Internal
class ReadableInternalTimeContext implements ProcessTableFunction.TimeContext<Long> {
+ protected long tableWatermark;
protected long currentWatermark;
protected @Nullable Long time;
- void setTime(long currentWatermark, @Nullable Long time) {
+ void setTime(long tableWatermark, long currentWatermark, @Nullable Long time) {
+ this.tableWatermark = tableWatermark;
this.currentWatermark = currentWatermark;
this.time = time;
}
@@ -50,6 +52,14 @@ class ReadableInternalTimeContext implements ProcessTableFunction.TimeContext<Lo
return time;
}
+ @Override
+ public Long tableWatermark() {
+ if (tableWatermark == Long.MIN_VALUE) {
+ return null;
+ }
+ return tableWatermark;
+ }
+
@Override
public Long currentWatermark() {
if (currentWatermark == Long.MIN_VALUE) {
diff --git a/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/RuntimeTableSemantics.java b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/RuntimeTableSemantics.java
index c5d4860d8f4..cabab4c6131 100644
--- a/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/RuntimeTableSemantics.java
+++ b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/RuntimeTableSemantics.java
@@ -38,6 +38,8 @@ public class RuntimeTableSemantics implements TableSemantics, Serializable {
private final int inputIndex;
private final DataType dataType;
private final int[] partitionByColumns;
+ private final int[] orderByColumns;
+ private final SortDirection[] orderByDirections;
private final RuntimeChangelogMode consumedChangelogMode;
private final boolean passColumnsThrough;
private final boolean hasSetSemantics;
@@ -50,6 +52,8 @@ public class RuntimeTableSemantics implements TableSemantics, Serializable {
int inputIndex,
DataType dataType,
int[] partitionByColumns,
+ int[] orderByColumns,
+ SortDirection[] orderByDirections,
RuntimeChangelogMode consumedChangelogMode,
boolean passColumnsThrough,
boolean hasSetSemantics,
@@ -58,6 +62,8 @@ public class RuntimeTableSemantics implements TableSemantics, Serializable {
this.inputIndex = inputIndex;
this.dataType = dataType;
this.partitionByColumns = partitionByColumns;
+ this.orderByColumns = orderByColumns;
+ this.orderByDirections = orderByDirections;
this.consumedChangelogMode = consumedChangelogMode;
this.passColumnsThrough = passColumnsThrough;
this.hasSetSemantics = hasSetSemantics;
@@ -99,7 +105,12 @@ public class RuntimeTableSemantics implements TableSemantics, Serializable {
@Override
public int[] orderByColumns() {
- return new int[0];
+ return orderByColumns;
+ }
+
+ @Override
+ public SortDirection[] orderByDirections() {
+ return orderByDirections;
}
@Override
diff --git a/flink-table/flink-table-runtime/src/test/java/org/apache/flink/table/runtime/operators/process/InputSortBufferTest.java b/flink-table/flink-table-runtime/src/test/java/org/apache/flink/table/runtime/operators/process/InputSortBufferTest.java
new file mode 100644
index 00000000000..906c8da94c9
--- /dev/null
+++ b/flink-table/flink-table-runtime/src/test/java/org/apache/flink/table/runtime/operators/process/InputSortBufferTest.java
@@ -0,0 +1,461 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.process;
+
+import org.apache.flink.api.common.state.KeyedStateStore;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.StateTtlConfig;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.runtime.generated.RecordComparator;
+import org.apache.flink.table.types.logical.BigIntType;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.table.types.logical.VarCharType;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyLong;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+/** Tests for {@link InputSortBuffer}. */
+class InputSortBufferTest {
+
+ private static final int TIME_COLUMN = 0;
+
+ private Map<Long, List<RowData>> backingMap;
+ private MapState<Long, List<RowData>> mapState;
+ private InternalTimerService<VoidNamespace> timerService;
+ private Set<Long> registeredTimers;
+ private List<RowData> emittedRows;
+ private InputSortBuffer.SortedRowConsumer rowConsumer;
+
+ @BeforeEach
+ void setUp() {
+ backingMap =
+ new TreeMap<>(); // Use TreeMap to maintain sorted order for ordered backend tests
+ mapState = createMockMapState();
+ timerService = createMockTimerService();
+ registeredTimers = new HashSet<>();
+ emittedRows = new ArrayList<>();
+ rowConsumer = emittedRows::add;
+ }
+
+ @Test
+ void testBasicElementBuffering() throws Exception {
+ // Setup
+ final InputSortBuffer buffer = createBuffer(null, "rocksdb");
+
+ // Execute
+ buffer.updateWatermark(50);
+ buffer.processElement(createRow(100, "A"));
+
+ // Verify
+ assertThat(backingMap).containsKey(100L);
+ assertThat(backingMap.get(100L)).hasSize(1);
+ assertThat(registeredTimers).contains(51L);
+ }
+
+ @Test
+ void testLateEventFiltering() throws Exception {
+ // Setup
+ final InputSortBuffer buffer = createBuffer(null, "rocksdb");
+
+ // Execute
+ buffer.updateWatermark(100);
+ buffer.processElement(createRow(50, "late"));
+
+ // Verify: Row not stored
+ assertThat(backingMap).doesNotContainKey(50L);
+ assertThat(emittedRows).isEmpty();
+ }
+
+ @Test
+ void testSortedEmissionOrderedBackend() throws Exception {
+ // Setup
+ final InputSortBuffer buffer = createBuffer(null, "rocksdb");
+
+ // Execute: Process out-of-order timestamps
+ buffer.updateWatermark(0);
+ buffer.processElement(createRow(200, "C"));
+ buffer.processElement(createRow(100, "A"));
+ buffer.processElement(createRow(150, "B"));
+
+ buffer.updateWatermark(200);
+ buffer.onEventTime(createTimer(200));
+
+ // Verify: Emitted in timestamp order
+ assertThat(emittedRows).hasSize(3);
+ assertThat(emittedRows.get(0).getLong(TIME_COLUMN)).isEqualTo(100);
+ assertThat(emittedRows.get(1).getLong(TIME_COLUMN)).isEqualTo(150);
+ assertThat(emittedRows.get(2).getLong(TIME_COLUMN)).isEqualTo(200);
+ assertThat(backingMap).isEmpty();
+ assertThat(registeredTimers).doesNotContain(200L); // Not re-registered
+ }
+
+ @Test
+ void testSortedEmissionUnorderedBackend() throws Exception {
+ // Setup
+ final InputSortBuffer buffer = createBuffer(null, "hashmap");
+
+ // Execute: Same as ordered backend test
+ buffer.updateWatermark(0);
+ buffer.processElement(createRow(200, "C"));
+ buffer.processElement(createRow(100, "A"));
+ buffer.processElement(createRow(150, "B"));
+
+ buffer.updateWatermark(200);
+ buffer.onEventTime(createTimer(200));
+
+ // Verify: Same result (manual sort path)
+ assertThat(emittedRows).hasSize(3);
+ assertThat(emittedRows.get(0).getLong(TIME_COLUMN)).isEqualTo(100);
+ assertThat(emittedRows.get(1).getLong(TIME_COLUMN)).isEqualTo(150);
+ assertThat(emittedRows.get(2).getLong(TIME_COLUMN)).isEqualTo(200);
+ }
+
+ @Test
+ void testSecondarySortingWithComparator() throws Exception {
+ // Setup: Comparator that sorts by field[1] (StringData) ascending
+ final RecordComparator comparator = createStringComparator();
+ final InputSortBuffer buffer = createBuffer(comparator, "rocksdb");
+
+ // Execute: Process rows with same timestamp but different IDs
+ buffer.updateWatermark(0);
+ buffer.processElement(createRow(100, "Z"));
+ buffer.processElement(createRow(100, "A"));
+ buffer.processElement(createRow(100, "M"));
+
+ buffer.updateWatermark(100);
+ buffer.onEventTime(createTimer(100));
+
+ // Verify: Emitted in sorted order by ID
+ assertThat(emittedRows).hasSize(3);
+ assertThat(emittedRows.get(0).getString(1).toString()).isEqualTo("A");
+ assertThat(emittedRows.get(1).getString(1).toString()).isEqualTo("M");
+ assertThat(emittedRows.get(2).getString(1).toString()).isEqualTo("Z");
+ }
+
+ @Test
+ void testNoComparatorInsertionOrder() throws Exception {
+ // Setup: No comparator
+ final InputSortBuffer buffer = createBuffer(null, "rocksdb");
+
+ // Execute: Multiple rows at same timestamp
+ buffer.updateWatermark(0);
+ final RowData row1 = createRow(100, "First");
+ final RowData row2 = createRow(100, "Second");
+ final RowData row3 = createRow(100, "Third");
+
+ buffer.processElement(row1);
+ buffer.processElement(row2);
+ buffer.processElement(row3);
+
+ buffer.updateWatermark(100);
+ buffer.onEventTime(createTimer(100));
+
+ // Verify: Emitted in insertion order
+ assertThat(emittedRows).hasSize(3);
+ assertThat(emittedRows.get(0)).isEqualTo(row1);
+ assertThat(emittedRows.get(1)).isEqualTo(row2);
+ assertThat(emittedRows.get(2)).isEqualTo(row3);
+ }
+
+ @Test
+ void testTimerReregistrationForOutstandingData() throws Exception {
+ // Setup
+ final InputSortBuffer buffer = createBuffer(null, "rocksdb");
+
+ // Execute: Set initial watermark, process data, then advance watermark
+ buffer.updateWatermark(50);
+ buffer.processElement(createRow(90, "A"));
+ buffer.processElement(createRow(150, "B"));
+
+ // Verify data is buffered
+ assertThat(backingMap).containsKey(90L);
+ assertThat(backingMap).containsKey(150L);
+
+ buffer.updateWatermark(100);
+ buffer.onEventTime(createTimer(100));
+
+ // Verify: Row(90) emitted, row(150) remains
+ assertThat(emittedRows).hasSize(1);
+ assertThat(emittedRows.get(0).getLong(TIME_COLUMN)).isEqualTo(90);
+ assertThat(backingMap).containsKey(150L);
+ assertThat(backingMap).doesNotContainKey(90L); // Removed after emission
+ // Timer registered twice initially at 51 (once for each processElement), then at 101 after
+ // processing
+ verify(timerService, times(2)).registerEventTimeTimer(eq(VoidNamespace.INSTANCE), eq(51L));
+ verify(timerService).registerEventTimeTimer(eq(VoidNamespace.INSTANCE), eq(101L));
+ }
+
+ @Test
+ void testNoTimerReregistrationWhenComplete() throws Exception {
+ // Setup
+ final InputSortBuffer buffer = createBuffer(null, "rocksdb");
+
+ // Execute: All data within watermark
+ buffer.processElement(createRow(90, "A"));
+ buffer.processElement(createRow(95, "B"));
+ buffer.updateWatermark(100);
+
+ buffer.onEventTime(createTimer(100));
+ registeredTimers.clear(); // Clear initial registration
+
+ // Verify: Both rows emitted, no re-registration
+ assertThat(emittedRows).hasSize(2);
+ assertThat(backingMap).isEmpty();
+ assertThat(registeredTimers).isEmpty();
+ }
+
+ @Test
+ void testEmptyStateOnTimerFire() throws Exception {
+ // Setup
+ final InputSortBuffer buffer = createBuffer(null, "rocksdb");
+
+ // Execute: Fire timer with empty state
+ buffer.updateWatermark(100);
+ buffer.onEventTime(createTimer(100));
+
+ // Verify: No crash, no emissions, no re-registration
+ assertThat(emittedRows).isEmpty();
+ assertThat(registeredTimers).doesNotContain(101L);
+ }
+
+ @Test
+ void testMultipleRowsAtSameTimestamp() throws Exception {
+ // Setup
+ final InputSortBuffer buffer = createBuffer(null, "rocksdb");
+
+ // Execute: 5 rows at same timestamp
+ buffer.updateWatermark(0);
+ for (int i = 0; i < 5; i++) {
+ buffer.processElement(createRow(100, "Row" + i));
+ }
+ buffer.updateWatermark(100);
+ buffer.onEventTime(createTimer(100));
+
+ // Verify: All 5 rows emitted
+ assertThat(emittedRows).hasSize(5);
+ assertThat(backingMap).isEmpty();
+ }
+
+ @Test
+ void testMultipleTimestampsInSingleBatch() throws Exception {
+ // Setup
+ final InputSortBuffer buffer = createBuffer(null, "rocksdb");
+
+ // Execute: Multiple timestamps below watermark
+ buffer.updateWatermark(0);
+ buffer.processElement(createRow(90, "A"));
+ buffer.processElement(createRow(100, "B"));
+ buffer.processElement(createRow(120, "C"));
+ buffer.updateWatermark(150);
+ buffer.onEventTime(createTimer(150));
+
+ // Verify: All emitted in one call
+ assertThat(emittedRows).hasSize(3);
+ assertThat(emittedRows.get(0).getLong(TIME_COLUMN)).isEqualTo(90);
+ assertThat(emittedRows.get(1).getLong(TIME_COLUMN)).isEqualTo(100);
+ assertThat(emittedRows.get(2).getLong(TIME_COLUMN)).isEqualTo(120);
+ }
+
+ @Test
+ void testIdempotentTimerRegistration() throws Exception {
+ // Setup
+ final InputSortBuffer buffer = createBuffer(null, "rocksdb");
+
+ // Execute: Process multiple rows (each triggers registration)
+ buffer.updateWatermark(50);
+ buffer.processElement(createRow(100, "A"));
+ buffer.processElement(createRow(101, "B"));
+ buffer.processElement(createRow(102, "C"));
+
+ // Verify: Timer registered 3 times (InternalTimerService deduplicates)
+ verify(timerService, times(3)).registerEventTimeTimer(eq(VoidNamespace.INSTANCE), eq(51L));
+ }
+
+ @Test
+ void testWatermarkUpdateWithoutElements() {
+ // Setup
+ final InputSortBuffer buffer = createBuffer(null, "rocksdb");
+
+ // Execute: Only watermark updates
+ buffer.updateWatermark(100);
+ buffer.updateWatermark(200);
+
+ // Verify: No state changes, no timers
+ assertThat(backingMap).isEmpty();
+ assertThat(registeredTimers).isEmpty();
+ }
+
+ @Test
+ void testLateEventAfterWatermarkAdvance() throws Exception {
+ // Setup
+ final InputSortBuffer buffer = createBuffer(null, "rocksdb");
+
+ // Execute: Process, fire timer, then same timestamp again
+ buffer.updateWatermark(0);
+ buffer.processElement(createRow(100, "First"));
+
+ buffer.updateWatermark(100);
+ buffer.onEventTime(createTimer(100));
+
+ emittedRows.clear();
+ buffer.processElement(createRow(100, "Late"));
+
+ // Verify: Second row dropped
+ assertThat(emittedRows).isEmpty();
+ assertThat(backingMap).doesNotContainKey(100L);
+ }
+
+ @Test
+ void testInitialWatermarkState() throws Exception {
+ // Setup: Initial state (triggeringWatermark = Long.MIN_VALUE)
+ final InputSortBuffer buffer = createBuffer(null, "rocksdb");
+
+ // Execute: Process row with time=0
+ buffer.processElement(createRow(0, "Zero"));
+
+ // Verify: Row accepted (0 > Long.MIN_VALUE)
+ assertThat(backingMap).containsKey(0L);
+ assertThat(registeredTimers).contains(Long.MIN_VALUE + 1);
+ }
+
+ // --------------------------------------------------------------------------------------------
+ // Helper methods
+ // --------------------------------------------------------------------------------------------
+
+ @SuppressWarnings("unchecked")
+ private InputSortBuffer createBuffer(RecordComparator comparator, String backendType) {
+ final KeyedStateStore stateStore = mock(KeyedStateStore.class);
+ when(stateStore.getMapState(any(MapStateDescriptor.class))).thenReturn(mapState);
+
+ final KeyedStateBackend<RowData> backend = mock(KeyedStateBackend.class);
+ when(backend.getBackendTypeIdentifier()).thenReturn(backendType);
+
+ // Row type: (BIGINT timestamp, VARCHAR id)
+ final RowType inputRowType = RowType.of(new BigIntType(), VarCharType.STRING_TYPE);
+
+ final InputSortBuffer buffer =
+ new InputSortBuffer(
+ 0, // inputIdx
+ inputRowType,
+ TIME_COLUMN,
+ comparator,
+ stateStore,
+ rowConsumer,
+ StateTtlConfig.DISABLED,
+ backend);
+
+ buffer.setTimerService(timerService);
+ return buffer;
+ }
+
+ private RowData createRow(long timestamp, String id) {
+ final GenericRowData row = new GenericRowData(2);
+ row.setField(0, timestamp);
+ row.setField(1, StringData.fromString(id));
+ return row;
+ }
+
+ private InternalTimer<RowData, VoidNamespace> createTimer(long timestamp) {
+ @SuppressWarnings("unchecked")
+ final InternalTimer<RowData, VoidNamespace> timer = mock(InternalTimer.class);
+ when(timer.getTimestamp()).thenReturn(timestamp);
+ when(timer.getNamespace()).thenReturn(VoidNamespace.INSTANCE);
+ return timer;
+ }
+
+ @SuppressWarnings({"unchecked", "SuspiciousMethodCalls"})
+ private MapState<Long, List<RowData>> createMockMapState() {
+ final MapState<Long, List<RowData>> state = mock(MapState.class);
+
+ try {
+ when(state.get(anyLong()))
+ .thenAnswer(invocation -> backingMap.get(invocation.getArgument(0)));
+
+ doAnswer(
+ invocation -> {
+ backingMap.put(
+ invocation.getArgument(0), invocation.getArgument(1));
+ return null;
+ })
+ .when(state)
+ .put(anyLong(), any());
+
+ when(state.keys()).thenAnswer(invocation -> backingMap.keySet());
+ when(state.entries()).thenAnswer(invocation -> backingMap.entrySet());
+
+ doAnswer(
+ invocation -> {
+ backingMap.remove(invocation.getArgument(0));
+ return null;
+ })
+ .when(state)
+ .remove(anyLong());
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+
+ return state;
+ }
+
+ @SuppressWarnings("unchecked")
+ private InternalTimerService<VoidNamespace> createMockTimerService() {
+ final InternalTimerService<VoidNamespace> buffer = mock(InternalTimerService.class);
+ doAnswer(
+ invocation -> {
+ registeredTimers.add(invocation.getArgument(1));
+ return null;
+ })
+ .when(buffer)
+ .registerEventTimeTimer(any(), anyLong());
+
+ return buffer;
+ }
+
+ private RecordComparator createStringComparator() {
+ return (row1, row2) -> {
+ final String s1 = row1.getString(1).toString();
+ final String s2 = row2.getString(1).toString();
+ return s1.compareTo(s2);
+ };
+ }
+}
diff --git a/flink-table/flink-table-runtime/src/test/java/org/apache/flink/table/runtime/operators/process/MailboxPartialWatermarkProcessorTest.java b/flink-table/flink-table-runtime/src/test/java/org/apache/flink/table/runtime/operators/process/MailboxPartialWatermarkProcessorTest.java
new file mode 100644
index 00000000000..8b543c3644a
--- /dev/null
+++ b/flink-table/flink-table-runtime/src/test/java/org/apache/flink/table/runtime/operators/process/MailboxPartialWatermarkProcessorTest.java
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.process;
+
+import org.apache.flink.api.common.operators.MailboxExecutor;
+import org.apache.flink.streaming.api.operators.InternalTimerServiceImpl;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor;
+import org.apache.flink.streaming.runtime.tasks.mailbox.MailboxExecutorImpl;
+import org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailbox;
+import org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.flink.util.function.ThrowingRunnable;
+
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyLong;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+/** Tests for {@link MailboxPartialWatermarkProcessor}. */
+class MailboxPartialWatermarkProcessorTest {
+
+ @Test
+ void testWatermarkAdvancementWithoutInterruption() throws Exception {
+ // Setup: Empty mailbox, single timer service
+ final TaskMailboxImpl mailbox = new TaskMailboxImpl();
+ final MailboxExecutor mailboxExecutor =
+ new MailboxExecutorImpl(mailbox, 0, StreamTaskActionExecutor.IMMEDIATE);
+
+ final InternalTimerServiceImpl<?, ?> timerService = createMockTimerService(true);
+ final List<Watermark> consumedWatermarks = new ArrayList<>();
+
+ final MailboxPartialWatermarkProcessor processor =
+ new MailboxPartialWatermarkProcessor(
+ "test", mailboxExecutor, List.of(timerService), consumedWatermarks::add);
+
+ // Execute
+ processor.emitWatermarkInsideMailbox(new Watermark(100));
+
+ // Verify
+ verify(timerService).tryAdvanceWatermark(eq(100L), any());
+ assertThat(consumedWatermarks).containsExactly(new Watermark(100));
+ assertThat(mailbox.hasMail()).isFalse();
+ }
+
+ @Test
+ void testWatermarkAdvancementWithInterruption() throws Exception {
+ // Setup: Timer service that interrupts first time
+ final TaskMailboxImpl mailbox = new TaskMailboxImpl();
+ final MailboxExecutor mailboxExecutor =
+ new MailboxExecutorImpl(mailbox, 0, StreamTaskActionExecutor.IMMEDIATE);
+
+ final AtomicInteger attemptCount = new AtomicInteger(0);
+ final InternalTimerServiceImpl<?, ?> timerService =
+ createMockTimerServiceWithAttempts(attemptCount, 2);
+
+ final List<Watermark> consumedWatermarks = new ArrayList<>();
+
+ final MailboxPartialWatermarkProcessor processor =
+ new MailboxPartialWatermarkProcessor(
+ "test", mailboxExecutor, List.of(timerService), consumedWatermarks::add);
+
+ // Execute
+ processor.emitWatermarkInsideMailbox(new Watermark(100));
+
+ // Verify: First attempt interrupted, continuation mail scheduled
+ assertThat(consumedWatermarks).isEmpty();
+ assertThat(mailbox.hasMail()).isTrue();
+ assertThat(attemptCount.get()).isEqualTo(1);
+
+ // Execute continuation mail
+ while (mailbox.hasMail()) {
+ mailbox.take(TaskMailbox.MIN_PRIORITY).run();
+ }
+
+ // Verify: Second attempt succeeded
+ assertThat(consumedWatermarks).containsExactly(new Watermark(100));
+ assertThat(attemptCount.get()).isEqualTo(2);
+ }
+
+ @Test
+ void testDuplicateSchedulingPrevention() throws Exception {
+ // Setup: Timer service that completes after 3 attempts
+ final TaskMailboxImpl mailbox = new TaskMailboxImpl();
+ final MailboxExecutor mailboxExecutor =
+ new MailboxExecutorImpl(mailbox, 0, StreamTaskActionExecutor.IMMEDIATE);
+
+ final AtomicInteger attemptCount = new AtomicInteger(0);
+ final InternalTimerServiceImpl<?, ?> timerService =
+ createMockTimerServiceWithAttempts(attemptCount, 3);
+ final List<Watermark> consumedWatermarks = new ArrayList<>();
+
+ final MailboxPartialWatermarkProcessor processor =
+ new MailboxPartialWatermarkProcessor(
+ "test", mailboxExecutor, List.of(timerService), consumedWatermarks::add);
+
+ // Execute twice rapidly
+ processor.emitWatermarkInsideMailbox(new Watermark(100));
+ processor.emitWatermarkInsideMailbox(new Watermark(100));
+
+ // Verify: Both calls attempt to advance, but only one continuation mail scheduled
+ assertThat(consumedWatermarks).isEmpty();
+ assertThat(attemptCount.get()).isEqualTo(2); // Both calls attempted
+
+ // Execute continuation mails until completion
+ while (mailbox.hasMail()) {
+ mailbox.take(TaskMailbox.MIN_PRIORITY).run();
+ }
+
+ // Verify: Completed after exactly 3 attempts total (2 initial + 1 continuation)
+ assertThat(attemptCount.get()).isEqualTo(3);
+ assertThat(consumedWatermarks).containsExactly(new Watermark(100));
+ }
+
+ @Test
+ void testWatermarkMonotonicity() throws Exception {
+ // Setup
+ final TaskMailboxImpl mailbox = new TaskMailboxImpl();
+ final MailboxExecutor mailboxExecutor =
+ new MailboxExecutorImpl(mailbox, 0, StreamTaskActionExecutor.IMMEDIATE);
+
+ final InternalTimerServiceImpl<?, ?> timerService = createMockTimerService(true);
+ final List<Watermark> consumedWatermarks = new ArrayList<>();
+
+ final MailboxPartialWatermarkProcessor processor =
+ new MailboxPartialWatermarkProcessor(
+ "test", mailboxExecutor, List.of(timerService), consumedWatermarks::add);
+
+ // Execute: Non-monotonic watermarks
+ processor.emitWatermarkInsideMailbox(new Watermark(100));
+ processor.emitWatermarkInsideMailbox(new Watermark(50));
+ processor.emitWatermarkInsideMailbox(new Watermark(75));
+ processor.emitWatermarkInsideMailbox(new Watermark(200));
+
+ // Verify: maxInputWatermark tracks maximum and timer service called with it
+ verify(timerService, times(3)).tryAdvanceWatermark(eq(100L), any());
+ verify(timerService, times(1)).tryAdvanceWatermark(eq(200L), any());
+ assertThat(consumedWatermarks)
+ .containsExactly(
+ new Watermark(100),
+ new Watermark(100), // 50 ignored
+ new Watermark(100), // 75 ignored
+ new Watermark(200));
+ }
+
+ @Test
+ void testMultipleTimerServices() throws Exception {
+ // Setup: Three timer services, second interrupts first
+ final TaskMailboxImpl mailbox = new TaskMailboxImpl();
+ final MailboxExecutor mailboxExecutor =
+ new MailboxExecutorImpl(mailbox, 0, StreamTaskActionExecutor.IMMEDIATE);
+
+ final InternalTimerServiceImpl<?, ?> service1 = createMockTimerService(true);
+ final AtomicInteger attemptCount = new AtomicInteger(0);
+ final InternalTimerServiceImpl<?, ?> service2 =
+ createMockTimerServiceWithAttempts(attemptCount, 2);
+ final InternalTimerServiceImpl<?, ?> service3 = createMockTimerService(true);
+
+ final List<Watermark> consumedWatermarks = new ArrayList<>();
+
+ final MailboxPartialWatermarkProcessor processor =
+ new MailboxPartialWatermarkProcessor(
+ "test",
+ mailboxExecutor,
+ Arrays.asList(service1, service2, service3),
+ consumedWatermarks::add);
+
+ // Execute
+ processor.emitWatermarkInsideMailbox(new Watermark(100));
+
+ // Verify: service1 called, service2 interrupted
+ verify(service1).tryAdvanceWatermark(eq(100L), any());
+ verify(service2).tryAdvanceWatermark(eq(100L), any());
+ verify(service3, times(0)).tryAdvanceWatermark(anyLong(), any());
+ assertThat(consumedWatermarks).isEmpty();
+
+ // Execute continuation
+ while (mailbox.hasMail()) {
+ mailbox.take(TaskMailbox.MIN_PRIORITY).run();
+ }
+
+ // Verify: All services called
+ verify(service1, times(2)).tryAdvanceWatermark(eq(100L), any());
+ verify(service2, times(2)).tryAdvanceWatermark(eq(100L), any());
+ verify(service3).tryAdvanceWatermark(eq(100L), any());
+ assertThat(consumedWatermarks).containsExactly(new Watermark(100));
+ }
+
+ @Test
+ void testEmptyTimerServicesList() throws Exception {
+ // Setup: Empty list
+ final TaskMailboxImpl mailbox = new TaskMailboxImpl();
+ final MailboxExecutor mailboxExecutor =
+ new MailboxExecutorImpl(mailbox, 0, StreamTaskActionExecutor.IMMEDIATE);
+
+ final List<Watermark> consumedWatermarks = new ArrayList<>();
+
+ final MailboxPartialWatermarkProcessor processor =
+ new MailboxPartialWatermarkProcessor(
+ "test", mailboxExecutor, List.of(), consumedWatermarks::add);
+
+ // Execute
+ processor.emitWatermarkInsideMailbox(new Watermark(100));
+
+ // Verify: Consumer immediately called
+ assertThat(consumedWatermarks).containsExactly(new Watermark(100));
+ assertThat(mailbox.hasMail()).isFalse();
+ }
+
+ @Test
+ void testConsumerExceptionPropagation() {
+ // Setup: Consumer that throws
+ final TaskMailboxImpl mailbox = new TaskMailboxImpl();
+ final MailboxExecutor mailboxExecutor =
+ new MailboxExecutorImpl(mailbox, 0, StreamTaskActionExecutor.IMMEDIATE);
+
+ final InternalTimerServiceImpl<?, ?> timerService = createMockTimerService(true);
+
+ final MailboxPartialWatermarkProcessor processor =
+ new MailboxPartialWatermarkProcessor(
+ "test",
+ mailboxExecutor,
+ List.of(timerService),
+ watermark -> {
+ throw new RuntimeException("Test exception");
+ });
+
+ // Execute and verify exception propagates
+ assertThatThrownBy(() -> processor.emitWatermarkInsideMailbox(new Watermark(100)))
+ .isInstanceOf(RuntimeException.class)
+ .hasMessage("Test exception");
+ }
+
+ @Test
+ void testContinuationMailIsDeferrable() throws Exception {
+ // Setup: Custom mailbox executor that tracks mail options
+ final AtomicBoolean deferrableUsed = new AtomicBoolean(false);
+ final List<ThrowingRunnable<? extends Exception>> scheduledMails = new ArrayList<>();
+ final MailboxExecutor mailboxExecutor =
+ createTrackingMailboxExecutor(deferrableUsed, scheduledMails);
+
+ final InternalTimerServiceImpl<?, ?> timerService = createMockTimerService(false);
+ final List<Watermark> consumedWatermarks = new ArrayList<>();
+
+ final MailboxPartialWatermarkProcessor processor =
+ new MailboxPartialWatermarkProcessor(
+ "test", mailboxExecutor, List.of(timerService), consumedWatermarks::add);
+
+ // Execute: Should schedule deferrable mail
+ processor.emitWatermarkInsideMailbox(new Watermark(100));
+
+ // Verify: Mail was scheduled as deferrable
+ assertThat(deferrableUsed.get()).isTrue();
+ // The consumer is not called since timer service never completes
+ assertThat(consumedWatermarks).isEmpty();
+ }
+
+ // --------------------------------------------------------------------------------------------
+ // Helper methods
+ // --------------------------------------------------------------------------------------------
+
+ private MailboxExecutor createTrackingMailboxExecutor(
+ AtomicBoolean deferrableUsed,
+ List<ThrowingRunnable<? extends Exception>> scheduledMails) {
+ return new MailboxExecutor() {
+ @Override
+ public void execute(
+ MailOptions mailOptions,
+ ThrowingRunnable<? extends Exception> command,
+ String descriptionFormat,
+ Object... descriptionArgs) {
+ if (mailOptions == MailOptions.deferrable()) {
+ deferrableUsed.set(true);
+ }
+ scheduledMails.add(command);
+ }
+
+ @Override
+ public boolean shouldInterrupt() {
+ return false;
+ }
+
+ @Override
+ public boolean tryYield() throws FlinkRuntimeException {
+ return false;
+ }
+
+ @Override
+ public void yield() throws FlinkRuntimeException {
+ // No-op for test
+ }
+ };
+ }
+
+ private InternalTimerServiceImpl<?, ?> createMockTimerService(boolean completes) {
+ @SuppressWarnings("unchecked")
+ final InternalTimerServiceImpl<Object, Object> mock = mock(InternalTimerServiceImpl.class);
+ try {
+ when(mock.tryAdvanceWatermark(anyLong(), any())).thenReturn(completes);
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ return mock;
+ }
+
+ private InternalTimerServiceImpl<?, ?> createMockTimerServiceWithAttempts(
+ AtomicInteger attemptCount, int succeedOnAttempt) {
+ @SuppressWarnings("unchecked")
+ final InternalTimerServiceImpl<Object, Object> mock = mock(InternalTimerServiceImpl.class);
+ try {
+ when(mock.tryAdvanceWatermark(anyLong(), any()))
+ .thenAnswer(
+ invocation -> {
+ int attempt = attemptCount.incrementAndGet();
+ return attempt >= succeedOnAttempt;
+ });
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ return mock;
+ }
+}
diff --git a/tools/maven/suppressions.xml b/tools/maven/suppressions.xml
index 9688d8ffb91..84cbbdd619d 100644
--- a/tools/maven/suppressions.xml
+++ b/tools/maven/suppressions.xml
@@ -44,7 +44,7 @@ under the License.
<!-- Legacy mockito usages -->
<!-- Updated this suppress files section due to FLINK-36063, which migrated many classes from flink-streaming-java to flink-runtime. This allows runtime access to stream graph. -->
- <suppress files="BufferDataOverWindowOperatorTest.java|CEPOperatorTest.java|CepRuntimeContextTest.java|CliFrontendListTest.java|CliFrontendPackageProgramTest.java|CliFrontendSavepointTest.java|DemultiplexingRecordDeserializerTest.java|DropwizardMeterWrapperTest.java|DynamicEventTimeSessionWindowsTest.java|DynamicProcessingTimeSessionWindowsTest.java|EmbeddedRocksDBStateBackendTest.java|EventTimeSessionWindowsTest.java|FlinkCalciteCatalogReaderTest.java|FlinkMeterWrapperTest.java|Hadoop [...]
+ <suppress files="BufferDataOverWindowOperatorTest.java|CEPOperatorTest.java|CepRuntimeContextTest.java|CliFrontendListTest.java|CliFrontendPackageProgramTest.java|CliFrontendSavepointTest.java|DemultiplexingRecordDeserializerTest.java|DropwizardMeterWrapperTest.java|DynamicEventTimeSessionWindowsTest.java|DynamicProcessingTimeSessionWindowsTest.java|EmbeddedRocksDBStateBackendTest.java|EventTimeSessionWindowsTest.java|FlinkCalciteCatalogReaderTest.java|FlinkMeterWrapperTest.java|Hadoop [...]
checks="IllegalImport"/>
<suppress files="org[\\/]apache[\\/]flink[\\/]formats[\\/]avro[\\/]generated[\\/].*.java" checks="[a-zA-Z0-9]*"/>