You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "autophagy (via GitHub)" <gi...@apache.org> on 2026/06/05 09:25:38 UTC
[PR] Timers [flink]
autophagy opened a new pull request, #28326:
URL: https://github.com/apache/flink/pull/28326
<!--
*Thank you very much for contributing to Apache Flink - we are happy that you want to help us improve Flink. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
*Please understand that we do not do this to make contributions to Flink a hassle. In order to uphold a high standard of quality for code contributions, while at the same time managing a large number of contributions, we need contributors to prepare the contributions well, and give reviewers enough contextual information for the review. Please also understand that contributions that do not follow this guide will take longer to review and thus typically be picked up with lower priority by the community.*
## Contribution Checklist
- Make sure that the pull request corresponds to a [JIRA issue](https://issues.apache.org/jira/projects/FLINK/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no JIRA issue.
- Name the pull request in the form "[FLINK-XXXX] [component] Title of the pull request", where *FLINK-XXXX* should be replaced by the actual issue number. Skip *component* if you are unsure about which is the best component.
Typo fixes that have no associated JIRA issue should be named following this pattern: `[hotfix] [docs] Fix typo in event time introduction` or `[hotfix] [javadocs] Expand JavaDoc for PuncuatedWatermarkGenerator`.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes. You can set up Azure Pipelines CI to do that following [this guide](https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines#AzurePipelines-Tutorial:SettingupAzurePipelinesforaforkoftheFlinkrepository).
- Each pull request should address only one issue, not mix up code from multiple issues.
- Each commit in the pull request has a meaningful commit message (including the JIRA id)
- Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
**(The sections below can be removed for hotfixes of typos)**
-->
## What is the purpose of the change
This PR adds support for `Context`, `OnTimerContext`, `TimeContext`, timer registration and firing, watermark management, and `rowtime` support to the PTF Test Harness.
When a PTF registers a timer, the timer is stored in the TimerManager and keyed by the partition row. When the user advances the watermark using
either `setWatermark` or `setWatermarkForTable`, the manager then fires all pending timers below or equal to the watermark in a deterministic order.
I moved per-invocation state stuff into an `InvocationContext` to make separation of concerns a little easier to follow.
Rows emitted from onTimer firings get the `rowtime` column, and the PTF rejects at the point of registration if the user tries to register a timer with `PASS_COLUMNS_THROUGH`
enabled (similar to live, per the `PROCESS_INVALID_PASS_THROUGH_TIMERS` test on live.)
One open question I have is that there's an edge case where if a user's `onTimer` registers a new timer at or before the current watermark, it then fires, and then registers,
then fires, etc. Should there be some sort of max depth check here?
## Brief change log
- Added support for `Context`, `OnTimerContext`, `TimeContext` in `ProcessTableFunctionTestHarness`
- Added support for timer registration, watermark tracking and timer firing in `ProcessTableFunctionTestHarness`
- Added an `InvocationContext` to capture per-invocation state to simplify the collector logic
- Reworked derriving the output type from the system inference to account for `on_time` and `uid`
- Fixed an issue in `createStateConverter` where the incorrect state converters were being used for Map/List state.
## Verifying this change
This change added tests and can be verified as follows:
- Added tests to `ProcessFunctionTestHarnessesTest` to cover timer firing, watermark advancement and context.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (yes)
- If yes, how is the feature documented? (docs)
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR, please
change the checkbox below to `[X]` followed by the name of the tool, and uncomment the
"Generated-by" line. See the ASF Generative Tooling Guidance for details:
https://www.apache.org/legal/generative-tooling.html
You are responsible for the quality and correctness of every change in this PR
regardless of the tooling used. Low-effort AI-generated PRs will be closed. See
AGENTS.md for the full guidance.
-->
- [x] Yes (please specify the tool below)
2.1.156 (Claude Code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39378][table] Add support for Context, timers and on_time to PTF test harness [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28326:
URL: https://github.com/apache/flink/pull/28326#discussion_r3395340887
##########
flink-table/flink-table-test-utils/src/test/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarnessTest.java:
##########
@@ -1728,4 +1741,888 @@ void testPartitionKeyValidationOnClearState() throws Exception {
harness.close();
}
+
+ // -------------------------------------------------------------------------
+ // Timer PTFs
+ // -------------------------------------------------------------------------
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class TimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ String name = input.getFieldAs("name");
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timeout-" + name, timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered-" + name));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("timer-fired-" + ctx.currentTimer()));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class UnnamedTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
Review Comment:
```suggestion
TimeContext<Instant> timeCtx = ctx.timeContext(Instant.class);
```
to check another codepath?
##########
flink-table/flink-table-test-utils/src/test/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarnessTest.java:
##########
@@ -1728,4 +1741,888 @@ void testPartitionKeyValidationOnClearState() throws Exception {
harness.close();
}
+
+ // -------------------------------------------------------------------------
+ // Timer PTFs
+ // -------------------------------------------------------------------------
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class TimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ String name = input.getFieldAs("name");
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timeout-" + name, timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered-" + name));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("timer-fired-" + ctx.currentTimer()));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class UnnamedTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime(timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ String timerName = ctx.currentTimer();
+ collect(Row.of("fired-unnamed-" + (timerName == null ? "null" : timerName)));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class TimerWithStatePTF extends ProcessTableFunction<Row> {
+ public static class CounterState {
+ public long count = 0L;
+ }
+
+ public void eval(
+ Context ctx,
+ @StateHint CounterState state,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ state.count++;
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("check", timeCtx.time().plus(Duration.ofSeconds(5)));
+ }
+
+ public void onTimer(OnTimerContext ctx, @StateHint CounterState state) {
+ collect(Row.of("count=" + state.count));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class PassThroughTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({
+ ArgumentTrait.SET_SEMANTIC_TABLE,
+ ArgumentTrait.PASS_COLUMNS_THROUGH,
+ ArgumentTrait.REQUIRE_ON_TIME
+ })
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timer", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("fired"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class NoOnTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timer", timeCtx.time().plus(Duration.ofSeconds(10)));
+ collect(Row.of("registered"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class MultipleOnTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timer", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("fired-no-state"));
+ }
+
+ public void onTimer(OnTimerContext ctx, @StateHint String state) {
+ collect(Row.of("fired-with-state"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class CascadingTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("first", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("eval"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ String name = ctx.currentTimer();
+ collect(Row.of("fired-" + name));
+ if ("first".equals(name)) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("second", timeCtx.time().minus(Duration.ofSeconds(2)));
+ }
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class MultiTableTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row leftTable,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row rightTable) {
+ if (leftTable != null) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("check", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("left"));
+ }
+ if (rightTable != null) {
+ collect(Row.of("right"));
+ }
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("timer-fired"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class ContextClearStatePTF extends ProcessTableFunction<Row> {
+ public static class CounterState {
+ public long count = 0L;
+ }
+
+ public void eval(
+ Context ctx,
+ @StateHint CounterState state,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ String action = input.getFieldAs("action");
+ if ("increment".equals(action)) {
+ if (state == null) {
+ state = new CounterState();
+ }
+ state.count++;
+ collect(Row.of("count=" + state.count));
+ } else if ("clear-state".equals(action)) {
+ ctx.clearState("state");
+ collect(Row.of("cleared"));
+ } else if ("clear-all-state".equals(action)) {
+ ctx.clearAllState();
+ collect(Row.of("cleared-all"));
+ } else if ("clear-all".equals(action)) {
+ ctx.clearAll();
+ collect(Row.of("cleared-everything"));
+ } else if ("register-timer".equals(action)) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("t", timeCtx.time().plus(Duration.ofHours(1)));
+ collect(Row.of("timer-registered"));
+ }
+ }
+
+ public void onTimer(OnTimerContext ctx, @StateHint CounterState state) {
+ collect(Row.of("timer-fired"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class PojoTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ TimedEvent input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("check", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered-" + input.key));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("fired-" + ctx.currentTimer()));
+ }
+ }
+
+ @DataTypeHint("ROW<partitionCols STRING, timeCol INT, changelogMode STRING>")
+ public static class ContextSemanticsIntrospectionPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TableSemantics semantics = ctx.tableSemanticsFor("input");
Review Comment:
would be good to do the same check also for `ctx.tableSemanticsFor()` being called from the `onTimer()` method.
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarness.java:
##########
@@ -303,38 +332,441 @@ public void clearStateForKey(String stateName, Row partitionKey) {
stateManager.clearStateForKey(stateName, partitionKey);
}
+ // -------------------------------------------------------------------------
+ // Watermark & Timer API
+ // -------------------------------------------------------------------------
+
+ /**
+ * Sets the watermark for all tables to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermark(LocalDateTime watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /** Sets the watermark for all tables to the given {@link Instant} and fires eligible timers. */
+ public void setWatermark(Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(watermark.toEpochMilli());
+ }
+
+ /**
+ * Sets the watermark for a specific table. Fires eligible timers if this advances the global
+ * watermark (the minimum across all tables).
+ */
+ public void setWatermarkForTable(String tableArgument, LocalDateTime watermark)
+ throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /**
+ * Sets the watermark for a specific table. Fires eligible timers if this advances the global
+ * watermark (the minimum across all tables).
+ */
+ public void setWatermarkForTable(String tableArgument, Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, watermark.toEpochMilli());
+ }
+
+ /** Returns all timers (both pending and fired), sorted by timestamp then name. */
+ public List<Timer> getAllTimers() {
+ return Stream.concat(
+ timerManager.getPendingTimers().stream(),
+ timerManager.getFiredTimers().stream())
+ .sorted()
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all timers (both pending and fired) for the given partition key. */
+ public List<Timer> getAllTimers(Row partitionKey) {
+ return getAllTimers().stream()
+ .filter(t -> partitionKey.equals(t.getKey()))
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all pending (not yet fired) timers, sorted by timestamp then name. */
+ public List<Timer> getPendingTimers() {
+ return timerManager.getPendingTimers();
+ }
+
+ /** Returns all pending timers for the given partition key. */
+ public List<Timer> getPendingTimers(Row partitionKey) {
+ return timerManager.getPendingTimers().stream()
+ .filter(t -> partitionKey.equals(t.getKey()))
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all pending timers with the given name. */
+ public List<Timer> getPendingTimers(String timerName) {
+ return timerManager.getPendingTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all timers that have fired, in the order they fired. */
+ public List<Timer> getFiredTimers() {
+ return timerManager.getFiredTimers();
+ }
+
+ /** Returns all fired timers for the given partition key. */
+ public List<Timer> getFiredTimers(Row partitionKey) {
+ return timerManager.getFiredTimers().stream()
+ .filter(t -> partitionKey.equals(t.getKey()))
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all fired timers with the given name. */
+ public List<Timer> getFiredTimers(String timerName) {
+ return timerManager.getFiredTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Clears the fired timer history. */
+ public void clearFiredTimers() {
+ timerManager.clearFiredTimers();
+ }
+
+ private void setWatermarkMillis(long millis) throws Exception {
+ checkState(isOpen, "Harness is not open");
+ for (TableArgumentInfo tableArg : ArgumentInfo.filterTableArguments(arguments)) {
+ timerManager.setTableWatermark(tableArg.name, millis);
+ }
+ timerManager.updateGlobalWatermarkAndFireTimers(this::fireTimer);
+ }
+
+ private void setWatermarkForTableMillis(String tableArgument, long millis) throws Exception {
+ checkState(isOpen, "Harness is not open");
+ checkNotNull(tableArgument, "tableArgument must not be null");
+ checkArgument(
+ argumentsByName.get(tableArgument) instanceof TableArgumentInfo,
+ "Unknown or non-table argument: %s",
+ tableArgument);
+ timerManager.setTableWatermark(tableArgument, millis);
+ timerManager.updateGlobalWatermarkAndFireTimers(this::fireTimer);
+ }
+
+ private void fireTimer(Timer timer) throws Exception {
+ if (onTimer == null) {
+ throw new IllegalStateException(
+ "Timer fired but no valid onTimer() method is defined in "
+ + function.getClass().getSimpleName());
+ }
+
+ currentInvocation = InvocationContext.forTimer(timer);
+
+ Map<String, Object> stateMap = stateManager.loadStateForKey(timer.partitionKey);
+
+ List<StateArgumentInfo> stateArgs = ArgumentInfo.filterStateArguments(arguments);
+ Object[] methodArgs = new Object[stateArgs.size()];
+ for (int i = 0; i < stateArgs.size(); i++) {
+ methodArgs[i] = stateMap.get(stateArgs.get(i).name);
+ }
+
+ try {
+ onTimer.invoke(function, new TestOnTimerContext(stateMap), methodArgs);
+ } catch (InvocationTargetException e) {
+ Throwable cause = e.getCause();
+ if (cause instanceof Exception) {
+ throw (Exception) cause;
+ }
+ throw new RuntimeException("onTimer() invocation failed", e);
+ } finally {
+ currentInvocation = null;
+ }
+
+ stateManager.updateStateForKey(timer.partitionKey, stateMap);
+ }
+
+ // -------------------------------------------------------------------------
+ // Context implementations
+ // -------------------------------------------------------------------------
+
+ private class TestContext implements ProcessTableFunction.Context {
+ final Map<String, Object> stateMap;
+
+ TestContext(Map<String, Object> stateMap) {
+ this.stateMap = stateMap;
+ }
+
+ @Override
+ public <TimeType> ProcessTableFunction.TimeContext<TimeType> timeContext(
+ Class<TimeType> conversionClass) {
+ return new TestTimeContext<>(conversionClass);
+ }
+
+ @Override
+ public TableSemantics tableSemanticsFor(String argName) {
+ List<String> tableArgNames =
+ ArgumentInfo.filterTableArguments(arguments).stream()
+ .map(t -> t.name)
+ .collect(Collectors.toList());
+ ArgumentInfo argInfo = argumentsByName.get(argName);
+ if (argInfo == null || !(argInfo instanceof TableArgumentInfo)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Argument '%s' is not a table argument. Available table arguments: %s",
+ argName, tableArgNames));
+ }
+ TableArgumentInfo tableArg = (TableArgumentInfo) argInfo;
+ int[] partitionIndices = getPartitionColumnIndices(tableArg);
+ int timeColumnIndex =
+ onTimeColumnName != null
+ ? getFieldNames(tableArg.dataType).indexOf(onTimeColumnName)
+ : -1;
+ return new TestHarnessTableSemantics(
+ tableArg.dataType, partitionIndices, timeColumnIndex);
+ }
+
+ @Override
+ public void clearState(String stateName) {
+ stateMap.remove(stateName);
+ }
+
+ @Override
+ public void clearAllState() {
+ stateMap.clear();
+ }
+
+ @Override
+ public void clearAllTimers() {
+ timerManager.clearAll(currentInvocation.partitionKey);
+ }
+
+ @Override
+ public void clearAll() {
+ stateMap.clear();
+ timerManager.clearAll(currentInvocation.partitionKey);
+ }
+
+ @Override
+ public ChangelogMode getChangelogMode() {
+ return ChangelogMode.insertOnly();
+ }
+ }
+
+ private class TestTimeContext<TimeType> implements ProcessTableFunction.TimeContext<TimeType> {
+ private final Class<TimeType> conversionClass;
+
+ TestTimeContext(Class<TimeType> conversionClass) {
+ this.conversionClass = conversionClass;
Review Comment:
Should we eagerly check here if `conversionClass` is a supported class?
##########
flink-table/flink-table-test-utils/src/test/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarnessTest.java:
##########
@@ -1728,4 +1741,888 @@ void testPartitionKeyValidationOnClearState() throws Exception {
harness.close();
}
+
+ // -------------------------------------------------------------------------
+ // Timer PTFs
+ // -------------------------------------------------------------------------
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class TimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ String name = input.getFieldAs("name");
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timeout-" + name, timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered-" + name));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("timer-fired-" + ctx.currentTimer()));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class UnnamedTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime(timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ String timerName = ctx.currentTimer();
+ collect(Row.of("fired-unnamed-" + (timerName == null ? "null" : timerName)));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class TimerWithStatePTF extends ProcessTableFunction<Row> {
+ public static class CounterState {
+ public long count = 0L;
+ }
+
+ public void eval(
+ Context ctx,
+ @StateHint CounterState state,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ state.count++;
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("check", timeCtx.time().plus(Duration.ofSeconds(5)));
+ }
+
+ public void onTimer(OnTimerContext ctx, @StateHint CounterState state) {
+ collect(Row.of("count=" + state.count));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class PassThroughTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({
+ ArgumentTrait.SET_SEMANTIC_TABLE,
+ ArgumentTrait.PASS_COLUMNS_THROUGH,
+ ArgumentTrait.REQUIRE_ON_TIME
+ })
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timer", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("fired"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class NoOnTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timer", timeCtx.time().plus(Duration.ofSeconds(10)));
+ collect(Row.of("registered"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class MultipleOnTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timer", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("fired-no-state"));
+ }
+
+ public void onTimer(OnTimerContext ctx, @StateHint String state) {
+ collect(Row.of("fired-with-state"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class CascadingTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("first", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("eval"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ String name = ctx.currentTimer();
+ collect(Row.of("fired-" + name));
+ if ("first".equals(name)) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("second", timeCtx.time().minus(Duration.ofSeconds(2)));
+ }
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class MultiTableTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row leftTable,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row rightTable) {
+ if (leftTable != null) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("check", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("left"));
+ }
+ if (rightTable != null) {
+ collect(Row.of("right"));
+ }
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("timer-fired"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class ContextClearStatePTF extends ProcessTableFunction<Row> {
+ public static class CounterState {
+ public long count = 0L;
+ }
+
+ public void eval(
+ Context ctx,
+ @StateHint CounterState state,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ String action = input.getFieldAs("action");
+ if ("increment".equals(action)) {
+ if (state == null) {
+ state = new CounterState();
+ }
+ state.count++;
+ collect(Row.of("count=" + state.count));
+ } else if ("clear-state".equals(action)) {
+ ctx.clearState("state");
+ collect(Row.of("cleared"));
+ } else if ("clear-all-state".equals(action)) {
+ ctx.clearAllState();
+ collect(Row.of("cleared-all"));
+ } else if ("clear-all".equals(action)) {
+ ctx.clearAll();
+ collect(Row.of("cleared-everything"));
+ } else if ("register-timer".equals(action)) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("t", timeCtx.time().plus(Duration.ofHours(1)));
+ collect(Row.of("timer-registered"));
+ }
+ }
+
+ public void onTimer(OnTimerContext ctx, @StateHint CounterState state) {
+ collect(Row.of("timer-fired"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class PojoTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ TimedEvent input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("check", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered-" + input.key));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("fired-" + ctx.currentTimer()));
+ }
+ }
+
+ @DataTypeHint("ROW<partitionCols STRING, timeCol INT, changelogMode STRING>")
+ public static class ContextSemanticsIntrospectionPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TableSemantics semantics = ctx.tableSemanticsFor("input");
+ collect(
+ Row.of(
+ java.util.Arrays.toString(semantics.partitionByColumns()),
+ semantics.timeColumn(),
+ ctx.getChangelogMode().toString()));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class ClearTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ String action = input.getFieldAs("action");
+ if ("register-named".equals(action)) {
+ timeCtx.registerOnTime("myTimer", timeCtx.time().plus(Duration.ofSeconds(5)));
+ } else if ("register-unnamed".equals(action)) {
+ timeCtx.registerOnTime(timeCtx.time().plus(Duration.ofSeconds(5)));
+ } else if ("clear-by-name".equals(action)) {
+ timeCtx.clearTimer("myTimer");
+ } else if ("clear-by-timestamp".equals(action)) {
+ timeCtx.clearTimer(timeCtx.time().plus(Duration.ofSeconds(4)));
+ } else if ("clear-all".equals(action)) {
+ ctx.clearAllTimers();
+ }
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(
+ Row.of(
+ "fired-"
+ + (ctx.currentTimer() != null
+ ? ctx.currentTimer()
+ : "unnamed")));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class OnTimePTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ String name = input.getFieldAs("name");
+ collect(Row.of("hello-" + name));
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Context Tests
+ // -------------------------------------------------------------------------
+
+ @Test
+ void testPerTableWatermarkTimerFiring() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(MultiTableTimerPTF.class)
+ .withTableArgument(
+ "leftTable", DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withTableArgument(
+ "rightTable",
+ DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("leftTable", "partition")
+ .withPartitionBy("rightTable", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ // Register a timer at t=5000 via the left table
+ harness.processElementForTable(
+ "leftTable", Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.clearOutput();
+
+ // Advance only left table past the timer — global watermark is still min(left, right)
+ // Since right has no watermark yet, global won't advance enough to fire
Review Comment:
This comment doesn't seem to be correct.
We are advancing both wms, but one is not advanced enough to trigger timers?
##########
flink-table/flink-table-test-utils/src/test/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarnessTest.java:
##########
Review Comment:
I don't think we need this PTF (and the corresponding test) anymore.
Everything it covers is covered many times by all the new tests.
##########
flink-table/flink-table-test-utils/src/test/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarnessTest.java:
##########
@@ -1728,4 +1741,888 @@ void testPartitionKeyValidationOnClearState() throws Exception {
harness.close();
}
+
+ // -------------------------------------------------------------------------
+ // Timer PTFs
+ // -------------------------------------------------------------------------
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class TimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ String name = input.getFieldAs("name");
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timeout-" + name, timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered-" + name));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("timer-fired-" + ctx.currentTimer()));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class UnnamedTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime(timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ String timerName = ctx.currentTimer();
+ collect(Row.of("fired-unnamed-" + (timerName == null ? "null" : timerName)));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class TimerWithStatePTF extends ProcessTableFunction<Row> {
+ public static class CounterState {
+ public long count = 0L;
+ }
+
+ public void eval(
+ Context ctx,
+ @StateHint CounterState state,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ state.count++;
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("check", timeCtx.time().plus(Duration.ofSeconds(5)));
+ }
+
+ public void onTimer(OnTimerContext ctx, @StateHint CounterState state) {
+ collect(Row.of("count=" + state.count));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class PassThroughTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({
+ ArgumentTrait.SET_SEMANTIC_TABLE,
+ ArgumentTrait.PASS_COLUMNS_THROUGH,
+ ArgumentTrait.REQUIRE_ON_TIME
+ })
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timer", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("fired"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class NoOnTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timer", timeCtx.time().plus(Duration.ofSeconds(10)));
+ collect(Row.of("registered"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class MultipleOnTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timer", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("fired-no-state"));
+ }
+
+ public void onTimer(OnTimerContext ctx, @StateHint String state) {
+ collect(Row.of("fired-with-state"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class CascadingTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("first", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("eval"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ String name = ctx.currentTimer();
+ collect(Row.of("fired-" + name));
+ if ("first".equals(name)) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("second", timeCtx.time().minus(Duration.ofSeconds(2)));
+ }
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class MultiTableTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row leftTable,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row rightTable) {
+ if (leftTable != null) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("check", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("left"));
+ }
+ if (rightTable != null) {
+ collect(Row.of("right"));
+ }
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("timer-fired"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class ContextClearStatePTF extends ProcessTableFunction<Row> {
+ public static class CounterState {
+ public long count = 0L;
+ }
+
+ public void eval(
+ Context ctx,
+ @StateHint CounterState state,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ String action = input.getFieldAs("action");
+ if ("increment".equals(action)) {
+ if (state == null) {
+ state = new CounterState();
+ }
+ state.count++;
+ collect(Row.of("count=" + state.count));
+ } else if ("clear-state".equals(action)) {
+ ctx.clearState("state");
+ collect(Row.of("cleared"));
+ } else if ("clear-all-state".equals(action)) {
+ ctx.clearAllState();
+ collect(Row.of("cleared-all"));
+ } else if ("clear-all".equals(action)) {
+ ctx.clearAll();
+ collect(Row.of("cleared-everything"));
+ } else if ("register-timer".equals(action)) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("t", timeCtx.time().plus(Duration.ofHours(1)));
+ collect(Row.of("timer-registered"));
+ }
+ }
+
+ public void onTimer(OnTimerContext ctx, @StateHint CounterState state) {
+ collect(Row.of("timer-fired"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class PojoTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ TimedEvent input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("check", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered-" + input.key));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("fired-" + ctx.currentTimer()));
+ }
+ }
+
+ @DataTypeHint("ROW<partitionCols STRING, timeCol INT, changelogMode STRING>")
+ public static class ContextSemanticsIntrospectionPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TableSemantics semantics = ctx.tableSemanticsFor("input");
+ collect(
+ Row.of(
+ java.util.Arrays.toString(semantics.partitionByColumns()),
+ semantics.timeColumn(),
+ ctx.getChangelogMode().toString()));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class ClearTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ String action = input.getFieldAs("action");
+ if ("register-named".equals(action)) {
+ timeCtx.registerOnTime("myTimer", timeCtx.time().plus(Duration.ofSeconds(5)));
+ } else if ("register-unnamed".equals(action)) {
+ timeCtx.registerOnTime(timeCtx.time().plus(Duration.ofSeconds(5)));
+ } else if ("clear-by-name".equals(action)) {
+ timeCtx.clearTimer("myTimer");
+ } else if ("clear-by-timestamp".equals(action)) {
+ timeCtx.clearTimer(timeCtx.time().plus(Duration.ofSeconds(4)));
+ } else if ("clear-all".equals(action)) {
+ ctx.clearAllTimers();
+ }
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(
+ Row.of(
+ "fired-"
+ + (ctx.currentTimer() != null
+ ? ctx.currentTimer()
+ : "unnamed")));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class OnTimePTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ String name = input.getFieldAs("name");
+ collect(Row.of("hello-" + name));
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Context Tests
+ // -------------------------------------------------------------------------
+
+ @Test
+ void testPerTableWatermarkTimerFiring() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(MultiTableTimerPTF.class)
+ .withTableArgument(
+ "leftTable", DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withTableArgument(
+ "rightTable",
+ DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("leftTable", "partition")
+ .withPartitionBy("rightTable", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ // Register a timer at t=5000 via the left table
+ harness.processElementForTable(
+ "leftTable", Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.clearOutput();
+
+ // Advance only left table past the timer — global watermark is still min(left, right)
+ // Since right has no watermark yet, global won't advance enough to fire
+ harness.setWatermarkForTable("rightTable", LocalDateTime.of(2025, 1, 1, 0, 0, 3));
Review Comment:
Do we also have tests that use the other supported WM data types?
##########
flink-table/flink-table-test-utils/src/test/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarnessTest.java:
##########
@@ -1728,4 +1741,888 @@ void testPartitionKeyValidationOnClearState() throws Exception {
harness.close();
}
+
+ // -------------------------------------------------------------------------
+ // Timer PTFs
+ // -------------------------------------------------------------------------
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class TimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ String name = input.getFieldAs("name");
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timeout-" + name, timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered-" + name));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("timer-fired-" + ctx.currentTimer()));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class UnnamedTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime(timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ String timerName = ctx.currentTimer();
+ collect(Row.of("fired-unnamed-" + (timerName == null ? "null" : timerName)));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class TimerWithStatePTF extends ProcessTableFunction<Row> {
+ public static class CounterState {
+ public long count = 0L;
+ }
+
+ public void eval(
+ Context ctx,
+ @StateHint CounterState state,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ state.count++;
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("check", timeCtx.time().plus(Duration.ofSeconds(5)));
+ }
+
+ public void onTimer(OnTimerContext ctx, @StateHint CounterState state) {
+ collect(Row.of("count=" + state.count));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class PassThroughTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({
+ ArgumentTrait.SET_SEMANTIC_TABLE,
+ ArgumentTrait.PASS_COLUMNS_THROUGH,
+ ArgumentTrait.REQUIRE_ON_TIME
+ })
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timer", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("fired"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class NoOnTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timer", timeCtx.time().plus(Duration.ofSeconds(10)));
+ collect(Row.of("registered"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class MultipleOnTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timer", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("fired-no-state"));
+ }
+
+ public void onTimer(OnTimerContext ctx, @StateHint String state) {
+ collect(Row.of("fired-with-state"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class CascadingTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("first", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("eval"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ String name = ctx.currentTimer();
+ collect(Row.of("fired-" + name));
+ if ("first".equals(name)) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("second", timeCtx.time().minus(Duration.ofSeconds(2)));
+ }
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class MultiTableTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row leftTable,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row rightTable) {
+ if (leftTable != null) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("check", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("left"));
+ }
+ if (rightTable != null) {
+ collect(Row.of("right"));
+ }
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("timer-fired"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class ContextClearStatePTF extends ProcessTableFunction<Row> {
+ public static class CounterState {
+ public long count = 0L;
+ }
+
+ public void eval(
+ Context ctx,
+ @StateHint CounterState state,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ String action = input.getFieldAs("action");
+ if ("increment".equals(action)) {
+ if (state == null) {
+ state = new CounterState();
+ }
+ state.count++;
+ collect(Row.of("count=" + state.count));
+ } else if ("clear-state".equals(action)) {
+ ctx.clearState("state");
+ collect(Row.of("cleared"));
+ } else if ("clear-all-state".equals(action)) {
+ ctx.clearAllState();
+ collect(Row.of("cleared-all"));
+ } else if ("clear-all".equals(action)) {
+ ctx.clearAll();
+ collect(Row.of("cleared-everything"));
+ } else if ("register-timer".equals(action)) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("t", timeCtx.time().plus(Duration.ofHours(1)));
+ collect(Row.of("timer-registered"));
+ }
+ }
+
+ public void onTimer(OnTimerContext ctx, @StateHint CounterState state) {
+ collect(Row.of("timer-fired"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class PojoTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ TimedEvent input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("check", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered-" + input.key));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("fired-" + ctx.currentTimer()));
+ }
+ }
+
+ @DataTypeHint("ROW<partitionCols STRING, timeCol INT, changelogMode STRING>")
+ public static class ContextSemanticsIntrospectionPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TableSemantics semantics = ctx.tableSemanticsFor("input");
+ collect(
+ Row.of(
+ java.util.Arrays.toString(semantics.partitionByColumns()),
+ semantics.timeColumn(),
+ ctx.getChangelogMode().toString()));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class ClearTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ String action = input.getFieldAs("action");
+ if ("register-named".equals(action)) {
+ timeCtx.registerOnTime("myTimer", timeCtx.time().plus(Duration.ofSeconds(5)));
+ } else if ("register-unnamed".equals(action)) {
+ timeCtx.registerOnTime(timeCtx.time().plus(Duration.ofSeconds(5)));
+ } else if ("clear-by-name".equals(action)) {
+ timeCtx.clearTimer("myTimer");
+ } else if ("clear-by-timestamp".equals(action)) {
+ timeCtx.clearTimer(timeCtx.time().plus(Duration.ofSeconds(4)));
+ } else if ("clear-all".equals(action)) {
+ ctx.clearAllTimers();
+ }
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(
+ Row.of(
+ "fired-"
+ + (ctx.currentTimer() != null
+ ? ctx.currentTimer()
+ : "unnamed")));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class OnTimePTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ String name = input.getFieldAs("name");
+ collect(Row.of("hello-" + name));
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Context Tests
+ // -------------------------------------------------------------------------
+
+ @Test
+ void testPerTableWatermarkTimerFiring() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(MultiTableTimerPTF.class)
+ .withTableArgument(
+ "leftTable", DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withTableArgument(
+ "rightTable",
+ DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("leftTable", "partition")
+ .withPartitionBy("rightTable", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ // Register a timer at t=5000 via the left table
+ harness.processElementForTable(
+ "leftTable", Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.clearOutput();
+
+ // Advance only left table past the timer — global watermark is still min(left, right)
+ // Since right has no watermark yet, global won't advance enough to fire
+ harness.setWatermarkForTable("rightTable", LocalDateTime.of(2025, 1, 1, 0, 0, 3));
+ harness.setWatermarkForTable("leftTable", LocalDateTime.of(2025, 1, 1, 0, 0, 10));
+ assertThat(harness.getFiredTimers()).isEmpty();
+
+ // Now advance right table past the timer — global watermark catches up, timer fires
+ harness.setWatermarkForTable("rightTable", LocalDateTime.of(2025, 1, 1, 0, 0, 10));
+ assertThat(harness.getFiredTimers()).hasSize(1);
+ // Both tables' partition keys are prepended (one "P1" per table)
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of(
+ "P1",
+ "P1",
+ "timer-fired",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 6)));
+ }
+ }
+
+ @Test
+ void testContextClearState() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(ContextClearStatePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of(
+ "ROW<partition STRING, action STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 2)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "count=1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)),
+ Row.of("P1", "count=2", LocalDateTime.of(2025, 1, 1, 0, 0, 2)));
+ harness.clearOutput();
+
+ harness.processElement(
+ Row.of("P1", "clear-state", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 4)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "cleared", LocalDateTime.of(2025, 1, 1, 0, 0, 3)),
+ Row.of("P1", "count=1", LocalDateTime.of(2025, 1, 1, 0, 0, 4)));
+ }
+ }
+
+ @Test
+ void testContextClearAllState() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(ContextClearStatePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of(
+ "ROW<partition STRING, action STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.clearOutput();
+
+ harness.processElement(
+ Row.of("P1", "clear-all-state", LocalDateTime.of(2025, 1, 1, 0, 0, 2)));
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "cleared-all", LocalDateTime.of(2025, 1, 1, 0, 0, 2)),
+ Row.of("P1", "count=1", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ }
+ }
+
+ @Test
+ void testContextClearAll() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(ContextClearStatePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of(
+ "ROW<partition STRING, action STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(
+ Row.of("P1", "register-timer", LocalDateTime.of(2025, 1, 1, 0, 0, 2)));
+ assertThat(harness.getPendingTimers()).hasSize(1);
+ harness.clearOutput();
+
+ harness.processElement(
+ Row.of("P1", "clear-all", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ assertThat(harness.getPendingTimers()).isEmpty();
+
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 4)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of(
+ "P1",
+ "cleared-everything",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 3)),
+ Row.of("P1", "count=1", LocalDateTime.of(2025, 1, 1, 0, 0, 4)));
+ }
+ }
+
+ @Test
+ void testContextTableSemanticsAndChangelogMode() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(ContextSemanticsIntrospectionPTF.class)
+ .withTableArgument(
+ "input", DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+
+ assertThat(harness.getOutput()).hasSize(1);
+ Row result = harness.getOutput().get(0);
+ // Field 0 is prepended partition key
+ assertThat(result.getFieldAs(0).toString()).isEqualTo("P1");
+ // partitionByColumns: [0] (index of "partition" column)
+ assertThat(result.getFieldAs(1).toString()).isEqualTo("[0]");
+ // timeColumn: 1 (index of "ts" column)
+ assertThat((int) result.getFieldAs(2)).isEqualTo(1);
+ // changelogMode: insert-only
+ assertThat(result.getFieldAs(3).toString())
+ .isEqualTo(ChangelogMode.insertOnly().toString());
+ }
+ }
+
+ @Test
+ void testPojoInputWithOnTimeColumn() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(PojoTimerPTF.class)
+ .withTableArgument("input")
+ .withPartitionBy("input", "key")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "registered-P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ assertThat(harness.getPendingTimers()).hasSize(1);
+ harness.clearOutput();
+
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 10));
+ assertThat(harness.getFiredTimers()).hasSize(1);
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "fired-check", LocalDateTime.of(2025, 1, 1, 0, 0, 6)));
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Time / On-Time Column Tests
+ // -------------------------------------------------------------------------
+
+ @Test
+ void testOnTimeColumnAppendedToEvalOutput() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(OnTimePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "hello-Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ }
+ }
+
+ @Test
+ void testOnTimeColumnWithMultipleRows() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(OnTimePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(Row.of("P1", "Bob", LocalDateTime.of(2025, 1, 1, 0, 0, 5)));
+ harness.processElement(Row.of("P2", "Carol", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "hello-Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)),
+ Row.of("P1", "hello-Bob", LocalDateTime.of(2025, 1, 1, 0, 0, 5)),
+ Row.of("P2", "hello-Carol", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ }
+ }
+
+ @Test
+ void testWatermarkAdvancesWithoutTimers() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(OnTimePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+
+ // Advancing watermark without timers registered should not produce additional output
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 10));
+ assertThat(harness.getOutput()).hasSize(1);
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Timer Tests
+ // -------------------------------------------------------------------------
+
+ @Test
+ void testNamedTimerRegistrationAndFiring() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(TimerPTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of(
+ "P1",
+ "registered-Alice",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.clearOutput();
+
+ assertThat(harness.getPendingTimers()).hasSize(1);
+ Timer timer = harness.getPendingTimers().get(0);
+ assertThat(timer.getName()).isEqualTo("timeout-Alice");
+ assertThat(timer.getTimestampAs(LocalDateTime.class))
+ .isEqualTo(LocalDateTime.of(2025, 1, 1, 0, 0, 6));
+
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 7));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of(
+ "P1",
+ "timer-fired-timeout-Alice",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 6)));
+
+ assertThat(harness.getPendingTimers()).isEmpty();
+ assertThat(harness.getFiredTimers()).hasSize(1);
+ }
+ }
+
+ @Test
+ void testUnnamedTimerFiring() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(UnnamedTimerPTF.class)
+ .withTableArgument(
+ "input", DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.clearOutput();
+
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 7));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of(
+ "P1",
+ "fired-unnamed-null",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 6)));
+ }
+ }
+
+ @Test
+ void testTimerReplacementSemantics() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(TimerPTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ // First eval: event at T+1s → timer at T+6s
+ // Second eval: event at T+6s → replaces timer to T+11s
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 6)));
+ harness.clearOutput();
+
+ assertThat(harness.getPendingTimers()).hasSize(1);
+
+ // Watermark at T+7s — timer at T+11s should NOT fire
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 7));
+ assertThat(harness.getOutput()).isEmpty();
+
+ // Watermark at T+12s — timer at T+11s should fire
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 12));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of(
+ "P1",
+ "timer-fired-timeout-Alice",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 11)));
+ }
+ }
+
+ @Test
+ void testMultipleTimersFiringOrder() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(TimerPTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", "Charlie", LocalDateTime.of(2025, 1, 1, 0, 0, 5)));
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(Row.of("P1", "Bob", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ harness.clearOutput();
+
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 11));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of(
+ "P1",
+ "timer-fired-timeout-Alice",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 6)),
+ Row.of(
+ "P1",
+ "timer-fired-timeout-Bob",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 8)),
+ Row.of(
+ "P1",
+ "timer-fired-timeout-Charlie",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 10)));
+ }
+ }
+
+ @Test
+ void testStateAccessInOnTimer() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(TimerWithStatePTF.class)
+ .withTableArgument(
+ "input", DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(Row.of("P2", LocalDateTime.of(2025, 1, 1, 0, 0, 2)));
+
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 8));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "count=3", LocalDateTime.of(2025, 1, 1, 0, 0, 6)),
+ Row.of("P2", "count=1", LocalDateTime.of(2025, 1, 1, 0, 0, 7)));
+
+ assertThat(harness.getFiredTimers(Row.of("P1"))).hasSize(1);
+ assertThat(harness.getFiredTimers(Row.of("P2"))).hasSize(1);
+ assertThat(harness.getPendingTimers(Row.of("P1"))).isEmpty();
+ assertThat(harness.getPendingTimers(Row.of("P2"))).isEmpty();
+ }
+ }
+
+ @Test
+ void testWatermarkCannotMoveBackward() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(TimerPTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 5));
+ assertThrows(
+ IllegalArgumentException.class,
+ () -> harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 2)));
+ }
+ }
+
+ @Test
+ void testPassThroughWithTimersIsRejected() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(PassThroughTimerPTF.class)
+ .withTableArgument(
+ "input", DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ assertThrows(
+ TableRuntimeException.class,
+ () ->
+ harness.processElement(
+ Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1))));
+ }
+ }
+
+ @Test
+ void testNoOnTimerMethodThrows() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(NoOnTimerPTF.class)
+ .withTableArgument(
+ "input", DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+
+ assertThrows(
Review Comment:
check if error message is meaningful for users?
##########
flink-table/flink-table-test-utils/src/test/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarnessTest.java:
##########
@@ -1728,4 +1741,888 @@ void testPartitionKeyValidationOnClearState() throws Exception {
harness.close();
}
+
+ // -------------------------------------------------------------------------
+ // Timer PTFs
+ // -------------------------------------------------------------------------
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class TimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ String name = input.getFieldAs("name");
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timeout-" + name, timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered-" + name));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("timer-fired-" + ctx.currentTimer()));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class UnnamedTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime(timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ String timerName = ctx.currentTimer();
+ collect(Row.of("fired-unnamed-" + (timerName == null ? "null" : timerName)));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class TimerWithStatePTF extends ProcessTableFunction<Row> {
+ public static class CounterState {
+ public long count = 0L;
+ }
+
+ public void eval(
+ Context ctx,
+ @StateHint CounterState state,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ state.count++;
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("check", timeCtx.time().plus(Duration.ofSeconds(5)));
+ }
+
+ public void onTimer(OnTimerContext ctx, @StateHint CounterState state) {
+ collect(Row.of("count=" + state.count));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class PassThroughTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({
+ ArgumentTrait.SET_SEMANTIC_TABLE,
+ ArgumentTrait.PASS_COLUMNS_THROUGH,
+ ArgumentTrait.REQUIRE_ON_TIME
+ })
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timer", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("fired"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class NoOnTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timer", timeCtx.time().plus(Duration.ofSeconds(10)));
+ collect(Row.of("registered"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class MultipleOnTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timer", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("fired-no-state"));
+ }
+
+ public void onTimer(OnTimerContext ctx, @StateHint String state) {
+ collect(Row.of("fired-with-state"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class CascadingTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("first", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("eval"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ String name = ctx.currentTimer();
+ collect(Row.of("fired-" + name));
+ if ("first".equals(name)) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("second", timeCtx.time().minus(Duration.ofSeconds(2)));
+ }
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class MultiTableTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row leftTable,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row rightTable) {
+ if (leftTable != null) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("check", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("left"));
+ }
+ if (rightTable != null) {
+ collect(Row.of("right"));
+ }
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("timer-fired"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class ContextClearStatePTF extends ProcessTableFunction<Row> {
+ public static class CounterState {
+ public long count = 0L;
+ }
+
+ public void eval(
+ Context ctx,
+ @StateHint CounterState state,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ String action = input.getFieldAs("action");
+ if ("increment".equals(action)) {
+ if (state == null) {
+ state = new CounterState();
+ }
+ state.count++;
+ collect(Row.of("count=" + state.count));
+ } else if ("clear-state".equals(action)) {
+ ctx.clearState("state");
+ collect(Row.of("cleared"));
+ } else if ("clear-all-state".equals(action)) {
+ ctx.clearAllState();
+ collect(Row.of("cleared-all"));
+ } else if ("clear-all".equals(action)) {
+ ctx.clearAll();
+ collect(Row.of("cleared-everything"));
+ } else if ("register-timer".equals(action)) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("t", timeCtx.time().plus(Duration.ofHours(1)));
+ collect(Row.of("timer-registered"));
+ }
+ }
+
+ public void onTimer(OnTimerContext ctx, @StateHint CounterState state) {
+ collect(Row.of("timer-fired"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class PojoTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ TimedEvent input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("check", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered-" + input.key));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("fired-" + ctx.currentTimer()));
+ }
+ }
+
+ @DataTypeHint("ROW<partitionCols STRING, timeCol INT, changelogMode STRING>")
+ public static class ContextSemanticsIntrospectionPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TableSemantics semantics = ctx.tableSemanticsFor("input");
+ collect(
+ Row.of(
+ java.util.Arrays.toString(semantics.partitionByColumns()),
+ semantics.timeColumn(),
+ ctx.getChangelogMode().toString()));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class ClearTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ String action = input.getFieldAs("action");
+ if ("register-named".equals(action)) {
+ timeCtx.registerOnTime("myTimer", timeCtx.time().plus(Duration.ofSeconds(5)));
+ } else if ("register-unnamed".equals(action)) {
+ timeCtx.registerOnTime(timeCtx.time().plus(Duration.ofSeconds(5)));
+ } else if ("clear-by-name".equals(action)) {
+ timeCtx.clearTimer("myTimer");
+ } else if ("clear-by-timestamp".equals(action)) {
+ timeCtx.clearTimer(timeCtx.time().plus(Duration.ofSeconds(4)));
+ } else if ("clear-all".equals(action)) {
+ ctx.clearAllTimers();
+ }
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(
+ Row.of(
+ "fired-"
+ + (ctx.currentTimer() != null
+ ? ctx.currentTimer()
+ : "unnamed")));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class OnTimePTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ String name = input.getFieldAs("name");
+ collect(Row.of("hello-" + name));
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Context Tests
+ // -------------------------------------------------------------------------
+
+ @Test
+ void testPerTableWatermarkTimerFiring() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(MultiTableTimerPTF.class)
+ .withTableArgument(
+ "leftTable", DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withTableArgument(
+ "rightTable",
+ DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("leftTable", "partition")
+ .withPartitionBy("rightTable", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ // Register a timer at t=5000 via the left table
+ harness.processElementForTable(
+ "leftTable", Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.clearOutput();
+
+ // Advance only left table past the timer — global watermark is still min(left, right)
+ // Since right has no watermark yet, global won't advance enough to fire
+ harness.setWatermarkForTable("rightTable", LocalDateTime.of(2025, 1, 1, 0, 0, 3));
+ harness.setWatermarkForTable("leftTable", LocalDateTime.of(2025, 1, 1, 0, 0, 10));
+ assertThat(harness.getFiredTimers()).isEmpty();
+
+ // Now advance right table past the timer — global watermark catches up, timer fires
+ harness.setWatermarkForTable("rightTable", LocalDateTime.of(2025, 1, 1, 0, 0, 10));
+ assertThat(harness.getFiredTimers()).hasSize(1);
+ // Both tables' partition keys are prepended (one "P1" per table)
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of(
+ "P1",
+ "P1",
+ "timer-fired",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 6)));
+ }
+ }
+
+ @Test
+ void testContextClearState() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(ContextClearStatePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of(
+ "ROW<partition STRING, action STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 2)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "count=1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)),
+ Row.of("P1", "count=2", LocalDateTime.of(2025, 1, 1, 0, 0, 2)));
+ harness.clearOutput();
+
+ harness.processElement(
+ Row.of("P1", "clear-state", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 4)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "cleared", LocalDateTime.of(2025, 1, 1, 0, 0, 3)),
+ Row.of("P1", "count=1", LocalDateTime.of(2025, 1, 1, 0, 0, 4)));
+ }
+ }
+
+ @Test
+ void testContextClearAllState() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(ContextClearStatePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of(
+ "ROW<partition STRING, action STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.clearOutput();
+
+ harness.processElement(
+ Row.of("P1", "clear-all-state", LocalDateTime.of(2025, 1, 1, 0, 0, 2)));
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "cleared-all", LocalDateTime.of(2025, 1, 1, 0, 0, 2)),
+ Row.of("P1", "count=1", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ }
+ }
+
+ @Test
+ void testContextClearAll() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(ContextClearStatePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of(
+ "ROW<partition STRING, action STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(
+ Row.of("P1", "register-timer", LocalDateTime.of(2025, 1, 1, 0, 0, 2)));
+ assertThat(harness.getPendingTimers()).hasSize(1);
+ harness.clearOutput();
+
+ harness.processElement(
+ Row.of("P1", "clear-all", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ assertThat(harness.getPendingTimers()).isEmpty();
+
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 4)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of(
+ "P1",
+ "cleared-everything",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 3)),
+ Row.of("P1", "count=1", LocalDateTime.of(2025, 1, 1, 0, 0, 4)));
+ }
+ }
+
+ @Test
+ void testContextTableSemanticsAndChangelogMode() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(ContextSemanticsIntrospectionPTF.class)
+ .withTableArgument(
+ "input", DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+
+ assertThat(harness.getOutput()).hasSize(1);
+ Row result = harness.getOutput().get(0);
+ // Field 0 is prepended partition key
+ assertThat(result.getFieldAs(0).toString()).isEqualTo("P1");
+ // partitionByColumns: [0] (index of "partition" column)
+ assertThat(result.getFieldAs(1).toString()).isEqualTo("[0]");
+ // timeColumn: 1 (index of "ts" column)
+ assertThat((int) result.getFieldAs(2)).isEqualTo(1);
+ // changelogMode: insert-only
+ assertThat(result.getFieldAs(3).toString())
+ .isEqualTo(ChangelogMode.insertOnly().toString());
+ }
+ }
+
+ @Test
+ void testPojoInputWithOnTimeColumn() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(PojoTimerPTF.class)
+ .withTableArgument("input")
+ .withPartitionBy("input", "key")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "registered-P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ assertThat(harness.getPendingTimers()).hasSize(1);
+ harness.clearOutput();
+
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 10));
+ assertThat(harness.getFiredTimers()).hasSize(1);
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "fired-check", LocalDateTime.of(2025, 1, 1, 0, 0, 6)));
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Time / On-Time Column Tests
+ // -------------------------------------------------------------------------
+
+ @Test
+ void testOnTimeColumnAppendedToEvalOutput() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(OnTimePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "hello-Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ }
+ }
+
+ @Test
+ void testOnTimeColumnWithMultipleRows() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(OnTimePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(Row.of("P1", "Bob", LocalDateTime.of(2025, 1, 1, 0, 0, 5)));
+ harness.processElement(Row.of("P2", "Carol", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "hello-Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)),
+ Row.of("P1", "hello-Bob", LocalDateTime.of(2025, 1, 1, 0, 0, 5)),
+ Row.of("P2", "hello-Carol", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ }
+ }
+
+ @Test
+ void testWatermarkAdvancesWithoutTimers() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(OnTimePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+
+ // Advancing watermark without timers registered should not produce additional output
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 10));
+ assertThat(harness.getOutput()).hasSize(1);
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Timer Tests
+ // -------------------------------------------------------------------------
+
+ @Test
+ void testNamedTimerRegistrationAndFiring() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(TimerPTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of(
+ "P1",
+ "registered-Alice",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.clearOutput();
+
+ assertThat(harness.getPendingTimers()).hasSize(1);
+ Timer timer = harness.getPendingTimers().get(0);
+ assertThat(timer.getName()).isEqualTo("timeout-Alice");
+ assertThat(timer.getTimestampAs(LocalDateTime.class))
+ .isEqualTo(LocalDateTime.of(2025, 1, 1, 0, 0, 6));
+
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 7));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of(
+ "P1",
+ "timer-fired-timeout-Alice",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 6)));
+
+ assertThat(harness.getPendingTimers()).isEmpty();
+ assertThat(harness.getFiredTimers()).hasSize(1);
+ }
+ }
+
+ @Test
+ void testUnnamedTimerFiring() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(UnnamedTimerPTF.class)
+ .withTableArgument(
+ "input", DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.clearOutput();
+
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 7));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of(
+ "P1",
+ "fired-unnamed-null",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 6)));
+ }
+ }
+
+ @Test
+ void testTimerReplacementSemantics() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(TimerPTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ // First eval: event at T+1s → timer at T+6s
+ // Second eval: event at T+6s → replaces timer to T+11s
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 6)));
+ harness.clearOutput();
+
+ assertThat(harness.getPendingTimers()).hasSize(1);
+
+ // Watermark at T+7s — timer at T+11s should NOT fire
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 7));
+ assertThat(harness.getOutput()).isEmpty();
+
+ // Watermark at T+12s — timer at T+11s should fire
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 12));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of(
+ "P1",
+ "timer-fired-timeout-Alice",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 11)));
+ }
+ }
+
+ @Test
+ void testMultipleTimersFiringOrder() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(TimerPTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", "Charlie", LocalDateTime.of(2025, 1, 1, 0, 0, 5)));
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(Row.of("P1", "Bob", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ harness.clearOutput();
+
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 11));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of(
+ "P1",
+ "timer-fired-timeout-Alice",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 6)),
+ Row.of(
+ "P1",
+ "timer-fired-timeout-Bob",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 8)),
+ Row.of(
+ "P1",
+ "timer-fired-timeout-Charlie",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 10)));
+ }
+ }
+
+ @Test
+ void testStateAccessInOnTimer() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(TimerWithStatePTF.class)
+ .withTableArgument(
+ "input", DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(Row.of("P2", LocalDateTime.of(2025, 1, 1, 0, 0, 2)));
+
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 8));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "count=3", LocalDateTime.of(2025, 1, 1, 0, 0, 6)),
+ Row.of("P2", "count=1", LocalDateTime.of(2025, 1, 1, 0, 0, 7)));
+
+ assertThat(harness.getFiredTimers(Row.of("P1"))).hasSize(1);
+ assertThat(harness.getFiredTimers(Row.of("P2"))).hasSize(1);
+ assertThat(harness.getPendingTimers(Row.of("P1"))).isEmpty();
+ assertThat(harness.getPendingTimers(Row.of("P2"))).isEmpty();
+ }
+ }
+
+ @Test
+ void testWatermarkCannotMoveBackward() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(TimerPTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 5));
+ assertThrows(
+ IllegalArgumentException.class,
+ () -> harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 2)));
+ }
+ }
+
+ @Test
+ void testPassThroughWithTimersIsRejected() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(PassThroughTimerPTF.class)
+ .withTableArgument(
+ "input", DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ assertThrows(
+ TableRuntimeException.class,
Review Comment:
check if error message is meaningful?
##########
flink-table/flink-table-test-utils/src/test/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarnessTest.java:
##########
@@ -1728,4 +1741,888 @@ void testPartitionKeyValidationOnClearState() throws Exception {
harness.close();
}
+
+ // -------------------------------------------------------------------------
+ // Timer PTFs
+ // -------------------------------------------------------------------------
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class TimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ String name = input.getFieldAs("name");
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timeout-" + name, timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered-" + name));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("timer-fired-" + ctx.currentTimer()));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class UnnamedTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime(timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ String timerName = ctx.currentTimer();
+ collect(Row.of("fired-unnamed-" + (timerName == null ? "null" : timerName)));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class TimerWithStatePTF extends ProcessTableFunction<Row> {
+ public static class CounterState {
+ public long count = 0L;
+ }
+
+ public void eval(
+ Context ctx,
+ @StateHint CounterState state,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ state.count++;
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("check", timeCtx.time().plus(Duration.ofSeconds(5)));
+ }
+
+ public void onTimer(OnTimerContext ctx, @StateHint CounterState state) {
+ collect(Row.of("count=" + state.count));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class PassThroughTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({
+ ArgumentTrait.SET_SEMANTIC_TABLE,
+ ArgumentTrait.PASS_COLUMNS_THROUGH,
+ ArgumentTrait.REQUIRE_ON_TIME
+ })
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timer", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("fired"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class NoOnTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timer", timeCtx.time().plus(Duration.ofSeconds(10)));
+ collect(Row.of("registered"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class MultipleOnTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timer", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("fired-no-state"));
+ }
+
+ public void onTimer(OnTimerContext ctx, @StateHint String state) {
+ collect(Row.of("fired-with-state"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class CascadingTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("first", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("eval"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ String name = ctx.currentTimer();
+ collect(Row.of("fired-" + name));
+ if ("first".equals(name)) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("second", timeCtx.time().minus(Duration.ofSeconds(2)));
+ }
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class MultiTableTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row leftTable,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row rightTable) {
+ if (leftTable != null) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("check", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("left"));
+ }
+ if (rightTable != null) {
+ collect(Row.of("right"));
+ }
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("timer-fired"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class ContextClearStatePTF extends ProcessTableFunction<Row> {
+ public static class CounterState {
+ public long count = 0L;
+ }
+
+ public void eval(
+ Context ctx,
+ @StateHint CounterState state,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ String action = input.getFieldAs("action");
+ if ("increment".equals(action)) {
+ if (state == null) {
+ state = new CounterState();
+ }
+ state.count++;
+ collect(Row.of("count=" + state.count));
+ } else if ("clear-state".equals(action)) {
+ ctx.clearState("state");
+ collect(Row.of("cleared"));
+ } else if ("clear-all-state".equals(action)) {
+ ctx.clearAllState();
+ collect(Row.of("cleared-all"));
+ } else if ("clear-all".equals(action)) {
+ ctx.clearAll();
+ collect(Row.of("cleared-everything"));
+ } else if ("register-timer".equals(action)) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("t", timeCtx.time().plus(Duration.ofHours(1)));
+ collect(Row.of("timer-registered"));
+ }
+ }
+
+ public void onTimer(OnTimerContext ctx, @StateHint CounterState state) {
+ collect(Row.of("timer-fired"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class PojoTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ TimedEvent input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("check", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered-" + input.key));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("fired-" + ctx.currentTimer()));
+ }
+ }
+
+ @DataTypeHint("ROW<partitionCols STRING, timeCol INT, changelogMode STRING>")
+ public static class ContextSemanticsIntrospectionPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TableSemantics semantics = ctx.tableSemanticsFor("input");
+ collect(
+ Row.of(
+ java.util.Arrays.toString(semantics.partitionByColumns()),
+ semantics.timeColumn(),
+ ctx.getChangelogMode().toString()));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class ClearTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ String action = input.getFieldAs("action");
+ if ("register-named".equals(action)) {
+ timeCtx.registerOnTime("myTimer", timeCtx.time().plus(Duration.ofSeconds(5)));
+ } else if ("register-unnamed".equals(action)) {
+ timeCtx.registerOnTime(timeCtx.time().plus(Duration.ofSeconds(5)));
+ } else if ("clear-by-name".equals(action)) {
+ timeCtx.clearTimer("myTimer");
+ } else if ("clear-by-timestamp".equals(action)) {
+ timeCtx.clearTimer(timeCtx.time().plus(Duration.ofSeconds(4)));
+ } else if ("clear-all".equals(action)) {
+ ctx.clearAllTimers();
+ }
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(
+ Row.of(
+ "fired-"
+ + (ctx.currentTimer() != null
+ ? ctx.currentTimer()
+ : "unnamed")));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class OnTimePTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ String name = input.getFieldAs("name");
+ collect(Row.of("hello-" + name));
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Context Tests
+ // -------------------------------------------------------------------------
+
+ @Test
+ void testPerTableWatermarkTimerFiring() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(MultiTableTimerPTF.class)
+ .withTableArgument(
+ "leftTable", DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withTableArgument(
+ "rightTable",
+ DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("leftTable", "partition")
+ .withPartitionBy("rightTable", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ // Register a timer at t=5000 via the left table
+ harness.processElementForTable(
+ "leftTable", Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.clearOutput();
+
+ // Advance only left table past the timer — global watermark is still min(left, right)
+ // Since right has no watermark yet, global won't advance enough to fire
+ harness.setWatermarkForTable("rightTable", LocalDateTime.of(2025, 1, 1, 0, 0, 3));
+ harness.setWatermarkForTable("leftTable", LocalDateTime.of(2025, 1, 1, 0, 0, 10));
+ assertThat(harness.getFiredTimers()).isEmpty();
+
+ // Now advance right table past the timer — global watermark catches up, timer fires
+ harness.setWatermarkForTable("rightTable", LocalDateTime.of(2025, 1, 1, 0, 0, 10));
+ assertThat(harness.getFiredTimers()).hasSize(1);
+ // Both tables' partition keys are prepended (one "P1" per table)
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of(
+ "P1",
+ "P1",
+ "timer-fired",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 6)));
+ }
+ }
+
+ @Test
+ void testContextClearState() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(ContextClearStatePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of(
+ "ROW<partition STRING, action STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 2)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "count=1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)),
+ Row.of("P1", "count=2", LocalDateTime.of(2025, 1, 1, 0, 0, 2)));
+ harness.clearOutput();
+
+ harness.processElement(
+ Row.of("P1", "clear-state", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 4)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "cleared", LocalDateTime.of(2025, 1, 1, 0, 0, 3)),
+ Row.of("P1", "count=1", LocalDateTime.of(2025, 1, 1, 0, 0, 4)));
+ }
+ }
+
+ @Test
+ void testContextClearAllState() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(ContextClearStatePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of(
+ "ROW<partition STRING, action STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.clearOutput();
+
+ harness.processElement(
+ Row.of("P1", "clear-all-state", LocalDateTime.of(2025, 1, 1, 0, 0, 2)));
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "cleared-all", LocalDateTime.of(2025, 1, 1, 0, 0, 2)),
+ Row.of("P1", "count=1", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ }
+ }
+
+ @Test
+ void testContextClearAll() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(ContextClearStatePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of(
+ "ROW<partition STRING, action STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(
+ Row.of("P1", "register-timer", LocalDateTime.of(2025, 1, 1, 0, 0, 2)));
+ assertThat(harness.getPendingTimers()).hasSize(1);
+ harness.clearOutput();
+
+ harness.processElement(
+ Row.of("P1", "clear-all", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ assertThat(harness.getPendingTimers()).isEmpty();
+
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 4)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of(
+ "P1",
+ "cleared-everything",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 3)),
+ Row.of("P1", "count=1", LocalDateTime.of(2025, 1, 1, 0, 0, 4)));
+ }
+ }
+
+ @Test
+ void testContextTableSemanticsAndChangelogMode() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(ContextSemanticsIntrospectionPTF.class)
+ .withTableArgument(
+ "input", DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+
+ assertThat(harness.getOutput()).hasSize(1);
+ Row result = harness.getOutput().get(0);
+ // Field 0 is prepended partition key
+ assertThat(result.getFieldAs(0).toString()).isEqualTo("P1");
+ // partitionByColumns: [0] (index of "partition" column)
+ assertThat(result.getFieldAs(1).toString()).isEqualTo("[0]");
+ // timeColumn: 1 (index of "ts" column)
+ assertThat((int) result.getFieldAs(2)).isEqualTo(1);
+ // changelogMode: insert-only
+ assertThat(result.getFieldAs(3).toString())
+ .isEqualTo(ChangelogMode.insertOnly().toString());
+ }
+ }
+
+ @Test
+ void testPojoInputWithOnTimeColumn() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(PojoTimerPTF.class)
+ .withTableArgument("input")
+ .withPartitionBy("input", "key")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "registered-P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ assertThat(harness.getPendingTimers()).hasSize(1);
+ harness.clearOutput();
+
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 10));
+ assertThat(harness.getFiredTimers()).hasSize(1);
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "fired-check", LocalDateTime.of(2025, 1, 1, 0, 0, 6)));
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Time / On-Time Column Tests
+ // -------------------------------------------------------------------------
+
+ @Test
+ void testOnTimeColumnAppendedToEvalOutput() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(OnTimePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "hello-Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ }
+ }
+
+ @Test
+ void testOnTimeColumnWithMultipleRows() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(OnTimePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(Row.of("P1", "Bob", LocalDateTime.of(2025, 1, 1, 0, 0, 5)));
+ harness.processElement(Row.of("P2", "Carol", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "hello-Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)),
+ Row.of("P1", "hello-Bob", LocalDateTime.of(2025, 1, 1, 0, 0, 5)),
+ Row.of("P2", "hello-Carol", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ }
+ }
+
+ @Test
+ void testWatermarkAdvancesWithoutTimers() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(OnTimePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+
+ // Advancing watermark without timers registered should not produce additional output
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 10));
+ assertThat(harness.getOutput()).hasSize(1);
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Timer Tests
+ // -------------------------------------------------------------------------
+
+ @Test
+ void testNamedTimerRegistrationAndFiring() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(TimerPTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of(
+ "P1",
+ "registered-Alice",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.clearOutput();
+
+ assertThat(harness.getPendingTimers()).hasSize(1);
+ Timer timer = harness.getPendingTimers().get(0);
+ assertThat(timer.getName()).isEqualTo("timeout-Alice");
+ assertThat(timer.getTimestampAs(LocalDateTime.class))
+ .isEqualTo(LocalDateTime.of(2025, 1, 1, 0, 0, 6));
+
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 7));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of(
+ "P1",
+ "timer-fired-timeout-Alice",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 6)));
+
+ assertThat(harness.getPendingTimers()).isEmpty();
+ assertThat(harness.getFiredTimers()).hasSize(1);
+ }
+ }
+
+ @Test
+ void testUnnamedTimerFiring() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(UnnamedTimerPTF.class)
+ .withTableArgument(
+ "input", DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.clearOutput();
+
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 7));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of(
+ "P1",
+ "fired-unnamed-null",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 6)));
+ }
+ }
+
+ @Test
+ void testTimerReplacementSemantics() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(TimerPTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ // First eval: event at T+1s → timer at T+6s
+ // Second eval: event at T+6s → replaces timer to T+11s
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 6)));
+ harness.clearOutput();
+
+ assertThat(harness.getPendingTimers()).hasSize(1);
+
+ // Watermark at T+7s — timer at T+11s should NOT fire
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 7));
+ assertThat(harness.getOutput()).isEmpty();
+
+ // Watermark at T+12s — timer at T+11s should fire
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 12));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of(
+ "P1",
+ "timer-fired-timeout-Alice",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 11)));
+ }
+ }
+
+ @Test
+ void testMultipleTimersFiringOrder() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(TimerPTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", "Charlie", LocalDateTime.of(2025, 1, 1, 0, 0, 5)));
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(Row.of("P1", "Bob", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ harness.clearOutput();
+
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 11));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of(
+ "P1",
+ "timer-fired-timeout-Alice",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 6)),
+ Row.of(
+ "P1",
+ "timer-fired-timeout-Bob",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 8)),
+ Row.of(
+ "P1",
+ "timer-fired-timeout-Charlie",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 10)));
+ }
+ }
+
+ @Test
+ void testStateAccessInOnTimer() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(TimerWithStatePTF.class)
+ .withTableArgument(
+ "input", DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(Row.of("P2", LocalDateTime.of(2025, 1, 1, 0, 0, 2)));
+
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 8));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "count=3", LocalDateTime.of(2025, 1, 1, 0, 0, 6)),
+ Row.of("P2", "count=1", LocalDateTime.of(2025, 1, 1, 0, 0, 7)));
+
+ assertThat(harness.getFiredTimers(Row.of("P1"))).hasSize(1);
+ assertThat(harness.getFiredTimers(Row.of("P2"))).hasSize(1);
+ assertThat(harness.getPendingTimers(Row.of("P1"))).isEmpty();
+ assertThat(harness.getPendingTimers(Row.of("P2"))).isEmpty();
+ }
+ }
+
+ @Test
+ void testWatermarkCannotMoveBackward() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(TimerPTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 5));
+ assertThrows(
+ IllegalArgumentException.class,
+ () -> harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 2)));
+ }
+ }
+
+ @Test
+ void testPassThroughWithTimersIsRejected() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(PassThroughTimerPTF.class)
+ .withTableArgument(
+ "input", DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ assertThrows(
+ TableRuntimeException.class,
+ () ->
+ harness.processElement(
+ Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1))));
+ }
+ }
+
+ @Test
+ void testNoOnTimerMethodThrows() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(NoOnTimerPTF.class)
+ .withTableArgument(
+ "input", DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+
+ assertThrows(
+ IllegalStateException.class,
+ () -> harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 1)));
+ }
+ }
+
+ @Test
+ void testMultipleOnTimerMethods() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(MultipleOnTimerPTF.class)
+ .withTableArgument(
+ "input", DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.clearOutput();
+
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 7));
+ assertThat(harness.getOutput()).hasSize(1);
Review Comment:
check that the right `onTimer()` method was called by checking the output row?
##########
flink-table/flink-table-test-utils/src/test/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarnessTest.java:
##########
@@ -1728,4 +1741,888 @@ void testPartitionKeyValidationOnClearState() throws Exception {
harness.close();
}
+
+ // -------------------------------------------------------------------------
+ // Timer PTFs
+ // -------------------------------------------------------------------------
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class TimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ String name = input.getFieldAs("name");
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timeout-" + name, timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered-" + name));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("timer-fired-" + ctx.currentTimer()));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class UnnamedTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime(timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ String timerName = ctx.currentTimer();
+ collect(Row.of("fired-unnamed-" + (timerName == null ? "null" : timerName)));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class TimerWithStatePTF extends ProcessTableFunction<Row> {
+ public static class CounterState {
+ public long count = 0L;
+ }
+
+ public void eval(
+ Context ctx,
+ @StateHint CounterState state,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ state.count++;
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("check", timeCtx.time().plus(Duration.ofSeconds(5)));
+ }
+
+ public void onTimer(OnTimerContext ctx, @StateHint CounterState state) {
+ collect(Row.of("count=" + state.count));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class PassThroughTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({
+ ArgumentTrait.SET_SEMANTIC_TABLE,
+ ArgumentTrait.PASS_COLUMNS_THROUGH,
+ ArgumentTrait.REQUIRE_ON_TIME
+ })
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timer", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("fired"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class NoOnTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timer", timeCtx.time().plus(Duration.ofSeconds(10)));
+ collect(Row.of("registered"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class MultipleOnTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timer", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("fired-no-state"));
+ }
+
+ public void onTimer(OnTimerContext ctx, @StateHint String state) {
+ collect(Row.of("fired-with-state"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class CascadingTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("first", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("eval"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ String name = ctx.currentTimer();
+ collect(Row.of("fired-" + name));
+ if ("first".equals(name)) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("second", timeCtx.time().minus(Duration.ofSeconds(2)));
+ }
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class MultiTableTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row leftTable,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row rightTable) {
+ if (leftTable != null) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("check", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("left"));
+ }
+ if (rightTable != null) {
+ collect(Row.of("right"));
+ }
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("timer-fired"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class ContextClearStatePTF extends ProcessTableFunction<Row> {
+ public static class CounterState {
+ public long count = 0L;
+ }
+
+ public void eval(
+ Context ctx,
+ @StateHint CounterState state,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ String action = input.getFieldAs("action");
+ if ("increment".equals(action)) {
+ if (state == null) {
+ state = new CounterState();
+ }
+ state.count++;
+ collect(Row.of("count=" + state.count));
+ } else if ("clear-state".equals(action)) {
+ ctx.clearState("state");
+ collect(Row.of("cleared"));
+ } else if ("clear-all-state".equals(action)) {
+ ctx.clearAllState();
+ collect(Row.of("cleared-all"));
+ } else if ("clear-all".equals(action)) {
+ ctx.clearAll();
+ collect(Row.of("cleared-everything"));
+ } else if ("register-timer".equals(action)) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("t", timeCtx.time().plus(Duration.ofHours(1)));
+ collect(Row.of("timer-registered"));
+ }
+ }
+
+ public void onTimer(OnTimerContext ctx, @StateHint CounterState state) {
+ collect(Row.of("timer-fired"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class PojoTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ TimedEvent input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("check", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered-" + input.key));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("fired-" + ctx.currentTimer()));
+ }
+ }
+
+ @DataTypeHint("ROW<partitionCols STRING, timeCol INT, changelogMode STRING>")
+ public static class ContextSemanticsIntrospectionPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TableSemantics semantics = ctx.tableSemanticsFor("input");
+ collect(
+ Row.of(
+ java.util.Arrays.toString(semantics.partitionByColumns()),
+ semantics.timeColumn(),
+ ctx.getChangelogMode().toString()));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class ClearTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ String action = input.getFieldAs("action");
+ if ("register-named".equals(action)) {
+ timeCtx.registerOnTime("myTimer", timeCtx.time().plus(Duration.ofSeconds(5)));
+ } else if ("register-unnamed".equals(action)) {
+ timeCtx.registerOnTime(timeCtx.time().plus(Duration.ofSeconds(5)));
+ } else if ("clear-by-name".equals(action)) {
+ timeCtx.clearTimer("myTimer");
+ } else if ("clear-by-timestamp".equals(action)) {
+ timeCtx.clearTimer(timeCtx.time().plus(Duration.ofSeconds(4)));
+ } else if ("clear-all".equals(action)) {
+ ctx.clearAllTimers();
+ }
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(
+ Row.of(
+ "fired-"
+ + (ctx.currentTimer() != null
+ ? ctx.currentTimer()
+ : "unnamed")));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class OnTimePTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ String name = input.getFieldAs("name");
+ collect(Row.of("hello-" + name));
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Context Tests
+ // -------------------------------------------------------------------------
+
+ @Test
+ void testPerTableWatermarkTimerFiring() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(MultiTableTimerPTF.class)
+ .withTableArgument(
+ "leftTable", DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withTableArgument(
+ "rightTable",
+ DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("leftTable", "partition")
+ .withPartitionBy("rightTable", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ // Register a timer at t=5000 via the left table
+ harness.processElementForTable(
+ "leftTable", Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.clearOutput();
+
+ // Advance only left table past the timer — global watermark is still min(left, right)
+ // Since right has no watermark yet, global won't advance enough to fire
+ harness.setWatermarkForTable("rightTable", LocalDateTime.of(2025, 1, 1, 0, 0, 3));
+ harness.setWatermarkForTable("leftTable", LocalDateTime.of(2025, 1, 1, 0, 0, 10));
+ assertThat(harness.getFiredTimers()).isEmpty();
+
+ // Now advance right table past the timer — global watermark catches up, timer fires
+ harness.setWatermarkForTable("rightTable", LocalDateTime.of(2025, 1, 1, 0, 0, 10));
+ assertThat(harness.getFiredTimers()).hasSize(1);
+ // Both tables' partition keys are prepended (one "P1" per table)
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of(
+ "P1",
+ "P1",
+ "timer-fired",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 6)));
+ }
+ }
+
+ @Test
+ void testContextClearState() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(ContextClearStatePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of(
+ "ROW<partition STRING, action STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 2)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "count=1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)),
+ Row.of("P1", "count=2", LocalDateTime.of(2025, 1, 1, 0, 0, 2)));
+ harness.clearOutput();
+
+ harness.processElement(
+ Row.of("P1", "clear-state", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 4)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "cleared", LocalDateTime.of(2025, 1, 1, 0, 0, 3)),
+ Row.of("P1", "count=1", LocalDateTime.of(2025, 1, 1, 0, 0, 4)));
+ }
+ }
+
+ @Test
+ void testContextClearAllState() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(ContextClearStatePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of(
+ "ROW<partition STRING, action STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.clearOutput();
+
+ harness.processElement(
+ Row.of("P1", "clear-all-state", LocalDateTime.of(2025, 1, 1, 0, 0, 2)));
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "cleared-all", LocalDateTime.of(2025, 1, 1, 0, 0, 2)),
+ Row.of("P1", "count=1", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ }
+ }
+
+ @Test
+ void testContextClearAll() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(ContextClearStatePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of(
+ "ROW<partition STRING, action STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(
+ Row.of("P1", "register-timer", LocalDateTime.of(2025, 1, 1, 0, 0, 2)));
+ assertThat(harness.getPendingTimers()).hasSize(1);
+ harness.clearOutput();
+
+ harness.processElement(
+ Row.of("P1", "clear-all", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ assertThat(harness.getPendingTimers()).isEmpty();
+
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 4)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of(
+ "P1",
+ "cleared-everything",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 3)),
+ Row.of("P1", "count=1", LocalDateTime.of(2025, 1, 1, 0, 0, 4)));
+ }
+ }
+
+ @Test
+ void testContextTableSemanticsAndChangelogMode() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(ContextSemanticsIntrospectionPTF.class)
+ .withTableArgument(
+ "input", DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+
+ assertThat(harness.getOutput()).hasSize(1);
+ Row result = harness.getOutput().get(0);
+ // Field 0 is prepended partition key
+ assertThat(result.getFieldAs(0).toString()).isEqualTo("P1");
+ // partitionByColumns: [0] (index of "partition" column)
+ assertThat(result.getFieldAs(1).toString()).isEqualTo("[0]");
+ // timeColumn: 1 (index of "ts" column)
+ assertThat((int) result.getFieldAs(2)).isEqualTo(1);
+ // changelogMode: insert-only
+ assertThat(result.getFieldAs(3).toString())
+ .isEqualTo(ChangelogMode.insertOnly().toString());
+ }
+ }
+
+ @Test
+ void testPojoInputWithOnTimeColumn() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(PojoTimerPTF.class)
+ .withTableArgument("input")
+ .withPartitionBy("input", "key")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "registered-P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ assertThat(harness.getPendingTimers()).hasSize(1);
+ harness.clearOutput();
+
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 10));
+ assertThat(harness.getFiredTimers()).hasSize(1);
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "fired-check", LocalDateTime.of(2025, 1, 1, 0, 0, 6)));
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Time / On-Time Column Tests
+ // -------------------------------------------------------------------------
+
+ @Test
+ void testOnTimeColumnAppendedToEvalOutput() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(OnTimePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "hello-Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ }
+ }
+
+ @Test
+ void testOnTimeColumnWithMultipleRows() throws Exception {
Review Comment:
Does this test subsume `testOnTimeColumnAppendedToEvalOutput()`?
##########
flink-table/flink-table-test-utils/src/test/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarnessTest.java:
##########
@@ -1728,4 +1741,888 @@ void testPartitionKeyValidationOnClearState() throws Exception {
harness.close();
}
+
+ // -------------------------------------------------------------------------
+ // Timer PTFs
+ // -------------------------------------------------------------------------
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class TimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ String name = input.getFieldAs("name");
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timeout-" + name, timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered-" + name));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("timer-fired-" + ctx.currentTimer()));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class UnnamedTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime(timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ String timerName = ctx.currentTimer();
+ collect(Row.of("fired-unnamed-" + (timerName == null ? "null" : timerName)));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class TimerWithStatePTF extends ProcessTableFunction<Row> {
+ public static class CounterState {
+ public long count = 0L;
+ }
+
+ public void eval(
+ Context ctx,
+ @StateHint CounterState state,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ state.count++;
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("check", timeCtx.time().plus(Duration.ofSeconds(5)));
+ }
+
+ public void onTimer(OnTimerContext ctx, @StateHint CounterState state) {
+ collect(Row.of("count=" + state.count));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class PassThroughTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({
+ ArgumentTrait.SET_SEMANTIC_TABLE,
+ ArgumentTrait.PASS_COLUMNS_THROUGH,
+ ArgumentTrait.REQUIRE_ON_TIME
+ })
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timer", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("fired"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class NoOnTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timer", timeCtx.time().plus(Duration.ofSeconds(10)));
+ collect(Row.of("registered"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class MultipleOnTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timer", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("fired-no-state"));
+ }
+
+ public void onTimer(OnTimerContext ctx, @StateHint String state) {
+ collect(Row.of("fired-with-state"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class CascadingTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("first", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("eval"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ String name = ctx.currentTimer();
+ collect(Row.of("fired-" + name));
+ if ("first".equals(name)) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("second", timeCtx.time().minus(Duration.ofSeconds(2)));
+ }
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class MultiTableTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row leftTable,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row rightTable) {
+ if (leftTable != null) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("check", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("left"));
+ }
+ if (rightTable != null) {
+ collect(Row.of("right"));
+ }
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("timer-fired"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class ContextClearStatePTF extends ProcessTableFunction<Row> {
+ public static class CounterState {
+ public long count = 0L;
+ }
+
+ public void eval(
+ Context ctx,
+ @StateHint CounterState state,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ String action = input.getFieldAs("action");
+ if ("increment".equals(action)) {
+ if (state == null) {
+ state = new CounterState();
+ }
+ state.count++;
+ collect(Row.of("count=" + state.count));
+ } else if ("clear-state".equals(action)) {
+ ctx.clearState("state");
+ collect(Row.of("cleared"));
+ } else if ("clear-all-state".equals(action)) {
+ ctx.clearAllState();
+ collect(Row.of("cleared-all"));
+ } else if ("clear-all".equals(action)) {
+ ctx.clearAll();
+ collect(Row.of("cleared-everything"));
+ } else if ("register-timer".equals(action)) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("t", timeCtx.time().plus(Duration.ofHours(1)));
+ collect(Row.of("timer-registered"));
+ }
+ }
+
+ public void onTimer(OnTimerContext ctx, @StateHint CounterState state) {
+ collect(Row.of("timer-fired"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class PojoTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ TimedEvent input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("check", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered-" + input.key));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("fired-" + ctx.currentTimer()));
+ }
+ }
+
+ @DataTypeHint("ROW<partitionCols STRING, timeCol INT, changelogMode STRING>")
+ public static class ContextSemanticsIntrospectionPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TableSemantics semantics = ctx.tableSemanticsFor("input");
+ collect(
+ Row.of(
+ java.util.Arrays.toString(semantics.partitionByColumns()),
+ semantics.timeColumn(),
+ ctx.getChangelogMode().toString()));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class ClearTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ String action = input.getFieldAs("action");
+ if ("register-named".equals(action)) {
+ timeCtx.registerOnTime("myTimer", timeCtx.time().plus(Duration.ofSeconds(5)));
+ } else if ("register-unnamed".equals(action)) {
+ timeCtx.registerOnTime(timeCtx.time().plus(Duration.ofSeconds(5)));
+ } else if ("clear-by-name".equals(action)) {
+ timeCtx.clearTimer("myTimer");
+ } else if ("clear-by-timestamp".equals(action)) {
+ timeCtx.clearTimer(timeCtx.time().plus(Duration.ofSeconds(4)));
+ } else if ("clear-all".equals(action)) {
+ ctx.clearAllTimers();
+ }
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(
+ Row.of(
+ "fired-"
+ + (ctx.currentTimer() != null
+ ? ctx.currentTimer()
+ : "unnamed")));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class OnTimePTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ String name = input.getFieldAs("name");
+ collect(Row.of("hello-" + name));
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Context Tests
+ // -------------------------------------------------------------------------
+
+ @Test
+ void testPerTableWatermarkTimerFiring() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(MultiTableTimerPTF.class)
+ .withTableArgument(
+ "leftTable", DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withTableArgument(
+ "rightTable",
+ DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("leftTable", "partition")
+ .withPartitionBy("rightTable", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ // Register a timer at t=5000 via the left table
+ harness.processElementForTable(
+ "leftTable", Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.clearOutput();
+
+ // Advance only left table past the timer — global watermark is still min(left, right)
+ // Since right has no watermark yet, global won't advance enough to fire
+ harness.setWatermarkForTable("rightTable", LocalDateTime.of(2025, 1, 1, 0, 0, 3));
+ harness.setWatermarkForTable("leftTable", LocalDateTime.of(2025, 1, 1, 0, 0, 10));
+ assertThat(harness.getFiredTimers()).isEmpty();
+
+ // Now advance right table past the timer — global watermark catches up, timer fires
+ harness.setWatermarkForTable("rightTable", LocalDateTime.of(2025, 1, 1, 0, 0, 10));
+ assertThat(harness.getFiredTimers()).hasSize(1);
+ // Both tables' partition keys are prepended (one "P1" per table)
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of(
+ "P1",
+ "P1",
+ "timer-fired",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 6)));
+ }
+ }
+
+ @Test
+ void testContextClearState() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(ContextClearStatePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of(
+ "ROW<partition STRING, action STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 2)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "count=1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)),
+ Row.of("P1", "count=2", LocalDateTime.of(2025, 1, 1, 0, 0, 2)));
+ harness.clearOutput();
+
+ harness.processElement(
+ Row.of("P1", "clear-state", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 4)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "cleared", LocalDateTime.of(2025, 1, 1, 0, 0, 3)),
+ Row.of("P1", "count=1", LocalDateTime.of(2025, 1, 1, 0, 0, 4)));
+ }
+ }
+
+ @Test
+ void testContextClearAllState() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(ContextClearStatePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of(
+ "ROW<partition STRING, action STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.clearOutput();
+
+ harness.processElement(
+ Row.of("P1", "clear-all-state", LocalDateTime.of(2025, 1, 1, 0, 0, 2)));
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "cleared-all", LocalDateTime.of(2025, 1, 1, 0, 0, 2)),
+ Row.of("P1", "count=1", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ }
+ }
+
+ @Test
+ void testContextClearAll() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(ContextClearStatePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of(
+ "ROW<partition STRING, action STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(
+ Row.of("P1", "register-timer", LocalDateTime.of(2025, 1, 1, 0, 0, 2)));
+ assertThat(harness.getPendingTimers()).hasSize(1);
+ harness.clearOutput();
+
+ harness.processElement(
+ Row.of("P1", "clear-all", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ assertThat(harness.getPendingTimers()).isEmpty();
+
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 4)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of(
+ "P1",
+ "cleared-everything",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 3)),
+ Row.of("P1", "count=1", LocalDateTime.of(2025, 1, 1, 0, 0, 4)));
+ }
+ }
+
+ @Test
+ void testContextTableSemanticsAndChangelogMode() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(ContextSemanticsIntrospectionPTF.class)
+ .withTableArgument(
+ "input", DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+
+ assertThat(harness.getOutput()).hasSize(1);
+ Row result = harness.getOutput().get(0);
+ // Field 0 is prepended partition key
+ assertThat(result.getFieldAs(0).toString()).isEqualTo("P1");
+ // partitionByColumns: [0] (index of "partition" column)
+ assertThat(result.getFieldAs(1).toString()).isEqualTo("[0]");
+ // timeColumn: 1 (index of "ts" column)
+ assertThat((int) result.getFieldAs(2)).isEqualTo(1);
+ // changelogMode: insert-only
+ assertThat(result.getFieldAs(3).toString())
+ .isEqualTo(ChangelogMode.insertOnly().toString());
+ }
+ }
+
+ @Test
+ void testPojoInputWithOnTimeColumn() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(PojoTimerPTF.class)
+ .withTableArgument("input")
+ .withPartitionBy("input", "key")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "registered-P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ assertThat(harness.getPendingTimers()).hasSize(1);
+ harness.clearOutput();
+
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 10));
+ assertThat(harness.getFiredTimers()).hasSize(1);
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "fired-check", LocalDateTime.of(2025, 1, 1, 0, 0, 6)));
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Time / On-Time Column Tests
+ // -------------------------------------------------------------------------
+
+ @Test
+ void testOnTimeColumnAppendedToEvalOutput() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(OnTimePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "hello-Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ }
+ }
+
+ @Test
+ void testOnTimeColumnWithMultipleRows() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(OnTimePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(Row.of("P1", "Bob", LocalDateTime.of(2025, 1, 1, 0, 0, 5)));
+ harness.processElement(Row.of("P2", "Carol", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "hello-Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)),
+ Row.of("P1", "hello-Bob", LocalDateTime.of(2025, 1, 1, 0, 0, 5)),
+ Row.of("P2", "hello-Carol", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ }
+ }
+
+ @Test
+ void testWatermarkAdvancesWithoutTimers() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(OnTimePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+
+ // Advancing watermark without timers registered should not produce additional output
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 10));
+ assertThat(harness.getOutput()).hasSize(1);
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Timer Tests
+ // -------------------------------------------------------------------------
+
+ @Test
+ void testNamedTimerRegistrationAndFiring() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(TimerPTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of(
+ "P1",
+ "registered-Alice",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.clearOutput();
+
+ assertThat(harness.getPendingTimers()).hasSize(1);
+ Timer timer = harness.getPendingTimers().get(0);
+ assertThat(timer.getName()).isEqualTo("timeout-Alice");
+ assertThat(timer.getTimestampAs(LocalDateTime.class))
+ .isEqualTo(LocalDateTime.of(2025, 1, 1, 0, 0, 6));
+
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 7));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of(
+ "P1",
+ "timer-fired-timeout-Alice",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 6)));
+
+ assertThat(harness.getPendingTimers()).isEmpty();
+ assertThat(harness.getFiredTimers()).hasSize(1);
+ }
+ }
+
+ @Test
+ void testUnnamedTimerFiring() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(UnnamedTimerPTF.class)
+ .withTableArgument(
+ "input", DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.clearOutput();
+
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 7));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of(
+ "P1",
+ "fired-unnamed-null",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 6)));
+ }
+ }
+
+ @Test
+ void testTimerReplacementSemantics() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(TimerPTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ // First eval: event at T+1s → timer at T+6s
+ // Second eval: event at T+6s → replaces timer to T+11s
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 6)));
+ harness.clearOutput();
+
+ assertThat(harness.getPendingTimers()).hasSize(1);
+
+ // Watermark at T+7s — timer at T+11s should NOT fire
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 7));
+ assertThat(harness.getOutput()).isEmpty();
+
+ // Watermark at T+12s — timer at T+11s should fire
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 12));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of(
+ "P1",
+ "timer-fired-timeout-Alice",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 11)));
+ }
+ }
+
+ @Test
+ void testMultipleTimersFiringOrder() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(TimerPTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", "Charlie", LocalDateTime.of(2025, 1, 1, 0, 0, 5)));
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(Row.of("P1", "Bob", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ harness.clearOutput();
+
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 11));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of(
+ "P1",
+ "timer-fired-timeout-Alice",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 6)),
+ Row.of(
+ "P1",
+ "timer-fired-timeout-Bob",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 8)),
+ Row.of(
+ "P1",
+ "timer-fired-timeout-Charlie",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 10)));
+ }
+ }
+
+ @Test
+ void testStateAccessInOnTimer() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(TimerWithStatePTF.class)
+ .withTableArgument(
+ "input", DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(Row.of("P2", LocalDateTime.of(2025, 1, 1, 0, 0, 2)));
+
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 8));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "count=3", LocalDateTime.of(2025, 1, 1, 0, 0, 6)),
+ Row.of("P2", "count=1", LocalDateTime.of(2025, 1, 1, 0, 0, 7)));
+
+ assertThat(harness.getFiredTimers(Row.of("P1"))).hasSize(1);
+ assertThat(harness.getFiredTimers(Row.of("P2"))).hasSize(1);
+ assertThat(harness.getPendingTimers(Row.of("P1"))).isEmpty();
+ assertThat(harness.getPendingTimers(Row.of("P2"))).isEmpty();
+ }
+ }
+
+ @Test
+ void testWatermarkCannotMoveBackward() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(TimerPTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 5));
Review Comment:
also check for `setWatermarkForTable()`?
##########
flink-table/flink-table-test-utils/src/test/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarnessTest.java:
##########
@@ -1728,4 +1741,888 @@ void testPartitionKeyValidationOnClearState() throws Exception {
harness.close();
}
+
+ // -------------------------------------------------------------------------
+ // Timer PTFs
+ // -------------------------------------------------------------------------
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class TimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ String name = input.getFieldAs("name");
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timeout-" + name, timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered-" + name));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("timer-fired-" + ctx.currentTimer()));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class UnnamedTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime(timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ String timerName = ctx.currentTimer();
+ collect(Row.of("fired-unnamed-" + (timerName == null ? "null" : timerName)));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class TimerWithStatePTF extends ProcessTableFunction<Row> {
+ public static class CounterState {
+ public long count = 0L;
+ }
+
+ public void eval(
+ Context ctx,
+ @StateHint CounterState state,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ state.count++;
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("check", timeCtx.time().plus(Duration.ofSeconds(5)));
+ }
+
+ public void onTimer(OnTimerContext ctx, @StateHint CounterState state) {
+ collect(Row.of("count=" + state.count));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class PassThroughTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({
+ ArgumentTrait.SET_SEMANTIC_TABLE,
+ ArgumentTrait.PASS_COLUMNS_THROUGH,
+ ArgumentTrait.REQUIRE_ON_TIME
+ })
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timer", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("fired"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class NoOnTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timer", timeCtx.time().plus(Duration.ofSeconds(10)));
+ collect(Row.of("registered"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class MultipleOnTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timer", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("fired-no-state"));
+ }
+
+ public void onTimer(OnTimerContext ctx, @StateHint String state) {
+ collect(Row.of("fired-with-state"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class CascadingTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("first", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("eval"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ String name = ctx.currentTimer();
+ collect(Row.of("fired-" + name));
+ if ("first".equals(name)) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("second", timeCtx.time().minus(Duration.ofSeconds(2)));
+ }
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class MultiTableTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row leftTable,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row rightTable) {
+ if (leftTable != null) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("check", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("left"));
+ }
+ if (rightTable != null) {
+ collect(Row.of("right"));
+ }
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("timer-fired"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class ContextClearStatePTF extends ProcessTableFunction<Row> {
+ public static class CounterState {
+ public long count = 0L;
+ }
+
+ public void eval(
+ Context ctx,
+ @StateHint CounterState state,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ String action = input.getFieldAs("action");
+ if ("increment".equals(action)) {
+ if (state == null) {
+ state = new CounterState();
+ }
+ state.count++;
+ collect(Row.of("count=" + state.count));
+ } else if ("clear-state".equals(action)) {
+ ctx.clearState("state");
+ collect(Row.of("cleared"));
+ } else if ("clear-all-state".equals(action)) {
+ ctx.clearAllState();
+ collect(Row.of("cleared-all"));
+ } else if ("clear-all".equals(action)) {
+ ctx.clearAll();
+ collect(Row.of("cleared-everything"));
+ } else if ("register-timer".equals(action)) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("t", timeCtx.time().plus(Duration.ofHours(1)));
+ collect(Row.of("timer-registered"));
+ }
+ }
+
+ public void onTimer(OnTimerContext ctx, @StateHint CounterState state) {
+ collect(Row.of("timer-fired"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class PojoTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ TimedEvent input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("check", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered-" + input.key));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("fired-" + ctx.currentTimer()));
+ }
+ }
+
+ @DataTypeHint("ROW<partitionCols STRING, timeCol INT, changelogMode STRING>")
+ public static class ContextSemanticsIntrospectionPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TableSemantics semantics = ctx.tableSemanticsFor("input");
+ collect(
+ Row.of(
+ java.util.Arrays.toString(semantics.partitionByColumns()),
+ semantics.timeColumn(),
+ ctx.getChangelogMode().toString()));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class ClearTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ String action = input.getFieldAs("action");
+ if ("register-named".equals(action)) {
+ timeCtx.registerOnTime("myTimer", timeCtx.time().plus(Duration.ofSeconds(5)));
+ } else if ("register-unnamed".equals(action)) {
+ timeCtx.registerOnTime(timeCtx.time().plus(Duration.ofSeconds(5)));
+ } else if ("clear-by-name".equals(action)) {
+ timeCtx.clearTimer("myTimer");
+ } else if ("clear-by-timestamp".equals(action)) {
+ timeCtx.clearTimer(timeCtx.time().plus(Duration.ofSeconds(4)));
+ } else if ("clear-all".equals(action)) {
+ ctx.clearAllTimers();
+ }
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(
+ Row.of(
+ "fired-"
+ + (ctx.currentTimer() != null
+ ? ctx.currentTimer()
+ : "unnamed")));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class OnTimePTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ String name = input.getFieldAs("name");
+ collect(Row.of("hello-" + name));
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Context Tests
+ // -------------------------------------------------------------------------
+
+ @Test
+ void testPerTableWatermarkTimerFiring() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(MultiTableTimerPTF.class)
+ .withTableArgument(
+ "leftTable", DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withTableArgument(
+ "rightTable",
+ DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("leftTable", "partition")
+ .withPartitionBy("rightTable", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ // Register a timer at t=5000 via the left table
+ harness.processElementForTable(
+ "leftTable", Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.clearOutput();
+
+ // Advance only left table past the timer — global watermark is still min(left, right)
+ // Since right has no watermark yet, global won't advance enough to fire
+ harness.setWatermarkForTable("rightTable", LocalDateTime.of(2025, 1, 1, 0, 0, 3));
+ harness.setWatermarkForTable("leftTable", LocalDateTime.of(2025, 1, 1, 0, 0, 10));
+ assertThat(harness.getFiredTimers()).isEmpty();
+
+ // Now advance right table past the timer — global watermark catches up, timer fires
+ harness.setWatermarkForTable("rightTable", LocalDateTime.of(2025, 1, 1, 0, 0, 10));
+ assertThat(harness.getFiredTimers()).hasSize(1);
+ // Both tables' partition keys are prepended (one "P1" per table)
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of(
+ "P1",
+ "P1",
+ "timer-fired",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 6)));
+ }
+ }
+
+ @Test
+ void testContextClearState() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(ContextClearStatePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of(
+ "ROW<partition STRING, action STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 2)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "count=1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)),
+ Row.of("P1", "count=2", LocalDateTime.of(2025, 1, 1, 0, 0, 2)));
+ harness.clearOutput();
+
+ harness.processElement(
+ Row.of("P1", "clear-state", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 4)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "cleared", LocalDateTime.of(2025, 1, 1, 0, 0, 3)),
+ Row.of("P1", "count=1", LocalDateTime.of(2025, 1, 1, 0, 0, 4)));
+ }
+ }
+
+ @Test
+ void testContextClearAllState() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(ContextClearStatePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of(
+ "ROW<partition STRING, action STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.clearOutput();
+
+ harness.processElement(
+ Row.of("P1", "clear-all-state", LocalDateTime.of(2025, 1, 1, 0, 0, 2)));
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "cleared-all", LocalDateTime.of(2025, 1, 1, 0, 0, 2)),
+ Row.of("P1", "count=1", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ }
+ }
+
+ @Test
+ void testContextClearAll() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(ContextClearStatePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of(
+ "ROW<partition STRING, action STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(
+ Row.of("P1", "register-timer", LocalDateTime.of(2025, 1, 1, 0, 0, 2)));
+ assertThat(harness.getPendingTimers()).hasSize(1);
+ harness.clearOutput();
+
+ harness.processElement(
+ Row.of("P1", "clear-all", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ assertThat(harness.getPendingTimers()).isEmpty();
+
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 4)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of(
+ "P1",
+ "cleared-everything",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 3)),
+ Row.of("P1", "count=1", LocalDateTime.of(2025, 1, 1, 0, 0, 4)));
+ }
+ }
+
+ @Test
+ void testContextTableSemanticsAndChangelogMode() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(ContextSemanticsIntrospectionPTF.class)
+ .withTableArgument(
+ "input", DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+
+ assertThat(harness.getOutput()).hasSize(1);
+ Row result = harness.getOutput().get(0);
+ // Field 0 is prepended partition key
+ assertThat(result.getFieldAs(0).toString()).isEqualTo("P1");
+ // partitionByColumns: [0] (index of "partition" column)
+ assertThat(result.getFieldAs(1).toString()).isEqualTo("[0]");
+ // timeColumn: 1 (index of "ts" column)
+ assertThat((int) result.getFieldAs(2)).isEqualTo(1);
+ // changelogMode: insert-only
+ assertThat(result.getFieldAs(3).toString())
+ .isEqualTo(ChangelogMode.insertOnly().toString());
+ }
+ }
+
+ @Test
+ void testPojoInputWithOnTimeColumn() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(PojoTimerPTF.class)
+ .withTableArgument("input")
+ .withPartitionBy("input", "key")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "registered-P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ assertThat(harness.getPendingTimers()).hasSize(1);
+ harness.clearOutput();
+
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 10));
+ assertThat(harness.getFiredTimers()).hasSize(1);
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "fired-check", LocalDateTime.of(2025, 1, 1, 0, 0, 6)));
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Time / On-Time Column Tests
+ // -------------------------------------------------------------------------
+
+ @Test
+ void testOnTimeColumnAppendedToEvalOutput() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(OnTimePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "hello-Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ }
+ }
+
+ @Test
+ void testOnTimeColumnWithMultipleRows() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(OnTimePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(Row.of("P1", "Bob", LocalDateTime.of(2025, 1, 1, 0, 0, 5)));
+ harness.processElement(Row.of("P2", "Carol", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "hello-Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)),
+ Row.of("P1", "hello-Bob", LocalDateTime.of(2025, 1, 1, 0, 0, 5)),
+ Row.of("P2", "hello-Carol", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ }
+ }
+
+ @Test
+ void testWatermarkAdvancesWithoutTimers() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(OnTimePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+
+ // Advancing watermark without timers registered should not produce additional output
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 10));
+ assertThat(harness.getOutput()).hasSize(1);
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Timer Tests
+ // -------------------------------------------------------------------------
+
+ @Test
+ void testNamedTimerRegistrationAndFiring() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(TimerPTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of(
+ "P1",
+ "registered-Alice",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.clearOutput();
+
+ assertThat(harness.getPendingTimers()).hasSize(1);
+ Timer timer = harness.getPendingTimers().get(0);
+ assertThat(timer.getName()).isEqualTo("timeout-Alice");
+ assertThat(timer.getTimestampAs(LocalDateTime.class))
+ .isEqualTo(LocalDateTime.of(2025, 1, 1, 0, 0, 6));
+
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 7));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of(
+ "P1",
+ "timer-fired-timeout-Alice",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 6)));
+
+ assertThat(harness.getPendingTimers()).isEmpty();
+ assertThat(harness.getFiredTimers()).hasSize(1);
Review Comment:
Also assert the specific fired timer (name + ts)?
##########
flink-table/flink-table-test-utils/src/test/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarnessTest.java:
##########
@@ -1728,4 +1741,888 @@ void testPartitionKeyValidationOnClearState() throws Exception {
harness.close();
}
+
+ // -------------------------------------------------------------------------
+ // Timer PTFs
+ // -------------------------------------------------------------------------
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class TimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ String name = input.getFieldAs("name");
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timeout-" + name, timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered-" + name));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("timer-fired-" + ctx.currentTimer()));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class UnnamedTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
Review Comment:
Also check with `Long` and `java.sql.Timestamp`?
##########
flink-table/flink-table-test-utils/src/test/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarnessTest.java:
##########
@@ -1728,4 +1741,888 @@ void testPartitionKeyValidationOnClearState() throws Exception {
harness.close();
}
+
+ // -------------------------------------------------------------------------
+ // Timer PTFs
+ // -------------------------------------------------------------------------
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class TimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ String name = input.getFieldAs("name");
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timeout-" + name, timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered-" + name));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("timer-fired-" + ctx.currentTimer()));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class UnnamedTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime(timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ String timerName = ctx.currentTimer();
+ collect(Row.of("fired-unnamed-" + (timerName == null ? "null" : timerName)));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class TimerWithStatePTF extends ProcessTableFunction<Row> {
+ public static class CounterState {
+ public long count = 0L;
+ }
+
+ public void eval(
+ Context ctx,
+ @StateHint CounterState state,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ state.count++;
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("check", timeCtx.time().plus(Duration.ofSeconds(5)));
+ }
+
+ public void onTimer(OnTimerContext ctx, @StateHint CounterState state) {
+ collect(Row.of("count=" + state.count));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class PassThroughTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({
+ ArgumentTrait.SET_SEMANTIC_TABLE,
+ ArgumentTrait.PASS_COLUMNS_THROUGH,
+ ArgumentTrait.REQUIRE_ON_TIME
+ })
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timer", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("fired"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class NoOnTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timer", timeCtx.time().plus(Duration.ofSeconds(10)));
+ collect(Row.of("registered"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class MultipleOnTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timer", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("fired-no-state"));
+ }
+
+ public void onTimer(OnTimerContext ctx, @StateHint String state) {
+ collect(Row.of("fired-with-state"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class CascadingTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("first", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("eval"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ String name = ctx.currentTimer();
+ collect(Row.of("fired-" + name));
+ if ("first".equals(name)) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("second", timeCtx.time().minus(Duration.ofSeconds(2)));
+ }
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class MultiTableTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row leftTable,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row rightTable) {
+ if (leftTable != null) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("check", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("left"));
+ }
+ if (rightTable != null) {
+ collect(Row.of("right"));
+ }
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("timer-fired"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class ContextClearStatePTF extends ProcessTableFunction<Row> {
+ public static class CounterState {
+ public long count = 0L;
+ }
+
+ public void eval(
+ Context ctx,
+ @StateHint CounterState state,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ String action = input.getFieldAs("action");
+ if ("increment".equals(action)) {
+ if (state == null) {
+ state = new CounterState();
+ }
+ state.count++;
+ collect(Row.of("count=" + state.count));
+ } else if ("clear-state".equals(action)) {
+ ctx.clearState("state");
+ collect(Row.of("cleared"));
+ } else if ("clear-all-state".equals(action)) {
+ ctx.clearAllState();
+ collect(Row.of("cleared-all"));
+ } else if ("clear-all".equals(action)) {
+ ctx.clearAll();
+ collect(Row.of("cleared-everything"));
+ } else if ("register-timer".equals(action)) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("t", timeCtx.time().plus(Duration.ofHours(1)));
+ collect(Row.of("timer-registered"));
+ }
+ }
+
+ public void onTimer(OnTimerContext ctx, @StateHint CounterState state) {
+ collect(Row.of("timer-fired"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class PojoTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ TimedEvent input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("check", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered-" + input.key));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("fired-" + ctx.currentTimer()));
+ }
+ }
+
+ @DataTypeHint("ROW<partitionCols STRING, timeCol INT, changelogMode STRING>")
+ public static class ContextSemanticsIntrospectionPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TableSemantics semantics = ctx.tableSemanticsFor("input");
+ collect(
+ Row.of(
+ java.util.Arrays.toString(semantics.partitionByColumns()),
+ semantics.timeColumn(),
+ ctx.getChangelogMode().toString()));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class ClearTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ String action = input.getFieldAs("action");
+ if ("register-named".equals(action)) {
+ timeCtx.registerOnTime("myTimer", timeCtx.time().plus(Duration.ofSeconds(5)));
+ } else if ("register-unnamed".equals(action)) {
+ timeCtx.registerOnTime(timeCtx.time().plus(Duration.ofSeconds(5)));
+ } else if ("clear-by-name".equals(action)) {
+ timeCtx.clearTimer("myTimer");
+ } else if ("clear-by-timestamp".equals(action)) {
+ timeCtx.clearTimer(timeCtx.time().plus(Duration.ofSeconds(4)));
+ } else if ("clear-all".equals(action)) {
+ ctx.clearAllTimers();
+ }
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(
+ Row.of(
+ "fired-"
+ + (ctx.currentTimer() != null
+ ? ctx.currentTimer()
+ : "unnamed")));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class OnTimePTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ String name = input.getFieldAs("name");
+ collect(Row.of("hello-" + name));
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Context Tests
+ // -------------------------------------------------------------------------
+
+ @Test
+ void testPerTableWatermarkTimerFiring() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(MultiTableTimerPTF.class)
+ .withTableArgument(
+ "leftTable", DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withTableArgument(
+ "rightTable",
+ DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("leftTable", "partition")
+ .withPartitionBy("rightTable", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ // Register a timer at t=5000 via the left table
+ harness.processElementForTable(
+ "leftTable", Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.clearOutput();
+
+ // Advance only left table past the timer — global watermark is still min(left, right)
+ // Since right has no watermark yet, global won't advance enough to fire
+ harness.setWatermarkForTable("rightTable", LocalDateTime.of(2025, 1, 1, 0, 0, 3));
+ harness.setWatermarkForTable("leftTable", LocalDateTime.of(2025, 1, 1, 0, 0, 10));
+ assertThat(harness.getFiredTimers()).isEmpty();
+
+ // Now advance right table past the timer — global watermark catches up, timer fires
+ harness.setWatermarkForTable("rightTable", LocalDateTime.of(2025, 1, 1, 0, 0, 10));
+ assertThat(harness.getFiredTimers()).hasSize(1);
+ // Both tables' partition keys are prepended (one "P1" per table)
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of(
+ "P1",
+ "P1",
+ "timer-fired",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 6)));
+ }
+ }
+
+ @Test
+ void testContextClearState() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(ContextClearStatePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of(
+ "ROW<partition STRING, action STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 2)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "count=1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)),
+ Row.of("P1", "count=2", LocalDateTime.of(2025, 1, 1, 0, 0, 2)));
+ harness.clearOutput();
+
+ harness.processElement(
+ Row.of("P1", "clear-state", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 4)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "cleared", LocalDateTime.of(2025, 1, 1, 0, 0, 3)),
+ Row.of("P1", "count=1", LocalDateTime.of(2025, 1, 1, 0, 0, 4)));
+ }
+ }
+
+ @Test
+ void testContextClearAllState() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(ContextClearStatePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of(
+ "ROW<partition STRING, action STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.clearOutput();
+
+ harness.processElement(
+ Row.of("P1", "clear-all-state", LocalDateTime.of(2025, 1, 1, 0, 0, 2)));
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "cleared-all", LocalDateTime.of(2025, 1, 1, 0, 0, 2)),
+ Row.of("P1", "count=1", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ }
+ }
+
+ @Test
+ void testContextClearAll() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(ContextClearStatePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of(
+ "ROW<partition STRING, action STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(
+ Row.of("P1", "register-timer", LocalDateTime.of(2025, 1, 1, 0, 0, 2)));
+ assertThat(harness.getPendingTimers()).hasSize(1);
+ harness.clearOutput();
+
+ harness.processElement(
+ Row.of("P1", "clear-all", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ assertThat(harness.getPendingTimers()).isEmpty();
+
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 4)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of(
+ "P1",
+ "cleared-everything",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 3)),
+ Row.of("P1", "count=1", LocalDateTime.of(2025, 1, 1, 0, 0, 4)));
+ }
+ }
+
+ @Test
+ void testContextTableSemanticsAndChangelogMode() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(ContextSemanticsIntrospectionPTF.class)
+ .withTableArgument(
+ "input", DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+
+ assertThat(harness.getOutput()).hasSize(1);
+ Row result = harness.getOutput().get(0);
+ // Field 0 is prepended partition key
+ assertThat(result.getFieldAs(0).toString()).isEqualTo("P1");
+ // partitionByColumns: [0] (index of "partition" column)
+ assertThat(result.getFieldAs(1).toString()).isEqualTo("[0]");
+ // timeColumn: 1 (index of "ts" column)
+ assertThat((int) result.getFieldAs(2)).isEqualTo(1);
+ // changelogMode: insert-only
+ assertThat(result.getFieldAs(3).toString())
+ .isEqualTo(ChangelogMode.insertOnly().toString());
+ }
+ }
+
+ @Test
+ void testPojoInputWithOnTimeColumn() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(PojoTimerPTF.class)
+ .withTableArgument("input")
+ .withPartitionBy("input", "key")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "registered-P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ assertThat(harness.getPendingTimers()).hasSize(1);
+ harness.clearOutput();
+
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 10));
+ assertThat(harness.getFiredTimers()).hasSize(1);
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "fired-check", LocalDateTime.of(2025, 1, 1, 0, 0, 6)));
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Time / On-Time Column Tests
+ // -------------------------------------------------------------------------
+
+ @Test
+ void testOnTimeColumnAppendedToEvalOutput() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(OnTimePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "hello-Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ }
+ }
+
+ @Test
+ void testOnTimeColumnWithMultipleRows() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(OnTimePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(Row.of("P1", "Bob", LocalDateTime.of(2025, 1, 1, 0, 0, 5)));
+ harness.processElement(Row.of("P2", "Carol", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "hello-Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)),
+ Row.of("P1", "hello-Bob", LocalDateTime.of(2025, 1, 1, 0, 0, 5)),
+ Row.of("P2", "hello-Carol", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ }
+ }
+
+ @Test
+ void testWatermarkAdvancesWithoutTimers() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(OnTimePTF.class)
Review Comment:
`onTimePTF` requires `on_time` definition.
Add a test with a harness that doesn't specify `on_time`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39378][table] Add support for Context, timers and on_time to PTF test harness [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28326:
URL: https://github.com/apache/flink/pull/28326#discussion_r3473614925
##########
flink-table/flink-table-test-utils/src/test/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarnessTest.java:
##########
@@ -2240,6 +2240,19 @@ void testWatermarkAdvancesWithoutTimers() throws Exception {
}
}
+ @Test
+ void testWatermarkAdvancesWithoutOnTimeColumn() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(PassthroughPTF.class)
+ .withTableArgument("input", DataTypes.of("ROW<value INT>"))
+ .build()) {
+
+ harness.processElement(Row.of(42));
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 10));
Review Comment:
Can the function access the current WM if no `on_time` attribute was specified?
Is that something that the harness needs to support?
For example, a function could:
* ask for the current WM
* register a timer for currentWm + x seconds (i.e., firing x event-time secs in the future)
* handle the timer
For this, the function would not really need an `on_time` attribute defined.
Not sure if the PTF framework supports this case, but if it does, the harness should support it too with the same behavior and there should be a test for it.
##########
flink-table/flink-table-test-utils/src/test/java/org/apache/flink/table/runtime/functions/TestHarnessTimerManagerTest.java:
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.types.Row;
+import org.apache.flink.util.function.ThrowingConsumer;
+
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+class TestHarnessTimerManagerTest {
+
+ private static final Row P1 = Row.of("P1");
+ private static final Row P2 = Row.of("P2");
+
+ private static final ThrowingConsumer<Timer, Exception> NOOP_FIRER = t -> {};
+
+ @Test
+ void testNamedTimerRegistration() {
+ TestHarnessTimerManager manager = new TestHarnessTimerManager();
+
+ manager.register(P1, 1000L, "timer-a");
+ manager.register(P1, 2000L, "timer-b");
+
+ assertThat(manager.getPendingTimers()).hasSize(2);
+ }
+
+ @Test
+ void testNamedTimerReplacementOverwritesTimestamp() throws Exception {
+ TestHarnessTimerManager manager = new TestHarnessTimerManager();
+ List<Timer> fired = new ArrayList<>();
+
+ manager.register(P1, 1000L, "timer-a");
+ manager.register(P1, 3000L, "timer-a");
+
+ assertThat(manager.getPendingTimers()).hasSize(1);
+ assertThat(manager.getPendingTimers().get(0).getTimestamp()).isEqualTo(3000L);
+
+ manager.setTableWatermark("table-a", 2000L);
+ manager.updateGlobalWatermarkAndFireTimers(fired::add);
+ assertThat(fired).isEmpty();
+
+ manager.setTableWatermark("table-a", 3000L);
+ manager.updateGlobalWatermarkAndFireTimers(fired::add);
+ assertThat(fired).hasSize(1);
+ assertThat(fired.get(0).getTimestamp()).isEqualTo(3000L);
+ }
+
+ @Test
+ void testNamedTimerReplacementIsPerPartition() {
+ TestHarnessTimerManager manager = new TestHarnessTimerManager();
+
+ manager.register(P1, 1000L, "timer-a");
+ manager.register(P2, 2000L, "timer-a");
+
+ assertThat(manager.getPendingTimers()).hasSize(2);
+ }
+
+ @Test
+ void testUnnamedTimerRegistration() {
+ TestHarnessTimerManager manager = new TestHarnessTimerManager();
+
+ manager.register(P1, 1000L, null);
+ manager.register(P1, 2000L, null);
+
+ assertThat(manager.getPendingTimers()).hasSize(2);
+ }
+
+ @Test
+ void testUnnamedTimerDeduplicationBySameTimestamp() {
+ TestHarnessTimerManager manager = new TestHarnessTimerManager();
+
+ manager.register(P1, 1000L, null);
+ manager.register(P1, 1000L, null);
+
+ assertThat(manager.getPendingTimers()).hasSize(1);
+ }
+
+ @Test
+ void testUnnamedTimerDeduplicationIsPerPartition() {
+ TestHarnessTimerManager manager = new TestHarnessTimerManager();
+
+ manager.register(P1, 1000L, null);
+ manager.register(P2, 1000L, null);
+
+ assertThat(manager.getPendingTimers()).hasSize(2);
+ }
+
+ @Test
+ void testNamedAndUnnamedTimersAreIndependent() {
+ TestHarnessTimerManager manager = new TestHarnessTimerManager();
+
+ manager.register(P1, 1000L, "timer-a");
+ manager.register(P1, 1000L, null);
+
+ assertThat(manager.getPendingTimers()).hasSize(2);
+ }
+
+ @Test
+ void testGlobalWatermarkIsMinAcrossTables() throws Exception {
+ TestHarnessTimerManager manager = new TestHarnessTimerManager();
+
+ manager.setTableWatermark("table-a", 1000L);
+ manager.setTableWatermark("table-b", 3000L);
+ manager.updateGlobalWatermarkAndFireTimers(NOOP_FIRER);
+
+ assertThat(manager.getGlobalWatermark()).isEqualTo(1000L);
+ }
+
+ @Test
+ void testGlobalWatermarkAdvancesWhenLaggingTableCatchesUp() throws Exception {
+ TestHarnessTimerManager manager = new TestHarnessTimerManager();
+
+ manager.setTableWatermark("table-a", 1000L);
+ manager.setTableWatermark("table-b", 3000L);
+ manager.updateGlobalWatermarkAndFireTimers(NOOP_FIRER);
+ assertThat(manager.getGlobalWatermark()).isEqualTo(1000L);
+
+ manager.setTableWatermark("table-a", 5000L);
+ manager.updateGlobalWatermarkAndFireTimers(NOOP_FIRER);
+ assertThat(manager.getGlobalWatermark()).isEqualTo(3000L);
+ }
+
+ @Test
+ void testPerTableWatermarkCannotMoveBackward() {
+ TestHarnessTimerManager manager = new TestHarnessTimerManager();
+
+ manager.setTableWatermark("table-a", 2000L);
+ assertThrows(
+ IllegalArgumentException.class, () -> manager.setTableWatermark("table-a", 1000L));
+ }
+
+ @Test
+ void testPerTableWatermarkTrackedIndependently() {
+ TestHarnessTimerManager manager = new TestHarnessTimerManager();
+
+ manager.setTableWatermark("table-a", 1000L);
+ manager.setTableWatermark("table-b", 3000L);
+
+ assertThat(manager.getWatermarkForTable("table-a")).isEqualTo(1000L);
+ assertThat(manager.getWatermarkForTable("table-b")).isEqualTo(3000L);
+ }
+
+ @Test
+ void testNewTableCanCauseGlobalWatermarkBackwardError() throws Exception {
+ TestHarnessTimerManager manager = new TestHarnessTimerManager();
+
+ manager.setTableWatermark("table-a", 5000L);
+ manager.updateGlobalWatermarkAndFireTimers(NOOP_FIRER);
+ assertThat(manager.getGlobalWatermark()).isEqualTo(5000L);
+
+ manager.setTableWatermark("table-b", 1000L);
+ assertThrows(
+ IllegalArgumentException.class,
+ () -> manager.updateGlobalWatermarkAndFireTimers(NOOP_FIRER));
+ }
+
+ @Test
+ void testTimerFiresWhenWatermarkAdvancesPastTimestamp() throws Exception {
+ TestHarnessTimerManager manager = new TestHarnessTimerManager();
+ List<Timer> fired = new ArrayList<>();
+
+ manager.register(P1, 1000L, "timer-a");
+ manager.setTableWatermark("table-a", 2000L);
+ manager.updateGlobalWatermarkAndFireTimers(fired::add);
+
+ assertThat(fired).hasSize(1);
+ assertThat(fired.get(0).getName()).isEqualTo("timer-a");
+ assertThat(manager.getPendingTimers()).isEmpty();
+ assertThat(manager.getFiredTimers()).hasSize(1);
+ }
+
+ @Test
+ void testTimerDoesNotFireBeforeWatermark() throws Exception {
+ TestHarnessTimerManager manager = new TestHarnessTimerManager();
+ List<Timer> fired = new ArrayList<>();
+
+ manager.register(P1, 3000L, "timer-a");
+ manager.setTableWatermark("table-a", 2000L);
Review Comment:
`manager.setTableWatermark("table-a", 2999L);`
##########
flink-table/flink-table-test-utils/src/test/java/org/apache/flink/table/runtime/functions/TestHarnessTimerManagerTest.java:
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.types.Row;
+import org.apache.flink.util.function.ThrowingConsumer;
+
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+class TestHarnessTimerManagerTest {
+
+ private static final Row P1 = Row.of("P1");
+ private static final Row P2 = Row.of("P2");
+
+ private static final ThrowingConsumer<Timer, Exception> NOOP_FIRER = t -> {};
+
+ @Test
+ void testNamedTimerRegistration() {
+ TestHarnessTimerManager manager = new TestHarnessTimerManager();
+
+ manager.register(P1, 1000L, "timer-a");
+ manager.register(P1, 2000L, "timer-b");
+
+ assertThat(manager.getPendingTimers()).hasSize(2);
+ }
+
+ @Test
+ void testNamedTimerReplacementOverwritesTimestamp() throws Exception {
+ TestHarnessTimerManager manager = new TestHarnessTimerManager();
+ List<Timer> fired = new ArrayList<>();
+
+ manager.register(P1, 1000L, "timer-a");
+ manager.register(P1, 3000L, "timer-a");
+
+ assertThat(manager.getPendingTimers()).hasSize(1);
+ assertThat(manager.getPendingTimers().get(0).getTimestamp()).isEqualTo(3000L);
+
+ manager.setTableWatermark("table-a", 2000L);
+ manager.updateGlobalWatermarkAndFireTimers(fired::add);
+ assertThat(fired).isEmpty();
+
+ manager.setTableWatermark("table-a", 3000L);
+ manager.updateGlobalWatermarkAndFireTimers(fired::add);
+ assertThat(fired).hasSize(1);
+ assertThat(fired.get(0).getTimestamp()).isEqualTo(3000L);
+ }
+
+ @Test
+ void testNamedTimerReplacementIsPerPartition() {
+ TestHarnessTimerManager manager = new TestHarnessTimerManager();
+
+ manager.register(P1, 1000L, "timer-a");
+ manager.register(P2, 2000L, "timer-a");
+
+ assertThat(manager.getPendingTimers()).hasSize(2);
+ }
+
+ @Test
+ void testUnnamedTimerRegistration() {
+ TestHarnessTimerManager manager = new TestHarnessTimerManager();
+
+ manager.register(P1, 1000L, null);
+ manager.register(P1, 2000L, null);
+
+ assertThat(manager.getPendingTimers()).hasSize(2);
+ }
+
+ @Test
+ void testUnnamedTimerDeduplicationBySameTimestamp() {
+ TestHarnessTimerManager manager = new TestHarnessTimerManager();
+
+ manager.register(P1, 1000L, null);
+ manager.register(P1, 1000L, null);
+
+ assertThat(manager.getPendingTimers()).hasSize(1);
+ }
+
+ @Test
+ void testUnnamedTimerDeduplicationIsPerPartition() {
+ TestHarnessTimerManager manager = new TestHarnessTimerManager();
+
+ manager.register(P1, 1000L, null);
+ manager.register(P2, 1000L, null);
+
+ assertThat(manager.getPendingTimers()).hasSize(2);
+ }
+
+ @Test
+ void testNamedAndUnnamedTimersAreIndependent() {
+ TestHarnessTimerManager manager = new TestHarnessTimerManager();
+
+ manager.register(P1, 1000L, "timer-a");
+ manager.register(P1, 1000L, null);
+
+ assertThat(manager.getPendingTimers()).hasSize(2);
+ }
+
+ @Test
+ void testGlobalWatermarkIsMinAcrossTables() throws Exception {
+ TestHarnessTimerManager manager = new TestHarnessTimerManager();
+
+ manager.setTableWatermark("table-a", 1000L);
+ manager.setTableWatermark("table-b", 3000L);
+ manager.updateGlobalWatermarkAndFireTimers(NOOP_FIRER);
+
+ assertThat(manager.getGlobalWatermark()).isEqualTo(1000L);
+ }
+
+ @Test
+ void testGlobalWatermarkAdvancesWhenLaggingTableCatchesUp() throws Exception {
+ TestHarnessTimerManager manager = new TestHarnessTimerManager();
+
+ manager.setTableWatermark("table-a", 1000L);
+ manager.setTableWatermark("table-b", 3000L);
+ manager.updateGlobalWatermarkAndFireTimers(NOOP_FIRER);
+ assertThat(manager.getGlobalWatermark()).isEqualTo(1000L);
+
+ manager.setTableWatermark("table-a", 5000L);
+ manager.updateGlobalWatermarkAndFireTimers(NOOP_FIRER);
+ assertThat(manager.getGlobalWatermark()).isEqualTo(3000L);
+ }
+
+ @Test
+ void testPerTableWatermarkCannotMoveBackward() {
+ TestHarnessTimerManager manager = new TestHarnessTimerManager();
+
+ manager.setTableWatermark("table-a", 2000L);
+ assertThrows(
+ IllegalArgumentException.class, () -> manager.setTableWatermark("table-a", 1000L));
+ }
+
+ @Test
+ void testPerTableWatermarkTrackedIndependently() {
+ TestHarnessTimerManager manager = new TestHarnessTimerManager();
+
+ manager.setTableWatermark("table-a", 1000L);
+ manager.setTableWatermark("table-b", 3000L);
+
+ assertThat(manager.getWatermarkForTable("table-a")).isEqualTo(1000L);
+ assertThat(manager.getWatermarkForTable("table-b")).isEqualTo(3000L);
+ }
+
+ @Test
+ void testNewTableCanCauseGlobalWatermarkBackwardError() throws Exception {
+ TestHarnessTimerManager manager = new TestHarnessTimerManager();
+
+ manager.setTableWatermark("table-a", 5000L);
+ manager.updateGlobalWatermarkAndFireTimers(NOOP_FIRER);
+ assertThat(manager.getGlobalWatermark()).isEqualTo(5000L);
+
+ manager.setTableWatermark("table-b", 1000L);
+ assertThrows(
+ IllegalArgumentException.class,
+ () -> manager.updateGlobalWatermarkAndFireTimers(NOOP_FIRER));
+ }
+
+ @Test
+ void testTimerFiresWhenWatermarkAdvancesPastTimestamp() throws Exception {
+ TestHarnessTimerManager manager = new TestHarnessTimerManager();
+ List<Timer> fired = new ArrayList<>();
+
+ manager.register(P1, 1000L, "timer-a");
+ manager.setTableWatermark("table-a", 2000L);
+ manager.updateGlobalWatermarkAndFireTimers(fired::add);
+
+ assertThat(fired).hasSize(1);
+ assertThat(fired.get(0).getName()).isEqualTo("timer-a");
+ assertThat(manager.getPendingTimers()).isEmpty();
+ assertThat(manager.getFiredTimers()).hasSize(1);
+ }
+
+ @Test
+ void testTimerDoesNotFireBeforeWatermark() throws Exception {
+ TestHarnessTimerManager manager = new TestHarnessTimerManager();
+ List<Timer> fired = new ArrayList<>();
+
+ manager.register(P1, 3000L, "timer-a");
+ manager.setTableWatermark("table-a", 2000L);
+ manager.updateGlobalWatermarkAndFireTimers(fired::add);
+
+ assertThat(fired).isEmpty();
+ assertThat(manager.getPendingTimers()).hasSize(1);
+ }
+
+ @Test
+ void testTimerFiresAtExactWatermark() throws Exception {
+ TestHarnessTimerManager manager = new TestHarnessTimerManager();
+ List<Timer> fired = new ArrayList<>();
+
+ manager.register(P1, 2000L, "timer-a");
+ manager.setTableWatermark("table-a", 2000L);
+ manager.updateGlobalWatermarkAndFireTimers(fired::add);
+
+ assertThat(fired).hasSize(1);
+ }
+
+ @Test
+ void testFiringOrderIsDeterministic() throws Exception {
+ TestHarnessTimerManager manager = new TestHarnessTimerManager();
+ List<String> firingOrder = new ArrayList<>();
+
+ manager.register(P1, 2000L, "b-timer");
+ manager.register(P1, 1000L, "c-timer");
+ manager.register(P1, 1000L, "a-timer");
+ manager.register(P2, 1000L, "a-timer");
+
+ manager.setTableWatermark("table-a", 3000L);
+ manager.updateGlobalWatermarkAndFireTimers(
+ t -> firingOrder.add(t.getKey() + ":" + t.getName()));
+
+ assertThat(firingOrder)
+ .containsExactly(
+ "+I[P1]:a-timer", "+I[P2]:a-timer", "+I[P1]:c-timer", "+I[P1]:b-timer");
+ }
+
+ @Test
+ void testCascadingTimerFiring() throws Exception {
+ TestHarnessTimerManager manager = new TestHarnessTimerManager();
+ List<String> fired = new ArrayList<>();
+
+ manager.register(P1, 1000L, "first");
+ manager.setTableWatermark("table-a", 3000L);
+ manager.updateGlobalWatermarkAndFireTimers(
+ t -> {
+ fired.add(t.getName());
+ if ("first".equals(t.getName())) {
+ manager.register(P1, 2000L, "within");
+ manager.register(P1, 5000L, "beyond");
+ }
+ });
+
+ assertThat(fired).containsExactly("first", "within");
+ assertThat(manager.getPendingTimers()).hasSize(1);
+ assertThat(manager.getPendingTimers().get(0).getName()).isEqualTo("beyond");
+ assertThat(manager.getFiredTimers()).hasSize(2);
+ }
+
+ @Test
+ void testTimerClearedDuringFiringCallbackDoesNotFire() throws Exception {
+ TestHarnessTimerManager manager = new TestHarnessTimerManager();
+ List<String> fired = new ArrayList<>();
+
+ manager.register(P1, 1000L, "first");
+ manager.register(P1, 1000L, "second");
+
+ manager.setTableWatermark("table-a", 2000L);
+ manager.updateGlobalWatermarkAndFireTimers(
+ t -> {
+ fired.add(t.getName());
+ if ("first".equals(t.getName())) {
+ manager.clearByName(P1, "second");
+ }
+ });
+
+ assertThat(fired).containsExactly("first");
Review Comment:
also assert that "second" is not pending?
##########
flink-table/flink-table-test-utils/src/test/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarnessTest.java:
##########
@@ -2455,11 +2457,13 @@ void testPassThroughWithTimersIsRejected() throws Exception {
.withOnTimeColumn("ts")
.build()) {
- assertThrows(
- TableRuntimeException.class,
- () ->
- harness.processElement(
- Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1))));
+ TableRuntimeException e =
+ assertThrows(
+ TableRuntimeException.class,
+ () ->
+ harness.processElement(
Review Comment:
Wouldn't it be better to throw when the harness is constructed?
All information should be present there already, correct?
Never mind, if this requires a major refactoring.
##########
flink-table/flink-table-test-utils/src/test/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarnessTest.java:
##########
@@ -1728,4 +1741,888 @@ void testPartitionKeyValidationOnClearState() throws Exception {
harness.close();
}
+
+ // -------------------------------------------------------------------------
+ // Timer PTFs
+ // -------------------------------------------------------------------------
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class TimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ String name = input.getFieldAs("name");
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timeout-" + name, timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered-" + name));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("timer-fired-" + ctx.currentTimer()));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class UnnamedTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime(timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ String timerName = ctx.currentTimer();
+ collect(Row.of("fired-unnamed-" + (timerName == null ? "null" : timerName)));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class TimerWithStatePTF extends ProcessTableFunction<Row> {
+ public static class CounterState {
+ public long count = 0L;
+ }
+
+ public void eval(
+ Context ctx,
+ @StateHint CounterState state,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ state.count++;
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("check", timeCtx.time().plus(Duration.ofSeconds(5)));
+ }
+
+ public void onTimer(OnTimerContext ctx, @StateHint CounterState state) {
+ collect(Row.of("count=" + state.count));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class PassThroughTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({
+ ArgumentTrait.SET_SEMANTIC_TABLE,
+ ArgumentTrait.PASS_COLUMNS_THROUGH,
+ ArgumentTrait.REQUIRE_ON_TIME
+ })
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timer", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("fired"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class NoOnTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timer", timeCtx.time().plus(Duration.ofSeconds(10)));
+ collect(Row.of("registered"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class MultipleOnTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timer", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("fired-no-state"));
+ }
+
+ public void onTimer(OnTimerContext ctx, @StateHint String state) {
+ collect(Row.of("fired-with-state"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class CascadingTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("first", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("eval"));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ String name = ctx.currentTimer();
+ collect(Row.of("fired-" + name));
+ if ("first".equals(name)) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("second", timeCtx.time().minus(Duration.ofSeconds(2)));
+ }
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class MultiTableTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row leftTable,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row rightTable) {
+ if (leftTable != null) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("check", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("left"));
+ }
+ if (rightTable != null) {
+ collect(Row.of("right"));
+ }
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("timer-fired"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class ContextClearStatePTF extends ProcessTableFunction<Row> {
+ public static class CounterState {
+ public long count = 0L;
+ }
+
+ public void eval(
+ Context ctx,
+ @StateHint CounterState state,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ String action = input.getFieldAs("action");
+ if ("increment".equals(action)) {
+ if (state == null) {
+ state = new CounterState();
+ }
+ state.count++;
+ collect(Row.of("count=" + state.count));
+ } else if ("clear-state".equals(action)) {
+ ctx.clearState("state");
+ collect(Row.of("cleared"));
+ } else if ("clear-all-state".equals(action)) {
+ ctx.clearAllState();
+ collect(Row.of("cleared-all"));
+ } else if ("clear-all".equals(action)) {
+ ctx.clearAll();
+ collect(Row.of("cleared-everything"));
+ } else if ("register-timer".equals(action)) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("t", timeCtx.time().plus(Duration.ofHours(1)));
+ collect(Row.of("timer-registered"));
+ }
+ }
+
+ public void onTimer(OnTimerContext ctx, @StateHint CounterState state) {
+ collect(Row.of("timer-fired"));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class PojoTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ TimedEvent input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("check", timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered-" + input.key));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("fired-" + ctx.currentTimer()));
+ }
+ }
+
+ @DataTypeHint("ROW<partitionCols STRING, timeCol INT, changelogMode STRING>")
+ public static class ContextSemanticsIntrospectionPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TableSemantics semantics = ctx.tableSemanticsFor("input");
+ collect(
+ Row.of(
+ java.util.Arrays.toString(semantics.partitionByColumns()),
+ semantics.timeColumn(),
+ ctx.getChangelogMode().toString()));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class ClearTimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ String action = input.getFieldAs("action");
+ if ("register-named".equals(action)) {
+ timeCtx.registerOnTime("myTimer", timeCtx.time().plus(Duration.ofSeconds(5)));
+ } else if ("register-unnamed".equals(action)) {
+ timeCtx.registerOnTime(timeCtx.time().plus(Duration.ofSeconds(5)));
+ } else if ("clear-by-name".equals(action)) {
+ timeCtx.clearTimer("myTimer");
+ } else if ("clear-by-timestamp".equals(action)) {
+ timeCtx.clearTimer(timeCtx.time().plus(Duration.ofSeconds(4)));
+ } else if ("clear-all".equals(action)) {
+ ctx.clearAllTimers();
+ }
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(
+ Row.of(
+ "fired-"
+ + (ctx.currentTimer() != null
+ ? ctx.currentTimer()
+ : "unnamed")));
+ }
+ }
+
+ @DataTypeHint("ROW<message STRING>")
+ public static class OnTimePTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ String name = input.getFieldAs("name");
+ collect(Row.of("hello-" + name));
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Context Tests
+ // -------------------------------------------------------------------------
+
+ @Test
+ void testPerTableWatermarkTimerFiring() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(MultiTableTimerPTF.class)
+ .withTableArgument(
+ "leftTable", DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withTableArgument(
+ "rightTable",
+ DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("leftTable", "partition")
+ .withPartitionBy("rightTable", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ // Register a timer at t=5000 via the left table
+ harness.processElementForTable(
+ "leftTable", Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.clearOutput();
+
+ // Advance only left table past the timer — global watermark is still min(left, right)
+ // Since right has no watermark yet, global won't advance enough to fire
+ harness.setWatermarkForTable("rightTable", LocalDateTime.of(2025, 1, 1, 0, 0, 3));
+ harness.setWatermarkForTable("leftTable", LocalDateTime.of(2025, 1, 1, 0, 0, 10));
+ assertThat(harness.getFiredTimers()).isEmpty();
+
+ // Now advance right table past the timer — global watermark catches up, timer fires
+ harness.setWatermarkForTable("rightTable", LocalDateTime.of(2025, 1, 1, 0, 0, 10));
+ assertThat(harness.getFiredTimers()).hasSize(1);
+ // Both tables' partition keys are prepended (one "P1" per table)
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of(
+ "P1",
+ "P1",
+ "timer-fired",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 6)));
+ }
+ }
+
+ @Test
+ void testContextClearState() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(ContextClearStatePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of(
+ "ROW<partition STRING, action STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 2)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "count=1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)),
+ Row.of("P1", "count=2", LocalDateTime.of(2025, 1, 1, 0, 0, 2)));
+ harness.clearOutput();
+
+ harness.processElement(
+ Row.of("P1", "clear-state", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 4)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "cleared", LocalDateTime.of(2025, 1, 1, 0, 0, 3)),
+ Row.of("P1", "count=1", LocalDateTime.of(2025, 1, 1, 0, 0, 4)));
+ }
+ }
+
+ @Test
+ void testContextClearAllState() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(ContextClearStatePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of(
+ "ROW<partition STRING, action STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.clearOutput();
+
+ harness.processElement(
+ Row.of("P1", "clear-all-state", LocalDateTime.of(2025, 1, 1, 0, 0, 2)));
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "cleared-all", LocalDateTime.of(2025, 1, 1, 0, 0, 2)),
+ Row.of("P1", "count=1", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ }
+ }
+
+ @Test
+ void testContextClearAll() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(ContextClearStatePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of(
+ "ROW<partition STRING, action STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(
+ Row.of("P1", "register-timer", LocalDateTime.of(2025, 1, 1, 0, 0, 2)));
+ assertThat(harness.getPendingTimers()).hasSize(1);
+ harness.clearOutput();
+
+ harness.processElement(
+ Row.of("P1", "clear-all", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ assertThat(harness.getPendingTimers()).isEmpty();
+
+ harness.processElement(
+ Row.of("P1", "increment", LocalDateTime.of(2025, 1, 1, 0, 0, 4)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of(
+ "P1",
+ "cleared-everything",
+ LocalDateTime.of(2025, 1, 1, 0, 0, 3)),
+ Row.of("P1", "count=1", LocalDateTime.of(2025, 1, 1, 0, 0, 4)));
+ }
+ }
+
+ @Test
+ void testContextTableSemanticsAndChangelogMode() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(ContextSemanticsIntrospectionPTF.class)
+ .withTableArgument(
+ "input", DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+
+ assertThat(harness.getOutput()).hasSize(1);
+ Row result = harness.getOutput().get(0);
+ // Field 0 is prepended partition key
+ assertThat(result.getFieldAs(0).toString()).isEqualTo("P1");
+ // partitionByColumns: [0] (index of "partition" column)
+ assertThat(result.getFieldAs(1).toString()).isEqualTo("[0]");
+ // timeColumn: 1 (index of "ts" column)
+ assertThat((int) result.getFieldAs(2)).isEqualTo(1);
+ // changelogMode: insert-only
+ assertThat(result.getFieldAs(3).toString())
+ .isEqualTo(ChangelogMode.insertOnly().toString());
+ }
+ }
+
+ @Test
+ void testPojoInputWithOnTimeColumn() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(PojoTimerPTF.class)
+ .withTableArgument("input")
+ .withPartitionBy("input", "key")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "registered-P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ assertThat(harness.getPendingTimers()).hasSize(1);
+ harness.clearOutput();
+
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 10));
+ assertThat(harness.getFiredTimers()).hasSize(1);
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "fired-check", LocalDateTime.of(2025, 1, 1, 0, 0, 6)));
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Time / On-Time Column Tests
+ // -------------------------------------------------------------------------
+
+ @Test
+ void testOnTimeColumnAppendedToEvalOutput() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(OnTimePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "hello-Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ }
+ }
+
+ @Test
+ void testOnTimeColumnWithMultipleRows() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(OnTimePTF.class)
+ .withTableArgument(
+ "input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(Row.of("P1", "Bob", LocalDateTime.of(2025, 1, 1, 0, 0, 5)));
+ harness.processElement(Row.of("P2", "Carol", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "hello-Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)),
+ Row.of("P1", "hello-Bob", LocalDateTime.of(2025, 1, 1, 0, 0, 5)),
+ Row.of("P2", "hello-Carol", LocalDateTime.of(2025, 1, 1, 0, 0, 3)));
+ }
+ }
+
+ @Test
+ void testWatermarkAdvancesWithoutTimers() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(OnTimePTF.class)
Review Comment:
I assume that you added `testWatermarkAdvancesWithoutOnTimeColumn()` to address this comment.
The new test is good, but IIRC I wanted to suggest to add a test that checks the correct error message of creating a harness for a function that requires `on_time` but with no `on_time` arg specified.
Sorry for the ambiguity
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39378][table] Add support for Context, timers and on_time to PTF test harness [flink]
Posted by "autophagy (via GitHub)" <gi...@apache.org>.
autophagy commented on code in PR #28326:
URL: https://github.com/apache/flink/pull/28326#discussion_r3381549067
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarness.java:
##########
@@ -543,6 +968,9 @@ public static class Builder<OUT> {
private final Map<String, TableArgumentConfiguration> tableArgs = new HashMap<>();
private final Map<String, PartitionConfiguration> partitionConfigs = new HashMap<>();
private final Map<String, StateArgumentConfiguration> stateArgs = new HashMap<>();
+ @Nullable private String onTimeColumnName = null;
+ private final Map<String, Long> initialWatermarks = new HashMap<>();
+ @Nullable private Long initialWatermarkForAll = null;
Review Comment:
Yeah, I agree - the initial watermark stuff isn't really useful as a builder option.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39378][table] Add support for Context, timers and on_time to PTF test harness [flink]
Posted by "autophagy (via GitHub)" <gi...@apache.org>.
autophagy commented on code in PR #28326:
URL: https://github.com/apache/flink/pull/28326#discussion_r3381985166
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarness.java:
##########
@@ -303,38 +332,419 @@ public void clearStateForKey(String stateName, Row partitionKey) {
stateManager.clearStateForKey(stateName, partitionKey);
}
+ // -------------------------------------------------------------------------
+ // Watermark & Timer API
+ // -------------------------------------------------------------------------
+
+ /**
+ * Sets the watermark for all tables to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermark(LocalDateTime watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /** Sets the watermark for all tables to the given {@link Instant} and fires eligible timers. */
+ public void setWatermark(Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(watermark.toEpochMilli());
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, LocalDateTime watermark)
+ throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link Instant} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, watermark.toEpochMilli());
+ }
+
+ /** Returns all timers (both pending and fired), sorted by timestamp then name. */
+ public List<Timer> getTimers() {
+ return Stream.concat(
+ timerManager.getPendingTimers().stream(),
+ timerManager.getFiredTimers().stream())
+ .sorted()
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all pending (not yet fired) timers, sorted by timestamp then name. */
+ public List<Timer> getPendingTimers() {
+ return timerManager.getPendingTimers();
+ }
+
+ /** Returns all pending timers with the given name. */
+ public List<Timer> getPendingTimers(String timerName) {
+ return timerManager.getPendingTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all timers that have fired, in the order they fired. */
+ public List<Timer> getFiredTimers() {
+ return timerManager.getFiredTimers();
+ }
+
+ /** Returns all fired timers with the given name. */
+ public List<Timer> getFiredTimers(String timerName) {
+ return timerManager.getFiredTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Clears the fired timer history. */
+ public void clearFiredTimers() {
+ timerManager.clearFiredTimers();
+ }
+
+ private void setWatermarkMillis(long millis) throws Exception {
+ checkState(isOpen, "Harness is not open");
+ for (TableArgumentInfo tableArg : ArgumentInfo.filterTableArguments(arguments)) {
+ timerManager.setTableWatermark(tableArg.name, millis);
+ }
+ timerManager.updateGlobalWatermarkAndFireTimers(this::fireTimer);
+ }
+
+ private void setWatermarkForTableMillis(String tableArgument, long millis) throws Exception {
+ checkState(isOpen, "Harness is not open");
+ checkNotNull(tableArgument, "tableArgument must not be null");
+ checkArgument(
+ argumentsByName.get(tableArgument) instanceof TableArgumentInfo,
+ "Unknown or non-table argument: %s",
+ tableArgument);
+ timerManager.setTableWatermark(tableArgument, millis);
+ timerManager.updateGlobalWatermarkAndFireTimers(this::fireTimer);
+ }
+
+ private void fireTimer(Timer timer) throws Exception {
+ if (onTimer == null) {
+ throw new IllegalStateException(
+ "Timer fired but no onTimer() method is defined in "
+ + function.getClass().getSimpleName());
+ }
+
+ currentInvocation = InvocationContext.forTimer(timer);
+
+ try {
+ Map<String, Object> stateMap = stateManager.loadStateForKey(timer.partitionKey);
+
+ List<StateArgumentInfo> stateArgs = ArgumentInfo.filterStateArguments(arguments);
+ Object[] methodArgs = new Object[stateArgs.size()];
+ for (int i = 0; i < stateArgs.size(); i++) {
+ methodArgs[i] = stateMap.get(stateArgs.get(i).name);
+ }
+
+ onTimer.invoke(function, new TestOnTimerContext(stateMap), methodArgs);
+ stateManager.updateStateForKey(timer.partitionKey, stateMap);
+ } catch (InvocationTargetException e) {
+ Throwable cause = e.getCause();
+ if (cause instanceof Exception) {
+ throw (Exception) cause;
+ }
+ throw new RuntimeException("onTimer() invocation failed", e);
+ } finally {
+ currentInvocation = null;
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Context implementations
+ // -------------------------------------------------------------------------
+
+ private class TestContext implements ProcessTableFunction.Context {
+ final Map<String, Object> stateMap;
+
+ TestContext(Map<String, Object> stateMap) {
+ this.stateMap = stateMap;
+ }
+
+ @Override
+ public <TimeType> ProcessTableFunction.TimeContext<TimeType> timeContext(
+ Class<TimeType> conversionClass) {
+ return new TestTimeContext<>(conversionClass);
+ }
+
+ @Override
+ public TableSemantics tableSemanticsFor(String argName) {
+ ArgumentInfo argInfo = argumentsByName.get(argName);
+ if (argInfo == null) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Argument '%s' not found. Available arguments: %s",
+ argName, argumentsByName.keySet()));
+ }
+ if (!(argInfo instanceof TableArgumentInfo)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Argument '%s' is not a table argument (type: %s)",
+ argName, argInfo.getClass().getSimpleName()));
+ }
+ TableArgumentInfo tableArg = (TableArgumentInfo) argInfo;
+ int[] partitionIndices = getPartitionColumnIndices(tableArg);
+ int timeColumn =
+ onTimeColumnName != null
+ ? getFieldNames(tableArg.dataType).indexOf(onTimeColumnName)
+ : -1;
+ return new TestHarnessTableSemantics(tableArg.dataType, partitionIndices, timeColumn);
+ }
+
+ @Override
+ public void clearState(String stateName) {
+ stateMap.remove(stateName);
+ }
+
+ @Override
+ public void clearAllState() {
+ stateMap.clear();
+ }
+
+ @Override
+ public void clearAllTimers() {
+ timerManager.clearAll(currentInvocation.partitionKey);
+ }
+
+ @Override
+ public void clearAll() {
+ stateMap.clear();
Review Comment:
See above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39378][table] Add support for Context, timers and on_time to PTF test harness [flink]
Posted by "autophagy (via GitHub)" <gi...@apache.org>.
autophagy commented on PR #28326:
URL: https://github.com/apache/flink/pull/28326#issuecomment-4809481630
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39378][table] Add support for Context, timers and on_time to PTF test harness [flink]
Posted by "autophagy (via GitHub)" <gi...@apache.org>.
autophagy commented on code in PR #28326:
URL: https://github.com/apache/flink/pull/28326#discussion_r3475762915
##########
flink-table/flink-table-test-utils/src/test/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarnessTest.java:
##########
@@ -2455,11 +2457,13 @@ void testPassThroughWithTimersIsRejected() throws Exception {
.withOnTimeColumn("ts")
.build()) {
- assertThrows(
- TableRuntimeException.class,
- () ->
- harness.processElement(
- Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1))));
+ TableRuntimeException e =
+ assertThrows(
+ TableRuntimeException.class,
+ () ->
+ harness.processElement(
Review Comment:
I thought so too, but there's a complication in that live flink also doesnt reject these PTFs at plan time but during execution, when you try to call `registerOnTime()`. So you could write a PTF that wouldn't be rejected and would run fine by having an `onTimer` method but not actually call `registerOnTime()`. If we rejected it at harness build, we would unnecessarily reject these PTFs that otherwise would run normally (even though a PTF with an onTimer but with no timer registration is, admittedly, goofy)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39378][table] Add support for Context, timers and on_time to PTF test harness [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28326:
URL: https://github.com/apache/flink/pull/28326#discussion_r3387142909
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarness.java:
##########
@@ -303,38 +332,419 @@ public void clearStateForKey(String stateName, Row partitionKey) {
stateManager.clearStateForKey(stateName, partitionKey);
}
+ // -------------------------------------------------------------------------
+ // Watermark & Timer API
+ // -------------------------------------------------------------------------
+
+ /**
+ * Sets the watermark for all tables to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermark(LocalDateTime watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /** Sets the watermark for all tables to the given {@link Instant} and fires eligible timers. */
+ public void setWatermark(Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(watermark.toEpochMilli());
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, LocalDateTime watermark)
+ throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link Instant} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, watermark.toEpochMilli());
+ }
+
+ /** Returns all timers (both pending and fired), sorted by timestamp then name. */
+ public List<Timer> getTimers() {
+ return Stream.concat(
+ timerManager.getPendingTimers().stream(),
+ timerManager.getFiredTimers().stream())
+ .sorted()
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all pending (not yet fired) timers, sorted by timestamp then name. */
+ public List<Timer> getPendingTimers() {
+ return timerManager.getPendingTimers();
+ }
+
+ /** Returns all pending timers with the given name. */
+ public List<Timer> getPendingTimers(String timerName) {
+ return timerManager.getPendingTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all timers that have fired, in the order they fired. */
+ public List<Timer> getFiredTimers() {
+ return timerManager.getFiredTimers();
+ }
+
+ /** Returns all fired timers with the given name. */
+ public List<Timer> getFiredTimers(String timerName) {
+ return timerManager.getFiredTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Clears the fired timer history. */
+ public void clearFiredTimers() {
+ timerManager.clearFiredTimers();
+ }
+
+ private void setWatermarkMillis(long millis) throws Exception {
+ checkState(isOpen, "Harness is not open");
+ for (TableArgumentInfo tableArg : ArgumentInfo.filterTableArguments(arguments)) {
+ timerManager.setTableWatermark(tableArg.name, millis);
+ }
+ timerManager.updateGlobalWatermarkAndFireTimers(this::fireTimer);
+ }
+
+ private void setWatermarkForTableMillis(String tableArgument, long millis) throws Exception {
+ checkState(isOpen, "Harness is not open");
+ checkNotNull(tableArgument, "tableArgument must not be null");
+ checkArgument(
+ argumentsByName.get(tableArgument) instanceof TableArgumentInfo,
+ "Unknown or non-table argument: %s",
+ tableArgument);
+ timerManager.setTableWatermark(tableArgument, millis);
+ timerManager.updateGlobalWatermarkAndFireTimers(this::fireTimer);
+ }
+
+ private void fireTimer(Timer timer) throws Exception {
+ if (onTimer == null) {
+ throw new IllegalStateException(
+ "Timer fired but no onTimer() method is defined in "
+ + function.getClass().getSimpleName());
+ }
+
+ currentInvocation = InvocationContext.forTimer(timer);
+
+ try {
+ Map<String, Object> stateMap = stateManager.loadStateForKey(timer.partitionKey);
+
+ List<StateArgumentInfo> stateArgs = ArgumentInfo.filterStateArguments(arguments);
+ Object[] methodArgs = new Object[stateArgs.size()];
+ for (int i = 0; i < stateArgs.size(); i++) {
+ methodArgs[i] = stateMap.get(stateArgs.get(i).name);
+ }
+
+ onTimer.invoke(function, new TestOnTimerContext(stateMap), methodArgs);
+ stateManager.updateStateForKey(timer.partitionKey, stateMap);
+ } catch (InvocationTargetException e) {
+ Throwable cause = e.getCause();
+ if (cause instanceof Exception) {
+ throw (Exception) cause;
+ }
+ throw new RuntimeException("onTimer() invocation failed", e);
+ } finally {
+ currentInvocation = null;
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Context implementations
+ // -------------------------------------------------------------------------
+
+ private class TestContext implements ProcessTableFunction.Context {
+ final Map<String, Object> stateMap;
+
+ TestContext(Map<String, Object> stateMap) {
+ this.stateMap = stateMap;
+ }
+
+ @Override
+ public <TimeType> ProcessTableFunction.TimeContext<TimeType> timeContext(
+ Class<TimeType> conversionClass) {
+ return new TestTimeContext<>(conversionClass);
+ }
+
+ @Override
+ public TableSemantics tableSemanticsFor(String argName) {
+ ArgumentInfo argInfo = argumentsByName.get(argName);
+ if (argInfo == null) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Argument '%s' not found. Available arguments: %s",
+ argName, argumentsByName.keySet()));
+ }
+ if (!(argInfo instanceof TableArgumentInfo)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Argument '%s' is not a table argument (type: %s)",
+ argName, argInfo.getClass().getSimpleName()));
+ }
+ TableArgumentInfo tableArg = (TableArgumentInfo) argInfo;
+ int[] partitionIndices = getPartitionColumnIndices(tableArg);
+ int timeColumn =
+ onTimeColumnName != null
+ ? getFieldNames(tableArg.dataType).indexOf(onTimeColumnName)
+ : -1;
+ return new TestHarnessTableSemantics(tableArg.dataType, partitionIndices, timeColumn);
+ }
+
+ @Override
+ public void clearState(String stateName) {
+ stateMap.remove(stateName);
+ }
+
+ @Override
+ public void clearAllState() {
+ stateMap.clear();
+ }
+
+ @Override
+ public void clearAllTimers() {
+ timerManager.clearAll(currentInvocation.partitionKey);
+ }
+
+ @Override
+ public void clearAll() {
+ stateMap.clear();
+ timerManager.clearAll(currentInvocation.partitionKey);
+ }
+
+ @Override
+ public ChangelogMode getChangelogMode() {
+ return ChangelogMode.insertOnly();
+ }
+ }
+
+ private class TestTimeContext<TimeType> implements ProcessTableFunction.TimeContext<TimeType> {
+ private final Class<TimeType> conversionClass;
+
+ TestTimeContext(Class<TimeType> conversionClass) {
+ this.conversionClass = conversionClass;
+ }
+
+ @Override
+ public TimeType time() {
+ InvocationContext ctx = currentInvocation;
+ if (ctx.isTimerInvocation()) {
+ return fromMillis(ctx.firingTimer.timestamp);
+ }
+ if (ctx.isEvalInvocation() && onTimeColumnName != null) {
+ ArgumentInfo argInfo = argumentsByName.get(ctx.tableArgumentName);
+ if (argInfo instanceof TableArgumentInfo) {
+ TableArgumentInfo tableArg = (TableArgumentInfo) argInfo;
+ if (!getFieldNames(tableArg.dataType).contains(onTimeColumnName)) {
+ return null;
+ }
+ }
+ Object timeValue = ctx.row.getField(onTimeColumnName);
+ if (timeValue == null) {
+ return null;
+ }
+ return fromMillis(toMillis(timeValue));
+ }
+ return null;
+ }
+
+ @Override
+ public TimeType tableWatermark() {
+ InvocationContext ctx = currentInvocation;
+ if (!ctx.isEvalInvocation()) {
+ return null;
+ }
+ Long wm = timerManager.getWatermarkForTable(ctx.tableArgumentName);
+ return wm != null ? fromMillis(wm) : null;
+ }
+
+ @Override
+ public TimeType currentWatermark() {
+ Long wm = timerManager.getGlobalWatermark();
+ return wm != null ? fromMillis(wm) : null;
+ }
+
+ @Override
+ public void registerOnTime(String name, TimeType time) {
+ checkTimersEnabled();
+ checkNotNull(name, "Timer name must not be null");
+ checkNotNull(time, "Timer timestamp must not be null");
+ timerManager.register(currentInvocation.partitionKey, toMillis(time), name);
+ }
+
+ @Override
+ public void registerOnTime(TimeType time) {
+ checkTimersEnabled();
+ checkNotNull(time, "Timer timestamp must not be null");
+ timerManager.register(currentInvocation.partitionKey, toMillis(time), null);
+ }
+
+ @Override
+ public void clearTimer(String name) {
+ checkNotNull(name, "Timer name must not be null");
+ timerManager.clearByName(currentInvocation.partitionKey, name);
+ }
+
+ @Override
+ public void clearTimer(TimeType time) {
+ checkNotNull(time, "Timer timestamp must not be null");
+ timerManager.clearByTimestamp(currentInvocation.partitionKey, toMillis(time));
+ }
+
+ @Override
+ public void clearAllTimers() {
+ timerManager.clearAll(currentInvocation.partitionKey);
+ }
+
+ private void checkTimersEnabled() {
+ boolean enabled =
+ ArgumentInfo.filterTableArguments(arguments).stream()
+ .anyMatch(
+ t ->
+ t.isSetSemantic
+ && t.prependStrategy
+ != OutputPrependStrategy.ALL_COLUMNS);
+ if (!enabled) {
+ throw new TableRuntimeException(
+ "Timers are not supported in the current PTF declaration. "
+ + "Note that only PTFs that take set semantic tables support timers. "
+ + "Also timers are not available for advanced traits such as "
+ + "supporting pass-through columns or updates.");
+ }
+ }
+
+ private TimeType fromMillis(long millis) {
+ return convertFromMillis(millis, conversionClass);
+ }
+
+ private long toMillis(Object time) {
+ if (time instanceof Long) {
+ return (Long) time;
+ } else if (time instanceof Instant) {
+ return ((Instant) time).toEpochMilli();
+ } else if (time instanceof LocalDateTime) {
+ return DateTimeUtils.toTimestampMillis((LocalDateTime) time);
+ } else if (time instanceof java.sql.Timestamp) {
+ return ((java.sql.Timestamp) time).getTime();
+ }
+ throw new IllegalArgumentException(
+ "Unsupported time type: " + time.getClass().getSimpleName());
+ }
+ }
+
+ private class TestOnTimerContext extends TestContext
+ implements ProcessTableFunction.OnTimerContext {
+ TestOnTimerContext(Map<String, Object> stateMap) {
+ super(stateMap);
+ }
+
+ @Override
+ public String currentTimer() {
+ if (currentInvocation.isTimerInvocation()) {
+ return currentInvocation.firingTimer.getName();
+ }
+ return null;
+ }
+ }
+
+ private static int[] getPartitionColumnIndices(TableArgumentInfo arg) {
+ if (arg.partitionColumnNames == null || arg.partitionColumnNames.length == 0) {
+ return new int[0];
+ }
+ List<String> fieldNames = getFieldNames(arg.dataType);
+ int[] indices = new int[arg.partitionColumnNames.length];
+ for (int i = 0; i < arg.partitionColumnNames.length; i++) {
+ String colName = arg.partitionColumnNames[i];
+ int index = fieldNames.indexOf(colName);
+ if (index < 0) {
+ throw new IllegalStateException(
+ "Partition column '"
+ + colName
+ + "' not found in table argument. "
+ + "Available fields: "
+ + fieldNames);
+ }
+ indices[i] = index;
+ }
+ return indices;
+ }
+
+ @Nullable
+ private Class<?> resolveRowtimeConversionClass(List<TableArgumentInfo> tableArguments) {
Review Comment:
thanks for checking! I think the harness should do the same then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39378][table] Add support for Context, timers and on_time to PTF test harness [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28326:
URL: https://github.com/apache/flink/pull/28326#discussion_r3387123212
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarness.java:
##########
@@ -303,38 +332,419 @@ public void clearStateForKey(String stateName, Row partitionKey) {
stateManager.clearStateForKey(stateName, partitionKey);
}
+ // -------------------------------------------------------------------------
+ // Watermark & Timer API
+ // -------------------------------------------------------------------------
+
+ /**
+ * Sets the watermark for all tables to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermark(LocalDateTime watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /** Sets the watermark for all tables to the given {@link Instant} and fires eligible timers. */
+ public void setWatermark(Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(watermark.toEpochMilli());
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, LocalDateTime watermark)
+ throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link Instant} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, watermark.toEpochMilli());
+ }
+
+ /** Returns all timers (both pending and fired), sorted by timestamp then name. */
+ public List<Timer> getTimers() {
+ return Stream.concat(
+ timerManager.getPendingTimers().stream(),
+ timerManager.getFiredTimers().stream())
+ .sorted()
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all pending (not yet fired) timers, sorted by timestamp then name. */
+ public List<Timer> getPendingTimers() {
+ return timerManager.getPendingTimers();
+ }
+
+ /** Returns all pending timers with the given name. */
+ public List<Timer> getPendingTimers(String timerName) {
+ return timerManager.getPendingTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all timers that have fired, in the order they fired. */
+ public List<Timer> getFiredTimers() {
+ return timerManager.getFiredTimers();
+ }
+
+ /** Returns all fired timers with the given name. */
+ public List<Timer> getFiredTimers(String timerName) {
+ return timerManager.getFiredTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Clears the fired timer history. */
+ public void clearFiredTimers() {
+ timerManager.clearFiredTimers();
+ }
+
+ private void setWatermarkMillis(long millis) throws Exception {
+ checkState(isOpen, "Harness is not open");
+ for (TableArgumentInfo tableArg : ArgumentInfo.filterTableArguments(arguments)) {
+ timerManager.setTableWatermark(tableArg.name, millis);
+ }
+ timerManager.updateGlobalWatermarkAndFireTimers(this::fireTimer);
+ }
+
+ private void setWatermarkForTableMillis(String tableArgument, long millis) throws Exception {
+ checkState(isOpen, "Harness is not open");
+ checkNotNull(tableArgument, "tableArgument must not be null");
+ checkArgument(
+ argumentsByName.get(tableArgument) instanceof TableArgumentInfo,
+ "Unknown or non-table argument: %s",
+ tableArgument);
+ timerManager.setTableWatermark(tableArgument, millis);
+ timerManager.updateGlobalWatermarkAndFireTimers(this::fireTimer);
+ }
+
+ private void fireTimer(Timer timer) throws Exception {
+ if (onTimer == null) {
+ throw new IllegalStateException(
+ "Timer fired but no onTimer() method is defined in "
+ + function.getClass().getSimpleName());
+ }
+
+ currentInvocation = InvocationContext.forTimer(timer);
+
+ try {
+ Map<String, Object> stateMap = stateManager.loadStateForKey(timer.partitionKey);
+
+ List<StateArgumentInfo> stateArgs = ArgumentInfo.filterStateArguments(arguments);
+ Object[] methodArgs = new Object[stateArgs.size()];
+ for (int i = 0; i < stateArgs.size(); i++) {
+ methodArgs[i] = stateMap.get(stateArgs.get(i).name);
+ }
+
+ onTimer.invoke(function, new TestOnTimerContext(stateMap), methodArgs);
+ stateManager.updateStateForKey(timer.partitionKey, stateMap);
+ } catch (InvocationTargetException e) {
+ Throwable cause = e.getCause();
+ if (cause instanceof Exception) {
+ throw (Exception) cause;
+ }
+ throw new RuntimeException("onTimer() invocation failed", e);
+ } finally {
+ currentInvocation = null;
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Context implementations
+ // -------------------------------------------------------------------------
+
+ private class TestContext implements ProcessTableFunction.Context {
+ final Map<String, Object> stateMap;
+
+ TestContext(Map<String, Object> stateMap) {
+ this.stateMap = stateMap;
+ }
+
+ @Override
+ public <TimeType> ProcessTableFunction.TimeContext<TimeType> timeContext(
+ Class<TimeType> conversionClass) {
+ return new TestTimeContext<>(conversionClass);
+ }
+
+ @Override
+ public TableSemantics tableSemanticsFor(String argName) {
+ ArgumentInfo argInfo = argumentsByName.get(argName);
+ if (argInfo == null) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Argument '%s' not found. Available arguments: %s",
+ argName, argumentsByName.keySet()));
+ }
+ if (!(argInfo instanceof TableArgumentInfo)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Argument '%s' is not a table argument (type: %s)",
+ argName, argInfo.getClass().getSimpleName()));
+ }
+ TableArgumentInfo tableArg = (TableArgumentInfo) argInfo;
+ int[] partitionIndices = getPartitionColumnIndices(tableArg);
+ int timeColumn =
+ onTimeColumnName != null
+ ? getFieldNames(tableArg.dataType).indexOf(onTimeColumnName)
+ : -1;
+ return new TestHarnessTableSemantics(tableArg.dataType, partitionIndices, timeColumn);
+ }
+
+ @Override
+ public void clearState(String stateName) {
+ stateMap.remove(stateName);
+ }
+
+ @Override
+ public void clearAllState() {
+ stateMap.clear();
Review Comment:
ah, sorry! Thought this was the full state...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39378][table] Add support for Context, timers and on_time to PTF test harness [flink]
Posted by "autophagy (via GitHub)" <gi...@apache.org>.
autophagy commented on code in PR #28326:
URL: https://github.com/apache/flink/pull/28326#discussion_r3382164630
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarness.java:
##########
@@ -711,25 +1197,67 @@ public ProcessTableFunctionTestHarness<OUT> build() throws Exception {
// Extract table arguments for output type derivation
// SystemTypeInference needs table semantics for pass-through column deduplication
- List<TableArgumentInfo> tableArgs = ArgumentInfo.filterTableArguments(arguments);
+ List<TableArgumentInfo> tableArgInfos = ArgumentInfo.filterTableArguments(arguments);
// Derive output schema using SystemTypeInference
DataType derivedOutputType =
deriveOutputTypeFromSystemInference(
- function, dataTypeFactory, systemTypeInference, tableArgs);
+ function,
+ dataTypeFactory,
+ systemTypeInference,
+ arguments,
+ tableArgInfos);
// Create output converter for PTF emissions
DataStructureConverter<Object, Object> harnessOutputConverter =
createPTFOutputConverter(derivedOutputType);
+ // Validate onTimeColumn configuration
+ if (onTimeColumnName != null) {
+ boolean foundInAnyTable =
+ tableArgInfos.stream()
+ .anyMatch(
+ t -> getFieldNames(t.dataType).contains(onTimeColumnName));
Review Comment:
We get all this for free from `SystemTypeInference`!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39378][table] Add support for Context, timers and on_time to PTF test harness [flink]
Posted by "autophagy (via GitHub)" <gi...@apache.org>.
autophagy commented on code in PR #28326:
URL: https://github.com/apache/flink/pull/28326#discussion_r3381567452
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarness.java:
##########
@@ -823,49 +1351,78 @@ private Method findEvalMethod() throws NoSuchMethodException {
"Multiple eval() methods found in "
+ functionClass.getSimpleName()
+ ". ProcessTableFunction must have exactly one eval() method.");
- } else {
- return evalMethod;
}
- }
- /**
- * Validates that the eval() method doesn't use unsupported features. Temporary, until
- * context is supported.
- */
- private void validateEvalMethodSupported(Method evalMethod, List<ArgumentInfo> arguments) {
- Parameter[] parameters = evalMethod.getParameters();
+ return evalMethod;
+ }
- for (int i = 0; i < parameters.length; i++) {
- Parameter param = parameters[i];
- Class<?> paramType = param.getType();
+ @Nullable
+ private static Method findOnTimerMethod(
+ Class<?> functionClass, List<ArgumentInfo> arguments) {
+ List<Method> candidates = ExtractionUtils.collectMethods(functionClass, "onTimer");
+ if (candidates.isEmpty()) {
Review Comment:
I had this initially, but digging into the way that live selects the ontimer function it does the same - it just uses the first one with a valid signature. I'll make sure this is the case though, it was a little while ago since I dug into this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39378][table] Add support for Context, timers and on_time to PTF test harness [flink]
Posted by "autophagy (via GitHub)" <gi...@apache.org>.
autophagy commented on code in PR #28326:
URL: https://github.com/apache/flink/pull/28326#discussion_r3382149944
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarness.java:
##########
@@ -1164,40 +1721,45 @@ private List<ArgumentInfo> extractAndValidateTypeInference(
}
/** Creates appropriate StateConverter for the given state data type. */
- private StateConverter createStateConverter(DataType stateDataType, ClassLoader classLoader)
- throws Exception {
- LogicalType logicalType = stateDataType.getLogicalType();
+ private StateConverter createStateConverter(
+ DataType stateDataType, ClassLoader classLoader) {
+ DataType resolvedType =
+ ListView.class.isAssignableFrom(stateDataType.getConversionClass())
+ || MapView.class.isAssignableFrom(
+ stateDataType.getConversionClass())
+ ? stateDataType.getChildren().get(0)
Review Comment:
Nope! It returns the KV data type, I lifted it from here https://github.com/apache/flink/blob/master/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/process/AbstractProcessTableOperator.java#L370. Just to sanity check, I stepped through it with a MapView state and it returned a `KeyValueDataType` of `MAP<STRING, INT>`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39378][table] Add support for Context, timers and on_time to PTF test harness [flink]
Posted by "autophagy (via GitHub)" <gi...@apache.org>.
autophagy commented on PR #28326:
URL: https://github.com/apache/flink/pull/28326#issuecomment-4809936510
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39378][table] Add support for Context, timers and on_time to PTF test harness [flink]
Posted by "autophagy (via GitHub)" <gi...@apache.org>.
autophagy commented on code in PR #28326:
URL: https://github.com/apache/flink/pull/28326#discussion_r3382050240
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarness.java:
##########
@@ -303,38 +332,419 @@ public void clearStateForKey(String stateName, Row partitionKey) {
stateManager.clearStateForKey(stateName, partitionKey);
}
+ // -------------------------------------------------------------------------
+ // Watermark & Timer API
+ // -------------------------------------------------------------------------
+
+ /**
+ * Sets the watermark for all tables to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermark(LocalDateTime watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /** Sets the watermark for all tables to the given {@link Instant} and fires eligible timers. */
+ public void setWatermark(Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(watermark.toEpochMilli());
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, LocalDateTime watermark)
+ throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link Instant} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, watermark.toEpochMilli());
+ }
+
+ /** Returns all timers (both pending and fired), sorted by timestamp then name. */
+ public List<Timer> getTimers() {
+ return Stream.concat(
+ timerManager.getPendingTimers().stream(),
+ timerManager.getFiredTimers().stream())
+ .sorted()
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all pending (not yet fired) timers, sorted by timestamp then name. */
+ public List<Timer> getPendingTimers() {
+ return timerManager.getPendingTimers();
+ }
+
+ /** Returns all pending timers with the given name. */
+ public List<Timer> getPendingTimers(String timerName) {
+ return timerManager.getPendingTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all timers that have fired, in the order they fired. */
+ public List<Timer> getFiredTimers() {
+ return timerManager.getFiredTimers();
+ }
+
+ /** Returns all fired timers with the given name. */
+ public List<Timer> getFiredTimers(String timerName) {
+ return timerManager.getFiredTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Clears the fired timer history. */
+ public void clearFiredTimers() {
+ timerManager.clearFiredTimers();
+ }
+
+ private void setWatermarkMillis(long millis) throws Exception {
+ checkState(isOpen, "Harness is not open");
+ for (TableArgumentInfo tableArg : ArgumentInfo.filterTableArguments(arguments)) {
+ timerManager.setTableWatermark(tableArg.name, millis);
+ }
+ timerManager.updateGlobalWatermarkAndFireTimers(this::fireTimer);
+ }
+
+ private void setWatermarkForTableMillis(String tableArgument, long millis) throws Exception {
+ checkState(isOpen, "Harness is not open");
+ checkNotNull(tableArgument, "tableArgument must not be null");
+ checkArgument(
+ argumentsByName.get(tableArgument) instanceof TableArgumentInfo,
+ "Unknown or non-table argument: %s",
+ tableArgument);
+ timerManager.setTableWatermark(tableArgument, millis);
+ timerManager.updateGlobalWatermarkAndFireTimers(this::fireTimer);
+ }
+
+ private void fireTimer(Timer timer) throws Exception {
+ if (onTimer == null) {
+ throw new IllegalStateException(
+ "Timer fired but no onTimer() method is defined in "
+ + function.getClass().getSimpleName());
+ }
+
+ currentInvocation = InvocationContext.forTimer(timer);
+
+ try {
+ Map<String, Object> stateMap = stateManager.loadStateForKey(timer.partitionKey);
+
+ List<StateArgumentInfo> stateArgs = ArgumentInfo.filterStateArguments(arguments);
+ Object[] methodArgs = new Object[stateArgs.size()];
+ for (int i = 0; i < stateArgs.size(); i++) {
+ methodArgs[i] = stateMap.get(stateArgs.get(i).name);
+ }
+
+ onTimer.invoke(function, new TestOnTimerContext(stateMap), methodArgs);
+ stateManager.updateStateForKey(timer.partitionKey, stateMap);
+ } catch (InvocationTargetException e) {
+ Throwable cause = e.getCause();
+ if (cause instanceof Exception) {
+ throw (Exception) cause;
+ }
+ throw new RuntimeException("onTimer() invocation failed", e);
+ } finally {
+ currentInvocation = null;
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Context implementations
+ // -------------------------------------------------------------------------
+
+ private class TestContext implements ProcessTableFunction.Context {
+ final Map<String, Object> stateMap;
+
+ TestContext(Map<String, Object> stateMap) {
+ this.stateMap = stateMap;
+ }
+
+ @Override
+ public <TimeType> ProcessTableFunction.TimeContext<TimeType> timeContext(
+ Class<TimeType> conversionClass) {
+ return new TestTimeContext<>(conversionClass);
+ }
+
+ @Override
+ public TableSemantics tableSemanticsFor(String argName) {
+ ArgumentInfo argInfo = argumentsByName.get(argName);
+ if (argInfo == null) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Argument '%s' not found. Available arguments: %s",
+ argName, argumentsByName.keySet()));
+ }
+ if (!(argInfo instanceof TableArgumentInfo)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Argument '%s' is not a table argument (type: %s)",
+ argName, argInfo.getClass().getSimpleName()));
+ }
+ TableArgumentInfo tableArg = (TableArgumentInfo) argInfo;
+ int[] partitionIndices = getPartitionColumnIndices(tableArg);
+ int timeColumn =
+ onTimeColumnName != null
+ ? getFieldNames(tableArg.dataType).indexOf(onTimeColumnName)
+ : -1;
+ return new TestHarnessTableSemantics(tableArg.dataType, partitionIndices, timeColumn);
+ }
+
+ @Override
+ public void clearState(String stateName) {
+ stateMap.remove(stateName);
+ }
+
+ @Override
+ public void clearAllState() {
+ stateMap.clear();
+ }
+
+ @Override
+ public void clearAllTimers() {
+ timerManager.clearAll(currentInvocation.partitionKey);
+ }
+
+ @Override
+ public void clearAll() {
+ stateMap.clear();
+ timerManager.clearAll(currentInvocation.partitionKey);
+ }
+
+ @Override
+ public ChangelogMode getChangelogMode() {
+ return ChangelogMode.insertOnly();
+ }
+ }
+
+ private class TestTimeContext<TimeType> implements ProcessTableFunction.TimeContext<TimeType> {
+ private final Class<TimeType> conversionClass;
+
+ TestTimeContext(Class<TimeType> conversionClass) {
+ this.conversionClass = conversionClass;
+ }
+
+ @Override
+ public TimeType time() {
+ InvocationContext ctx = currentInvocation;
+ if (ctx.isTimerInvocation()) {
+ return fromMillis(ctx.firingTimer.timestamp);
+ }
+ if (ctx.isEvalInvocation() && onTimeColumnName != null) {
+ ArgumentInfo argInfo = argumentsByName.get(ctx.tableArgumentName);
+ if (argInfo instanceof TableArgumentInfo) {
+ TableArgumentInfo tableArg = (TableArgumentInfo) argInfo;
+ if (!getFieldNames(tableArg.dataType).contains(onTimeColumnName)) {
+ return null;
+ }
+ }
+ Object timeValue = ctx.row.getField(onTimeColumnName);
+ if (timeValue == null) {
+ return null;
+ }
+ return fromMillis(toMillis(timeValue));
+ }
+ return null;
+ }
+
+ @Override
+ public TimeType tableWatermark() {
+ InvocationContext ctx = currentInvocation;
+ if (!ctx.isEvalInvocation()) {
+ return null;
+ }
+ Long wm = timerManager.getWatermarkForTable(ctx.tableArgumentName);
+ return wm != null ? fromMillis(wm) : null;
+ }
+
+ @Override
+ public TimeType currentWatermark() {
+ Long wm = timerManager.getGlobalWatermark();
+ return wm != null ? fromMillis(wm) : null;
+ }
+
+ @Override
+ public void registerOnTime(String name, TimeType time) {
+ checkTimersEnabled();
+ checkNotNull(name, "Timer name must not be null");
+ checkNotNull(time, "Timer timestamp must not be null");
+ timerManager.register(currentInvocation.partitionKey, toMillis(time), name);
+ }
+
+ @Override
+ public void registerOnTime(TimeType time) {
+ checkTimersEnabled();
+ checkNotNull(time, "Timer timestamp must not be null");
+ timerManager.register(currentInvocation.partitionKey, toMillis(time), null);
+ }
+
+ @Override
+ public void clearTimer(String name) {
+ checkNotNull(name, "Timer name must not be null");
+ timerManager.clearByName(currentInvocation.partitionKey, name);
+ }
+
+ @Override
+ public void clearTimer(TimeType time) {
+ checkNotNull(time, "Timer timestamp must not be null");
+ timerManager.clearByTimestamp(currentInvocation.partitionKey, toMillis(time));
+ }
+
+ @Override
+ public void clearAllTimers() {
+ timerManager.clearAll(currentInvocation.partitionKey);
+ }
+
+ private void checkTimersEnabled() {
+ boolean enabled =
+ ArgumentInfo.filterTableArguments(arguments).stream()
+ .anyMatch(
+ t ->
+ t.isSetSemantic
+ && t.prependStrategy
+ != OutputPrependStrategy.ALL_COLUMNS);
+ if (!enabled) {
+ throw new TableRuntimeException(
+ "Timers are not supported in the current PTF declaration. "
+ + "Note that only PTFs that take set semantic tables support timers. "
+ + "Also timers are not available for advanced traits such as "
+ + "supporting pass-through columns or updates.");
+ }
+ }
+
+ private TimeType fromMillis(long millis) {
+ return convertFromMillis(millis, conversionClass);
+ }
+
+ private long toMillis(Object time) {
+ if (time instanceof Long) {
+ return (Long) time;
+ } else if (time instanceof Instant) {
+ return ((Instant) time).toEpochMilli();
+ } else if (time instanceof LocalDateTime) {
+ return DateTimeUtils.toTimestampMillis((LocalDateTime) time);
+ } else if (time instanceof java.sql.Timestamp) {
+ return ((java.sql.Timestamp) time).getTime();
+ }
+ throw new IllegalArgumentException(
+ "Unsupported time type: " + time.getClass().getSimpleName());
+ }
+ }
+
+ private class TestOnTimerContext extends TestContext
+ implements ProcessTableFunction.OnTimerContext {
+ TestOnTimerContext(Map<String, Object> stateMap) {
+ super(stateMap);
+ }
+
+ @Override
+ public String currentTimer() {
+ if (currentInvocation.isTimerInvocation()) {
+ return currentInvocation.firingTimer.getName();
+ }
+ return null;
+ }
+ }
+
+ private static int[] getPartitionColumnIndices(TableArgumentInfo arg) {
+ if (arg.partitionColumnNames == null || arg.partitionColumnNames.length == 0) {
+ return new int[0];
+ }
+ List<String> fieldNames = getFieldNames(arg.dataType);
+ int[] indices = new int[arg.partitionColumnNames.length];
+ for (int i = 0; i < arg.partitionColumnNames.length; i++) {
+ String colName = arg.partitionColumnNames[i];
+ int index = fieldNames.indexOf(colName);
+ if (index < 0) {
+ throw new IllegalStateException(
+ "Partition column '"
+ + colName
+ + "' not found in table argument. "
+ + "Available fields: "
+ + fieldNames);
+ }
+ indices[i] = index;
+ }
+ return indices;
+ }
+
+ @Nullable
+ private Class<?> resolveRowtimeConversionClass(List<TableArgumentInfo> tableArguments) {
Review Comment:
Looking at live, it looks like the planner rejects this. It goes over all the columns types for the `on_time` attributes, gets their `LogicalTypeRoot` and rejects it if they are different
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39378][table] Add support for Context, timers and on_time to PTF test harness [flink]
Posted by "autophagy (via GitHub)" <gi...@apache.org>.
autophagy commented on code in PR #28326:
URL: https://github.com/apache/flink/pull/28326#discussion_r3382094328
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarness.java:
##########
@@ -823,49 +1351,78 @@ private Method findEvalMethod() throws NoSuchMethodException {
"Multiple eval() methods found in "
+ functionClass.getSimpleName()
+ ". ProcessTableFunction must have exactly one eval() method.");
- } else {
- return evalMethod;
}
- }
- /**
- * Validates that the eval() method doesn't use unsupported features. Temporary, until
- * context is supported.
- */
- private void validateEvalMethodSupported(Method evalMethod, List<ArgumentInfo> arguments) {
- Parameter[] parameters = evalMethod.getParameters();
+ return evalMethod;
+ }
- for (int i = 0; i < parameters.length; i++) {
- Parameter param = parameters[i];
- Class<?> paramType = param.getType();
+ @Nullable
+ private static Method findOnTimerMethod(
+ Class<?> functionClass, List<ArgumentInfo> arguments) {
+ List<Method> candidates = ExtractionUtils.collectMethods(functionClass, "onTimer");
+ if (candidates.isEmpty()) {
Review Comment:
Yep, looking at `UserDefinedFunctionHelper.validateClassForRuntime`, it basically tries to see if any of the ontimer methods match with `.anyMatch()`. The runtime tries to find a method with the context & state parameters, if it doesnt find one it tries just with the state params, and then fails. The harness works the same way - we _could_ output a warning if theres multiple `onTimer` methods but failing would diverge from what the runtime does
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Timers [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #28326:
URL: https://github.com/apache/flink/pull/28326#issuecomment-4630127735
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b05a72732f1701c42134ca77cf93cc22afca70ef",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b05a72732f1701c42134ca77cf93cc22afca70ef",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b05a72732f1701c42134ca77cf93cc22afca70ef UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39378][table] Add support for Context, timers and on_time to PTF test harness [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28326:
URL: https://github.com/apache/flink/pull/28326#discussion_r3387176444
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarness.java:
##########
@@ -823,49 +1351,78 @@ private Method findEvalMethod() throws NoSuchMethodException {
"Multiple eval() methods found in "
+ functionClass.getSimpleName()
+ ". ProcessTableFunction must have exactly one eval() method.");
- } else {
- return evalMethod;
}
- }
- /**
- * Validates that the eval() method doesn't use unsupported features. Temporary, until
- * context is supported.
- */
- private void validateEvalMethodSupported(Method evalMethod, List<ArgumentInfo> arguments) {
- Parameter[] parameters = evalMethod.getParameters();
+ return evalMethod;
+ }
- for (int i = 0; i < parameters.length; i++) {
- Parameter param = parameters[i];
- Class<?> paramType = param.getType();
+ @Nullable
+ private static Method findOnTimerMethod(
+ Class<?> functionClass, List<ArgumentInfo> arguments) {
+ List<Method> candidates = ExtractionUtils.collectMethods(functionClass, "onTimer");
+ if (candidates.isEmpty()) {
Review Comment:
I see, then let's keep it like this. We don't need to be stricter than the runtime (although, I don't think it makes sense to have multiple `onTimer()` methods 🙄 ).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39378][table] Add support for Context, timers and on_time to PTF test harness [flink]
Posted by "autophagy (via GitHub)" <gi...@apache.org>.
autophagy commented on PR #28326:
URL: https://github.com/apache/flink/pull/28326#issuecomment-4809775497
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39378][table] Add support for Context, timers and on_time to PTF test harness [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28326:
URL: https://github.com/apache/flink/pull/28326#discussion_r3479635881
##########
flink-table/flink-table-test-utils/src/test/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarnessTest.java:
##########
@@ -2455,11 +2457,13 @@ void testPassThroughWithTimersIsRejected() throws Exception {
.withOnTimeColumn("ts")
.build()) {
- assertThrows(
- TableRuntimeException.class,
- () ->
- harness.processElement(
- Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1))));
+ TableRuntimeException e =
+ assertThrows(
+ TableRuntimeException.class,
+ () ->
+ harness.processElement(
Review Comment:
Thanks for the explanation!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39378][table] Add support for Context, timers and on_time to PTF test harness [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on PR #28326:
URL: https://github.com/apache/flink/pull/28326#issuecomment-4808277515
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39378][table] Add support for Context, timers and on_time to PTF test harness [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske merged PR #28326:
URL: https://github.com/apache/flink/pull/28326
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39378][table] Add support for Context, timers and on_time to PTF test harness [flink]
Posted by "autophagy (via GitHub)" <gi...@apache.org>.
autophagy commented on PR #28326:
URL: https://github.com/apache/flink/pull/28326#issuecomment-4809353255
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39378][table] Add support for Context, timers and on_time to PTF test harness [flink]
Posted by "autophagy (via GitHub)" <gi...@apache.org>.
autophagy commented on code in PR #28326:
URL: https://github.com/apache/flink/pull/28326#discussion_r3475684066
##########
flink-table/flink-table-test-utils/src/test/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarnessTest.java:
##########
@@ -2240,6 +2240,19 @@ void testWatermarkAdvancesWithoutTimers() throws Exception {
}
}
+ @Test
+ void testWatermarkAdvancesWithoutOnTimeColumn() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(PassthroughPTF.class)
+ .withTableArgument("input", DataTypes.of("ROW<value INT>"))
+ .build()) {
+
+ harness.processElement(Row.of(42));
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 10));
Review Comment:
Yep, this is something that the PTF framework supports! There's even a test PTF for it in the live tests:
```
public static class OptionalOnTimeFunction extends AppendProcessTableFunctionBase {
public void eval(Context ctx, @ArgumentHint(SET_SEMANTIC_TABLE) Row r) {
final TimeContext<Long> timeCtx = ctx.timeContext(Long.class);
collectEvalEvent(timeCtx, r);
Long wm = timeCtx.currentWatermark();
// Register at wm+1 to always target the immediate next watermark: the timer fires
// exactly once per watermark advance, and each new row re-registers the timer for the
// following watermark step, demonstrating repeated timer re-registration.
long timer = wm == null || wm < 0 ? 1 : wm + 1;
collectCreateTimer(timeCtx, "t", timer);
}
public void onTimer(OnTimerContext ctx) {
final TimeContext<Long> timeCtx = ctx.timeContext(Long.class);
collectOnTimerEvent(ctx);
}
}
```
The harness should already handle this correctly but I dont think we have an explicit test case for it - i'll add one!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39378][table] Add support for Context, timers and on_time to PTF test harness [flink]
Posted by "autophagy (via GitHub)" <gi...@apache.org>.
autophagy commented on code in PR #28326:
URL: https://github.com/apache/flink/pull/28326#discussion_r3381544600
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarness.java:
##########
@@ -303,38 +332,419 @@ public void clearStateForKey(String stateName, Row partitionKey) {
stateManager.clearStateForKey(stateName, partitionKey);
}
+ // -------------------------------------------------------------------------
+ // Watermark & Timer API
+ // -------------------------------------------------------------------------
+
+ /**
+ * Sets the watermark for all tables to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermark(LocalDateTime watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /** Sets the watermark for all tables to the given {@link Instant} and fires eligible timers. */
+ public void setWatermark(Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(watermark.toEpochMilli());
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, LocalDateTime watermark)
+ throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link Instant} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, watermark.toEpochMilli());
+ }
+
+ /** Returns all timers (both pending and fired), sorted by timestamp then name. */
+ public List<Timer> getTimers() {
+ return Stream.concat(
+ timerManager.getPendingTimers().stream(),
+ timerManager.getFiredTimers().stream())
+ .sorted()
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all pending (not yet fired) timers, sorted by timestamp then name. */
+ public List<Timer> getPendingTimers() {
+ return timerManager.getPendingTimers();
+ }
+
+ /** Returns all pending timers with the given name. */
+ public List<Timer> getPendingTimers(String timerName) {
+ return timerManager.getPendingTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all timers that have fired, in the order they fired. */
+ public List<Timer> getFiredTimers() {
+ return timerManager.getFiredTimers();
+ }
+
+ /** Returns all fired timers with the given name. */
+ public List<Timer> getFiredTimers(String timerName) {
+ return timerManager.getFiredTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Clears the fired timer history. */
+ public void clearFiredTimers() {
+ timerManager.clearFiredTimers();
+ }
+
+ private void setWatermarkMillis(long millis) throws Exception {
+ checkState(isOpen, "Harness is not open");
+ for (TableArgumentInfo tableArg : ArgumentInfo.filterTableArguments(arguments)) {
+ timerManager.setTableWatermark(tableArg.name, millis);
+ }
+ timerManager.updateGlobalWatermarkAndFireTimers(this::fireTimer);
+ }
+
+ private void setWatermarkForTableMillis(String tableArgument, long millis) throws Exception {
+ checkState(isOpen, "Harness is not open");
+ checkNotNull(tableArgument, "tableArgument must not be null");
+ checkArgument(
+ argumentsByName.get(tableArgument) instanceof TableArgumentInfo,
+ "Unknown or non-table argument: %s",
+ tableArgument);
+ timerManager.setTableWatermark(tableArgument, millis);
+ timerManager.updateGlobalWatermarkAndFireTimers(this::fireTimer);
+ }
+
+ private void fireTimer(Timer timer) throws Exception {
+ if (onTimer == null) {
+ throw new IllegalStateException(
+ "Timer fired but no onTimer() method is defined in "
+ + function.getClass().getSimpleName());
+ }
+
+ currentInvocation = InvocationContext.forTimer(timer);
+
+ try {
+ Map<String, Object> stateMap = stateManager.loadStateForKey(timer.partitionKey);
+
+ List<StateArgumentInfo> stateArgs = ArgumentInfo.filterStateArguments(arguments);
+ Object[] methodArgs = new Object[stateArgs.size()];
+ for (int i = 0; i < stateArgs.size(); i++) {
+ methodArgs[i] = stateMap.get(stateArgs.get(i).name);
+ }
+
+ onTimer.invoke(function, new TestOnTimerContext(stateMap), methodArgs);
+ stateManager.updateStateForKey(timer.partitionKey, stateMap);
+ } catch (InvocationTargetException e) {
+ Throwable cause = e.getCause();
+ if (cause instanceof Exception) {
+ throw (Exception) cause;
+ }
+ throw new RuntimeException("onTimer() invocation failed", e);
+ } finally {
+ currentInvocation = null;
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Context implementations
+ // -------------------------------------------------------------------------
+
+ private class TestContext implements ProcessTableFunction.Context {
+ final Map<String, Object> stateMap;
+
+ TestContext(Map<String, Object> stateMap) {
+ this.stateMap = stateMap;
+ }
+
+ @Override
+ public <TimeType> ProcessTableFunction.TimeContext<TimeType> timeContext(
+ Class<TimeType> conversionClass) {
+ return new TestTimeContext<>(conversionClass);
+ }
+
+ @Override
+ public TableSemantics tableSemanticsFor(String argName) {
+ ArgumentInfo argInfo = argumentsByName.get(argName);
+ if (argInfo == null) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Argument '%s' not found. Available arguments: %s",
+ argName, argumentsByName.keySet()));
+ }
+ if (!(argInfo instanceof TableArgumentInfo)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Argument '%s' is not a table argument (type: %s)",
+ argName, argInfo.getClass().getSimpleName()));
+ }
+ TableArgumentInfo tableArg = (TableArgumentInfo) argInfo;
+ int[] partitionIndices = getPartitionColumnIndices(tableArg);
+ int timeColumn =
+ onTimeColumnName != null
+ ? getFieldNames(tableArg.dataType).indexOf(onTimeColumnName)
+ : -1;
+ return new TestHarnessTableSemantics(tableArg.dataType, partitionIndices, timeColumn);
+ }
+
+ @Override
+ public void clearState(String stateName) {
+ stateMap.remove(stateName);
+ }
+
+ @Override
+ public void clearAllState() {
+ stateMap.clear();
+ }
+
+ @Override
+ public void clearAllTimers() {
+ timerManager.clearAll(currentInvocation.partitionKey);
+ }
+
+ @Override
+ public void clearAll() {
+ stateMap.clear();
+ timerManager.clearAll(currentInvocation.partitionKey);
+ }
+
+ @Override
+ public ChangelogMode getChangelogMode() {
+ return ChangelogMode.insertOnly();
+ }
+ }
+
+ private class TestTimeContext<TimeType> implements ProcessTableFunction.TimeContext<TimeType> {
+ private final Class<TimeType> conversionClass;
+
+ TestTimeContext(Class<TimeType> conversionClass) {
+ this.conversionClass = conversionClass;
+ }
+
+ @Override
+ public TimeType time() {
+ InvocationContext ctx = currentInvocation;
+ if (ctx.isTimerInvocation()) {
+ return fromMillis(ctx.firingTimer.timestamp);
+ }
+ if (ctx.isEvalInvocation() && onTimeColumnName != null) {
+ ArgumentInfo argInfo = argumentsByName.get(ctx.tableArgumentName);
+ if (argInfo instanceof TableArgumentInfo) {
+ TableArgumentInfo tableArg = (TableArgumentInfo) argInfo;
+ if (!getFieldNames(tableArg.dataType).contains(onTimeColumnName)) {
+ return null;
+ }
+ }
+ Object timeValue = ctx.row.getField(onTimeColumnName);
+ if (timeValue == null) {
+ return null;
+ }
+ return fromMillis(toMillis(timeValue));
+ }
+ return null;
+ }
+
+ @Override
+ public TimeType tableWatermark() {
+ InvocationContext ctx = currentInvocation;
+ if (!ctx.isEvalInvocation()) {
+ return null;
+ }
+ Long wm = timerManager.getWatermarkForTable(ctx.tableArgumentName);
+ return wm != null ? fromMillis(wm) : null;
+ }
+
+ @Override
+ public TimeType currentWatermark() {
+ Long wm = timerManager.getGlobalWatermark();
+ return wm != null ? fromMillis(wm) : null;
+ }
+
+ @Override
+ public void registerOnTime(String name, TimeType time) {
+ checkTimersEnabled();
+ checkNotNull(name, "Timer name must not be null");
+ checkNotNull(time, "Timer timestamp must not be null");
+ timerManager.register(currentInvocation.partitionKey, toMillis(time), name);
+ }
+
+ @Override
+ public void registerOnTime(TimeType time) {
+ checkTimersEnabled();
+ checkNotNull(time, "Timer timestamp must not be null");
+ timerManager.register(currentInvocation.partitionKey, toMillis(time), null);
+ }
+
+ @Override
+ public void clearTimer(String name) {
+ checkNotNull(name, "Timer name must not be null");
+ timerManager.clearByName(currentInvocation.partitionKey, name);
+ }
+
+ @Override
+ public void clearTimer(TimeType time) {
+ checkNotNull(time, "Timer timestamp must not be null");
+ timerManager.clearByTimestamp(currentInvocation.partitionKey, toMillis(time));
+ }
+
+ @Override
+ public void clearAllTimers() {
+ timerManager.clearAll(currentInvocation.partitionKey);
+ }
+
+ private void checkTimersEnabled() {
+ boolean enabled =
+ ArgumentInfo.filterTableArguments(arguments).stream()
+ .anyMatch(
+ t ->
+ t.isSetSemantic
+ && t.prependStrategy
+ != OutputPrependStrategy.ALL_COLUMNS);
+ if (!enabled) {
+ throw new TableRuntimeException(
+ "Timers are not supported in the current PTF declaration. "
+ + "Note that only PTFs that take set semantic tables support timers. "
+ + "Also timers are not available for advanced traits such as "
+ + "supporting pass-through columns or updates.");
+ }
+ }
+
+ private TimeType fromMillis(long millis) {
+ return convertFromMillis(millis, conversionClass);
+ }
+
+ private long toMillis(Object time) {
+ if (time instanceof Long) {
+ return (Long) time;
+ } else if (time instanceof Instant) {
+ return ((Instant) time).toEpochMilli();
+ } else if (time instanceof LocalDateTime) {
+ return DateTimeUtils.toTimestampMillis((LocalDateTime) time);
+ } else if (time instanceof java.sql.Timestamp) {
+ return ((java.sql.Timestamp) time).getTime();
+ }
+ throw new IllegalArgumentException(
+ "Unsupported time type: " + time.getClass().getSimpleName());
+ }
+ }
+
+ private class TestOnTimerContext extends TestContext
+ implements ProcessTableFunction.OnTimerContext {
+ TestOnTimerContext(Map<String, Object> stateMap) {
+ super(stateMap);
+ }
+
+ @Override
+ public String currentTimer() {
+ if (currentInvocation.isTimerInvocation()) {
+ return currentInvocation.firingTimer.getName();
+ }
+ return null;
+ }
+ }
+
+ private static int[] getPartitionColumnIndices(TableArgumentInfo arg) {
+ if (arg.partitionColumnNames == null || arg.partitionColumnNames.length == 0) {
+ return new int[0];
+ }
+ List<String> fieldNames = getFieldNames(arg.dataType);
+ int[] indices = new int[arg.partitionColumnNames.length];
+ for (int i = 0; i < arg.partitionColumnNames.length; i++) {
+ String colName = arg.partitionColumnNames[i];
+ int index = fieldNames.indexOf(colName);
+ if (index < 0) {
+ throw new IllegalStateException(
+ "Partition column '"
+ + colName
+ + "' not found in table argument. "
+ + "Available fields: "
+ + fieldNames);
+ }
+ indices[i] = index;
+ }
+ return indices;
+ }
+
+ @Nullable
+ private Class<?> resolveRowtimeConversionClass(List<TableArgumentInfo> tableArguments) {
Review Comment:
I thought that this was the case, but it's worth verifying against live, yeah.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39378][table] Add support for Context, timers and on_time to PTF test harness [flink]
Posted by "autophagy (via GitHub)" <gi...@apache.org>.
autophagy commented on code in PR #28326:
URL: https://github.com/apache/flink/pull/28326#discussion_r3381984056
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarness.java:
##########
@@ -303,38 +332,419 @@ public void clearStateForKey(String stateName, Row partitionKey) {
stateManager.clearStateForKey(stateName, partitionKey);
}
+ // -------------------------------------------------------------------------
+ // Watermark & Timer API
+ // -------------------------------------------------------------------------
+
+ /**
+ * Sets the watermark for all tables to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermark(LocalDateTime watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /** Sets the watermark for all tables to the given {@link Instant} and fires eligible timers. */
+ public void setWatermark(Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(watermark.toEpochMilli());
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, LocalDateTime watermark)
+ throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link Instant} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, watermark.toEpochMilli());
+ }
+
+ /** Returns all timers (both pending and fired), sorted by timestamp then name. */
+ public List<Timer> getTimers() {
+ return Stream.concat(
+ timerManager.getPendingTimers().stream(),
+ timerManager.getFiredTimers().stream())
+ .sorted()
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all pending (not yet fired) timers, sorted by timestamp then name. */
+ public List<Timer> getPendingTimers() {
+ return timerManager.getPendingTimers();
+ }
+
+ /** Returns all pending timers with the given name. */
+ public List<Timer> getPendingTimers(String timerName) {
+ return timerManager.getPendingTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all timers that have fired, in the order they fired. */
+ public List<Timer> getFiredTimers() {
+ return timerManager.getFiredTimers();
+ }
+
+ /** Returns all fired timers with the given name. */
+ public List<Timer> getFiredTimers(String timerName) {
+ return timerManager.getFiredTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Clears the fired timer history. */
+ public void clearFiredTimers() {
+ timerManager.clearFiredTimers();
+ }
+
+ private void setWatermarkMillis(long millis) throws Exception {
+ checkState(isOpen, "Harness is not open");
+ for (TableArgumentInfo tableArg : ArgumentInfo.filterTableArguments(arguments)) {
+ timerManager.setTableWatermark(tableArg.name, millis);
+ }
+ timerManager.updateGlobalWatermarkAndFireTimers(this::fireTimer);
+ }
+
+ private void setWatermarkForTableMillis(String tableArgument, long millis) throws Exception {
+ checkState(isOpen, "Harness is not open");
+ checkNotNull(tableArgument, "tableArgument must not be null");
+ checkArgument(
+ argumentsByName.get(tableArgument) instanceof TableArgumentInfo,
+ "Unknown or non-table argument: %s",
+ tableArgument);
+ timerManager.setTableWatermark(tableArgument, millis);
+ timerManager.updateGlobalWatermarkAndFireTimers(this::fireTimer);
+ }
+
+ private void fireTimer(Timer timer) throws Exception {
+ if (onTimer == null) {
+ throw new IllegalStateException(
+ "Timer fired but no onTimer() method is defined in "
+ + function.getClass().getSimpleName());
+ }
+
+ currentInvocation = InvocationContext.forTimer(timer);
+
+ try {
+ Map<String, Object> stateMap = stateManager.loadStateForKey(timer.partitionKey);
+
+ List<StateArgumentInfo> stateArgs = ArgumentInfo.filterStateArguments(arguments);
+ Object[] methodArgs = new Object[stateArgs.size()];
+ for (int i = 0; i < stateArgs.size(); i++) {
+ methodArgs[i] = stateMap.get(stateArgs.get(i).name);
+ }
+
+ onTimer.invoke(function, new TestOnTimerContext(stateMap), methodArgs);
+ stateManager.updateStateForKey(timer.partitionKey, stateMap);
+ } catch (InvocationTargetException e) {
+ Throwable cause = e.getCause();
+ if (cause instanceof Exception) {
+ throw (Exception) cause;
+ }
+ throw new RuntimeException("onTimer() invocation failed", e);
+ } finally {
+ currentInvocation = null;
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Context implementations
+ // -------------------------------------------------------------------------
+
+ private class TestContext implements ProcessTableFunction.Context {
+ final Map<String, Object> stateMap;
+
+ TestContext(Map<String, Object> stateMap) {
+ this.stateMap = stateMap;
+ }
+
+ @Override
+ public <TimeType> ProcessTableFunction.TimeContext<TimeType> timeContext(
+ Class<TimeType> conversionClass) {
+ return new TestTimeContext<>(conversionClass);
+ }
+
+ @Override
+ public TableSemantics tableSemanticsFor(String argName) {
+ ArgumentInfo argInfo = argumentsByName.get(argName);
+ if (argInfo == null) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Argument '%s' not found. Available arguments: %s",
+ argName, argumentsByName.keySet()));
+ }
+ if (!(argInfo instanceof TableArgumentInfo)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Argument '%s' is not a table argument (type: %s)",
+ argName, argInfo.getClass().getSimpleName()));
+ }
+ TableArgumentInfo tableArg = (TableArgumentInfo) argInfo;
+ int[] partitionIndices = getPartitionColumnIndices(tableArg);
+ int timeColumn =
+ onTimeColumnName != null
+ ? getFieldNames(tableArg.dataType).indexOf(onTimeColumnName)
+ : -1;
+ return new TestHarnessTableSemantics(tableArg.dataType, partitionIndices, timeColumn);
+ }
+
+ @Override
+ public void clearState(String stateName) {
+ stateMap.remove(stateName);
+ }
+
+ @Override
+ public void clearAllState() {
+ stateMap.clear();
Review Comment:
It currently is partitioned by key! The state map passed into the context is the state map for that particular partition key, since the PTFs mutate them in place we have to pass them in to the PTF, and then write them back into the state manager again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39378][table] Add support for Context, timers and on_time to PTF test harness [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28326:
URL: https://github.com/apache/flink/pull/28326#discussion_r3387142909
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarness.java:
##########
@@ -303,38 +332,419 @@ public void clearStateForKey(String stateName, Row partitionKey) {
stateManager.clearStateForKey(stateName, partitionKey);
}
+ // -------------------------------------------------------------------------
+ // Watermark & Timer API
+ // -------------------------------------------------------------------------
+
+ /**
+ * Sets the watermark for all tables to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermark(LocalDateTime watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /** Sets the watermark for all tables to the given {@link Instant} and fires eligible timers. */
+ public void setWatermark(Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(watermark.toEpochMilli());
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, LocalDateTime watermark)
+ throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link Instant} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, watermark.toEpochMilli());
+ }
+
+ /** Returns all timers (both pending and fired), sorted by timestamp then name. */
+ public List<Timer> getTimers() {
+ return Stream.concat(
+ timerManager.getPendingTimers().stream(),
+ timerManager.getFiredTimers().stream())
+ .sorted()
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all pending (not yet fired) timers, sorted by timestamp then name. */
+ public List<Timer> getPendingTimers() {
+ return timerManager.getPendingTimers();
+ }
+
+ /** Returns all pending timers with the given name. */
+ public List<Timer> getPendingTimers(String timerName) {
+ return timerManager.getPendingTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all timers that have fired, in the order they fired. */
+ public List<Timer> getFiredTimers() {
+ return timerManager.getFiredTimers();
+ }
+
+ /** Returns all fired timers with the given name. */
+ public List<Timer> getFiredTimers(String timerName) {
+ return timerManager.getFiredTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Clears the fired timer history. */
+ public void clearFiredTimers() {
+ timerManager.clearFiredTimers();
+ }
+
+ private void setWatermarkMillis(long millis) throws Exception {
+ checkState(isOpen, "Harness is not open");
+ for (TableArgumentInfo tableArg : ArgumentInfo.filterTableArguments(arguments)) {
+ timerManager.setTableWatermark(tableArg.name, millis);
+ }
+ timerManager.updateGlobalWatermarkAndFireTimers(this::fireTimer);
+ }
+
+ private void setWatermarkForTableMillis(String tableArgument, long millis) throws Exception {
+ checkState(isOpen, "Harness is not open");
+ checkNotNull(tableArgument, "tableArgument must not be null");
+ checkArgument(
+ argumentsByName.get(tableArgument) instanceof TableArgumentInfo,
+ "Unknown or non-table argument: %s",
+ tableArgument);
+ timerManager.setTableWatermark(tableArgument, millis);
+ timerManager.updateGlobalWatermarkAndFireTimers(this::fireTimer);
+ }
+
+ private void fireTimer(Timer timer) throws Exception {
+ if (onTimer == null) {
+ throw new IllegalStateException(
+ "Timer fired but no onTimer() method is defined in "
+ + function.getClass().getSimpleName());
+ }
+
+ currentInvocation = InvocationContext.forTimer(timer);
+
+ try {
+ Map<String, Object> stateMap = stateManager.loadStateForKey(timer.partitionKey);
+
+ List<StateArgumentInfo> stateArgs = ArgumentInfo.filterStateArguments(arguments);
+ Object[] methodArgs = new Object[stateArgs.size()];
+ for (int i = 0; i < stateArgs.size(); i++) {
+ methodArgs[i] = stateMap.get(stateArgs.get(i).name);
+ }
+
+ onTimer.invoke(function, new TestOnTimerContext(stateMap), methodArgs);
+ stateManager.updateStateForKey(timer.partitionKey, stateMap);
+ } catch (InvocationTargetException e) {
+ Throwable cause = e.getCause();
+ if (cause instanceof Exception) {
+ throw (Exception) cause;
+ }
+ throw new RuntimeException("onTimer() invocation failed", e);
+ } finally {
+ currentInvocation = null;
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Context implementations
+ // -------------------------------------------------------------------------
+
+ private class TestContext implements ProcessTableFunction.Context {
+ final Map<String, Object> stateMap;
+
+ TestContext(Map<String, Object> stateMap) {
+ this.stateMap = stateMap;
+ }
+
+ @Override
+ public <TimeType> ProcessTableFunction.TimeContext<TimeType> timeContext(
+ Class<TimeType> conversionClass) {
+ return new TestTimeContext<>(conversionClass);
+ }
+
+ @Override
+ public TableSemantics tableSemanticsFor(String argName) {
+ ArgumentInfo argInfo = argumentsByName.get(argName);
+ if (argInfo == null) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Argument '%s' not found. Available arguments: %s",
+ argName, argumentsByName.keySet()));
+ }
+ if (!(argInfo instanceof TableArgumentInfo)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Argument '%s' is not a table argument (type: %s)",
+ argName, argInfo.getClass().getSimpleName()));
+ }
+ TableArgumentInfo tableArg = (TableArgumentInfo) argInfo;
+ int[] partitionIndices = getPartitionColumnIndices(tableArg);
+ int timeColumn =
+ onTimeColumnName != null
+ ? getFieldNames(tableArg.dataType).indexOf(onTimeColumnName)
+ : -1;
+ return new TestHarnessTableSemantics(tableArg.dataType, partitionIndices, timeColumn);
+ }
+
+ @Override
+ public void clearState(String stateName) {
+ stateMap.remove(stateName);
+ }
+
+ @Override
+ public void clearAllState() {
+ stateMap.clear();
+ }
+
+ @Override
+ public void clearAllTimers() {
+ timerManager.clearAll(currentInvocation.partitionKey);
+ }
+
+ @Override
+ public void clearAll() {
+ stateMap.clear();
+ timerManager.clearAll(currentInvocation.partitionKey);
+ }
+
+ @Override
+ public ChangelogMode getChangelogMode() {
+ return ChangelogMode.insertOnly();
+ }
+ }
+
+ private class TestTimeContext<TimeType> implements ProcessTableFunction.TimeContext<TimeType> {
+ private final Class<TimeType> conversionClass;
+
+ TestTimeContext(Class<TimeType> conversionClass) {
+ this.conversionClass = conversionClass;
+ }
+
+ @Override
+ public TimeType time() {
+ InvocationContext ctx = currentInvocation;
+ if (ctx.isTimerInvocation()) {
+ return fromMillis(ctx.firingTimer.timestamp);
+ }
+ if (ctx.isEvalInvocation() && onTimeColumnName != null) {
+ ArgumentInfo argInfo = argumentsByName.get(ctx.tableArgumentName);
+ if (argInfo instanceof TableArgumentInfo) {
+ TableArgumentInfo tableArg = (TableArgumentInfo) argInfo;
+ if (!getFieldNames(tableArg.dataType).contains(onTimeColumnName)) {
+ return null;
+ }
+ }
+ Object timeValue = ctx.row.getField(onTimeColumnName);
+ if (timeValue == null) {
+ return null;
+ }
+ return fromMillis(toMillis(timeValue));
+ }
+ return null;
+ }
+
+ @Override
+ public TimeType tableWatermark() {
+ InvocationContext ctx = currentInvocation;
+ if (!ctx.isEvalInvocation()) {
+ return null;
+ }
+ Long wm = timerManager.getWatermarkForTable(ctx.tableArgumentName);
+ return wm != null ? fromMillis(wm) : null;
+ }
+
+ @Override
+ public TimeType currentWatermark() {
+ Long wm = timerManager.getGlobalWatermark();
+ return wm != null ? fromMillis(wm) : null;
+ }
+
+ @Override
+ public void registerOnTime(String name, TimeType time) {
+ checkTimersEnabled();
+ checkNotNull(name, "Timer name must not be null");
+ checkNotNull(time, "Timer timestamp must not be null");
+ timerManager.register(currentInvocation.partitionKey, toMillis(time), name);
+ }
+
+ @Override
+ public void registerOnTime(TimeType time) {
+ checkTimersEnabled();
+ checkNotNull(time, "Timer timestamp must not be null");
+ timerManager.register(currentInvocation.partitionKey, toMillis(time), null);
+ }
+
+ @Override
+ public void clearTimer(String name) {
+ checkNotNull(name, "Timer name must not be null");
+ timerManager.clearByName(currentInvocation.partitionKey, name);
+ }
+
+ @Override
+ public void clearTimer(TimeType time) {
+ checkNotNull(time, "Timer timestamp must not be null");
+ timerManager.clearByTimestamp(currentInvocation.partitionKey, toMillis(time));
+ }
+
+ @Override
+ public void clearAllTimers() {
+ timerManager.clearAll(currentInvocation.partitionKey);
+ }
+
+ private void checkTimersEnabled() {
+ boolean enabled =
+ ArgumentInfo.filterTableArguments(arguments).stream()
+ .anyMatch(
+ t ->
+ t.isSetSemantic
+ && t.prependStrategy
+ != OutputPrependStrategy.ALL_COLUMNS);
+ if (!enabled) {
+ throw new TableRuntimeException(
+ "Timers are not supported in the current PTF declaration. "
+ + "Note that only PTFs that take set semantic tables support timers. "
+ + "Also timers are not available for advanced traits such as "
+ + "supporting pass-through columns or updates.");
+ }
+ }
+
+ private TimeType fromMillis(long millis) {
+ return convertFromMillis(millis, conversionClass);
+ }
+
+ private long toMillis(Object time) {
+ if (time instanceof Long) {
+ return (Long) time;
+ } else if (time instanceof Instant) {
+ return ((Instant) time).toEpochMilli();
+ } else if (time instanceof LocalDateTime) {
+ return DateTimeUtils.toTimestampMillis((LocalDateTime) time);
+ } else if (time instanceof java.sql.Timestamp) {
+ return ((java.sql.Timestamp) time).getTime();
+ }
+ throw new IllegalArgumentException(
+ "Unsupported time type: " + time.getClass().getSimpleName());
+ }
+ }
+
+ private class TestOnTimerContext extends TestContext
+ implements ProcessTableFunction.OnTimerContext {
+ TestOnTimerContext(Map<String, Object> stateMap) {
+ super(stateMap);
+ }
+
+ @Override
+ public String currentTimer() {
+ if (currentInvocation.isTimerInvocation()) {
+ return currentInvocation.firingTimer.getName();
+ }
+ return null;
+ }
+ }
+
+ private static int[] getPartitionColumnIndices(TableArgumentInfo arg) {
+ if (arg.partitionColumnNames == null || arg.partitionColumnNames.length == 0) {
+ return new int[0];
+ }
+ List<String> fieldNames = getFieldNames(arg.dataType);
+ int[] indices = new int[arg.partitionColumnNames.length];
+ for (int i = 0; i < arg.partitionColumnNames.length; i++) {
+ String colName = arg.partitionColumnNames[i];
+ int index = fieldNames.indexOf(colName);
+ if (index < 0) {
+ throw new IllegalStateException(
+ "Partition column '"
+ + colName
+ + "' not found in table argument. "
+ + "Available fields: "
+ + fieldNames);
+ }
+ indices[i] = index;
+ }
+ return indices;
+ }
+
+ @Nullable
+ private Class<?> resolveRowtimeConversionClass(List<TableArgumentInfo> tableArguments) {
Review Comment:
thanks for checking!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39378][table] Add support for Context, timers and on_time to PTF test harness [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28326:
URL: https://github.com/apache/flink/pull/28326#discussion_r3371943350
##########
docs/content.zh/docs/dev/table/functions/ptfs.md:
##########
@@ -2465,6 +2465,115 @@ void testStateMutation() throws Exception {
{{< /tab >}}
{{< /tabs >}}
+#### Testing with Timers and Context
+
+The harness supports the `Context` parameter, timer registration via `TimeContext`, and `onTimer`
+callbacks. Use `.withOnTimeColumn()` to configure the event time column and `.setWatermark()` to
+advance watermarks and fire eligible timers.
+
+{{< tabs "timer-testing" >}}
+{{< tab "Java" >}}
+```java
+// A PTF that registers a named timer 5 seconds after each event, and emits when it fires.
+@DataTypeHint("ROW<message STRING>")
+public class TimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ String name = input.getFieldAs("name");
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timeout-" + name, timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered-" + name));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("timer-fired-" + ctx.currentTimer()));
+ }
+}
+
+@Test
+void testTimerRegistrationAndFiring() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(TimerPTF.class)
+ .withTableArgument("input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+
+ // Verify the timer was registered
+ assertThat(harness.getPendingTimers()).hasSize(1);
+ assertThat(harness.getPendingTimers().get(0).getName()).isEqualTo("timeout-Alice");
+
+ // Advance watermark past the timer's timestamp to fire it
+ harness.clearOutput();
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 7));
+
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "timer-fired-timeout-Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 6)));
+
+ assertThat(harness.getPendingTimers()).isEmpty();
+ assertThat(harness.getFiredTimers()).hasSize(1);
+ }
+}
+```
+{{< /tab >}}
+{{< /tabs >}}
+
+**Timers with State**: State persisted during `eval()` is accessible in `onTimer()`:
+
+{{< tabs "timer-state-testing" >}}
+{{< tab "Java" >}}
+```java
+@DataTypeHint("ROW<message STRING>")
+public class TimerWithStatePTF extends ProcessTableFunction<Row> {
+ public static class CounterState {
+ public long count = 0L;
+ }
+
+ public void eval(
+ Context ctx,
+ @StateHint CounterState state,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ state.count++;
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("check", timeCtx.time().plus(Duration.ofSeconds(5)));
+ }
+
+ public void onTimer(OnTimerContext ctx, @StateHint CounterState state) {
+ collect(Row.of("count=" + state.count));
+ }
+}
+
+@Test
+void testTimerWithState() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(TimerWithStatePTF.class)
+ .withTableArgument("input",
+ DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
Review Comment:
```suggestion
harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 2)));
```
show that two timers can be fired from one wm advancement?
##########
docs/content.zh/docs/dev/table/functions/ptfs.md:
##########
@@ -2465,6 +2465,115 @@ void testStateMutation() throws Exception {
{{< /tab >}}
{{< /tabs >}}
+#### Testing with Timers and Context
+
+The harness supports the `Context` parameter, timer registration via `TimeContext`, and `onTimer`
+callbacks. Use `.withOnTimeColumn()` to configure the event time column and `.setWatermark()` to
+advance watermarks and fire eligible timers.
+
+{{< tabs "timer-testing" >}}
+{{< tab "Java" >}}
+```java
+// A PTF that registers a named timer 5 seconds after each event, and emits when it fires.
+@DataTypeHint("ROW<message STRING>")
+public class TimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ String name = input.getFieldAs("name");
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timeout-" + name, timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered-" + name));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("timer-fired-" + ctx.currentTimer()));
+ }
+}
+
+@Test
+void testTimerRegistrationAndFiring() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(TimerPTF.class)
+ .withTableArgument("input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+
+ // Verify the timer was registered
+ assertThat(harness.getPendingTimers()).hasSize(1);
+ assertThat(harness.getPendingTimers().get(0).getName()).isEqualTo("timeout-Alice");
+
+ // Advance watermark past the timer's timestamp to fire it
+ harness.clearOutput();
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 7));
+
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "timer-fired-timeout-Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 6)));
+
+ assertThat(harness.getPendingTimers()).isEmpty();
+ assertThat(harness.getFiredTimers()).hasSize(1);
+ }
+}
+```
+{{< /tab >}}
+{{< /tabs >}}
+
+**Timers with State**: State persisted during `eval()` is accessible in `onTimer()`:
+
+{{< tabs "timer-state-testing" >}}
+{{< tab "Java" >}}
+```java
+@DataTypeHint("ROW<message STRING>")
+public class TimerWithStatePTF extends ProcessTableFunction<Row> {
+ public static class CounterState {
+ public long count = 0L;
+ }
+
+ public void eval(
+ Context ctx,
+ @StateHint CounterState state,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ state.count++;
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("check", timeCtx.time().plus(Duration.ofSeconds(5)));
+ }
+
+ public void onTimer(OnTimerContext ctx, @StateHint CounterState state) {
+ collect(Row.of("count=" + state.count));
+ }
+}
+
+@Test
+void testTimerWithState() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(TimerWithStatePTF.class)
+ .withTableArgument("input",
+ DataTypes.of("ROW<partition STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+ harness.processElement(Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+
+ harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 7));
+ assertThat(harness.getOutput())
+ .containsExactly(
+ Row.of("P1", "count=3", LocalDateTime.of(2025, 1, 1, 0, 0, 6)));
Review Comment:
```suggestion
harness.setWatermark(LocalDateTime.of(2025, 1, 1, 0, 0, 8));
assertThat(harness.getOutput())
.containsExactly(
Row.of("P1", "count=2", LocalDateTime.of(2025, 1, 1, 0, 0, 6)),
Row.of("P1", "count=3", LocalDateTime.of(2025, 1, 1, 0, 0, 7)),);
```
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarness.java:
##########
@@ -303,38 +332,419 @@ public void clearStateForKey(String stateName, Row partitionKey) {
stateManager.clearStateForKey(stateName, partitionKey);
}
+ // -------------------------------------------------------------------------
+ // Watermark & Timer API
+ // -------------------------------------------------------------------------
+
+ /**
+ * Sets the watermark for all tables to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermark(LocalDateTime watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /** Sets the watermark for all tables to the given {@link Instant} and fires eligible timers. */
+ public void setWatermark(Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(watermark.toEpochMilli());
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, LocalDateTime watermark)
+ throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link Instant} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, watermark.toEpochMilli());
+ }
+
+ /** Returns all timers (both pending and fired), sorted by timestamp then name. */
+ public List<Timer> getTimers() {
+ return Stream.concat(
+ timerManager.getPendingTimers().stream(),
+ timerManager.getFiredTimers().stream())
+ .sorted()
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all pending (not yet fired) timers, sorted by timestamp then name. */
+ public List<Timer> getPendingTimers() {
+ return timerManager.getPendingTimers();
+ }
+
+ /** Returns all pending timers with the given name. */
+ public List<Timer> getPendingTimers(String timerName) {
+ return timerManager.getPendingTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all timers that have fired, in the order they fired. */
+ public List<Timer> getFiredTimers() {
+ return timerManager.getFiredTimers();
+ }
+
+ /** Returns all fired timers with the given name. */
+ public List<Timer> getFiredTimers(String timerName) {
+ return timerManager.getFiredTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Clears the fired timer history. */
+ public void clearFiredTimers() {
+ timerManager.clearFiredTimers();
+ }
+
+ private void setWatermarkMillis(long millis) throws Exception {
+ checkState(isOpen, "Harness is not open");
+ for (TableArgumentInfo tableArg : ArgumentInfo.filterTableArguments(arguments)) {
+ timerManager.setTableWatermark(tableArg.name, millis);
+ }
+ timerManager.updateGlobalWatermarkAndFireTimers(this::fireTimer);
+ }
+
+ private void setWatermarkForTableMillis(String tableArgument, long millis) throws Exception {
+ checkState(isOpen, "Harness is not open");
+ checkNotNull(tableArgument, "tableArgument must not be null");
+ checkArgument(
+ argumentsByName.get(tableArgument) instanceof TableArgumentInfo,
+ "Unknown or non-table argument: %s",
+ tableArgument);
+ timerManager.setTableWatermark(tableArgument, millis);
+ timerManager.updateGlobalWatermarkAndFireTimers(this::fireTimer);
+ }
+
+ private void fireTimer(Timer timer) throws Exception {
+ if (onTimer == null) {
+ throw new IllegalStateException(
+ "Timer fired but no onTimer() method is defined in "
Review Comment:
nit
```suggestion
"Timer fired but no valid onTimer() method is defined in "
```
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/InvocationContext.java:
##########
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.types.Row;
+
+import javax.annotation.Nullable;
+
+/** Captures the per-invocation state for an eval() or onTimer() call in the test harness. */
Review Comment:
expand the comment a bit and add that `partitionKey` is always set and for `eval()` `row` and `tableArgumentName` are set and for `onTimer()` `firingTimer`?
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarness.java:
##########
@@ -303,38 +332,419 @@ public void clearStateForKey(String stateName, Row partitionKey) {
stateManager.clearStateForKey(stateName, partitionKey);
}
+ // -------------------------------------------------------------------------
+ // Watermark & Timer API
+ // -------------------------------------------------------------------------
+
+ /**
+ * Sets the watermark for all tables to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermark(LocalDateTime watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /** Sets the watermark for all tables to the given {@link Instant} and fires eligible timers. */
+ public void setWatermark(Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(watermark.toEpochMilli());
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, LocalDateTime watermark)
+ throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link Instant} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, watermark.toEpochMilli());
+ }
+
+ /** Returns all timers (both pending and fired), sorted by timestamp then name. */
+ public List<Timer> getTimers() {
+ return Stream.concat(
+ timerManager.getPendingTimers().stream(),
+ timerManager.getFiredTimers().stream())
+ .sorted()
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all pending (not yet fired) timers, sorted by timestamp then name. */
+ public List<Timer> getPendingTimers() {
+ return timerManager.getPendingTimers();
+ }
+
+ /** Returns all pending timers with the given name. */
+ public List<Timer> getPendingTimers(String timerName) {
+ return timerManager.getPendingTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all timers that have fired, in the order they fired. */
+ public List<Timer> getFiredTimers() {
+ return timerManager.getFiredTimers();
+ }
+
+ /** Returns all fired timers with the given name. */
+ public List<Timer> getFiredTimers(String timerName) {
+ return timerManager.getFiredTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Clears the fired timer history. */
+ public void clearFiredTimers() {
+ timerManager.clearFiredTimers();
+ }
+
+ private void setWatermarkMillis(long millis) throws Exception {
+ checkState(isOpen, "Harness is not open");
+ for (TableArgumentInfo tableArg : ArgumentInfo.filterTableArguments(arguments)) {
+ timerManager.setTableWatermark(tableArg.name, millis);
+ }
+ timerManager.updateGlobalWatermarkAndFireTimers(this::fireTimer);
+ }
+
+ private void setWatermarkForTableMillis(String tableArgument, long millis) throws Exception {
+ checkState(isOpen, "Harness is not open");
+ checkNotNull(tableArgument, "tableArgument must not be null");
+ checkArgument(
+ argumentsByName.get(tableArgument) instanceof TableArgumentInfo,
+ "Unknown or non-table argument: %s",
+ tableArgument);
+ timerManager.setTableWatermark(tableArgument, millis);
+ timerManager.updateGlobalWatermarkAndFireTimers(this::fireTimer);
+ }
+
+ private void fireTimer(Timer timer) throws Exception {
+ if (onTimer == null) {
+ throw new IllegalStateException(
+ "Timer fired but no onTimer() method is defined in "
+ + function.getClass().getSimpleName());
+ }
+
+ currentInvocation = InvocationContext.forTimer(timer);
+
+ try {
+ Map<String, Object> stateMap = stateManager.loadStateForKey(timer.partitionKey);
+
+ List<StateArgumentInfo> stateArgs = ArgumentInfo.filterStateArguments(arguments);
+ Object[] methodArgs = new Object[stateArgs.size()];
+ for (int i = 0; i < stateArgs.size(); i++) {
+ methodArgs[i] = stateMap.get(stateArgs.get(i).name);
+ }
+
+ onTimer.invoke(function, new TestOnTimerContext(stateMap), methodArgs);
+ stateManager.updateStateForKey(timer.partitionKey, stateMap);
+ } catch (InvocationTargetException e) {
+ Throwable cause = e.getCause();
+ if (cause instanceof Exception) {
+ throw (Exception) cause;
+ }
+ throw new RuntimeException("onTimer() invocation failed", e);
+ } finally {
+ currentInvocation = null;
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Context implementations
+ // -------------------------------------------------------------------------
+
+ private class TestContext implements ProcessTableFunction.Context {
+ final Map<String, Object> stateMap;
+
+ TestContext(Map<String, Object> stateMap) {
+ this.stateMap = stateMap;
+ }
+
+ @Override
+ public <TimeType> ProcessTableFunction.TimeContext<TimeType> timeContext(
+ Class<TimeType> conversionClass) {
+ return new TestTimeContext<>(conversionClass);
+ }
+
+ @Override
+ public TableSemantics tableSemanticsFor(String argName) {
+ ArgumentInfo argInfo = argumentsByName.get(argName);
+ if (argInfo == null) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Argument '%s' not found. Available arguments: %s",
+ argName, argumentsByName.keySet()));
+ }
+ if (!(argInfo instanceof TableArgumentInfo)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Argument '%s' is not a table argument (type: %s)",
+ argName, argInfo.getClass().getSimpleName()));
+ }
+ TableArgumentInfo tableArg = (TableArgumentInfo) argInfo;
+ int[] partitionIndices = getPartitionColumnIndices(tableArg);
+ int timeColumn =
Review Comment:
nit
```suggestion
int timeColumnIndex =
```
##########
docs/content.zh/docs/dev/table/functions/ptfs.md:
##########
@@ -2465,6 +2465,115 @@ void testStateMutation() throws Exception {
{{< /tab >}}
{{< /tabs >}}
+#### Testing with Timers and Context
+
+The harness supports the `Context` parameter, timer registration via `TimeContext`, and `onTimer`
+callbacks. Use `.withOnTimeColumn()` to configure the event time column and `.setWatermark()` to
+advance watermarks and fire eligible timers.
+
+{{< tabs "timer-testing" >}}
+{{< tab "Java" >}}
+```java
+// A PTF that registers a named timer 5 seconds after each event, and emits when it fires.
+@DataTypeHint("ROW<message STRING>")
+public class TimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ String name = input.getFieldAs("name");
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timeout-" + name, timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered-" + name));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("timer-fired-" + ctx.currentTimer()));
+ }
+}
+
+@Test
+void testTimerRegistrationAndFiring() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(TimerPTF.class)
+ .withTableArgument("input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+
+ // Verify the timer was registered
+ assertThat(harness.getPendingTimers()).hasSize(1);
+ assertThat(harness.getPendingTimers().get(0).getName()).isEqualTo("timeout-Alice");
Review Comment:
is it possible to assert the timestamp of a timer as well?
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarness.java:
##########
@@ -303,38 +332,419 @@ public void clearStateForKey(String stateName, Row partitionKey) {
stateManager.clearStateForKey(stateName, partitionKey);
}
+ // -------------------------------------------------------------------------
+ // Watermark & Timer API
+ // -------------------------------------------------------------------------
+
+ /**
+ * Sets the watermark for all tables to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermark(LocalDateTime watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /** Sets the watermark for all tables to the given {@link Instant} and fires eligible timers. */
+ public void setWatermark(Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(watermark.toEpochMilli());
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, LocalDateTime watermark)
+ throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link Instant} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, watermark.toEpochMilli());
+ }
+
+ /** Returns all timers (both pending and fired), sorted by timestamp then name. */
+ public List<Timer> getTimers() {
+ return Stream.concat(
+ timerManager.getPendingTimers().stream(),
+ timerManager.getFiredTimers().stream())
+ .sorted()
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all pending (not yet fired) timers, sorted by timestamp then name. */
+ public List<Timer> getPendingTimers() {
+ return timerManager.getPendingTimers();
+ }
+
+ /** Returns all pending timers with the given name. */
+ public List<Timer> getPendingTimers(String timerName) {
+ return timerManager.getPendingTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all timers that have fired, in the order they fired. */
+ public List<Timer> getFiredTimers() {
+ return timerManager.getFiredTimers();
+ }
+
+ /** Returns all fired timers with the given name. */
+ public List<Timer> getFiredTimers(String timerName) {
+ return timerManager.getFiredTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Clears the fired timer history. */
+ public void clearFiredTimers() {
+ timerManager.clearFiredTimers();
+ }
+
+ private void setWatermarkMillis(long millis) throws Exception {
+ checkState(isOpen, "Harness is not open");
+ for (TableArgumentInfo tableArg : ArgumentInfo.filterTableArguments(arguments)) {
+ timerManager.setTableWatermark(tableArg.name, millis);
+ }
+ timerManager.updateGlobalWatermarkAndFireTimers(this::fireTimer);
+ }
+
+ private void setWatermarkForTableMillis(String tableArgument, long millis) throws Exception {
+ checkState(isOpen, "Harness is not open");
+ checkNotNull(tableArgument, "tableArgument must not be null");
+ checkArgument(
+ argumentsByName.get(tableArgument) instanceof TableArgumentInfo,
+ "Unknown or non-table argument: %s",
+ tableArgument);
+ timerManager.setTableWatermark(tableArgument, millis);
+ timerManager.updateGlobalWatermarkAndFireTimers(this::fireTimer);
+ }
+
+ private void fireTimer(Timer timer) throws Exception {
+ if (onTimer == null) {
+ throw new IllegalStateException(
+ "Timer fired but no onTimer() method is defined in "
+ + function.getClass().getSimpleName());
+ }
+
+ currentInvocation = InvocationContext.forTimer(timer);
+
+ try {
+ Map<String, Object> stateMap = stateManager.loadStateForKey(timer.partitionKey);
+
+ List<StateArgumentInfo> stateArgs = ArgumentInfo.filterStateArguments(arguments);
+ Object[] methodArgs = new Object[stateArgs.size()];
+ for (int i = 0; i < stateArgs.size(); i++) {
+ methodArgs[i] = stateMap.get(stateArgs.get(i).name);
+ }
+
+ onTimer.invoke(function, new TestOnTimerContext(stateMap), methodArgs);
+ stateManager.updateStateForKey(timer.partitionKey, stateMap);
+ } catch (InvocationTargetException e) {
+ Throwable cause = e.getCause();
+ if (cause instanceof Exception) {
+ throw (Exception) cause;
+ }
+ throw new RuntimeException("onTimer() invocation failed", e);
+ } finally {
+ currentInvocation = null;
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Context implementations
+ // -------------------------------------------------------------------------
+
+ private class TestContext implements ProcessTableFunction.Context {
+ final Map<String, Object> stateMap;
+
+ TestContext(Map<String, Object> stateMap) {
+ this.stateMap = stateMap;
+ }
+
+ @Override
+ public <TimeType> ProcessTableFunction.TimeContext<TimeType> timeContext(
+ Class<TimeType> conversionClass) {
+ return new TestTimeContext<>(conversionClass);
+ }
+
+ @Override
+ public TableSemantics tableSemanticsFor(String argName) {
+ ArgumentInfo argInfo = argumentsByName.get(argName);
+ if (argInfo == null) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Argument '%s' not found. Available arguments: %s",
+ argName, argumentsByName.keySet()));
Review Comment:
Filter arguments already down to table args?
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarness.java:
##########
@@ -655,6 +1083,56 @@ public Builder<OUT> withPartitionBy(String argumentName, String... columnNames)
return this;
}
+ // ---------------------------------------------------------------------
+ // Timer & Watermark Configuration
+ // ---------------------------------------------------------------------
+
+ /**
+ * Configures the on-time column name for the function.
+ *
+ * @param columnName The column that carries event time
+ */
+ public Builder<OUT> withOnTimeColumn(String columnName) {
+ checkNotNull(columnName, "columnName must not be null");
+ this.onTimeColumnName = columnName;
+ return this;
+ }
+
+ // ---------------------------------------------------------------------
+ // Watermark
+ // ---------------------------------------------------------------------
+
+ /** Sets the initial watermark for all table arguments. */
+ public Builder<OUT> withInitialWatermark(LocalDateTime watermark) {
+ checkNotNull(watermark, "watermark must not be null");
+ this.initialWatermarkForAll = DateTimeUtils.toTimestampMillis(watermark);
+ return this;
+ }
+
+ /** Sets the initial watermark for all table arguments. */
+ public Builder<OUT> withInitialWatermark(Instant watermark) {
+ checkNotNull(watermark, "watermark must not be null");
+ this.initialWatermarkForAll = watermark.toEpochMilli();
+ return this;
+ }
+
+ /** Sets the initial watermark for a specific table argument. */
+ public Builder<OUT> withInitialWatermarkForTable(
+ String tableArgument, LocalDateTime watermark) {
+ checkNotNull(tableArgument, "tableArgument must not be null");
+ checkNotNull(watermark, "watermark must not be null");
+ initialWatermarks.put(tableArgument, DateTimeUtils.toTimestampMillis(watermark));
+ return this;
+ }
+
+ /** Sets the initial watermark for a specific table argument. */
+ public Builder<OUT> withInitialWatermarkForTable(String tableArgument, Instant watermark) {
+ checkNotNull(tableArgument, "tableArgument must not be null");
+ checkNotNull(watermark, "watermark must not be null");
+ initialWatermarks.put(tableArgument, watermark.toEpochMilli());
+ return this;
+ }
Review Comment:
see above
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ResolvedMethod.java:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import javax.annotation.Nullable;
+
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+
+/**
+ * A resolved PTF method (eval or onTimer) paired with whether its first parameter accepts a
+ * Context.
+ */
+class ResolvedMethod {
+ final Method method;
+ final boolean takesContext;
+
+ static ResolvedMethod of(Method method, Class<?> contextClass) {
Review Comment:
```suggestion
static ResolvedMethod<T> of(Method method, Class<T> contextClass) {
```
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarness.java:
##########
@@ -823,49 +1351,78 @@ private Method findEvalMethod() throws NoSuchMethodException {
"Multiple eval() methods found in "
+ functionClass.getSimpleName()
+ ". ProcessTableFunction must have exactly one eval() method.");
- } else {
- return evalMethod;
}
- }
- /**
- * Validates that the eval() method doesn't use unsupported features. Temporary, until
- * context is supported.
- */
- private void validateEvalMethodSupported(Method evalMethod, List<ArgumentInfo> arguments) {
- Parameter[] parameters = evalMethod.getParameters();
+ return evalMethod;
+ }
- for (int i = 0; i < parameters.length; i++) {
- Parameter param = parameters[i];
- Class<?> paramType = param.getType();
+ @Nullable
+ private static Method findOnTimerMethod(
+ Class<?> functionClass, List<ArgumentInfo> arguments) {
+ List<Method> candidates = ExtractionUtils.collectMethods(functionClass, "onTimer");
+ if (candidates.isEmpty()) {
Review Comment:
shouldn't we also fail if we have more than one candidate?
How would the framework decide which `onTimer` method to call?
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/TestHarnessTimerManager.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.function.ThrowingConsumer;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Timer and watermark manager for {@link ProcessTableFunctionTestHarness}.
+ *
+ * <p>Handles timer registration, clearing, watermark tracking, and determining which timers are
+ * eligible to fire.
+ */
+@Internal
+class TestHarnessTimerManager {
+
+ private final Map<Row, Set<Timer>> pendingTimersByPartition = new HashMap<>();
+ private final List<Timer> firedTimers = new ArrayList<>();
+ private final Map<String, Long> watermarkByTable = new HashMap<>();
+ @Nullable private Long globalWatermark;
+
+ TestHarnessTimerManager() {}
+
+ // -------------------------------------------------------------------------
+ // Watermark
+ // -------------------------------------------------------------------------
+
+ void setTableWatermark(String tableName, long absoluteMillis) {
+ Long current = watermarkByTable.get(tableName);
+ if (current != null && absoluteMillis < current) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Cannot move watermark backward for table '%s': current=%d, new=%d",
+ tableName, current, absoluteMillis));
+ }
+ watermarkByTable.put(tableName, absoluteMillis);
+ }
+
+ void updateGlobalWatermarkAndFireTimers(ThrowingConsumer<Timer, Exception> firer)
+ throws Exception {
+ if (watermarkByTable.isEmpty()) {
+ return;
+ }
+ long newGlobalWatermark = Collections.min(watermarkByTable.values());
+ if (globalWatermark != null && newGlobalWatermark < globalWatermark) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Cannot move global watermark backward: current=%d, new=%d",
+ globalWatermark, newGlobalWatermark));
+ }
+ globalWatermark = newGlobalWatermark;
+ fireEligibleTimers(newGlobalWatermark, firer);
+ }
+
+ @Nullable
+ Long getGlobalWatermark() {
+ return globalWatermark;
+ }
+
+ @Nullable
+ Long getWatermarkForTable(String tableName) {
+ return watermarkByTable.get(tableName);
+ }
+
+ // -------------------------------------------------------------------------
+ // Timer Registration
+ // -------------------------------------------------------------------------
+
+ void register(Row partitionKey, long timestampMillis, @Nullable String name) {
+ Set<Timer> timerSet =
+ pendingTimersByPartition.computeIfAbsent(partitionKey, k -> new HashSet<>());
+
+ if (name != null) {
+ timerSet.removeIf(t -> name.equals(t.name));
+ }
Review Comment:
I think we should also remove unnamed timers.
There can only be one timer per timestamp and name (also if name = `null`).
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarness.java:
##########
@@ -711,25 +1197,67 @@ public ProcessTableFunctionTestHarness<OUT> build() throws Exception {
// Extract table arguments for output type derivation
// SystemTypeInference needs table semantics for pass-through column deduplication
- List<TableArgumentInfo> tableArgs = ArgumentInfo.filterTableArguments(arguments);
+ List<TableArgumentInfo> tableArgInfos = ArgumentInfo.filterTableArguments(arguments);
// Derive output schema using SystemTypeInference
DataType derivedOutputType =
deriveOutputTypeFromSystemInference(
- function, dataTypeFactory, systemTypeInference, tableArgs);
+ function,
+ dataTypeFactory,
+ systemTypeInference,
+ arguments,
+ tableArgInfos);
// Create output converter for PTF emissions
DataStructureConverter<Object, Object> harnessOutputConverter =
createPTFOutputConverter(derivedOutputType);
+ // Validate onTimeColumn configuration
+ if (onTimeColumnName != null) {
+ boolean foundInAnyTable =
+ tableArgInfos.stream()
+ .anyMatch(
+ t -> getFieldNames(t.dataType).contains(onTimeColumnName));
Review Comment:
Do we also need to check if the tables with `on_time` column have set semantics (and partitioning)?
##########
docs/content.zh/docs/dev/table/functions/ptfs.md:
##########
@@ -2465,6 +2465,115 @@ void testStateMutation() throws Exception {
{{< /tab >}}
{{< /tabs >}}
+#### Testing with Timers and Context
+
+The harness supports the `Context` parameter, timer registration via `TimeContext`, and `onTimer`
+callbacks. Use `.withOnTimeColumn()` to configure the event time column and `.setWatermark()` to
+advance watermarks and fire eligible timers.
+
+{{< tabs "timer-testing" >}}
+{{< tab "Java" >}}
+```java
+// A PTF that registers a named timer 5 seconds after each event, and emits when it fires.
+@DataTypeHint("ROW<message STRING>")
+public class TimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ String name = input.getFieldAs("name");
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timeout-" + name, timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered-" + name));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("timer-fired-" + ctx.currentTimer()));
+ }
+}
+
+@Test
+void testTimerRegistrationAndFiring() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(TimerPTF.class)
+ .withTableArgument("input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+
+ // Verify the timer was registered
+ assertThat(harness.getPendingTimers()).hasSize(1);
+ assertThat(harness.getPendingTimers().get(0).getName()).isEqualTo("timeout-Alice");
+
+ // Advance watermark past the timer's timestamp to fire it
+ harness.clearOutput();
Review Comment:
the output should still be empty, no?
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarness.java:
##########
@@ -303,38 +332,419 @@ public void clearStateForKey(String stateName, Row partitionKey) {
stateManager.clearStateForKey(stateName, partitionKey);
}
+ // -------------------------------------------------------------------------
+ // Watermark & Timer API
+ // -------------------------------------------------------------------------
+
+ /**
+ * Sets the watermark for all tables to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermark(LocalDateTime watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /** Sets the watermark for all tables to the given {@link Instant} and fires eligible timers. */
+ public void setWatermark(Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(watermark.toEpochMilli());
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link LocalDateTime} and fires eligible
+ * timers.
Review Comment:
nit:
I would rephrase the `fires eligible timers` part a bit because this only happens the WMs on the other tables have been advanced far enough as well.
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarness.java:
##########
@@ -303,38 +332,419 @@ public void clearStateForKey(String stateName, Row partitionKey) {
stateManager.clearStateForKey(stateName, partitionKey);
}
+ // -------------------------------------------------------------------------
+ // Watermark & Timer API
+ // -------------------------------------------------------------------------
+
+ /**
+ * Sets the watermark for all tables to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermark(LocalDateTime watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /** Sets the watermark for all tables to the given {@link Instant} and fires eligible timers. */
+ public void setWatermark(Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(watermark.toEpochMilli());
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, LocalDateTime watermark)
+ throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link Instant} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, watermark.toEpochMilli());
+ }
+
+ /** Returns all timers (both pending and fired), sorted by timestamp then name. */
+ public List<Timer> getTimers() {
+ return Stream.concat(
+ timerManager.getPendingTimers().stream(),
+ timerManager.getFiredTimers().stream())
+ .sorted()
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all pending (not yet fired) timers, sorted by timestamp then name. */
+ public List<Timer> getPendingTimers() {
+ return timerManager.getPendingTimers();
+ }
+
+ /** Returns all pending timers with the given name. */
+ public List<Timer> getPendingTimers(String timerName) {
+ return timerManager.getPendingTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all timers that have fired, in the order they fired. */
+ public List<Timer> getFiredTimers() {
+ return timerManager.getFiredTimers();
+ }
+
+ /** Returns all fired timers with the given name. */
+ public List<Timer> getFiredTimers(String timerName) {
Review Comment:
also add a method to get fired timers by partition key?
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarness.java:
##########
@@ -303,38 +332,419 @@ public void clearStateForKey(String stateName, Row partitionKey) {
stateManager.clearStateForKey(stateName, partitionKey);
}
+ // -------------------------------------------------------------------------
+ // Watermark & Timer API
+ // -------------------------------------------------------------------------
+
+ /**
+ * Sets the watermark for all tables to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermark(LocalDateTime watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /** Sets the watermark for all tables to the given {@link Instant} and fires eligible timers. */
+ public void setWatermark(Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(watermark.toEpochMilli());
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, LocalDateTime watermark)
+ throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link Instant} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, watermark.toEpochMilli());
+ }
+
+ /** Returns all timers (both pending and fired), sorted by timestamp then name. */
+ public List<Timer> getTimers() {
+ return Stream.concat(
+ timerManager.getPendingTimers().stream(),
+ timerManager.getFiredTimers().stream())
+ .sorted()
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all pending (not yet fired) timers, sorted by timestamp then name. */
+ public List<Timer> getPendingTimers() {
+ return timerManager.getPendingTimers();
+ }
+
+ /** Returns all pending timers with the given name. */
+ public List<Timer> getPendingTimers(String timerName) {
Review Comment:
Maybe also add a `getPendingTimers(Row partitionKey)` method?
##########
docs/content.zh/docs/dev/table/functions/ptfs.md:
##########
@@ -2582,8 +2691,5 @@ void testPOJO() throws Exception {
### PTF Features Unsupported by the TestHarness
-- `Context` parameter
-- Timers (`onTimer`)
-- `on_time` / `rowtime`
Review Comment:
🙌
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarness.java:
##########
@@ -303,38 +332,419 @@ public void clearStateForKey(String stateName, Row partitionKey) {
stateManager.clearStateForKey(stateName, partitionKey);
}
+ // -------------------------------------------------------------------------
+ // Watermark & Timer API
+ // -------------------------------------------------------------------------
+
+ /**
+ * Sets the watermark for all tables to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermark(LocalDateTime watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /** Sets the watermark for all tables to the given {@link Instant} and fires eligible timers. */
+ public void setWatermark(Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(watermark.toEpochMilli());
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, LocalDateTime watermark)
+ throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link Instant} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, watermark.toEpochMilli());
+ }
+
+ /** Returns all timers (both pending and fired), sorted by timestamp then name. */
+ public List<Timer> getTimers() {
Review Comment:
nit:
```suggestion
public List<Timer> getAllTimers() {
```
I think, returning fired timers might be unexpected. So using "all" might highlight this.
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarness.java:
##########
@@ -303,38 +332,419 @@ public void clearStateForKey(String stateName, Row partitionKey) {
stateManager.clearStateForKey(stateName, partitionKey);
}
+ // -------------------------------------------------------------------------
+ // Watermark & Timer API
+ // -------------------------------------------------------------------------
+
+ /**
+ * Sets the watermark for all tables to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermark(LocalDateTime watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /** Sets the watermark for all tables to the given {@link Instant} and fires eligible timers. */
+ public void setWatermark(Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(watermark.toEpochMilli());
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, LocalDateTime watermark)
+ throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link Instant} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, watermark.toEpochMilli());
+ }
+
+ /** Returns all timers (both pending and fired), sorted by timestamp then name. */
+ public List<Timer> getTimers() {
+ return Stream.concat(
+ timerManager.getPendingTimers().stream(),
+ timerManager.getFiredTimers().stream())
+ .sorted()
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all pending (not yet fired) timers, sorted by timestamp then name. */
+ public List<Timer> getPendingTimers() {
+ return timerManager.getPendingTimers();
+ }
+
+ /** Returns all pending timers with the given name. */
+ public List<Timer> getPendingTimers(String timerName) {
+ return timerManager.getPendingTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all timers that have fired, in the order they fired. */
+ public List<Timer> getFiredTimers() {
+ return timerManager.getFiredTimers();
+ }
+
+ /** Returns all fired timers with the given name. */
+ public List<Timer> getFiredTimers(String timerName) {
+ return timerManager.getFiredTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Clears the fired timer history. */
+ public void clearFiredTimers() {
+ timerManager.clearFiredTimers();
+ }
+
+ private void setWatermarkMillis(long millis) throws Exception {
+ checkState(isOpen, "Harness is not open");
+ for (TableArgumentInfo tableArg : ArgumentInfo.filterTableArguments(arguments)) {
+ timerManager.setTableWatermark(tableArg.name, millis);
+ }
+ timerManager.updateGlobalWatermarkAndFireTimers(this::fireTimer);
+ }
+
+ private void setWatermarkForTableMillis(String tableArgument, long millis) throws Exception {
+ checkState(isOpen, "Harness is not open");
+ checkNotNull(tableArgument, "tableArgument must not be null");
+ checkArgument(
+ argumentsByName.get(tableArgument) instanceof TableArgumentInfo,
+ "Unknown or non-table argument: %s",
+ tableArgument);
+ timerManager.setTableWatermark(tableArgument, millis);
+ timerManager.updateGlobalWatermarkAndFireTimers(this::fireTimer);
+ }
+
+ private void fireTimer(Timer timer) throws Exception {
+ if (onTimer == null) {
+ throw new IllegalStateException(
+ "Timer fired but no onTimer() method is defined in "
+ + function.getClass().getSimpleName());
+ }
+
+ currentInvocation = InvocationContext.forTimer(timer);
+
+ try {
+ Map<String, Object> stateMap = stateManager.loadStateForKey(timer.partitionKey);
+
+ List<StateArgumentInfo> stateArgs = ArgumentInfo.filterStateArguments(arguments);
+ Object[] methodArgs = new Object[stateArgs.size()];
+ for (int i = 0; i < stateArgs.size(); i++) {
+ methodArgs[i] = stateMap.get(stateArgs.get(i).name);
+ }
+
+ onTimer.invoke(function, new TestOnTimerContext(stateMap), methodArgs);
+ stateManager.updateStateForKey(timer.partitionKey, stateMap);
Review Comment:
move out of `try` block as well?
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarness.java:
##########
@@ -303,38 +332,419 @@ public void clearStateForKey(String stateName, Row partitionKey) {
stateManager.clearStateForKey(stateName, partitionKey);
}
+ // -------------------------------------------------------------------------
+ // Watermark & Timer API
+ // -------------------------------------------------------------------------
+
+ /**
+ * Sets the watermark for all tables to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermark(LocalDateTime watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /** Sets the watermark for all tables to the given {@link Instant} and fires eligible timers. */
+ public void setWatermark(Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(watermark.toEpochMilli());
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, LocalDateTime watermark)
+ throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link Instant} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, watermark.toEpochMilli());
+ }
+
+ /** Returns all timers (both pending and fired), sorted by timestamp then name. */
+ public List<Timer> getTimers() {
+ return Stream.concat(
+ timerManager.getPendingTimers().stream(),
+ timerManager.getFiredTimers().stream())
+ .sorted()
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all pending (not yet fired) timers, sorted by timestamp then name. */
+ public List<Timer> getPendingTimers() {
+ return timerManager.getPendingTimers();
+ }
+
+ /** Returns all pending timers with the given name. */
+ public List<Timer> getPendingTimers(String timerName) {
+ return timerManager.getPendingTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all timers that have fired, in the order they fired. */
+ public List<Timer> getFiredTimers() {
+ return timerManager.getFiredTimers();
+ }
+
+ /** Returns all fired timers with the given name. */
+ public List<Timer> getFiredTimers(String timerName) {
+ return timerManager.getFiredTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Clears the fired timer history. */
+ public void clearFiredTimers() {
+ timerManager.clearFiredTimers();
+ }
+
+ private void setWatermarkMillis(long millis) throws Exception {
+ checkState(isOpen, "Harness is not open");
+ for (TableArgumentInfo tableArg : ArgumentInfo.filterTableArguments(arguments)) {
+ timerManager.setTableWatermark(tableArg.name, millis);
+ }
+ timerManager.updateGlobalWatermarkAndFireTimers(this::fireTimer);
+ }
+
+ private void setWatermarkForTableMillis(String tableArgument, long millis) throws Exception {
+ checkState(isOpen, "Harness is not open");
+ checkNotNull(tableArgument, "tableArgument must not be null");
+ checkArgument(
+ argumentsByName.get(tableArgument) instanceof TableArgumentInfo,
+ "Unknown or non-table argument: %s",
+ tableArgument);
+ timerManager.setTableWatermark(tableArgument, millis);
+ timerManager.updateGlobalWatermarkAndFireTimers(this::fireTimer);
+ }
+
+ private void fireTimer(Timer timer) throws Exception {
+ if (onTimer == null) {
+ throw new IllegalStateException(
+ "Timer fired but no onTimer() method is defined in "
+ + function.getClass().getSimpleName());
+ }
+
+ currentInvocation = InvocationContext.forTimer(timer);
+
+ try {
+ Map<String, Object> stateMap = stateManager.loadStateForKey(timer.partitionKey);
+
+ List<StateArgumentInfo> stateArgs = ArgumentInfo.filterStateArguments(arguments);
+ Object[] methodArgs = new Object[stateArgs.size()];
+ for (int i = 0; i < stateArgs.size(); i++) {
+ methodArgs[i] = stateMap.get(stateArgs.get(i).name);
+ }
+
+ onTimer.invoke(function, new TestOnTimerContext(stateMap), methodArgs);
+ stateManager.updateStateForKey(timer.partitionKey, stateMap);
+ } catch (InvocationTargetException e) {
+ Throwable cause = e.getCause();
+ if (cause instanceof Exception) {
+ throw (Exception) cause;
+ }
+ throw new RuntimeException("onTimer() invocation failed", e);
+ } finally {
+ currentInvocation = null;
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Context implementations
+ // -------------------------------------------------------------------------
+
+ private class TestContext implements ProcessTableFunction.Context {
+ final Map<String, Object> stateMap;
+
+ TestContext(Map<String, Object> stateMap) {
+ this.stateMap = stateMap;
+ }
+
+ @Override
+ public <TimeType> ProcessTableFunction.TimeContext<TimeType> timeContext(
+ Class<TimeType> conversionClass) {
+ return new TestTimeContext<>(conversionClass);
+ }
+
+ @Override
+ public TableSemantics tableSemanticsFor(String argName) {
+ ArgumentInfo argInfo = argumentsByName.get(argName);
+ if (argInfo == null) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Argument '%s' not found. Available arguments: %s",
+ argName, argumentsByName.keySet()));
+ }
+ if (!(argInfo instanceof TableArgumentInfo)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Argument '%s' is not a table argument (type: %s)",
+ argName, argInfo.getClass().getSimpleName()));
+ }
+ TableArgumentInfo tableArg = (TableArgumentInfo) argInfo;
+ int[] partitionIndices = getPartitionColumnIndices(tableArg);
+ int timeColumn =
+ onTimeColumnName != null
+ ? getFieldNames(tableArg.dataType).indexOf(onTimeColumnName)
+ : -1;
+ return new TestHarnessTableSemantics(tableArg.dataType, partitionIndices, timeColumn);
+ }
+
+ @Override
+ public void clearState(String stateName) {
+ stateMap.remove(stateName);
+ }
+
+ @Override
+ public void clearAllState() {
+ stateMap.clear();
Review Comment:
Should be scoped to the partitionKey (like `clearAllTimers()`)?
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarness.java:
##########
@@ -303,38 +332,419 @@ public void clearStateForKey(String stateName, Row partitionKey) {
stateManager.clearStateForKey(stateName, partitionKey);
}
+ // -------------------------------------------------------------------------
+ // Watermark & Timer API
+ // -------------------------------------------------------------------------
+
+ /**
+ * Sets the watermark for all tables to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermark(LocalDateTime watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /** Sets the watermark for all tables to the given {@link Instant} and fires eligible timers. */
+ public void setWatermark(Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(watermark.toEpochMilli());
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, LocalDateTime watermark)
+ throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link Instant} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, watermark.toEpochMilli());
+ }
+
+ /** Returns all timers (both pending and fired), sorted by timestamp then name. */
+ public List<Timer> getTimers() {
+ return Stream.concat(
+ timerManager.getPendingTimers().stream(),
+ timerManager.getFiredTimers().stream())
+ .sorted()
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all pending (not yet fired) timers, sorted by timestamp then name. */
+ public List<Timer> getPendingTimers() {
+ return timerManager.getPendingTimers();
+ }
+
+ /** Returns all pending timers with the given name. */
+ public List<Timer> getPendingTimers(String timerName) {
+ return timerManager.getPendingTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all timers that have fired, in the order they fired. */
+ public List<Timer> getFiredTimers() {
+ return timerManager.getFiredTimers();
+ }
+
+ /** Returns all fired timers with the given name. */
+ public List<Timer> getFiredTimers(String timerName) {
+ return timerManager.getFiredTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Clears the fired timer history. */
+ public void clearFiredTimers() {
+ timerManager.clearFiredTimers();
+ }
+
+ private void setWatermarkMillis(long millis) throws Exception {
+ checkState(isOpen, "Harness is not open");
+ for (TableArgumentInfo tableArg : ArgumentInfo.filterTableArguments(arguments)) {
+ timerManager.setTableWatermark(tableArg.name, millis);
+ }
+ timerManager.updateGlobalWatermarkAndFireTimers(this::fireTimer);
+ }
+
+ private void setWatermarkForTableMillis(String tableArgument, long millis) throws Exception {
+ checkState(isOpen, "Harness is not open");
+ checkNotNull(tableArgument, "tableArgument must not be null");
+ checkArgument(
+ argumentsByName.get(tableArgument) instanceof TableArgumentInfo,
+ "Unknown or non-table argument: %s",
+ tableArgument);
+ timerManager.setTableWatermark(tableArgument, millis);
+ timerManager.updateGlobalWatermarkAndFireTimers(this::fireTimer);
+ }
+
+ private void fireTimer(Timer timer) throws Exception {
+ if (onTimer == null) {
+ throw new IllegalStateException(
+ "Timer fired but no onTimer() method is defined in "
+ + function.getClass().getSimpleName());
+ }
+
+ currentInvocation = InvocationContext.forTimer(timer);
+
+ try {
+ Map<String, Object> stateMap = stateManager.loadStateForKey(timer.partitionKey);
+
+ List<StateArgumentInfo> stateArgs = ArgumentInfo.filterStateArguments(arguments);
+ Object[] methodArgs = new Object[stateArgs.size()];
+ for (int i = 0; i < stateArgs.size(); i++) {
+ methodArgs[i] = stateMap.get(stateArgs.get(i).name);
+ }
+
+ onTimer.invoke(function, new TestOnTimerContext(stateMap), methodArgs);
+ stateManager.updateStateForKey(timer.partitionKey, stateMap);
+ } catch (InvocationTargetException e) {
+ Throwable cause = e.getCause();
+ if (cause instanceof Exception) {
+ throw (Exception) cause;
+ }
+ throw new RuntimeException("onTimer() invocation failed", e);
+ } finally {
+ currentInvocation = null;
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Context implementations
+ // -------------------------------------------------------------------------
+
+ private class TestContext implements ProcessTableFunction.Context {
+ final Map<String, Object> stateMap;
+
+ TestContext(Map<String, Object> stateMap) {
+ this.stateMap = stateMap;
+ }
+
+ @Override
+ public <TimeType> ProcessTableFunction.TimeContext<TimeType> timeContext(
+ Class<TimeType> conversionClass) {
+ return new TestTimeContext<>(conversionClass);
+ }
+
+ @Override
+ public TableSemantics tableSemanticsFor(String argName) {
+ ArgumentInfo argInfo = argumentsByName.get(argName);
+ if (argInfo == null) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Argument '%s' not found. Available arguments: %s",
+ argName, argumentsByName.keySet()));
+ }
+ if (!(argInfo instanceof TableArgumentInfo)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Argument '%s' is not a table argument (type: %s)",
+ argName, argInfo.getClass().getSimpleName()));
+ }
+ TableArgumentInfo tableArg = (TableArgumentInfo) argInfo;
+ int[] partitionIndices = getPartitionColumnIndices(tableArg);
+ int timeColumn =
+ onTimeColumnName != null
+ ? getFieldNames(tableArg.dataType).indexOf(onTimeColumnName)
+ : -1;
+ return new TestHarnessTableSemantics(tableArg.dataType, partitionIndices, timeColumn);
+ }
+
+ @Override
+ public void clearState(String stateName) {
+ stateMap.remove(stateName);
Review Comment:
I think this should be scoped to the partitionKey (like `clearAllTimers()`)
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarness.java:
##########
@@ -303,38 +332,419 @@ public void clearStateForKey(String stateName, Row partitionKey) {
stateManager.clearStateForKey(stateName, partitionKey);
}
+ // -------------------------------------------------------------------------
+ // Watermark & Timer API
+ // -------------------------------------------------------------------------
+
+ /**
+ * Sets the watermark for all tables to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermark(LocalDateTime watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /** Sets the watermark for all tables to the given {@link Instant} and fires eligible timers. */
+ public void setWatermark(Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(watermark.toEpochMilli());
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, LocalDateTime watermark)
+ throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link Instant} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, watermark.toEpochMilli());
+ }
+
+ /** Returns all timers (both pending and fired), sorted by timestamp then name. */
+ public List<Timer> getTimers() {
+ return Stream.concat(
+ timerManager.getPendingTimers().stream(),
+ timerManager.getFiredTimers().stream())
+ .sorted()
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all pending (not yet fired) timers, sorted by timestamp then name. */
+ public List<Timer> getPendingTimers() {
+ return timerManager.getPendingTimers();
+ }
+
+ /** Returns all pending timers with the given name. */
+ public List<Timer> getPendingTimers(String timerName) {
+ return timerManager.getPendingTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all timers that have fired, in the order they fired. */
+ public List<Timer> getFiredTimers() {
+ return timerManager.getFiredTimers();
+ }
+
+ /** Returns all fired timers with the given name. */
+ public List<Timer> getFiredTimers(String timerName) {
+ return timerManager.getFiredTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Clears the fired timer history. */
+ public void clearFiredTimers() {
+ timerManager.clearFiredTimers();
+ }
+
+ private void setWatermarkMillis(long millis) throws Exception {
+ checkState(isOpen, "Harness is not open");
+ for (TableArgumentInfo tableArg : ArgumentInfo.filterTableArguments(arguments)) {
+ timerManager.setTableWatermark(tableArg.name, millis);
+ }
+ timerManager.updateGlobalWatermarkAndFireTimers(this::fireTimer);
+ }
+
+ private void setWatermarkForTableMillis(String tableArgument, long millis) throws Exception {
+ checkState(isOpen, "Harness is not open");
+ checkNotNull(tableArgument, "tableArgument must not be null");
+ checkArgument(
+ argumentsByName.get(tableArgument) instanceof TableArgumentInfo,
+ "Unknown or non-table argument: %s",
+ tableArgument);
+ timerManager.setTableWatermark(tableArgument, millis);
+ timerManager.updateGlobalWatermarkAndFireTimers(this::fireTimer);
+ }
+
+ private void fireTimer(Timer timer) throws Exception {
+ if (onTimer == null) {
+ throw new IllegalStateException(
+ "Timer fired but no onTimer() method is defined in "
+ + function.getClass().getSimpleName());
+ }
+
+ currentInvocation = InvocationContext.forTimer(timer);
+
+ try {
+ Map<String, Object> stateMap = stateManager.loadStateForKey(timer.partitionKey);
+
+ List<StateArgumentInfo> stateArgs = ArgumentInfo.filterStateArguments(arguments);
+ Object[] methodArgs = new Object[stateArgs.size()];
+ for (int i = 0; i < stateArgs.size(); i++) {
+ methodArgs[i] = stateMap.get(stateArgs.get(i).name);
+ }
Review Comment:
move this out of the `try` block? It's not throwing anything that's being caught.
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarness.java:
##########
@@ -303,38 +332,419 @@ public void clearStateForKey(String stateName, Row partitionKey) {
stateManager.clearStateForKey(stateName, partitionKey);
}
+ // -------------------------------------------------------------------------
+ // Watermark & Timer API
+ // -------------------------------------------------------------------------
+
+ /**
+ * Sets the watermark for all tables to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermark(LocalDateTime watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /** Sets the watermark for all tables to the given {@link Instant} and fires eligible timers. */
+ public void setWatermark(Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(watermark.toEpochMilli());
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, LocalDateTime watermark)
+ throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link Instant} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, watermark.toEpochMilli());
+ }
+
+ /** Returns all timers (both pending and fired), sorted by timestamp then name. */
+ public List<Timer> getTimers() {
+ return Stream.concat(
+ timerManager.getPendingTimers().stream(),
+ timerManager.getFiredTimers().stream())
+ .sorted()
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all pending (not yet fired) timers, sorted by timestamp then name. */
+ public List<Timer> getPendingTimers() {
+ return timerManager.getPendingTimers();
+ }
+
+ /** Returns all pending timers with the given name. */
+ public List<Timer> getPendingTimers(String timerName) {
+ return timerManager.getPendingTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all timers that have fired, in the order they fired. */
+ public List<Timer> getFiredTimers() {
+ return timerManager.getFiredTimers();
+ }
+
+ /** Returns all fired timers with the given name. */
+ public List<Timer> getFiredTimers(String timerName) {
+ return timerManager.getFiredTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Clears the fired timer history. */
+ public void clearFiredTimers() {
+ timerManager.clearFiredTimers();
+ }
+
+ private void setWatermarkMillis(long millis) throws Exception {
+ checkState(isOpen, "Harness is not open");
+ for (TableArgumentInfo tableArg : ArgumentInfo.filterTableArguments(arguments)) {
+ timerManager.setTableWatermark(tableArg.name, millis);
+ }
+ timerManager.updateGlobalWatermarkAndFireTimers(this::fireTimer);
+ }
+
+ private void setWatermarkForTableMillis(String tableArgument, long millis) throws Exception {
+ checkState(isOpen, "Harness is not open");
+ checkNotNull(tableArgument, "tableArgument must not be null");
+ checkArgument(
+ argumentsByName.get(tableArgument) instanceof TableArgumentInfo,
+ "Unknown or non-table argument: %s",
+ tableArgument);
+ timerManager.setTableWatermark(tableArgument, millis);
+ timerManager.updateGlobalWatermarkAndFireTimers(this::fireTimer);
+ }
+
+ private void fireTimer(Timer timer) throws Exception {
+ if (onTimer == null) {
+ throw new IllegalStateException(
+ "Timer fired but no onTimer() method is defined in "
+ + function.getClass().getSimpleName());
+ }
+
+ currentInvocation = InvocationContext.forTimer(timer);
+
+ try {
+ Map<String, Object> stateMap = stateManager.loadStateForKey(timer.partitionKey);
+
+ List<StateArgumentInfo> stateArgs = ArgumentInfo.filterStateArguments(arguments);
+ Object[] methodArgs = new Object[stateArgs.size()];
+ for (int i = 0; i < stateArgs.size(); i++) {
+ methodArgs[i] = stateMap.get(stateArgs.get(i).name);
+ }
+
+ onTimer.invoke(function, new TestOnTimerContext(stateMap), methodArgs);
+ stateManager.updateStateForKey(timer.partitionKey, stateMap);
+ } catch (InvocationTargetException e) {
+ Throwable cause = e.getCause();
+ if (cause instanceof Exception) {
+ throw (Exception) cause;
+ }
+ throw new RuntimeException("onTimer() invocation failed", e);
+ } finally {
+ currentInvocation = null;
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Context implementations
+ // -------------------------------------------------------------------------
+
+ private class TestContext implements ProcessTableFunction.Context {
+ final Map<String, Object> stateMap;
+
+ TestContext(Map<String, Object> stateMap) {
+ this.stateMap = stateMap;
+ }
+
+ @Override
+ public <TimeType> ProcessTableFunction.TimeContext<TimeType> timeContext(
+ Class<TimeType> conversionClass) {
+ return new TestTimeContext<>(conversionClass);
+ }
+
+ @Override
+ public TableSemantics tableSemanticsFor(String argName) {
+ ArgumentInfo argInfo = argumentsByName.get(argName);
+ if (argInfo == null) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Argument '%s' not found. Available arguments: %s",
+ argName, argumentsByName.keySet()));
+ }
+ if (!(argInfo instanceof TableArgumentInfo)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Argument '%s' is not a table argument (type: %s)",
+ argName, argInfo.getClass().getSimpleName()));
+ }
+ TableArgumentInfo tableArg = (TableArgumentInfo) argInfo;
+ int[] partitionIndices = getPartitionColumnIndices(tableArg);
+ int timeColumn =
+ onTimeColumnName != null
+ ? getFieldNames(tableArg.dataType).indexOf(onTimeColumnName)
+ : -1;
+ return new TestHarnessTableSemantics(tableArg.dataType, partitionIndices, timeColumn);
+ }
+
+ @Override
+ public void clearState(String stateName) {
+ stateMap.remove(stateName);
+ }
+
+ @Override
+ public void clearAllState() {
+ stateMap.clear();
+ }
+
+ @Override
+ public void clearAllTimers() {
+ timerManager.clearAll(currentInvocation.partitionKey);
+ }
+
+ @Override
+ public void clearAll() {
+ stateMap.clear();
Review Comment:
Should be scoped to the partitionKey (like `clearAllTimers()`)?
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarness.java:
##########
@@ -543,6 +968,9 @@ public static class Builder<OUT> {
private final Map<String, TableArgumentConfiguration> tableArgs = new HashMap<>();
private final Map<String, PartitionConfiguration> partitionConfigs = new HashMap<>();
private final Map<String, StateArgumentConfiguration> stateArgs = new HashMap<>();
+ @Nullable private String onTimeColumnName = null;
+ private final Map<String, Long> initialWatermarks = new HashMap<>();
+ @Nullable private Long initialWatermarkForAll = null;
Review Comment:
Wondering if we really need these.
Users could just call the `setWatermark()` methods right after creating / opening the harness.
That would have the same effect and wouldn't really add complexity.
So, IMO we could drop these from the builder. WDYT?
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarness.java:
##########
@@ -303,38 +332,419 @@ public void clearStateForKey(String stateName, Row partitionKey) {
stateManager.clearStateForKey(stateName, partitionKey);
}
+ // -------------------------------------------------------------------------
+ // Watermark & Timer API
+ // -------------------------------------------------------------------------
+
+ /**
+ * Sets the watermark for all tables to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermark(LocalDateTime watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /** Sets the watermark for all tables to the given {@link Instant} and fires eligible timers. */
+ public void setWatermark(Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(watermark.toEpochMilli());
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, LocalDateTime watermark)
+ throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link Instant} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, watermark.toEpochMilli());
+ }
+
+ /** Returns all timers (both pending and fired), sorted by timestamp then name. */
+ public List<Timer> getTimers() {
+ return Stream.concat(
+ timerManager.getPendingTimers().stream(),
+ timerManager.getFiredTimers().stream())
+ .sorted()
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all pending (not yet fired) timers, sorted by timestamp then name. */
+ public List<Timer> getPendingTimers() {
+ return timerManager.getPendingTimers();
+ }
+
+ /** Returns all pending timers with the given name. */
+ public List<Timer> getPendingTimers(String timerName) {
+ return timerManager.getPendingTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all timers that have fired, in the order they fired. */
+ public List<Timer> getFiredTimers() {
+ return timerManager.getFiredTimers();
+ }
+
+ /** Returns all fired timers with the given name. */
+ public List<Timer> getFiredTimers(String timerName) {
+ return timerManager.getFiredTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Clears the fired timer history. */
+ public void clearFiredTimers() {
+ timerManager.clearFiredTimers();
+ }
+
+ private void setWatermarkMillis(long millis) throws Exception {
+ checkState(isOpen, "Harness is not open");
+ for (TableArgumentInfo tableArg : ArgumentInfo.filterTableArguments(arguments)) {
+ timerManager.setTableWatermark(tableArg.name, millis);
+ }
+ timerManager.updateGlobalWatermarkAndFireTimers(this::fireTimer);
+ }
+
+ private void setWatermarkForTableMillis(String tableArgument, long millis) throws Exception {
+ checkState(isOpen, "Harness is not open");
+ checkNotNull(tableArgument, "tableArgument must not be null");
+ checkArgument(
+ argumentsByName.get(tableArgument) instanceof TableArgumentInfo,
+ "Unknown or non-table argument: %s",
+ tableArgument);
+ timerManager.setTableWatermark(tableArgument, millis);
+ timerManager.updateGlobalWatermarkAndFireTimers(this::fireTimer);
+ }
+
+ private void fireTimer(Timer timer) throws Exception {
+ if (onTimer == null) {
+ throw new IllegalStateException(
+ "Timer fired but no onTimer() method is defined in "
+ + function.getClass().getSimpleName());
+ }
+
+ currentInvocation = InvocationContext.forTimer(timer);
+
+ try {
+ Map<String, Object> stateMap = stateManager.loadStateForKey(timer.partitionKey);
+
+ List<StateArgumentInfo> stateArgs = ArgumentInfo.filterStateArguments(arguments);
+ Object[] methodArgs = new Object[stateArgs.size()];
+ for (int i = 0; i < stateArgs.size(); i++) {
+ methodArgs[i] = stateMap.get(stateArgs.get(i).name);
+ }
+
+ onTimer.invoke(function, new TestOnTimerContext(stateMap), methodArgs);
+ stateManager.updateStateForKey(timer.partitionKey, stateMap);
+ } catch (InvocationTargetException e) {
+ Throwable cause = e.getCause();
+ if (cause instanceof Exception) {
+ throw (Exception) cause;
+ }
+ throw new RuntimeException("onTimer() invocation failed", e);
+ } finally {
+ currentInvocation = null;
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Context implementations
+ // -------------------------------------------------------------------------
+
+ private class TestContext implements ProcessTableFunction.Context {
+ final Map<String, Object> stateMap;
+
+ TestContext(Map<String, Object> stateMap) {
+ this.stateMap = stateMap;
+ }
+
+ @Override
+ public <TimeType> ProcessTableFunction.TimeContext<TimeType> timeContext(
+ Class<TimeType> conversionClass) {
+ return new TestTimeContext<>(conversionClass);
+ }
+
+ @Override
+ public TableSemantics tableSemanticsFor(String argName) {
+ ArgumentInfo argInfo = argumentsByName.get(argName);
+ if (argInfo == null) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Argument '%s' not found. Available arguments: %s",
+ argName, argumentsByName.keySet()));
+ }
+ if (!(argInfo instanceof TableArgumentInfo)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Argument '%s' is not a table argument (type: %s)",
+ argName, argInfo.getClass().getSimpleName()));
+ }
+ TableArgumentInfo tableArg = (TableArgumentInfo) argInfo;
+ int[] partitionIndices = getPartitionColumnIndices(tableArg);
+ int timeColumn =
+ onTimeColumnName != null
+ ? getFieldNames(tableArg.dataType).indexOf(onTimeColumnName)
+ : -1;
+ return new TestHarnessTableSemantics(tableArg.dataType, partitionIndices, timeColumn);
+ }
+
+ @Override
+ public void clearState(String stateName) {
+ stateMap.remove(stateName);
+ }
+
+ @Override
+ public void clearAllState() {
+ stateMap.clear();
+ }
+
+ @Override
+ public void clearAllTimers() {
+ timerManager.clearAll(currentInvocation.partitionKey);
+ }
+
+ @Override
+ public void clearAll() {
+ stateMap.clear();
+ timerManager.clearAll(currentInvocation.partitionKey);
+ }
+
+ @Override
+ public ChangelogMode getChangelogMode() {
+ return ChangelogMode.insertOnly();
+ }
+ }
+
+ private class TestTimeContext<TimeType> implements ProcessTableFunction.TimeContext<TimeType> {
+ private final Class<TimeType> conversionClass;
+
+ TestTimeContext(Class<TimeType> conversionClass) {
+ this.conversionClass = conversionClass;
+ }
+
+ @Override
+ public TimeType time() {
+ InvocationContext ctx = currentInvocation;
+ if (ctx.isTimerInvocation()) {
+ return fromMillis(ctx.firingTimer.timestamp);
+ }
+ if (ctx.isEvalInvocation() && onTimeColumnName != null) {
+ ArgumentInfo argInfo = argumentsByName.get(ctx.tableArgumentName);
+ if (argInfo instanceof TableArgumentInfo) {
+ TableArgumentInfo tableArg = (TableArgumentInfo) argInfo;
+ if (!getFieldNames(tableArg.dataType).contains(onTimeColumnName)) {
+ return null;
+ }
+ }
+ Object timeValue = ctx.row.getField(onTimeColumnName);
+ if (timeValue == null) {
+ return null;
+ }
+ return fromMillis(toMillis(timeValue));
+ }
+ return null;
+ }
+
+ @Override
+ public TimeType tableWatermark() {
+ InvocationContext ctx = currentInvocation;
+ if (!ctx.isEvalInvocation()) {
+ return null;
+ }
+ Long wm = timerManager.getWatermarkForTable(ctx.tableArgumentName);
+ return wm != null ? fromMillis(wm) : null;
+ }
+
+ @Override
+ public TimeType currentWatermark() {
+ Long wm = timerManager.getGlobalWatermark();
+ return wm != null ? fromMillis(wm) : null;
+ }
+
+ @Override
+ public void registerOnTime(String name, TimeType time) {
+ checkTimersEnabled();
+ checkNotNull(name, "Timer name must not be null");
+ checkNotNull(time, "Timer timestamp must not be null");
+ timerManager.register(currentInvocation.partitionKey, toMillis(time), name);
+ }
+
+ @Override
+ public void registerOnTime(TimeType time) {
+ checkTimersEnabled();
+ checkNotNull(time, "Timer timestamp must not be null");
+ timerManager.register(currentInvocation.partitionKey, toMillis(time), null);
+ }
+
+ @Override
+ public void clearTimer(String name) {
+ checkNotNull(name, "Timer name must not be null");
+ timerManager.clearByName(currentInvocation.partitionKey, name);
+ }
+
+ @Override
+ public void clearTimer(TimeType time) {
+ checkNotNull(time, "Timer timestamp must not be null");
+ timerManager.clearByTimestamp(currentInvocation.partitionKey, toMillis(time));
+ }
+
+ @Override
+ public void clearAllTimers() {
+ timerManager.clearAll(currentInvocation.partitionKey);
+ }
+
+ private void checkTimersEnabled() {
+ boolean enabled =
+ ArgumentInfo.filterTableArguments(arguments).stream()
+ .anyMatch(
+ t ->
+ t.isSetSemantic
+ && t.prependStrategy
+ != OutputPrependStrategy.ALL_COLUMNS);
+ if (!enabled) {
+ throw new TableRuntimeException(
+ "Timers are not supported in the current PTF declaration. "
+ + "Note that only PTFs that take set semantic tables support timers. "
+ + "Also timers are not available for advanced traits such as "
+ + "supporting pass-through columns or updates.");
+ }
+ }
+
+ private TimeType fromMillis(long millis) {
+ return convertFromMillis(millis, conversionClass);
+ }
+
+ private long toMillis(Object time) {
+ if (time instanceof Long) {
+ return (Long) time;
+ } else if (time instanceof Instant) {
+ return ((Instant) time).toEpochMilli();
+ } else if (time instanceof LocalDateTime) {
+ return DateTimeUtils.toTimestampMillis((LocalDateTime) time);
+ } else if (time instanceof java.sql.Timestamp) {
+ return ((java.sql.Timestamp) time).getTime();
+ }
+ throw new IllegalArgumentException(
+ "Unsupported time type: " + time.getClass().getSimpleName());
+ }
+ }
+
+ private class TestOnTimerContext extends TestContext
+ implements ProcessTableFunction.OnTimerContext {
+ TestOnTimerContext(Map<String, Object> stateMap) {
Review Comment:
nit
```suggestion
TestOnTimerContext(Map<String, Object> stateMap) {
```
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ResolvedMethod.java:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import javax.annotation.Nullable;
+
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+
+/**
+ * A resolved PTF method (eval or onTimer) paired with whether its first parameter accepts a
+ * Context.
+ */
+class ResolvedMethod {
Review Comment:
I think this class could be typed on the contextClass.
```suggestion
class ResolvedMethod<T extends ProcessTableFunction.Context> {
```
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarness.java:
##########
@@ -303,38 +332,419 @@ public void clearStateForKey(String stateName, Row partitionKey) {
stateManager.clearStateForKey(stateName, partitionKey);
}
+ // -------------------------------------------------------------------------
+ // Watermark & Timer API
+ // -------------------------------------------------------------------------
+
+ /**
+ * Sets the watermark for all tables to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermark(LocalDateTime watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /** Sets the watermark for all tables to the given {@link Instant} and fires eligible timers. */
+ public void setWatermark(Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(watermark.toEpochMilli());
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, LocalDateTime watermark)
+ throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link Instant} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, watermark.toEpochMilli());
+ }
+
+ /** Returns all timers (both pending and fired), sorted by timestamp then name. */
+ public List<Timer> getTimers() {
+ return Stream.concat(
+ timerManager.getPendingTimers().stream(),
+ timerManager.getFiredTimers().stream())
+ .sorted()
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all pending (not yet fired) timers, sorted by timestamp then name. */
+ public List<Timer> getPendingTimers() {
+ return timerManager.getPendingTimers();
+ }
+
+ /** Returns all pending timers with the given name. */
+ public List<Timer> getPendingTimers(String timerName) {
+ return timerManager.getPendingTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all timers that have fired, in the order they fired. */
+ public List<Timer> getFiredTimers() {
+ return timerManager.getFiredTimers();
+ }
+
+ /** Returns all fired timers with the given name. */
+ public List<Timer> getFiredTimers(String timerName) {
+ return timerManager.getFiredTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Clears the fired timer history. */
+ public void clearFiredTimers() {
+ timerManager.clearFiredTimers();
+ }
+
+ private void setWatermarkMillis(long millis) throws Exception {
+ checkState(isOpen, "Harness is not open");
+ for (TableArgumentInfo tableArg : ArgumentInfo.filterTableArguments(arguments)) {
+ timerManager.setTableWatermark(tableArg.name, millis);
+ }
+ timerManager.updateGlobalWatermarkAndFireTimers(this::fireTimer);
+ }
+
+ private void setWatermarkForTableMillis(String tableArgument, long millis) throws Exception {
+ checkState(isOpen, "Harness is not open");
+ checkNotNull(tableArgument, "tableArgument must not be null");
+ checkArgument(
+ argumentsByName.get(tableArgument) instanceof TableArgumentInfo,
+ "Unknown or non-table argument: %s",
+ tableArgument);
+ timerManager.setTableWatermark(tableArgument, millis);
+ timerManager.updateGlobalWatermarkAndFireTimers(this::fireTimer);
+ }
+
+ private void fireTimer(Timer timer) throws Exception {
+ if (onTimer == null) {
+ throw new IllegalStateException(
+ "Timer fired but no onTimer() method is defined in "
+ + function.getClass().getSimpleName());
+ }
+
+ currentInvocation = InvocationContext.forTimer(timer);
+
+ try {
+ Map<String, Object> stateMap = stateManager.loadStateForKey(timer.partitionKey);
+
+ List<StateArgumentInfo> stateArgs = ArgumentInfo.filterStateArguments(arguments);
+ Object[] methodArgs = new Object[stateArgs.size()];
+ for (int i = 0; i < stateArgs.size(); i++) {
+ methodArgs[i] = stateMap.get(stateArgs.get(i).name);
+ }
+
+ onTimer.invoke(function, new TestOnTimerContext(stateMap), methodArgs);
+ stateManager.updateStateForKey(timer.partitionKey, stateMap);
+ } catch (InvocationTargetException e) {
+ Throwable cause = e.getCause();
+ if (cause instanceof Exception) {
+ throw (Exception) cause;
+ }
+ throw new RuntimeException("onTimer() invocation failed", e);
+ } finally {
+ currentInvocation = null;
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Context implementations
+ // -------------------------------------------------------------------------
+
+ private class TestContext implements ProcessTableFunction.Context {
+ final Map<String, Object> stateMap;
+
+ TestContext(Map<String, Object> stateMap) {
+ this.stateMap = stateMap;
+ }
+
+ @Override
+ public <TimeType> ProcessTableFunction.TimeContext<TimeType> timeContext(
+ Class<TimeType> conversionClass) {
+ return new TestTimeContext<>(conversionClass);
+ }
+
+ @Override
+ public TableSemantics tableSemanticsFor(String argName) {
+ ArgumentInfo argInfo = argumentsByName.get(argName);
+ if (argInfo == null) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Argument '%s' not found. Available arguments: %s",
+ argName, argumentsByName.keySet()));
+ }
+ if (!(argInfo instanceof TableArgumentInfo)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Argument '%s' is not a table argument (type: %s)",
+ argName, argInfo.getClass().getSimpleName()));
+ }
+ TableArgumentInfo tableArg = (TableArgumentInfo) argInfo;
+ int[] partitionIndices = getPartitionColumnIndices(tableArg);
+ int timeColumn =
+ onTimeColumnName != null
+ ? getFieldNames(tableArg.dataType).indexOf(onTimeColumnName)
+ : -1;
+ return new TestHarnessTableSemantics(tableArg.dataType, partitionIndices, timeColumn);
+ }
+
+ @Override
+ public void clearState(String stateName) {
+ stateMap.remove(stateName);
+ }
+
+ @Override
+ public void clearAllState() {
+ stateMap.clear();
+ }
+
+ @Override
+ public void clearAllTimers() {
+ timerManager.clearAll(currentInvocation.partitionKey);
+ }
+
+ @Override
+ public void clearAll() {
+ stateMap.clear();
+ timerManager.clearAll(currentInvocation.partitionKey);
+ }
+
+ @Override
+ public ChangelogMode getChangelogMode() {
+ return ChangelogMode.insertOnly();
+ }
+ }
+
+ private class TestTimeContext<TimeType> implements ProcessTableFunction.TimeContext<TimeType> {
+ private final Class<TimeType> conversionClass;
+
+ TestTimeContext(Class<TimeType> conversionClass) {
+ this.conversionClass = conversionClass;
+ }
+
+ @Override
+ public TimeType time() {
+ InvocationContext ctx = currentInvocation;
+ if (ctx.isTimerInvocation()) {
+ return fromMillis(ctx.firingTimer.timestamp);
+ }
+ if (ctx.isEvalInvocation() && onTimeColumnName != null) {
+ ArgumentInfo argInfo = argumentsByName.get(ctx.tableArgumentName);
+ if (argInfo instanceof TableArgumentInfo) {
+ TableArgumentInfo tableArg = (TableArgumentInfo) argInfo;
+ if (!getFieldNames(tableArg.dataType).contains(onTimeColumnName)) {
+ return null;
+ }
+ }
+ Object timeValue = ctx.row.getField(onTimeColumnName);
+ if (timeValue == null) {
+ return null;
+ }
+ return fromMillis(toMillis(timeValue));
+ }
+ return null;
+ }
+
+ @Override
+ public TimeType tableWatermark() {
+ InvocationContext ctx = currentInvocation;
+ if (!ctx.isEvalInvocation()) {
+ return null;
+ }
+ Long wm = timerManager.getWatermarkForTable(ctx.tableArgumentName);
+ return wm != null ? fromMillis(wm) : null;
+ }
+
+ @Override
+ public TimeType currentWatermark() {
+ Long wm = timerManager.getGlobalWatermark();
+ return wm != null ? fromMillis(wm) : null;
+ }
+
+ @Override
+ public void registerOnTime(String name, TimeType time) {
+ checkTimersEnabled();
+ checkNotNull(name, "Timer name must not be null");
+ checkNotNull(time, "Timer timestamp must not be null");
+ timerManager.register(currentInvocation.partitionKey, toMillis(time), name);
+ }
+
+ @Override
+ public void registerOnTime(TimeType time) {
+ checkTimersEnabled();
+ checkNotNull(time, "Timer timestamp must not be null");
+ timerManager.register(currentInvocation.partitionKey, toMillis(time), null);
+ }
+
+ @Override
+ public void clearTimer(String name) {
+ checkNotNull(name, "Timer name must not be null");
+ timerManager.clearByName(currentInvocation.partitionKey, name);
+ }
+
+ @Override
+ public void clearTimer(TimeType time) {
+ checkNotNull(time, "Timer timestamp must not be null");
+ timerManager.clearByTimestamp(currentInvocation.partitionKey, toMillis(time));
+ }
+
+ @Override
+ public void clearAllTimers() {
+ timerManager.clearAll(currentInvocation.partitionKey);
+ }
+
+ private void checkTimersEnabled() {
+ boolean enabled =
+ ArgumentInfo.filterTableArguments(arguments).stream()
+ .anyMatch(
+ t ->
+ t.isSetSemantic
+ && t.prependStrategy
+ != OutputPrependStrategy.ALL_COLUMNS);
+ if (!enabled) {
+ throw new TableRuntimeException(
+ "Timers are not supported in the current PTF declaration. "
+ + "Note that only PTFs that take set semantic tables support timers. "
+ + "Also timers are not available for advanced traits such as "
+ + "supporting pass-through columns or updates.");
+ }
+ }
+
+ private TimeType fromMillis(long millis) {
+ return convertFromMillis(millis, conversionClass);
+ }
+
+ private long toMillis(Object time) {
+ if (time instanceof Long) {
+ return (Long) time;
+ } else if (time instanceof Instant) {
+ return ((Instant) time).toEpochMilli();
+ } else if (time instanceof LocalDateTime) {
+ return DateTimeUtils.toTimestampMillis((LocalDateTime) time);
+ } else if (time instanceof java.sql.Timestamp) {
+ return ((java.sql.Timestamp) time).getTime();
+ }
+ throw new IllegalArgumentException(
+ "Unsupported time type: " + time.getClass().getSimpleName());
+ }
+ }
+
+ private class TestOnTimerContext extends TestContext
+ implements ProcessTableFunction.OnTimerContext {
+ TestOnTimerContext(Map<String, Object> stateMap) {
+ super(stateMap);
+ }
+
+ @Override
+ public String currentTimer() {
+ if (currentInvocation.isTimerInvocation()) {
+ return currentInvocation.firingTimer.getName();
+ }
+ return null;
+ }
+ }
+
+ private static int[] getPartitionColumnIndices(TableArgumentInfo arg) {
+ if (arg.partitionColumnNames == null || arg.partitionColumnNames.length == 0) {
+ return new int[0];
+ }
+ List<String> fieldNames = getFieldNames(arg.dataType);
+ int[] indices = new int[arg.partitionColumnNames.length];
+ for (int i = 0; i < arg.partitionColumnNames.length; i++) {
+ String colName = arg.partitionColumnNames[i];
+ int index = fieldNames.indexOf(colName);
+ if (index < 0) {
+ throw new IllegalStateException(
+ "Partition column '"
+ + colName
+ + "' not found in table argument. "
+ + "Available fields: "
+ + fieldNames);
+ }
+ indices[i] = index;
+ }
+ return indices;
+ }
+
+ @Nullable
+ private Class<?> resolveRowtimeConversionClass(List<TableArgumentInfo> tableArguments) {
Review Comment:
Is it guaranteed (or required) that all table args need to have the same type / class for the `on_time` attribute? Or could table A use `Long` and table B use `Instant`?
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ResolvedMethod.java:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import javax.annotation.Nullable;
+
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+
+/**
+ * A resolved PTF method (eval or onTimer) paired with whether its first parameter accepts a
+ * Context.
+ */
+class ResolvedMethod {
+ final Method method;
+ final boolean takesContext;
+
+ static ResolvedMethod of(Method method, Class<?> contextClass) {
+ boolean takesContext =
+ method.getParameterTypes().length > 0
+ && contextClass.isAssignableFrom(method.getParameterTypes()[0]);
+ return new ResolvedMethod(method, takesContext);
+ }
+
+ private ResolvedMethod(Method method, boolean takesContext) {
+ this.method = method;
+ this.takesContext = takesContext;
+ }
+
+ void invoke(Object target, @Nullable Object context, Object[] methodArgs)
Review Comment:
```suggestion
void invoke(Object target, @Nullable T context, Object[] methodArgs)
```
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarness.java:
##########
@@ -711,25 +1197,67 @@ public ProcessTableFunctionTestHarness<OUT> build() throws Exception {
// Extract table arguments for output type derivation
// SystemTypeInference needs table semantics for pass-through column deduplication
- List<TableArgumentInfo> tableArgs = ArgumentInfo.filterTableArguments(arguments);
+ List<TableArgumentInfo> tableArgInfos = ArgumentInfo.filterTableArguments(arguments);
// Derive output schema using SystemTypeInference
DataType derivedOutputType =
deriveOutputTypeFromSystemInference(
- function, dataTypeFactory, systemTypeInference, tableArgs);
+ function,
+ dataTypeFactory,
+ systemTypeInference,
+ arguments,
+ tableArgInfos);
// Create output converter for PTF emissions
DataStructureConverter<Object, Object> harnessOutputConverter =
createPTFOutputConverter(derivedOutputType);
+ // Validate onTimeColumn configuration
+ if (onTimeColumnName != null) {
+ boolean foundInAnyTable =
+ tableArgInfos.stream()
+ .anyMatch(
+ t -> getFieldNames(t.dataType).contains(onTimeColumnName));
Review Comment:
should we check that the field is a valid timestamp type and also that the field has the same type in all table args that have this field?
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarness.java:
##########
@@ -1386,25 +1948,76 @@ private ProcessTableFunction<OUT> instantiateFunction() throws IllegalArgumentEx
/**
* Derives the output schema using SystemTypeInference, including field name deduplication.
+ *
+ * <p>SystemTypeInference's staticArgs list includes both user-declared arguments and
+ * system-injected arguments (on_time, uid). The CallContext we build must mirror this full
+ * list positionally — each index maps to a staticArg. User args get their resolved
+ * DataType; system args get placeholder types (DESCRIPTOR for on_time, STRING for uid).
+ * Table semantics are attached at the positions of table arguments so SystemTypeInference
+ * can perform pass-through column deduplication.
*/
private DataType deriveOutputTypeFromSystemInference(
ProcessTableFunction<OUT> function,
DataTypeFactory dataTypeFactory,
TypeInference systemTypeInference,
- List<TableArgumentInfo> arguments) {
+ List<ArgumentInfo> allArguments,
+ List<TableArgumentInfo> tableArguments) {
- List<DataType> argumentDataTypes = new ArrayList<>();
- for (TableArgumentInfo arg : arguments) {
- argumentDataTypes.add(arg.dataType);
+ Optional<List<StaticArgument>> staticArgsOpt = systemTypeInference.getStaticArguments();
+ List<StaticArgument> staticArgs =
+ staticArgsOpt.orElseThrow(
+ () ->
+ new IllegalStateException(
+ "SystemTypeInference has no static arguments"));
Review Comment:
nit
```suggestion
List<StaticArgument> staticArgs =
systemTypeInference.getStaticArguments().orElseThrow(
() ->
new IllegalStateException(
"SystemTypeInference has no static arguments"));
```
Can be folded?
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/TestHarnessTimerManager.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.function.ThrowingConsumer;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Timer and watermark manager for {@link ProcessTableFunctionTestHarness}.
+ *
+ * <p>Handles timer registration, clearing, watermark tracking, and determining which timers are
+ * eligible to fire.
+ */
+@Internal
+class TestHarnessTimerManager {
Review Comment:
It might make sense to add a separate test for this class.
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarness.java:
##########
@@ -1164,40 +1721,45 @@ private List<ArgumentInfo> extractAndValidateTypeInference(
}
/** Creates appropriate StateConverter for the given state data type. */
- private StateConverter createStateConverter(DataType stateDataType, ClassLoader classLoader)
- throws Exception {
- LogicalType logicalType = stateDataType.getLogicalType();
+ private StateConverter createStateConverter(
+ DataType stateDataType, ClassLoader classLoader) {
+ DataType resolvedType =
+ ListView.class.isAssignableFrom(stateDataType.getConversionClass())
+ || MapView.class.isAssignableFrom(
+ stateDataType.getConversionClass())
+ ? stateDataType.getChildren().get(0)
Review Comment:
Woundn't this just return the key type for `MapView`?
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarness.java:
##########
@@ -303,38 +332,419 @@ public void clearStateForKey(String stateName, Row partitionKey) {
stateManager.clearStateForKey(stateName, partitionKey);
}
+ // -------------------------------------------------------------------------
+ // Watermark & Timer API
+ // -------------------------------------------------------------------------
+
+ /**
+ * Sets the watermark for all tables to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermark(LocalDateTime watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /** Sets the watermark for all tables to the given {@link Instant} and fires eligible timers. */
+ public void setWatermark(Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(watermark.toEpochMilli());
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, LocalDateTime watermark)
+ throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link Instant} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, watermark.toEpochMilli());
+ }
+
+ /** Returns all timers (both pending and fired), sorted by timestamp then name. */
+ public List<Timer> getTimers() {
Review Comment:
Also have a method `getAllTimers(Row partitionKey)`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39378][table] Add support for Context, timers and on_time to PTF test harness [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28326:
URL: https://github.com/apache/flink/pull/28326#discussion_r3387147000
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarness.java:
##########
@@ -711,25 +1197,67 @@ public ProcessTableFunctionTestHarness<OUT> build() throws Exception {
// Extract table arguments for output type derivation
// SystemTypeInference needs table semantics for pass-through column deduplication
- List<TableArgumentInfo> tableArgs = ArgumentInfo.filterTableArguments(arguments);
+ List<TableArgumentInfo> tableArgInfos = ArgumentInfo.filterTableArguments(arguments);
// Derive output schema using SystemTypeInference
DataType derivedOutputType =
deriveOutputTypeFromSystemInference(
- function, dataTypeFactory, systemTypeInference, tableArgs);
+ function,
+ dataTypeFactory,
+ systemTypeInference,
+ arguments,
+ tableArgInfos);
// Create output converter for PTF emissions
DataStructureConverter<Object, Object> harnessOutputConverter =
createPTFOutputConverter(derivedOutputType);
+ // Validate onTimeColumn configuration
+ if (onTimeColumnName != null) {
+ boolean foundInAnyTable =
+ tableArgInfos.stream()
+ .anyMatch(
+ t -> getFieldNames(t.dataType).contains(onTimeColumnName));
Review Comment:
🙌
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39378][table] Add support for Context, timers and on_time to PTF test harness [flink]
Posted by "autophagy (via GitHub)" <gi...@apache.org>.
autophagy commented on code in PR #28326:
URL: https://github.com/apache/flink/pull/28326#discussion_r3382256359
##########
docs/content.zh/docs/dev/table/functions/ptfs.md:
##########
@@ -2465,6 +2465,115 @@ void testStateMutation() throws Exception {
{{< /tab >}}
{{< /tabs >}}
+#### Testing with Timers and Context
+
+The harness supports the `Context` parameter, timer registration via `TimeContext`, and `onTimer`
+callbacks. Use `.withOnTimeColumn()` to configure the event time column and `.setWatermark()` to
+advance watermarks and fire eligible timers.
+
+{{< tabs "timer-testing" >}}
+{{< tab "Java" >}}
+```java
+// A PTF that registers a named timer 5 seconds after each event, and emits when it fires.
+@DataTypeHint("ROW<message STRING>")
+public class TimerPTF extends ProcessTableFunction<Row> {
+ public void eval(
+ Context ctx,
+ @ArgumentHint({ArgumentTrait.SET_SEMANTIC_TABLE, ArgumentTrait.REQUIRE_ON_TIME})
+ Row input) {
+ String name = input.getFieldAs("name");
+ TimeContext<LocalDateTime> timeCtx = ctx.timeContext(LocalDateTime.class);
+ timeCtx.registerOnTime("timeout-" + name, timeCtx.time().plus(Duration.ofSeconds(5)));
+ collect(Row.of("registered-" + name));
+ }
+
+ public void onTimer(OnTimerContext ctx) {
+ collect(Row.of("timer-fired-" + ctx.currentTimer()));
+ }
+}
+
+@Test
+void testTimerRegistrationAndFiring() throws Exception {
+ try (ProcessTableFunctionTestHarness<Row> harness =
+ ProcessTableFunctionTestHarness.ofClass(TimerPTF.class)
+ .withTableArgument("input",
+ DataTypes.of("ROW<partition STRING, name STRING, ts TIMESTAMP(3)>"))
+ .withPartitionBy("input", "partition")
+ .withOnTimeColumn("ts")
+ .build()) {
+
+ harness.processElement(Row.of("P1", "Alice", LocalDateTime.of(2025, 1, 1, 0, 0, 1)));
+
+ // Verify the timer was registered
+ assertThat(harness.getPendingTimers()).hasSize(1);
+ assertThat(harness.getPendingTimers().get(0).getName()).isEqualTo("timeout-Alice");
+
+ // Advance watermark past the timer's timestamp to fire it
+ harness.clearOutput();
Review Comment:
It'll have 1 row in it (we collect one upon timer registration). But i'll assert that its there and then clear, just so its not confusing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39378][table] Add support for Context, timers and on_time to PTF test harness [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28326:
URL: https://github.com/apache/flink/pull/28326#discussion_r3387180458
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarness.java:
##########
@@ -1164,40 +1721,45 @@ private List<ArgumentInfo> extractAndValidateTypeInference(
}
/** Creates appropriate StateConverter for the given state data type. */
- private StateConverter createStateConverter(DataType stateDataType, ClassLoader classLoader)
- throws Exception {
- LogicalType logicalType = stateDataType.getLogicalType();
+ private StateConverter createStateConverter(
+ DataType stateDataType, ClassLoader classLoader) {
+ DataType resolvedType =
+ ListView.class.isAssignableFrom(stateDataType.getConversionClass())
+ || MapView.class.isAssignableFrom(
+ stateDataType.getConversionClass())
+ ? stateDataType.getChildren().get(0)
Review Comment:
thanks for checking!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39378][table] Add support for Context, timers and on_time to PTF test harness [flink]
Posted by "autophagy (via GitHub)" <gi...@apache.org>.
autophagy commented on code in PR #28326:
URL: https://github.com/apache/flink/pull/28326#discussion_r3381984056
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarness.java:
##########
@@ -303,38 +332,419 @@ public void clearStateForKey(String stateName, Row partitionKey) {
stateManager.clearStateForKey(stateName, partitionKey);
}
+ // -------------------------------------------------------------------------
+ // Watermark & Timer API
+ // -------------------------------------------------------------------------
+
+ /**
+ * Sets the watermark for all tables to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermark(LocalDateTime watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /** Sets the watermark for all tables to the given {@link Instant} and fires eligible timers. */
+ public void setWatermark(Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(watermark.toEpochMilli());
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, LocalDateTime watermark)
+ throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link Instant} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, watermark.toEpochMilli());
+ }
+
+ /** Returns all timers (both pending and fired), sorted by timestamp then name. */
+ public List<Timer> getTimers() {
+ return Stream.concat(
+ timerManager.getPendingTimers().stream(),
+ timerManager.getFiredTimers().stream())
+ .sorted()
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all pending (not yet fired) timers, sorted by timestamp then name. */
+ public List<Timer> getPendingTimers() {
+ return timerManager.getPendingTimers();
+ }
+
+ /** Returns all pending timers with the given name. */
+ public List<Timer> getPendingTimers(String timerName) {
+ return timerManager.getPendingTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all timers that have fired, in the order they fired. */
+ public List<Timer> getFiredTimers() {
+ return timerManager.getFiredTimers();
+ }
+
+ /** Returns all fired timers with the given name. */
+ public List<Timer> getFiredTimers(String timerName) {
+ return timerManager.getFiredTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Clears the fired timer history. */
+ public void clearFiredTimers() {
+ timerManager.clearFiredTimers();
+ }
+
+ private void setWatermarkMillis(long millis) throws Exception {
+ checkState(isOpen, "Harness is not open");
+ for (TableArgumentInfo tableArg : ArgumentInfo.filterTableArguments(arguments)) {
+ timerManager.setTableWatermark(tableArg.name, millis);
+ }
+ timerManager.updateGlobalWatermarkAndFireTimers(this::fireTimer);
+ }
+
+ private void setWatermarkForTableMillis(String tableArgument, long millis) throws Exception {
+ checkState(isOpen, "Harness is not open");
+ checkNotNull(tableArgument, "tableArgument must not be null");
+ checkArgument(
+ argumentsByName.get(tableArgument) instanceof TableArgumentInfo,
+ "Unknown or non-table argument: %s",
+ tableArgument);
+ timerManager.setTableWatermark(tableArgument, millis);
+ timerManager.updateGlobalWatermarkAndFireTimers(this::fireTimer);
+ }
+
+ private void fireTimer(Timer timer) throws Exception {
+ if (onTimer == null) {
+ throw new IllegalStateException(
+ "Timer fired but no onTimer() method is defined in "
+ + function.getClass().getSimpleName());
+ }
+
+ currentInvocation = InvocationContext.forTimer(timer);
+
+ try {
+ Map<String, Object> stateMap = stateManager.loadStateForKey(timer.partitionKey);
+
+ List<StateArgumentInfo> stateArgs = ArgumentInfo.filterStateArguments(arguments);
+ Object[] methodArgs = new Object[stateArgs.size()];
+ for (int i = 0; i < stateArgs.size(); i++) {
+ methodArgs[i] = stateMap.get(stateArgs.get(i).name);
+ }
+
+ onTimer.invoke(function, new TestOnTimerContext(stateMap), methodArgs);
+ stateManager.updateStateForKey(timer.partitionKey, stateMap);
+ } catch (InvocationTargetException e) {
+ Throwable cause = e.getCause();
+ if (cause instanceof Exception) {
+ throw (Exception) cause;
+ }
+ throw new RuntimeException("onTimer() invocation failed", e);
+ } finally {
+ currentInvocation = null;
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Context implementations
+ // -------------------------------------------------------------------------
+
+ private class TestContext implements ProcessTableFunction.Context {
+ final Map<String, Object> stateMap;
+
+ TestContext(Map<String, Object> stateMap) {
+ this.stateMap = stateMap;
+ }
+
+ @Override
+ public <TimeType> ProcessTableFunction.TimeContext<TimeType> timeContext(
+ Class<TimeType> conversionClass) {
+ return new TestTimeContext<>(conversionClass);
+ }
+
+ @Override
+ public TableSemantics tableSemanticsFor(String argName) {
+ ArgumentInfo argInfo = argumentsByName.get(argName);
+ if (argInfo == null) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Argument '%s' not found. Available arguments: %s",
+ argName, argumentsByName.keySet()));
+ }
+ if (!(argInfo instanceof TableArgumentInfo)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Argument '%s' is not a table argument (type: %s)",
+ argName, argInfo.getClass().getSimpleName()));
+ }
+ TableArgumentInfo tableArg = (TableArgumentInfo) argInfo;
+ int[] partitionIndices = getPartitionColumnIndices(tableArg);
+ int timeColumn =
+ onTimeColumnName != null
+ ? getFieldNames(tableArg.dataType).indexOf(onTimeColumnName)
+ : -1;
+ return new TestHarnessTableSemantics(tableArg.dataType, partitionIndices, timeColumn);
+ }
+
+ @Override
+ public void clearState(String stateName) {
+ stateMap.remove(stateName);
+ }
+
+ @Override
+ public void clearAllState() {
+ stateMap.clear();
Review Comment:
It currently is scoped to the partition key, as the `statemap` passed into the context is the state map for the particular partition key in question. However it is an inconsistency, we use the timer manager for managing timers but not the state manager for managing state (it gets passed in on context creation).
I'll refactor this so that instead of the state map for the particular partition key being passed in to the context on creation, we use the state manager, like the timer manager.
##########
flink-table/flink-table-test-utils/src/main/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarness.java:
##########
@@ -303,38 +332,419 @@ public void clearStateForKey(String stateName, Row partitionKey) {
stateManager.clearStateForKey(stateName, partitionKey);
}
+ // -------------------------------------------------------------------------
+ // Watermark & Timer API
+ // -------------------------------------------------------------------------
+
+ /**
+ * Sets the watermark for all tables to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermark(LocalDateTime watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /** Sets the watermark for all tables to the given {@link Instant} and fires eligible timers. */
+ public void setWatermark(Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkMillis(watermark.toEpochMilli());
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link LocalDateTime} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, LocalDateTime watermark)
+ throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, DateTimeUtils.toTimestampMillis(watermark));
+ }
+
+ /**
+ * Sets the watermark for a specific table to the given {@link Instant} and fires eligible
+ * timers.
+ */
+ public void setWatermarkForTable(String tableArgument, Instant watermark) throws Exception {
+ checkNotNull(watermark, "watermark must not be null");
+ setWatermarkForTableMillis(tableArgument, watermark.toEpochMilli());
+ }
+
+ /** Returns all timers (both pending and fired), sorted by timestamp then name. */
+ public List<Timer> getTimers() {
+ return Stream.concat(
+ timerManager.getPendingTimers().stream(),
+ timerManager.getFiredTimers().stream())
+ .sorted()
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all pending (not yet fired) timers, sorted by timestamp then name. */
+ public List<Timer> getPendingTimers() {
+ return timerManager.getPendingTimers();
+ }
+
+ /** Returns all pending timers with the given name. */
+ public List<Timer> getPendingTimers(String timerName) {
+ return timerManager.getPendingTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Returns all timers that have fired, in the order they fired. */
+ public List<Timer> getFiredTimers() {
+ return timerManager.getFiredTimers();
+ }
+
+ /** Returns all fired timers with the given name. */
+ public List<Timer> getFiredTimers(String timerName) {
+ return timerManager.getFiredTimers().stream()
+ .filter(t -> timerName.equals(t.getName()))
+ .collect(Collectors.toList());
+ }
+
+ /** Clears the fired timer history. */
+ public void clearFiredTimers() {
+ timerManager.clearFiredTimers();
+ }
+
+ private void setWatermarkMillis(long millis) throws Exception {
+ checkState(isOpen, "Harness is not open");
+ for (TableArgumentInfo tableArg : ArgumentInfo.filterTableArguments(arguments)) {
+ timerManager.setTableWatermark(tableArg.name, millis);
+ }
+ timerManager.updateGlobalWatermarkAndFireTimers(this::fireTimer);
+ }
+
+ private void setWatermarkForTableMillis(String tableArgument, long millis) throws Exception {
+ checkState(isOpen, "Harness is not open");
+ checkNotNull(tableArgument, "tableArgument must not be null");
+ checkArgument(
+ argumentsByName.get(tableArgument) instanceof TableArgumentInfo,
+ "Unknown or non-table argument: %s",
+ tableArgument);
+ timerManager.setTableWatermark(tableArgument, millis);
+ timerManager.updateGlobalWatermarkAndFireTimers(this::fireTimer);
+ }
+
+ private void fireTimer(Timer timer) throws Exception {
+ if (onTimer == null) {
+ throw new IllegalStateException(
+ "Timer fired but no onTimer() method is defined in "
+ + function.getClass().getSimpleName());
+ }
+
+ currentInvocation = InvocationContext.forTimer(timer);
+
+ try {
+ Map<String, Object> stateMap = stateManager.loadStateForKey(timer.partitionKey);
+
+ List<StateArgumentInfo> stateArgs = ArgumentInfo.filterStateArguments(arguments);
+ Object[] methodArgs = new Object[stateArgs.size()];
+ for (int i = 0; i < stateArgs.size(); i++) {
+ methodArgs[i] = stateMap.get(stateArgs.get(i).name);
+ }
+
+ onTimer.invoke(function, new TestOnTimerContext(stateMap), methodArgs);
+ stateManager.updateStateForKey(timer.partitionKey, stateMap);
+ } catch (InvocationTargetException e) {
+ Throwable cause = e.getCause();
+ if (cause instanceof Exception) {
+ throw (Exception) cause;
+ }
+ throw new RuntimeException("onTimer() invocation failed", e);
+ } finally {
+ currentInvocation = null;
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Context implementations
+ // -------------------------------------------------------------------------
+
+ private class TestContext implements ProcessTableFunction.Context {
+ final Map<String, Object> stateMap;
+
+ TestContext(Map<String, Object> stateMap) {
+ this.stateMap = stateMap;
+ }
+
+ @Override
+ public <TimeType> ProcessTableFunction.TimeContext<TimeType> timeContext(
+ Class<TimeType> conversionClass) {
+ return new TestTimeContext<>(conversionClass);
+ }
+
+ @Override
+ public TableSemantics tableSemanticsFor(String argName) {
+ ArgumentInfo argInfo = argumentsByName.get(argName);
+ if (argInfo == null) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Argument '%s' not found. Available arguments: %s",
+ argName, argumentsByName.keySet()));
+ }
+ if (!(argInfo instanceof TableArgumentInfo)) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Argument '%s' is not a table argument (type: %s)",
+ argName, argInfo.getClass().getSimpleName()));
+ }
+ TableArgumentInfo tableArg = (TableArgumentInfo) argInfo;
+ int[] partitionIndices = getPartitionColumnIndices(tableArg);
+ int timeColumn =
+ onTimeColumnName != null
+ ? getFieldNames(tableArg.dataType).indexOf(onTimeColumnName)
+ : -1;
+ return new TestHarnessTableSemantics(tableArg.dataType, partitionIndices, timeColumn);
+ }
+
+ @Override
+ public void clearState(String stateName) {
+ stateMap.remove(stateName);
Review Comment:
See above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39378][table] Add support for Context, timers and on_time to PTF test harness [flink]
Posted by "fhueske (via GitHub)" <gi...@apache.org>.
fhueske commented on code in PR #28326:
URL: https://github.com/apache/flink/pull/28326#discussion_r3479635881
##########
flink-table/flink-table-test-utils/src/test/java/org/apache/flink/table/runtime/functions/ProcessTableFunctionTestHarnessTest.java:
##########
@@ -2455,11 +2457,13 @@ void testPassThroughWithTimersIsRejected() throws Exception {
.withOnTimeColumn("ts")
.build()) {
- assertThrows(
- TableRuntimeException.class,
- () ->
- harness.processElement(
- Row.of("P1", LocalDateTime.of(2025, 1, 1, 0, 0, 1))));
+ TableRuntimeException e =
+ assertThrows(
+ TableRuntimeException.class,
+ () ->
+ harness.processElement(
Review Comment:
Thanks for the explanation!
Agreed that the harness should behave like the actual runtime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org