You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "twalthr (via GitHub)" <gi...@apache.org> on 2025/03/05 07:24:12 UTC

[PR] [FLINK-37302][table] Support PTFs with time and timer services [flink]

twalthr opened a new pull request, #26250:
URL: https://github.com/apache/flink/pull/26250

   ## What is the purpose of the change
   
   *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)*
   
   
   ## Brief change log
   
   *(for example:)*
     - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact*
     - *Deployments RPC transmits only the blob storage reference*
     - *TaskManagers retrieve the TaskInfo from the blob cache*
   
   
   ## Verifying this change
   
   Please make sure both new and modified tests in this PR follow [the conventions for tests defined in our code quality guide](https://flink.apache.org/how-to-contribute/code-style-and-quality-common/#7-testing).
   
   *(Please pick either of the following options)*
   
   This change is a trivial rework / code cleanup without any test coverage.
   
   *(or)*
   
   This change is already covered by existing tests, such as *(please describe tests)*.
   
   *(or)*
   
   This change added tests and can be verified as follows:
   
   *(example:)*
     - *Added integration tests for end-to-end deployment with large payloads (100MB)*
     - *Extended integration test for recovery after master (JobManager) failure*
     - *Added test that validates that TaskInfo is transferred only once across recoveries*
     - *Manually verified the change by running a 4 node cluster with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.*
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): (yes / no)
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no)
     - The serializers: (yes / no / don't know)
     - The runtime per-record code paths (performance sensitive): (yes / no / don't know)
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
     - The S3 file system connector: (yes / no / don't know)
   
   ## Documentation
   
     - Does this pull request introduce a new feature? (yes / no)
     - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37302][table] Support PTFs with time and timer services [flink]

Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #26250:
URL: https://github.com/apache/flink/pull/26250#discussion_r1980864849


##########
flink-table/flink-table-api-java/src/test/java/org/apache/flink/table/test/program/TestStep.java:
##########
@@ -50,6 +50,7 @@ enum TestKind {
         SINK_WITHOUT_DATA,
         SINK_WITH_DATA,
         SINK_WITH_RESTORE_DATA,
+        FAILING_SQL

Review Comment:
   Do we want to distinguish when sql is failing?
   Like parsing/validation/something else?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37302][table] Support PTFs with time and timer services [flink]

Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #26250:
URL: https://github.com/apache/flink/pull/26250#discussion_r1980888148


##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/ProcessTableFunction.java:
##########
@@ -285,6 +288,76 @@
  * }
  * }</pre>
  *
+ * <h1>Time and Timers</h1>
+ *
+ * <p>A PTF supports event time natively. Time-based services are available via {@link
+ * Context#timeContext(Class)}.
+ *
+ * <h2>Time</h2>
+ *
+ * <p>Every PTF takes an optional {@code on_time} argument. The {@code on_time} argument in the function call declares the time attribute column for which
+ * a watermark has been declared. When processing a table's row, this timestamp can be accessed via
+ * {@link TimeContext#time()} and the watermark via {@link TimeContext#currentWatermark()}
+ * respectively.
+ *
+ * <p>Specifying an {@code on_time} argument in the function call instructs the framework to return
+ * a {@code rowtime} column in the function's output for subsequent time-based operations.
+ *
+ * <p>The {@link ArgumentTrait#REQUIRE_ON_TIME} makes the {@code on_time} argument mandatory if
+ * necessary.
+ *
+ * <h2>Timers</h2>
+ *
+ * <p>A PTF that takes set semantic tables can support timers. Timers allow for continuing the
+ * processing at a later point in time. This makes waiting, synchronization, or timeouts possible. A timer
+ * fires for the registered time when the watermark progresses the logical clock.
+ *
+ * <p>Timers can be named ({@link TimeContext#registerOnTime(String, Object)}) or unnamed ({@link
+ * TimeContext#registerOnTime(Object)}). The name of a timer can be useful for replacing or deleting
+ * an existing timer, or for identifying multiple timers via {@link OnTimerContext#currentTimer()} when they fire.
+ *
+ * <p>An {@code onTimer()} method must be declared next to the eval() method for reacting to timer
+ * events. The signature of the onTimer() method must contain an optional {@link OnTimerContext} followed by all
+ * state entries (as declared in the eval() method).
+ *
+ * <p>Flink takes care of storing and restoring timers during failures or restarts. Thus, timers are
+ * a special kind of state. Similarly, timers are scoped to a virtual processor defined by the
+ * PARTITION BY clause. A timer can only be registered and deleted in the current virtual processor.
+ *
+ * <pre>{@code
+ * // a function that waits for a second event or timeouts after 60 seconds
+ * class TimerFunction extends ProcessTableFunction<String> {
+ *   public static class SeenState {
+ *     public String seen = null;
+ *   }
+ *
+ *   public void eval(
+ *       Context ctx,
+ *       @StateHint SeenState memory,
+ *       @ArgumentHint( { TABLE_AS_SET, REQUIRE_ON_TIME } ) Row input) {
+ *     TimeContext<Instant> timeCtx = ctx.timeContext(Instant.class);
+ *     if (memory.seen == null) {
+ *       memory.seen = input.getField(0).toString();
+ *       timeCtx.registerOnTimer("timeout", timeCtx.time().plusSeconds(60));
+ *     } else {
+ *       collect("Second event arrived for: " + memory.seen)
+ *       ctx.clearAll();
+ *     }
+ *   }
+ *
+ *   public void onTimer(SeenState memory) {
+ *     collect("Timeout for: " + memory.seen)
+ *   }
+ * }
+ *
+ * <h2>Efficiency and Design Principles</h2>
+ *
+ * <p>Registering too many timers might affect performance. An ever-growing timer state can happen
+ * by an unlimited number of partitions (i.e. an open keyspace) or even within a partition. Thus,

Review Comment:
   Out of curiosity: is there any exporting metric showing amount of timers
   or are there plans to add something like this?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37302][table] Support PTFs with time and timer services [flink]

Posted by "twalthr (via GitHub)" <gi...@apache.org>.
twalthr commented on code in PR #26250:
URL: https://github.com/apache/flink/pull/26250#discussion_r1983424932


##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/ProcessTableFunction.java:
##########
@@ -351,9 +439,158 @@ public interface Context {
         /**
          * Clears all state entries within the virtual partition once the eval() method returns.
          *
-         * <p>Semantically this is equal to calling {@link #clearState(String)} on all state
+         * <p>Semantically, this is equal to calling {@link #clearState(String)} on all state
          * entries.
          */
         void clearAllState();
+
+        /** Clears all timers within the virtual partition. */
+        void clearAllTimers();
+
+        /** Clears the virtual partition including timers and state. */
+        void clearAll();
+    }
+
+    /**
+     * A context that gives access to Flink's concepts of time and timers.
+     *
+     * <p>An event can have an event-time timestamp assigned. The timestamp can be accessed using
+     * the {@link #time()} method.
+     *
+     * <p>Timers allow for continuing the processing at a later point in time. This makes waiting,
+     * synchronization, or timeouts possible. A timer fires for the registered time when the
+     * watermark progresses the logical clock.
+     *
+     * <p>Flink takes care of storing and restoring timers during failures or restarts. Thus, timers
+     * are a special kind of state. Similarly, timers are scoped to a virtual processor defined by
+     * the PARTITION BY clause. A timer can only be registered and deleted in the current virtual
+     * processor.
+     *
+     * @param <TimeType> conversion class of timestamps, see {@link Context#timeContext(Class)}
+     */
+    @PublicEvolving
+    public interface TimeContext<TimeType> {
+
+        /**
+         * Returns the timestamp of the currently processed event.
+         *
+         * <p>An event can be either the row of a table or a firing timer:
+         *
+         * <h1>Row event timestamp</h1>
+         *
+         * <p>The timestamp of the row currently being processed within the {@code eval()} method.
+         *
+         * <p>Powered by the function call's {@code on_time} argument, this method will return the
+         * content of the referenced time attribute column. Returns {@code null} if the {@code
+         * on_time} argument doesn't reference a time attribute column in the currently processed
+         * table.
+         *
+         * <h1>Timer event timestamp</h1>
+         *
+         * <p>The timestamp of the firing timer currently being processed within the {@code
+         * onTimer()} method.
+         *
+         * @return the event-time timestamp, or {@code null} if no timestamp is present
+         */
+        TimeType time();
+
+        /**
+         * Returns the current event-time watermark.
+         *
+         * <p>Watermarks are generated in sources and sent through the topology for advancing the
+         * logical clock in each Flink subtask. The current watermark of a Flink subtask is the
+         * global minimum watermark of all inputs (i.e. across all parallel inputs and table
+         * partitions).
+         *
+         * <p>This method returns the current watermark of the Flink subtask that evaluates the PTF.
+         * Thus, the returned timestamp represents the entire Flink subtask, independent of the
+         * currently processed partition. This behavior is similar to a call to {@code SELECT
+         * CURRENT_WATERMARK(...)} in SQL.
+         *
+         * <p>If a watermark was not received from all inputs, the method returns {@code null}.
+         *
+         * <p>In case this method is called within the {@code onTimer()} method, the returned
+         * watermark is the triggering watermark that currently fires the timer.
+         *
+         * @return the current watermark of the Flink subtask, or {@code null} if no common logical
+         *     time could be determined from the inputs
+         */
+        TimeType currentWatermark();
+
+        /**
+         * Registers a timer under the given name.
+         *
+         * <p>The timer fires when the {@link #currentWatermark()} advances the logical clock of the
+         * Flink subtask to a timestamp later or equal to the desired timestamp. In other words: A
+         * timer only fires if a watermark was received from all inputs and the timestamp is smaller
+         * or equal to the minimum of all received watermarks.
+         *
+         * <p>Timers can be named for distinguishing them in the {@code onTimer()} method.
+         * Registering a timer under the same name twice will replace an existing timer.

Review Comment:
   In general this is a valid request. However, this was not in the FLIP and it might cause issues when implementing async PTFs. Those would need to communicate back just for this information. Currently an async PTF implementation could simply collect the modifications and send them back after the eval() returns.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37302][table] Support PTFs with time and timer services [flink]

Posted by "twalthr (via GitHub)" <gi...@apache.org>.
twalthr commented on code in PR #26250:
URL: https://github.com/apache/flink/pull/26250#discussion_r1983419808


##########
flink-table/flink-table-api-java/src/test/java/org/apache/flink/table/test/program/TestStep.java:
##########
@@ -50,6 +50,7 @@ enum TestKind {
         SINK_WITHOUT_DATA,
         SINK_WITH_DATA,
         SINK_WITH_RESTORE_DATA,
+        FAILING_SQL

Review Comment:
   I generalized the approach. But would start with validation and runtime for now. All other exception types rather hint to a wrong exception design.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37302][table] Support PTFs with time and timer services [flink]

Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #26250:
URL: https://github.com/apache/flink/pull/26250#discussion_r1980888148


##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/ProcessTableFunction.java:
##########
@@ -285,6 +288,76 @@
  * }
  * }</pre>
  *
+ * <h1>Time and Timers</h1>
+ *
+ * <p>A PTF supports event time natively. Time-based services are available via {@link
+ * Context#timeContext(Class)}.
+ *
+ * <h2>Time</h2>
+ *
+ * <p>Every PTF takes an optional {@code on_time} argument. The {@code on_time} argument in the function call declares the time attribute column for which
+ * a watermark has been declared. When processing a table's row, this timestamp can be accessed via
+ * {@link TimeContext#time()} and the watermark via {@link TimeContext#currentWatermark()}
+ * respectively.
+ *
+ * <p>Specifying an {@code on_time} argument in the function call instructs the framework to return
+ * a {@code rowtime} column in the function's output for subsequent time-based operations.
+ *
+ * <p>The {@link ArgumentTrait#REQUIRE_ON_TIME} makes the {@code on_time} argument mandatory if
+ * necessary.
+ *
+ * <h2>Timers</h2>
+ *
+ * <p>A PTF that takes set semantic tables can support timers. Timers allow for continuing the
+ * processing at a later point in time. This makes waiting, synchronization, or timeouts possible. A timer
+ * fires for the registered time when the watermark progresses the logical clock.
+ *
+ * <p>Timers can be named ({@link TimeContext#registerOnTime(String, Object)}) or unnamed ({@link
+ * TimeContext#registerOnTime(Object)}). The name of a timer can be useful for replacing or deleting
+ * an existing timer, or for identifying multiple timers via {@link OnTimerContext#currentTimer()} when they fire.
+ *
+ * <p>An {@code onTimer()} method must be declared next to the eval() method for reacting to timer
+ * events. The signature of the onTimer() method must contain an optional {@link OnTimerContext} followed by all
+ * state entries (as declared in the eval() method).
+ *
+ * <p>Flink takes care of storing and restoring timers during failures or restarts. Thus, timers are
+ * a special kind of state. Similarly, timers are scoped to a virtual processor defined by the
+ * PARTITION BY clause. A timer can only be registered and deleted in the current virtual processor.
+ *
+ * <pre>{@code
+ * // a function that waits for a second event or timeouts after 60 seconds
+ * class TimerFunction extends ProcessTableFunction<String> {
+ *   public static class SeenState {
+ *     public String seen = null;
+ *   }
+ *
+ *   public void eval(
+ *       Context ctx,
+ *       @StateHint SeenState memory,
+ *       @ArgumentHint( { TABLE_AS_SET, REQUIRE_ON_TIME } ) Row input) {
+ *     TimeContext<Instant> timeCtx = ctx.timeContext(Instant.class);
+ *     if (memory.seen == null) {
+ *       memory.seen = input.getField(0).toString();
+ *       timeCtx.registerOnTimer("timeout", timeCtx.time().plusSeconds(60));
+ *     } else {
+ *       collect("Second event arrived for: " + memory.seen)
+ *       ctx.clearAll();
+ *     }
+ *   }
+ *
+ *   public void onTimer(SeenState memory) {
+ *     collect("Timeout for: " + memory.seen)
+ *   }
+ * }
+ *
+ * <h2>Efficiency and Design Principles</h2>
+ *
+ * <p>Registering too many timers might affect performance. An ever-growing timer state can happen
+ * by an unlimited number of partitions (i.e. an open keyspace) or even within a partition. Thus,

Review Comment:
   Out of curiosity: is there any exported metric showing amount of timers
   or are there plans to add something like this?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37302][table] Support PTFs with time and timer services [flink]

Posted by "twalthr (via GitHub)" <gi...@apache.org>.
twalthr commented on code in PR #26250:
URL: https://github.com/apache/flink/pull/26250#discussion_r1983420963


##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/ProcessTableFunction.java:
##########
@@ -285,6 +288,76 @@
  * }
  * }</pre>
  *
+ * <h1>Time and Timers</h1>
+ *
+ * <p>A PTF supports event time natively. Time-based services are available via {@link
+ * Context#timeContext(Class)}.
+ *
+ * <h2>Time</h2>
+ *
+ * <p>Every PTF takes an optional {@code on_time} argument. The {@code on_time} argument in the function call declares the time attribute column for which
+ * a watermark has been declared. When processing a table's row, this timestamp can be accessed via
+ * {@link TimeContext#time()} and the watermark via {@link TimeContext#currentWatermark()}
+ * respectively.
+ *
+ * <p>Specifying an {@code on_time} argument in the function call instructs the framework to return
+ * a {@code rowtime} column in the function's output for subsequent time-based operations.
+ *
+ * <p>The {@link ArgumentTrait#REQUIRE_ON_TIME} makes the {@code on_time} argument mandatory if
+ * necessary.
+ *
+ * <h2>Timers</h2>
+ *
+ * <p>A PTF that takes set semantic tables can support timers. Timers allow for continuing the
+ * processing at a later point in time. This makes waiting, synchronization, or timeouts possible. A timer
+ * fires for the registered time when the watermark progresses the logical clock.
+ *
+ * <p>Timers can be named ({@link TimeContext#registerOnTime(String, Object)}) or unnamed ({@link
+ * TimeContext#registerOnTime(Object)}). The name of a timer can be useful for replacing or deleting
+ * an existing timer, or for identifying multiple timers via {@link OnTimerContext#currentTimer()} when they fire.
+ *
+ * <p>An {@code onTimer()} method must be declared next to the eval() method for reacting to timer
+ * events. The signature of the onTimer() method must contain an optional {@link OnTimerContext} followed by all
+ * state entries (as declared in the eval() method).
+ *
+ * <p>Flink takes care of storing and restoring timers during failures or restarts. Thus, timers are
+ * a special kind of state. Similarly, timers are scoped to a virtual processor defined by the
+ * PARTITION BY clause. A timer can only be registered and deleted in the current virtual processor.
+ *
+ * <pre>{@code
+ * // a function that waits for a second event or timeouts after 60 seconds
+ * class TimerFunction extends ProcessTableFunction<String> {
+ *   public static class SeenState {
+ *     public String seen = null;
+ *   }
+ *
+ *   public void eval(
+ *       Context ctx,
+ *       @StateHint SeenState memory,
+ *       @ArgumentHint( { TABLE_AS_SET, REQUIRE_ON_TIME } ) Row input) {
+ *     TimeContext<Instant> timeCtx = ctx.timeContext(Instant.class);
+ *     if (memory.seen == null) {
+ *       memory.seen = input.getField(0).toString();
+ *       timeCtx.registerOnTimer("timeout", timeCtx.time().plusSeconds(60));
+ *     } else {
+ *       collect("Second event arrived for: " + memory.seen)
+ *       ctx.clearAll();
+ *     }
+ *   }
+ *
+ *   public void onTimer(SeenState memory) {
+ *     collect("Timeout for: " + memory.seen)
+ *   }
+ * }
+ *
+ * <h2>Efficiency and Design Principles</h2>
+ *
+ * <p>Registering too many timers might affect performance. An ever-growing timer state can happen
+ * by an unlimited number of partitions (i.e. an open keyspace) or even within a partition. Thus,

Review Comment:
   I don't know but if it is missing, then its a StreamOperator responsibility.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37302][table] Support PTFs with time and timer services [flink]

Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #26250:
URL: https://github.com/apache/flink/pull/26250#discussion_r1980914361


##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/SystemTypeInference.java:
##########
@@ -275,6 +294,149 @@ private List<Field> deriveFunctionOutputFields(DataType functionDataType) {
                     .mapToObj(pos -> DataTypes.FIELD(fieldNames.get(pos), fieldTypes.get(pos)))
                     .collect(Collectors.toList());
         }
+
+        private List<Field> deriveRowtimeField(CallContext callContext) {
+            if (this.functionKind != FunctionKind.PROCESS_TABLE) {
+                return List.of();
+            }
+            final List<DataType> args = callContext.getArgumentDataTypes();
+
+            // Check if on_time is defined and non-empty
+            final int onTimePos = args.size() - 2;
+            final Set<String> onTimeFields =
+                    callContext
+                            .getArgumentValue(onTimePos, ColumnList.class)
+                            .map(ColumnList::getNames)
+                            .map(Set::copyOf)
+                            .orElse(Set.of());
+
+            final Set<String> usedOnTimeFields = new HashSet<>();
+
+            final List<LogicalType> onTimeColumns =
+                    IntStream.range(0, staticArgs.size())
+                            .mapToObj(
+                                    pos -> {
+                                        final StaticArgument staticArg = staticArgs.get(pos);
+                                        if (!staticArg.is(StaticArgumentTrait.TABLE)) {
+                                            return null;
+                                        }
+                                        final RowType rowType =
+                                                (RowType) args.get(pos).getLogicalType();
+                                        final int onTimeColumn =
+                                                findUniqueOnTimeColumn(
+                                                        staticArg.getName(), rowType, onTimeFields);
+                                        if (onTimeColumn >= 0) {
+                                            usedOnTimeFields.add(
+                                                    rowType.getFieldNames().get(onTimeColumn));
+                                            return rowType.getTypeAt(onTimeColumn);
+                                        }
+                                        if (staticArg.is(StaticArgumentTrait.REQUIRE_ON_TIME)) {
+                                            throw new ValidationException(
+                                                    String.format(
+                                                            "Table argument '%s' requires a time attribute. "
+                                                                    + "Please provide one using the implicit `on_time` argument. "
+                                                                    + "For example: myFunction(..., on_time => DESCRIPTOR(`my_timestamp`)",
+                                                            staticArg.getName()));
+                                        }
+                                        return null;
+                                    })
+                            .filter(Objects::nonNull)
+                            .collect(Collectors.toList());
+
+            final Set<String> unusedOnTimeFields = new HashSet<>(onTimeFields);
+            unusedOnTimeFields.removeAll(usedOnTimeFields);
+            if (!unusedOnTimeFields.isEmpty()) {
+                throw new ValidationException(
+                        "Invalid time attribute declaration. "
+                                + "Each column in the `on_time` argument must reference at least one "
+                                + "column in one of the table arguments. Unknown references: "
+                                + unusedOnTimeFields);
+            }
+
+            if (onTimeColumns.isEmpty()) {
+                return List.of();
+            }
+
+            // Don't allow mixtures of time attribute roots
+            if (onTimeColumns.stream().map(LogicalType::getTypeRoot).distinct().count() > 1) {
+                throw new ValidationException(
+                        "Invalid time attribute declaration. "
+                                + "All columns in the `on_time` argument must reference the same data type kind.");

Review Comment:
   should we be more verbose here putting faced types in exception message?



##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/SystemTypeInference.java:
##########
@@ -275,6 +294,149 @@ private List<Field> deriveFunctionOutputFields(DataType functionDataType) {
                     .mapToObj(pos -> DataTypes.FIELD(fieldNames.get(pos), fieldTypes.get(pos)))
                     .collect(Collectors.toList());
         }
+
+        private List<Field> deriveRowtimeField(CallContext callContext) {
+            if (this.functionKind != FunctionKind.PROCESS_TABLE) {
+                return List.of();
+            }
+            final List<DataType> args = callContext.getArgumentDataTypes();
+
+            // Check if on_time is defined and non-empty
+            final int onTimePos = args.size() - 2;
+            final Set<String> onTimeFields =
+                    callContext
+                            .getArgumentValue(onTimePos, ColumnList.class)
+                            .map(ColumnList::getNames)
+                            .map(Set::copyOf)
+                            .orElse(Set.of());
+
+            final Set<String> usedOnTimeFields = new HashSet<>();
+
+            final List<LogicalType> onTimeColumns =
+                    IntStream.range(0, staticArgs.size())
+                            .mapToObj(
+                                    pos -> {
+                                        final StaticArgument staticArg = staticArgs.get(pos);
+                                        if (!staticArg.is(StaticArgumentTrait.TABLE)) {
+                                            return null;
+                                        }
+                                        final RowType rowType =
+                                                (RowType) args.get(pos).getLogicalType();
+                                        final int onTimeColumn =
+                                                findUniqueOnTimeColumn(
+                                                        staticArg.getName(), rowType, onTimeFields);
+                                        if (onTimeColumn >= 0) {
+                                            usedOnTimeFields.add(
+                                                    rowType.getFieldNames().get(onTimeColumn));
+                                            return rowType.getTypeAt(onTimeColumn);
+                                        }
+                                        if (staticArg.is(StaticArgumentTrait.REQUIRE_ON_TIME)) {
+                                            throw new ValidationException(
+                                                    String.format(
+                                                            "Table argument '%s' requires a time attribute. "
+                                                                    + "Please provide one using the implicit `on_time` argument. "
+                                                                    + "For example: myFunction(..., on_time => DESCRIPTOR(`my_timestamp`)",
+                                                            staticArg.getName()));
+                                        }
+                                        return null;
+                                    })
+                            .filter(Objects::nonNull)
+                            .collect(Collectors.toList());
+
+            final Set<String> unusedOnTimeFields = new HashSet<>(onTimeFields);
+            unusedOnTimeFields.removeAll(usedOnTimeFields);
+            if (!unusedOnTimeFields.isEmpty()) {
+                throw new ValidationException(
+                        "Invalid time attribute declaration. "
+                                + "Each column in the `on_time` argument must reference at least one "
+                                + "column in one of the table arguments. Unknown references: "
+                                + unusedOnTimeFields);
+            }
+
+            if (onTimeColumns.isEmpty()) {
+                return List.of();
+            }
+
+            // Don't allow mixtures of time attribute roots
+            if (onTimeColumns.stream().map(LogicalType::getTypeRoot).distinct().count() > 1) {
+                throw new ValidationException(
+                        "Invalid time attribute declaration. "
+                                + "All columns in the `on_time` argument must reference the same data type kind.");

Review Comment:
   I wonder whether we should be more verbose here putting faced types in exception message?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37302][table] Support PTFs with time and timer services [flink]

Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #26250:
URL: https://github.com/apache/flink/pull/26250#issuecomment-2700117306

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "6c348b520bf423faf396b024914295bf71ddd911",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6c348b520bf423faf396b024914295bf71ddd911",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 6c348b520bf423faf396b024914295bf71ddd911 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37302][table] Support PTFs with time and timer services [flink]

Posted by "twalthr (via GitHub)" <gi...@apache.org>.
twalthr merged PR #26250:
URL: https://github.com/apache/flink/pull/26250


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37302][table] Support PTFs with time and timer services [flink]

Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #26250:
URL: https://github.com/apache/flink/pull/26250#discussion_r1980859071


##########
flink-table/flink-table-api-java/src/test/java/org/apache/flink/table/test/program/FailingSqlTestStep.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.test.program;
+
+import org.apache.flink.table.api.TableEnvironment;
+import org.apache.flink.table.api.TableRuntimeException;
+
+import static org.apache.flink.core.testutils.FlinkAssertions.anyCauseMatches;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+/**
+ * Test step for executing SQL that will fail eventually with a {@link TableRuntimeException}..

Review Comment:
   ```suggestion
    * Test step for executing SQL that will fail eventually with a {@link TableRuntimeException}.
   ```
   One dot?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37302][table] Support PTFs with time and timer services [flink]

Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #26250:
URL: https://github.com/apache/flink/pull/26250#discussion_r1980901441


##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/ProcessTableFunction.java:
##########
@@ -351,9 +439,158 @@ public interface Context {
         /**
          * Clears all state entries within the virtual partition once the eval() method returns.
          *
-         * <p>Semantically this is equal to calling {@link #clearState(String)} on all state
+         * <p>Semantically, this is equal to calling {@link #clearState(String)} on all state
          * entries.
          */
         void clearAllState();
+
+        /** Clears all timers within the virtual partition. */
+        void clearAllTimers();
+
+        /** Clears the virtual partition including timers and state. */
+        void clearAll();
+    }
+
+    /**
+     * A context that gives access to Flink's concepts of time and timers.
+     *
+     * <p>An event can have an event-time timestamp assigned. The timestamp can be accessed using
+     * the {@link #time()} method.
+     *
+     * <p>Timers allow for continuing the processing at a later point in time. This makes waiting,
+     * synchronization, or timeouts possible. A timer fires for the registered time when the
+     * watermark progresses the logical clock.
+     *
+     * <p>Flink takes care of storing and restoring timers during failures or restarts. Thus, timers
+     * are a special kind of state. Similarly, timers are scoped to a virtual processor defined by
+     * the PARTITION BY clause. A timer can only be registered and deleted in the current virtual
+     * processor.
+     *
+     * @param <TimeType> conversion class of timestamps, see {@link Context#timeContext(Class)}
+     */
+    @PublicEvolving
+    public interface TimeContext<TimeType> {
+
+        /**
+         * Returns the timestamp of the currently processed event.
+         *
+         * <p>An event can be either the row of a table or a firing timer:
+         *
+         * <h1>Row event timestamp</h1>
+         *
+         * <p>The timestamp of the row currently being processed within the {@code eval()} method.
+         *
+         * <p>Powered by the function call's {@code on_time} argument, this method will return the
+         * content of the referenced time attribute column. Returns {@code null} if the {@code
+         * on_time} argument doesn't reference a time attribute column in the currently processed
+         * table.
+         *
+         * <h1>Timer event timestamp</h1>
+         *
+         * <p>The timestamp of the firing timer currently being processed within the {@code
+         * onTimer()} method.
+         *
+         * @return the event-time timestamp, or {@code null} if no timestamp is present
+         */
+        TimeType time();
+
+        /**
+         * Returns the current event-time watermark.
+         *
+         * <p>Watermarks are generated in sources and sent through the topology for advancing the
+         * logical clock in each Flink subtask. The current watermark of a Flink subtask is the
+         * global minimum watermark of all inputs (i.e. across all parallel inputs and table
+         * partitions).
+         *
+         * <p>This method returns the current watermark of the Flink subtask that evaluates the PTF.
+         * Thus, the returned timestamp represents the entire Flink subtask, independent of the
+         * currently processed partition. This behavior is similar to a call to {@code SELECT
+         * CURRENT_WATERMARK(...)} in SQL.
+         *
+         * <p>If a watermark was not received from all inputs, the method returns {@code null}.
+         *
+         * <p>In case this method is called within the {@code onTimer()} method, the returned
+         * watermark is the triggering watermark that currently fires the timer.
+         *
+         * @return the current watermark of the Flink subtask, or {@code null} if no common logical
+         *     time could be determined from the inputs
+         */
+        TimeType currentWatermark();
+
+        /**
+         * Registers a timer under the given name.
+         *
+         * <p>The timer fires when the {@link #currentWatermark()} advances the logical clock of the
+         * Flink subtask to a timestamp later or equal to the desired timestamp. In other words: A
+         * timer only fires if a watermark was received from all inputs and the timestamp is smaller
+         * or equal to the minimum of all received watermarks.
+         *
+         * <p>Timers can be named for distinguishing them in the {@code onTimer()} method.
+         * Registering a timer under the same name twice will replace an existing timer.
+         *
+         * <p>Note: Because only PTFs taking set semantic tables support state, and timers are a
+         * special kind of state, at least one {@link ArgumentTrait#TABLE_AS_SET} table argument
+         * must be declared.
+         *
+         * @param name identifier of the timer
+         * @param time timestamp when the timer should fire
+         */
+        void registerOnTime(String name, TimeType time);
+
+        /**
+         * Registers a timer.
+         *
+         * <p>The timer fires when the {@link #currentWatermark()} advances the logical clock of the
+         * Flink subtask to a timestamp later or equal to the desired timestamp. In other words: A
+         * timer only fires if a watermark was received from all inputs and the timestamp is smaller
+         * or equal to the minimum of all received watermarks.
+         *
+         * <p>Only one timer can be registered for a given time.
+         *
+         * <p>Note: Because only PTFs taking set semantic tables support state, and timers are a
+         * special kind of state, at least one {@link ArgumentTrait#TABLE_AS_SET} table argument
+         * must be declared.
+         *
+         * @param time timestamp when the timer should fire
+         */
+        void registerOnTime(TimeType time);
+
+        /**
+         * Clears a timer that was previously registered under the given name.
+         *
+         * <p>The call is ignored if no timer can be found.
+         *
+         * @param name identifier of the timer
+         */
+        void clearTimer(String name);

Review Comment:
   Similar question about returning the value associated with `name` like in case of `Map#remove` for instance



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37302][table] Support PTFs with time and timer services [flink]

Posted by "twalthr (via GitHub)" <gi...@apache.org>.
twalthr commented on code in PR #26250:
URL: https://github.com/apache/flink/pull/26250#discussion_r1983146115


##########
flink-table/flink-table-api-java/src/test/java/org/apache/flink/table/test/program/FailingSqlTestStep.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.test.program;
+
+import org.apache.flink.table.api.TableEnvironment;
+import org.apache.flink.table.api.TableRuntimeException;
+
+import static org.apache.flink.core.testutils.FlinkAssertions.anyCauseMatches;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+/**
+ * Test step for executing SQL that will fail eventually with a {@link TableRuntimeException}..
+ *
+ * <p>Note: Not every runner supports generic SQL statements. Sometimes the runner would like to
+ * enrich properties e.g. of a CREATE TABLE. Use this step with caution.

Review Comment:
   This is what `org.apache.flink.table.test.program.TableTestProgramRunner#supportedSetupSteps` and `org.apache.flink.table.test.program.TableTestProgramRunner#supportedRunSteps` is good for.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37302][table] Support PTFs with time and timer services [flink]

Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #26250:
URL: https://github.com/apache/flink/pull/26250#discussion_r1980862517


##########
flink-table/flink-table-api-java/src/test/java/org/apache/flink/table/test/program/FailingSqlTestStep.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.test.program;
+
+import org.apache.flink.table.api.TableEnvironment;
+import org.apache.flink.table.api.TableRuntimeException;
+
+import static org.apache.flink.core.testutils.FlinkAssertions.anyCauseMatches;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+/**
+ * Test step for executing SQL that will fail eventually with a {@link TableRuntimeException}..
+ *
+ * <p>Note: Not every runner supports generic SQL statements. Sometimes the runner would like to
+ * enrich properties e.g. of a CREATE TABLE. Use this step with caution.

Review Comment:
   I wonder if there is a way to know the answer whether runner supports generic SQL statements or not only by looking at code/comments or there is any api which could help to answer such question?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37302][table] Support PTFs with time and timer services [flink]

Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #26250:
URL: https://github.com/apache/flink/pull/26250#discussion_r1980897961


##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/ProcessTableFunction.java:
##########
@@ -351,9 +439,158 @@ public interface Context {
         /**
          * Clears all state entries within the virtual partition once the eval() method returns.
          *
-         * <p>Semantically this is equal to calling {@link #clearState(String)} on all state
+         * <p>Semantically, this is equal to calling {@link #clearState(String)} on all state
          * entries.
          */
         void clearAllState();
+
+        /** Clears all timers within the virtual partition. */
+        void clearAllTimers();
+
+        /** Clears the virtual partition including timers and state. */
+        void clearAll();
+    }
+
+    /**
+     * A context that gives access to Flink's concepts of time and timers.
+     *
+     * <p>An event can have an event-time timestamp assigned. The timestamp can be accessed using
+     * the {@link #time()} method.
+     *
+     * <p>Timers allow for continuing the processing at a later point in time. This makes waiting,
+     * synchronization, or timeouts possible. A timer fires for the registered time when the
+     * watermark progresses the logical clock.
+     *
+     * <p>Flink takes care of storing and restoring timers during failures or restarts. Thus, timers
+     * are a special kind of state. Similarly, timers are scoped to a virtual processor defined by
+     * the PARTITION BY clause. A timer can only be registered and deleted in the current virtual
+     * processor.
+     *
+     * @param <TimeType> conversion class of timestamps, see {@link Context#timeContext(Class)}
+     */
+    @PublicEvolving
+    public interface TimeContext<TimeType> {
+
+        /**
+         * Returns the timestamp of the currently processed event.
+         *
+         * <p>An event can be either the row of a table or a firing timer:
+         *
+         * <h1>Row event timestamp</h1>
+         *
+         * <p>The timestamp of the row currently being processed within the {@code eval()} method.
+         *
+         * <p>Powered by the function call's {@code on_time} argument, this method will return the
+         * content of the referenced time attribute column. Returns {@code null} if the {@code
+         * on_time} argument doesn't reference a time attribute column in the currently processed
+         * table.
+         *
+         * <h1>Timer event timestamp</h1>
+         *
+         * <p>The timestamp of the firing timer currently being processed within the {@code
+         * onTimer()} method.
+         *
+         * @return the event-time timestamp, or {@code null} if no timestamp is present
+         */
+        TimeType time();
+
+        /**
+         * Returns the current event-time watermark.
+         *
+         * <p>Watermarks are generated in sources and sent through the topology for advancing the
+         * logical clock in each Flink subtask. The current watermark of a Flink subtask is the
+         * global minimum watermark of all inputs (i.e. across all parallel inputs and table
+         * partitions).
+         *
+         * <p>This method returns the current watermark of the Flink subtask that evaluates the PTF.
+         * Thus, the returned timestamp represents the entire Flink subtask, independent of the
+         * currently processed partition. This behavior is similar to a call to {@code SELECT
+         * CURRENT_WATERMARK(...)} in SQL.
+         *
+         * <p>If a watermark was not received from all inputs, the method returns {@code null}.
+         *
+         * <p>In case this method is called within the {@code onTimer()} method, the returned
+         * watermark is the triggering watermark that currently fires the timer.
+         *
+         * @return the current watermark of the Flink subtask, or {@code null} if no common logical
+         *     time could be determined from the inputs
+         */
+        TimeType currentWatermark();
+
+        /**
+         * Registers a timer under the given name.
+         *
+         * <p>The timer fires when the {@link #currentWatermark()} advances the logical clock of the
+         * Flink subtask to a timestamp later or equal to the desired timestamp. In other words: A
+         * timer only fires if a watermark was received from all inputs and the timestamp is smaller
+         * or equal to the minimum of all received watermarks.
+         *
+         * <p>Timers can be named for distinguishing them in the {@code onTimer()} method.
+         * Registering a timer under the same name twice will replace an existing timer.

Review Comment:
   Should it then return for instance `boolean`or timer object indicating whether existing timer was replaced or not? Similar to for example `put` method in `java.util.Map`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org