You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "yunfengzhou-hub (via GitHub)" <gi...@apache.org> on 2023/07/03 10:29:06 UTC

[GitHub] [flink] yunfengzhou-hub opened a new pull request, #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

yunfengzhou-hub opened a new pull request, #22931:
URL: https://github.com/apache/flink/pull/22931

   ## What is the purpose of the change
   
   This pull request adds support that a Flink job can change its checkpointing interval depending on the process backlog status of the job.
   
   
   ## Brief change log
   
   - Adds configuration `execution.checkpointing.interval-during-backlog`
   - Adds public method `SplitEnumeratorContext#setIsProcessingBacklog`
   - Makes HybridSource reports processing backlog depending on source stages
   - Supports changing checkpoint interval based on process backlog in CheckpointCoordinator
   
   ## Verifying this change
   
   This change added tests and can be verified as follows:
   
   - Added integration tests for changing checkpointing intervals.
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): no
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: yes
     - The serializers: no
     - The runtime per-record code paths (performance sensitive): no
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
     - The S3 file system connector: no
   
   ## Documentation
   
     - Does this pull request introduce a new feature? yes
     - If yes, how is the feature documented? docs & JavaDocs
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] lindong28 commented on a diff in pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "lindong28 (via GitHub)" <gi...@apache.org>.
lindong28 commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1278993694


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -183,6 +189,15 @@ public class CheckpointCoordinator {
     /** A handle to the current periodic trigger, to cancel it when necessary. */
     private ScheduledFuture<?> currentPeriodicTrigger;
 
+    /**
+     * The timestamp (via {@link Clock#relativeTimeMillis()}) when the next checkpoint will be
+     * triggered.
+     *
+     * <p>If it's value is {@link Long#MAX_VALUE}, it means there is not a next checkpoint
+     * scheduled.
+     */
+    private volatile long nextCheckpointTriggeringRelativeTime;

Review Comment:
   As discussed offline, we agree it is useful to annotate this variable with `@GuardedBy("lock")`. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] lindong28 commented on a diff in pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "lindong28 (via GitHub)" <gi...@apache.org>.
lindong28 commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1270207481


##########
flink-runtime/src/main/java/org/apache/flink/runtime/operators/coordination/OperatorCoordinatorHolder.java:
##########
@@ -164,6 +167,15 @@ public void lazyInitialize(
         context.lazyInitialize(
                 globalFailureHandler, mainThreadExecutor, operatorCoordinatorMetricGroup);
 
+        OperatorCoordinator rootCoordinator = coordinator;
+        if (coordinator instanceof RecreateOnResetOperatorCoordinator) {
+            rootCoordinator =
+                    ((RecreateOnResetOperatorCoordinator) rootCoordinator).getInternalCoordinator();
+        }
+        if (rootCoordinator instanceof SourceCoordinator) {
+            ((SourceCoordinator<?, ?>) rootCoordinator).lazyInitialize(checkpointCoordinator);

Review Comment:
   I left another comment regarding how to simplify the implementation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] lindong28 commented on a diff in pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "lindong28 (via GitHub)" <gi...@apache.org>.
lindong28 commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1271414814


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -1963,7 +2020,14 @@ public void startCheckpointScheduler() {
             stopCheckpointScheduler();
 
             periodicScheduling = true;
-            currentPeriodicTrigger = scheduleTriggerWithDelay(getRandomInitDelay());
+            if (baseIntervalDuringBacklog != baseInterval) {

Review Comment:
   The code that enforces the checkpoint-interval-during-backlog is way more complicated than it should be. For example, it would not be necessary to have two parallel periodic tasks scheduled at the same time.
   
   It should be much simpler and more readable to do the following:
   
   ```
   - Use a variable to maintain the absolute time of last checkpoint.
   - Let method `getNextCheckpointDelay()` return the time to wait from now until we should trigger the next checkpoint. T
   - Let method `scheduleAndMaybeTriggerCheckpoint()` do the following:
     - Invoke `triggerCheckpoint()` if `getNextCheckpointDelay()` <= 0
     - Schedule `scheduleAndMaybeTriggerCheckpoint()` with delay=`getNextCheckpointDelay()`.
   - `startCheckpointScheduler()` should schedule `scheduleAndMaybeTriggerCheckpointI()` with delay=`getNextCheckpointDelay()`
   - Whenever isBacklog status changes, we can cancel the pending scheduled task and re-schedule it based on `getNextCheckpointDelay()`.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] lindong28 merged pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "lindong28 (via GitHub)" <gi...@apache.org>.
lindong28 merged PR #22931:
URL: https://github.com/apache/flink/pull/22931


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-32514] Support configuring checkpointing interval during process backlog [flink]

Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan commented on PR #22931:
URL: https://github.com/apache/flink/pull/22931#issuecomment-3917666220

   I'm curious, was there any reason why this feature was not implemented "outside" of `CheckpointScheduler`?
   
   I mean something like this:
   1. `SourceCoordinator` decides if there's backlog and communicates that to `CheckpointCoordinatorDeActivator`
   2. `CheckpointCoordinatorDeActivator` chooses an appropriate interval and sets it on `CheckpointScheduler`
   3. If it's > 0 then starts scheduling (as it does now)
   4. Once backlog is consumed, `SourceCoordinator` communicates that to `CheckpointCoordinatorDeActivator`
   5. CheckpointCoordinatorDeActivator` stops the scheduler, updates the interval, and starts it again
   
   IMO, that would move much of the complexity out of `CheckpointScheduler` / `CheckpointDecider` and probably avoid race conditions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] lindong28 commented on a diff in pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "lindong28 (via GitHub)" <gi...@apache.org>.
lindong28 commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1271414814


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -1963,7 +2020,14 @@ public void startCheckpointScheduler() {
             stopCheckpointScheduler();
 
             periodicScheduling = true;
-            currentPeriodicTrigger = scheduleTriggerWithDelay(getRandomInitDelay());
+            if (baseIntervalDuringBacklog != baseInterval) {

Review Comment:
   The code that enforces the checkpoint-interval-during-backlog is way more complicated than it should be. The resulting checkpoint trigger time can be wrong (I can explain this offline). And it is not necessary to have two parallel periodic tasks scheduled at the same time.
   
   It should be much simpler and more readable to do the following:
   
   - Let `next-checkpoint-time` denote the absolute time when the next checkpoint is expected to be triggered. Its value should only increase when a checkpoint is triggered.
   - Let method `triggerAndScheduleCheckpoint()` do the following:
     - Invoke `triggerCheckpoint()`.
     - Set `next-checkpoint-time = now + interval`.
     - Schedule `triggerAndScheduleCheckpoint()` with delay = interval.
    - `startCheckpointScheduler()` should schedule `triggerAndScheduleCheckpoint()` with delay=interval.
   - Whenever isBacklog changes from true to false, we should check if `next-checkpoint-time` should be smaller. If yes, update `next-checkpoint-time`, cancel the yet-to-be-scheduled task, and re-schedule `startCheckpointScheduler` with delay = interval-during-backlog.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] lindong28 commented on a diff in pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "lindong28 (via GitHub)" <gi...@apache.org>.
lindong28 commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1271906245


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -426,6 +446,36 @@ public boolean isShutdown() {
         return shutdown;
     }
 
+    /**
+     * Reports whether a source operator is currently processing backlog.
+     *
+     * <p>If any source operator is processing backlog, the checkpoint interval would be decided by
+     * {@code execution.checkpointing.interval-during-backlog} instead of {@code
+     * execution.checkpointing.interval}.
+     *
+     * <p>If a source has not invoked this method, the source is considered to have
+     * isProcessingBacklog=false. If a source operator has invoked this method multiple times, the
+     * last reported value is used.
+     *
+     * @param operatorID the operator ID of the source operator.
+     * @param isProcessingBacklog whether the source operator is processing backlog.
+     */
+    public void setIsProcessingBacklog(OperatorID operatorID, boolean isProcessingBacklog) {
+        if (isProcessingBacklog) {
+            backlogOperators.add(operatorID);
+        } else {
+            backlogOperators.remove(operatorID);
+        }
+
+        if (getCurrentCheckpointInterval() != Long.MAX_VALUE) {
+            long newNextCheckpointTriggeringTime =
+                    clock.absoluteTimeMillis() + getCurrentCheckpointInterval();

Review Comment:
   It is inconsistent to use `absoluteTimeMillis()` here and `relativeTimeMillis()` within `rescheduleTrigger()`.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -1915,6 +1965,14 @@ public CompletedCheckpointStore getCheckpointStore() {
         return completedCheckpointStore;
     }
 
+    /**
+     * Gets the checkpoint interval. Its value might vary depending on whether there is processing
+     * backlog.
+     */
+    private long getCurrentCheckpointInterval() {
+        return backlogOperators.isEmpty() ? baseInterval : baseIntervalDuringBacklog;

Review Comment:
   Can we exclude operators that have finished?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -426,6 +446,36 @@ public boolean isShutdown() {
         return shutdown;
     }
 
+    /**
+     * Reports whether a source operator is currently processing backlog.
+     *
+     * <p>If any source operator is processing backlog, the checkpoint interval would be decided by
+     * {@code execution.checkpointing.interval-during-backlog} instead of {@code
+     * execution.checkpointing.interval}.
+     *
+     * <p>If a source has not invoked this method, the source is considered to have
+     * isProcessingBacklog=false. If a source operator has invoked this method multiple times, the
+     * last reported value is used.
+     *
+     * @param operatorID the operator ID of the source operator.
+     * @param isProcessingBacklog whether the source operator is processing backlog.
+     */
+    public void setIsProcessingBacklog(OperatorID operatorID, boolean isProcessingBacklog) {
+        if (isProcessingBacklog) {
+            backlogOperators.add(operatorID);
+        } else {
+            backlogOperators.remove(operatorID);
+        }
+
+        if (getCurrentCheckpointInterval() != Long.MAX_VALUE) {
+            long newNextCheckpointTriggeringTime =
+                    clock.absoluteTimeMillis() + getCurrentCheckpointInterval();
+            if (newNextCheckpointTriggeringTime < nextCheckpointTriggeringRelativeTime) {
+                rescheduleTrigger(getCurrentCheckpointInterval());

Review Comment:
   Would it be simpler to re-use the `newNextCheckpointTriggeringTime` obtained above rather than getting the timestamp twice (with slightly different values)?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -426,6 +446,36 @@ public boolean isShutdown() {
         return shutdown;
     }
 
+    /**
+     * Reports whether a source operator is currently processing backlog.
+     *
+     * <p>If any source operator is processing backlog, the checkpoint interval would be decided by
+     * {@code execution.checkpointing.interval-during-backlog} instead of {@code
+     * execution.checkpointing.interval}.
+     *
+     * <p>If a source has not invoked this method, the source is considered to have
+     * isProcessingBacklog=false. If a source operator has invoked this method multiple times, the
+     * last reported value is used.
+     *
+     * @param operatorID the operator ID of the source operator.
+     * @param isProcessingBacklog whether the source operator is processing backlog.
+     */
+    public void setIsProcessingBacklog(OperatorID operatorID, boolean isProcessingBacklog) {
+        if (isProcessingBacklog) {
+            backlogOperators.add(operatorID);
+        } else {
+            backlogOperators.remove(operatorID);
+        }
+
+        if (getCurrentCheckpointInterval() != Long.MAX_VALUE) {
+            long newNextCheckpointTriggeringTime =
+                    clock.absoluteTimeMillis() + getCurrentCheckpointInterval();
+            if (newNextCheckpointTriggeringTime < nextCheckpointTriggeringRelativeTime) {

Review Comment:
   Is the following sequence of events possible:
   - newNextCheckpointTriggeringTime = newNextCheckpointTriggeringTime - 1 ms.
   - Right before currentPeriodicTrigger.cancel() is invoked, checkpoint is triggered.
   - Then we schedule/trigger another checkpoint almost immediately after the last checkpoint is triggered.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -181,7 +188,16 @@ public class CheckpointCoordinator {
     private JobStatusListener jobStatusListener;
 
     /** A handle to the current periodic trigger, to cancel it when necessary. */
-    private ScheduledFuture<?> currentPeriodicTrigger;
+    private Future<?> currentPeriodicTrigger;

Review Comment:
   It seems that `currentPeriodicTrigger` is always a `ScheduledFuture`. To avoid back-and-forth code change, would it be better to keep the existing code unless there is a good reason to change it?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -181,7 +188,16 @@ public class CheckpointCoordinator {
     private JobStatusListener jobStatusListener;
 
     /** A handle to the current periodic trigger, to cancel it when necessary. */
-    private ScheduledFuture<?> currentPeriodicTrigger;
+    private Future<?> currentPeriodicTrigger;
+
+    /**
+     * The timestamp (via {@link Clock#relativeTimeMillis()}) when the next checkpoint will be
+     * triggered.
+     *
+     * <p>If it's value is {@link Long#MAX_VALUE}, it means there is not a next checkpoint
+     * scheduled.
+     */
+    private long nextCheckpointTriggeringRelativeTime;

Review Comment:
   Given that `nextCheckpointTriggeringRelativeTime` might be written by different threads, we should mark it as `volatile`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] yunfengzhou-hub commented on a diff in pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "yunfengzhou-hub (via GitHub)" <gi...@apache.org>.
yunfengzhou-hub commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1279088716


##########
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/VertexEndOfDataListener.java:
##########
@@ -51,6 +51,12 @@ public void recordTaskEndOfData(ExecutionAttemptID executionAttemptID) {
         subtaskStatus.set(executionAttemptID.getSubtaskIndex());
     }
 
+    public boolean areAllTasksOfJobVertexEndOfData(JobVertexID jobVertexID) {
+        BitSet subtaskStatus = tasksReachedEndOfData.get(jobVertexID);
+        return subtaskStatus.cardinality()

Review Comment:
   This code path had been covered by CheckpointIntervalDuringBacklogITCase#testExcludeFinishedOperatorBacklogStatus, but it seems to be a flaky test where it depends on the order of vertices returned by Map#iterator whether exceptions would be thrown. I'll fix this bug anyway.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] yunfengzhou-hub commented on a diff in pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "yunfengzhou-hub (via GitHub)" <gi...@apache.org>.
yunfengzhou-hub commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1267763705


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -291,13 +313,27 @@ public CheckpointCoordinator(
 
         // it does not make sense to schedule checkpoints more often then the desired
         // time between checkpoints
-        long baseInterval = chkConfig.getCheckpointInterval();
-        if (baseInterval < minPauseBetweenCheckpoints) {
-            baseInterval = minPauseBetweenCheckpoints;
+        long checkpointInterval = chkConfig.getCheckpointInterval();
+        if (checkpointInterval < minPauseBetweenCheckpoints) {
+            checkpointInterval = minPauseBetweenCheckpoints;
+        }
+
+        long checkpointIntervalDuringBacklog = chkConfig.getCheckpointIntervalDuringBacklog();
+        if (checkpointIntervalDuringBacklog < minPauseBetweenCheckpoints) {
+            checkpointIntervalDuringBacklog = minPauseBetweenCheckpoints;
+        }
+
+        if (checkpointIntervalDuringBacklog < checkpointInterval) {
+            throw new IllegalArgumentException(
+                    "execution.checkpoint.interval-during-backlog must either be 0, "
+                            + "or be larger than or equal to execution.checkpoint.interval.");
         }
 
         this.job = checkNotNull(job);
-        this.baseInterval = baseInterval;
+        this.checkpointInterval = checkpointInterval;
+        this.checkpointIntervalDuringBacklog = checkpointIntervalDuringBacklog;
+        this.lastCheckpointTriggeringRelativeTime = NO_CHECKPOINT;

Review Comment:
   It might be better to initialize this variable into -1L in accordance with `OperatorCoordinator.NO_CHECKPOINT`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-32514] Support configuring checkpointing interval during process backlog [flink]

Posted by "lindong28 (via GitHub)" <gi...@apache.org>.
lindong28 commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1456726081


##########
flink-runtime/src/main/java/org/apache/flink/runtime/operators/coordination/OperatorCoordinatorHolder.java:
##########
@@ -143,12 +144,14 @@ private OperatorCoordinatorHolder(
 
     public void lazyInitialize(
             GlobalFailureHandler globalFailureHandler,
-            ComponentMainThreadExecutor mainThreadExecutor) {
+            ComponentMainThreadExecutor mainThreadExecutor,
+            @Nullable CheckpointCoordinator checkpointCoordinator) {
 
         this.globalFailureHandler = globalFailureHandler;
         this.mainThreadExecutor = mainThreadExecutor;
+        context.lazyInitialize(globalFailureHandler, mainThreadExecutor, checkpointCoordinator);
 
-        context.lazyInitialize(globalFailureHandler, mainThreadExecutor);
+        context.lazyInitialize(globalFailureHandler, mainThreadExecutor, checkpointCoordinator);

Review Comment:
   @XComp Thanks for catching this. No, we didn't do it intentionally. I believe it is introduced when Yunfeng rebased the PR. I didn't catch this issue because I didn't go over the entire PR end-to-end very carefully when there is only minor remaining comments in the last 2 rounds of review.
   
   It seems that the extra invocation of `lazyInitialize()` would not introduce any visible performance or correctness issue. Maybe one of us can fix it in our next PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] yunfengzhou-hub commented on a diff in pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "yunfengzhou-hub (via GitHub)" <gi...@apache.org>.
yunfengzhou-hub commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1267760838


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -2079,11 +2149,19 @@ private final class ScheduledTrigger implements Runnable {
 
         @Override
         public void run() {
+            long currentTime = clock.relativeTimeMillis();
+            if (lastCheckpointTriggeringRelativeTime != NO_CHECKPOINT
+                    && currentTime - lastCheckpointTriggeringRelativeTime < baseInterval) {

Review Comment:
   With the current implementation in which `lastCheckpointTriggeringRelativeTime` could be `Long.MIN_VALUE`, `currentTime - lastCheckpointTriggeringRelativeTime` could cause stack overflow.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] yunfengzhou-hub commented on a diff in pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "yunfengzhou-hub (via GitHub)" <gi...@apache.org>.
yunfengzhou-hub commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1267866662


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -426,6 +462,41 @@ public boolean isShutdown() {
         return shutdown;
     }
 
+    /**
+     * Reports whether a source operator is currently processing backlog.
+     *
+     * <p>If any source operator is processing backlog, the checkpoint interval would be decided by
+     * {@code execution.checkpointing.interval-during-backlog} instead of {@code
+     * execution.checkpointing.interval}.
+     *
+     * <p>If a source has not invoked this method, the source is considered to have
+     * isProcessingBacklog=false. If a source operator has invoked this method multiple times, the
+     * last reported value is used.
+     *
+     * @param operatorID the operator ID of the source operator.
+     * @param isProcessingBacklog whether the source operator is processing backlog.
+     */
+    public void setIsProcessingBacklog(OperatorID operatorID, boolean isProcessingBacklog) {
+        boolean wasBacklog = !backlogOperators.isEmpty();
+        if (isProcessingBacklog) {
+            backlogOperators.add(operatorID);
+        } else {
+            backlogOperators.remove(operatorID);
+        }
+
+        if (wasBacklog && backlogOperators.isEmpty()) {
+            baseInterval = checkpointInterval;
+        } else if (!wasBacklog && !backlogOperators.isEmpty()) {
+            baseInterval = checkpointIntervalDuringBacklog;
+        } else {
+            return;
+        }
+
+        if (baseInterval != Long.MAX_VALUE) {
+            scheduleTriggerWithDelay(getRandomInitDelay());

Review Comment:
   The existing behavior introduce random initial delay only once, partially because it only needs to schedule periodic checkpoint once. Thus I don't think there is a suitable enough existing behavior to follow. Maybe we can discuss offline.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] lindong28 commented on a diff in pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "lindong28 (via GitHub)" <gi...@apache.org>.
lindong28 commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1280153092


##########
flink-runtime/src/main/java/org/apache/flink/runtime/source/event/EndOfDataEvent.java:
##########
@@ -0,0 +1,24 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.source.event;
+
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+
+/** A source event notifying coordinator that a source operator has reached the end of data. */
+public class EndOfDataEvent implements OperatorEvent {}

Review Comment:
   Do we still need this class?



##########
flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinatorTest.java:
##########
@@ -2317,18 +2318,21 @@ private void testMaxConcurrentAttempts(int maxConcurrentAttempts) {
                     new AcknowledgeCheckpoint(graph.getJobID(), attemptID1, 1L),
                     TASK_MANAGER_LOCATION_INFO);
 
-            final Collection<ScheduledFuture<?>> periodicScheduledTasks =
-                    manuallyTriggeredScheduledExecutor.getActivePeriodicScheduledTask();
-            assertThat(periodicScheduledTasks.size()).isOne();
+            final Collection<ScheduledFuture<?>> nonPeriodicScheduledTasks =
+                    manuallyTriggeredScheduledExecutor.getActiveNonPeriodicScheduledRunnable(
+                            CheckpointCoordinator.ScheduledTrigger.class);

Review Comment:
   Passing a class as the parameter to test util method appears very hacky. Is there a way to avoid doing this?
   
   For example, can we just verify that the expected instance of runnable is found in the list of the scheduled tasks?



##########
flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/CheckpointRequestDeciderTest.java:
##########
@@ -284,7 +284,7 @@ private CheckpointRequestDecider decider(
                 maxQueued);
     }
 
-    private static final LongConsumer NO_OP = unused -> {};
+    private static final BiConsumer<Long, Long> NO_OP = (unused, unused2) -> {};

Review Comment:
   It seems hack to name variables as unused2. How about using this:
   
   `(currentTimeMillis, tillNextMillis) -> {}`



##########
flink-runtime/src/main/java/org/apache/flink/runtime/operators/coordination/OperatorCoordinatorHolder.java:
##########
@@ -572,6 +577,7 @@ private static final class LazyInitializedCoordinatorContext
         private GlobalFailureHandler globalFailureHandler;
         private Executor schedulerExecutor;
         private OperatorCoordinatorMetricGroup metricGroup;
+        private CheckpointCoordinator checkpointCoordinator;

Review Comment:
   Given that it can be null, can we mark it nullable?
   
   Same for related method parameter.



##########
flink-core/src/test/java/org/apache/flink/util/concurrent/ManuallyTriggeredScheduledExecutor.java:
##########
@@ -94,6 +94,11 @@ public Collection<ScheduledFuture<?>> getActiveNonPeriodicScheduledTask() {
         return execService.getActiveNonPeriodicScheduledTask();
     }
 
+    public Collection<ScheduledFuture<?>> getActiveNonPeriodicScheduledRunnable(

Review Comment:
   Should we replace `Runnable` with `Task` for consistency with the existing method names?



##########
flink-tests/src/test/java/org/apache/flink/test/util/NumberSequenceSourceBlockableByCheckpoint.java:
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.test.util;
+
+import org.apache.flink.api.connector.source.ReaderOutput;
+import org.apache.flink.api.connector.source.SourceReader;
+import org.apache.flink.api.connector.source.SourceReaderContext;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.flink.api.connector.source.SplitEnumerator;
+import org.apache.flink.api.connector.source.SplitEnumeratorContext;
+import org.apache.flink.api.connector.source.lib.NumberSequenceSource;
+import org.apache.flink.api.connector.source.lib.util.IteratorSourceEnumerator;
+import org.apache.flink.api.connector.source.lib.util.IteratorSourceReader;
+import org.apache.flink.api.connector.source.lib.util.IteratorSourceSplit;
+import org.apache.flink.core.io.InputStatus;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.ArrayDeque;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Queue;
+
+/**
+ * A {@link NumberSequenceSource} that can be blocked by checkpoints. This source can be used to

Review Comment:
   It is hard to understand when/how to use this class by just reading this doc. For example, how do the values of `isBlockByCheckpoint` and `isBacklog` affect the behavior of this Source.
   
   Can you update the doc to clarify its semantics?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-32514] Support configuring checkpointing interval during process backlog [flink]

Posted by "XComp (via GitHub)" <gi...@apache.org>.
XComp commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1455086469


##########
flink-runtime/src/main/java/org/apache/flink/runtime/operators/coordination/OperatorCoordinatorHolder.java:
##########
@@ -143,12 +144,14 @@ private OperatorCoordinatorHolder(
 
     public void lazyInitialize(
             GlobalFailureHandler globalFailureHandler,
-            ComponentMainThreadExecutor mainThreadExecutor) {
+            ComponentMainThreadExecutor mainThreadExecutor,
+            @Nullable CheckpointCoordinator checkpointCoordinator) {
 
         this.globalFailureHandler = globalFailureHandler;
         this.mainThreadExecutor = mainThreadExecutor;
+        context.lazyInitialize(globalFailureHandler, mainThreadExecutor, checkpointCoordinator);
 
-        context.lazyInitialize(globalFailureHandler, mainThreadExecutor);
+        context.lazyInitialize(globalFailureHandler, mainThreadExecutor, checkpointCoordinator);

Review Comment:
   @lindong28 @yunfengzhou-hub Is initializing the `OperatorCoordinatorHolder` twice done deliberately here? :thinking: 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] lindong28 commented on a diff in pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "lindong28 (via GitHub)" <gi...@apache.org>.
lindong28 commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1271381779


##########
flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/tasks/CheckpointCoordinatorConfiguration.java:
##########
@@ -278,6 +290,16 @@ public CheckpointCoordinatorConfiguration build() {
         public CheckpointCoordinatorConfigurationBuilder setCheckpointInterval(
                 long checkpointInterval) {
             this.checkpointInterval = checkpointInterval;
+            if (!isCheckpointIntervalDuringBacklogManuallyConfigured) {
+                this.checkpointIntervalDuringBacklog = checkpointInterval;
+            }
+            return this;
+        }
+
+        public CheckpointCoordinatorConfigurationBuilder setCheckpointIntervalDuringBacklog(
+                long checkpointIntervalDuringBacklog) {
+            this.checkpointIntervalDuringBacklog = checkpointIntervalDuringBacklog;

Review Comment:
   Currently, the interaction between `checkpointInterval` and `checkpointIntervalDuringBacklog` is encoded both here and in `CheckpointConfig#getCheckpointIntervalDuringBacklog`. Can we encode it in only one place?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -296,8 +307,20 @@ public CheckpointCoordinator(
             baseInterval = minPauseBetweenCheckpoints;
         }
 
+        long baseIntervalDuringBacklog = chkConfig.getCheckpointIntervalDuringBacklog();
+        if (baseIntervalDuringBacklog < minPauseBetweenCheckpoints) {

Review Comment:
   This logic is pretty much the same as the logic used for baseInterval. Instead of duplicating this logic here, would it be simpler to to the following:
   
   - Determine the value for checkpoint interval (w.r.t. minPauseBetweenCheckpoints,  min checkpoint interval etc.)
   - If checkpoint-interval-during-backlog is explicitly set, throw exception if it is smaller than the checkpoint interval.
   - Otherwise, set checkpoint-interval-during-backlog to use the same value as checkpoint interval.
   
   And it is preferred to put all these logic in the same method/class.
   
   It might be reasonable to put all the logic of determining checkpoint-interval-during-backlog in `CheckpointConfig#getCheckpointIntervalDuringBacklog`. And we can move the logic of determining checkpoint interval to `CheckpointConfig#getCheckpointInterval` if needed.



##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamingJobGraphGenerator.java:
##########
@@ -1911,6 +1911,12 @@ private void configureCheckpointing() {
             interval = Long.MAX_VALUE;
         }
 
+        long intervalDuringBacklog = cfg.getCheckpointIntervalDuringBacklog();
+        if (intervalDuringBacklog < MINIMAL_CHECKPOINT_TIME) {

Review Comment:
   Given that `intervalDuringBacklog` should always be >= `interval`, do we still need the logic here? 



##########
flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/tasks/CheckpointCoordinatorConfiguration.java:
##########
@@ -119,6 +123,7 @@ private CheckpointCoordinatorConfiguration(
                 "maxConcurrentCheckpoints can't be > 1 if UnalignedCheckpoints enabled");
 
         this.checkpointInterval = checkpointInterval;
+        this.checkpointIntervalDuringBacklog = checkpointIntervalDuringBacklog;

Review Comment:
   Would it be simpler to use Optional<Long> as the type of `checkpointIntervalDuringBacklog` to specify whether it is explicitly set by user?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -1963,7 +2020,14 @@ public void startCheckpointScheduler() {
             stopCheckpointScheduler();
 
             periodicScheduling = true;
-            currentPeriodicTrigger = scheduleTriggerWithDelay(getRandomInitDelay());
+            if (baseIntervalDuringBacklog != baseInterval) {

Review Comment:
   The code that enforces the checkpoint-interval-during-backlog is far more complicated than it should be. For example, it would not be necessary to have two parallel periodic tasks scheduled at the same time.
   
   It should be much simpler and more readable to do the following:
   
   ```
   - Use a variable to maintain the absolute time of last checkpoint.
   - Let method `getNextCheckpointDelay()` return the time to wait from now until we should trigger the next checkpoint. T
   - Let method `scheduleAndMaybeTriggerCheckpoint()` do the following:
     - Invoke `triggerCheckpoint()` if `getNextCheckpointDelay()` <= 0
     - Schedule `scheduleAndMaybeTriggerCheckpoint()` with delay=`getNextCheckpointDelay()`.
   - `startCheckpointScheduler()` should schedule `scheduleAndMaybeTriggerCheckpointI()` with delay=`getNextCheckpointDelay()`
   - Whenever isBacklog status changes, we should cancel the pending scheduled task and re-schedule it based on `getNextCheckpointDelay()`.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] lindong28 commented on a diff in pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "lindong28 (via GitHub)" <gi...@apache.org>.
lindong28 commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1271414814


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -1963,7 +2020,14 @@ public void startCheckpointScheduler() {
             stopCheckpointScheduler();
 
             periodicScheduling = true;
-            currentPeriodicTrigger = scheduleTriggerWithDelay(getRandomInitDelay());
+            if (baseIntervalDuringBacklog != baseInterval) {

Review Comment:
   The code that enforces the checkpoint-interval-during-backlog is way more complicated than it should be. For example, it would not be necessary to have two parallel periodic tasks scheduled at the same time.
   
   It should be much simpler and more readable to do the following:
   
   ```
   - Use a variable to maintain the absolute time of the last successful checkpoint trigger.
   - Let method `getNextCheckpointDelay()` return the time to wait from now until we should trigger the next checkpoint.
   - Let method `scheduleAndMaybeTriggerCheckpoint()` do the following:
     - Invoke `triggerCheckpoint()` if `getNextCheckpointDelay()` <= 0
     - Schedule `scheduleAndMaybeTriggerCheckpoint()` with delay=`getNextCheckpointDelay()`.
   - `startCheckpointScheduler()` should schedule `scheduleAndMaybeTriggerCheckpointI()` with delay=`getNextCheckpointDelay()`
   - Whenever isBacklog status changes, we can cancel the pending scheduled task and re-schedule it based on `getNextCheckpointDelay()`.
   ```
   
   One important advantage of the approach described above is that all the logic of determining checkpoint trigger time can be found in `getNextCheckpointDelay()`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] lindong28 commented on a diff in pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "lindong28 (via GitHub)" <gi...@apache.org>.
lindong28 commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1271414814


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -1963,7 +2020,14 @@ public void startCheckpointScheduler() {
             stopCheckpointScheduler();
 
             periodicScheduling = true;
-            currentPeriodicTrigger = scheduleTriggerWithDelay(getRandomInitDelay());
+            if (baseIntervalDuringBacklog != baseInterval) {

Review Comment:
   The code that enforces the checkpoint-interval-during-backlog is way more complicated than it should be. The resulting checkpoint trigger time can be wrong (I can explain this offline). And it is not necessary to have two parallel periodic tasks scheduled at the same time.
   
   It should be much simpler and more readable to do the following:
   
   ```
   - Use a variable to maintain the absolute time of the last successful checkpoint trigger.
   - Let method `getNextCheckpointDelay()` return the time to wait from now until we should trigger the next checkpoint.
   - Let method `scheduleAndMaybeTriggerCheckpoint()` do the following:
     - Invoke `triggerCheckpoint()` if `getNextCheckpointDelay()` <= 0
     - Schedule `scheduleAndMaybeTriggerCheckpoint()` with delay=`getNextCheckpointDelay()`.
   - `startCheckpointScheduler()` should schedule `scheduleAndMaybeTriggerCheckpointI()` with delay=`getNextCheckpointDelay()`
   - Whenever isBacklog status changes, we can cancel the pending scheduled task and re-schedule it based on `getNextCheckpointDelay()`.
   ```
   
   One important advantage of the approach described above is that all the logic of determining checkpoint trigger time can be found in `getNextCheckpointDelay()`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] lindong28 commented on a diff in pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "lindong28 (via GitHub)" <gi...@apache.org>.
lindong28 commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1271414814


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -1963,7 +2020,14 @@ public void startCheckpointScheduler() {
             stopCheckpointScheduler();
 
             periodicScheduling = true;
-            currentPeriodicTrigger = scheduleTriggerWithDelay(getRandomInitDelay());
+            if (baseIntervalDuringBacklog != baseInterval) {

Review Comment:
   The code that enforces the checkpoint-interval-during-backlog is way more complicated than it should be. The interval is incorrect (I can explain this offline). It is not necessary to have two parallel periodic tasks scheduled at the same time.
   
   It should be much simpler and more readable to do the following:
   
   ```
   - Use a variable to maintain the absolute time of the last successful checkpoint trigger.
   - Let method `getNextCheckpointDelay()` return the time to wait from now until we should trigger the next checkpoint.
   - Let method `scheduleAndMaybeTriggerCheckpoint()` do the following:
     - Invoke `triggerCheckpoint()` if `getNextCheckpointDelay()` <= 0
     - Schedule `scheduleAndMaybeTriggerCheckpoint()` with delay=`getNextCheckpointDelay()`.
   - `startCheckpointScheduler()` should schedule `scheduleAndMaybeTriggerCheckpointI()` with delay=`getNextCheckpointDelay()`
   - Whenever isBacklog status changes, we can cancel the pending scheduled task and re-schedule it based on `getNextCheckpointDelay()`.
   ```
   
   One important advantage of the approach described above is that all the logic of determining checkpoint trigger time can be found in `getNextCheckpointDelay()`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] lindong28 commented on pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "lindong28 (via GitHub)" <gi...@apache.org>.
lindong28 commented on PR #22931:
URL: https://github.com/apache/flink/pull/22931#issuecomment-1691248952

   Thanks for the update. LGTM.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] yunfengzhou-hub commented on a diff in pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "yunfengzhou-hub (via GitHub)" <gi...@apache.org>.
yunfengzhou-hub commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1282534739


##########
flink-tests/src/test/java/org/apache/flink/test/util/NumberSequenceSourceBlockableByCheckpoint.java:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.test.util;
+
+import org.apache.flink.api.connector.source.ReaderOutput;
+import org.apache.flink.api.connector.source.SourceReader;
+import org.apache.flink.api.connector.source.SourceReaderContext;
+import org.apache.flink.api.connector.source.SplitEnumerator;
+import org.apache.flink.api.connector.source.SplitEnumeratorContext;
+import org.apache.flink.api.connector.source.lib.NumberSequenceSource;
+import org.apache.flink.api.connector.source.lib.util.IteratorSourceEnumerator;
+import org.apache.flink.api.connector.source.lib.util.IteratorSourceReader;
+import org.apache.flink.api.connector.source.lib.util.IteratorSourceSplit;
+import org.apache.flink.core.io.InputStatus;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayDeque;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Queue;
+
+/**
+ * A {@link NumberSequenceSource} that can be blocked by checkpoints. If {@link
+ * #isBlockByCheckpoint} is set to true, this source would guarantee that a checkpoint has been
+ * invoked before the source ends its input.
+ */
+public class NumberSequenceSourceBlockableByCheckpoint extends NumberSequenceSource {

Review Comment:
   I have kept `TestingNumberSequenceSource`'s behavior as it had been. But given that this PR is changing this class from a private class to a public class in test utils package, the name `TestingNumberSequenceSource` might not provide enough information about its function. Thus I named it as `NumberSequenceSourceWithWaitForCheckpoint`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] yunfengzhou-hub commented on a diff in pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "yunfengzhou-hub (via GitHub)" <gi...@apache.org>.
yunfengzhou-hub commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1267760838


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -2079,11 +2149,19 @@ private final class ScheduledTrigger implements Runnable {
 
         @Override
         public void run() {
+            long currentTime = clock.relativeTimeMillis();
+            if (lastCheckpointTriggeringRelativeTime != NO_CHECKPOINT
+                    && currentTime - lastCheckpointTriggeringRelativeTime < baseInterval) {

Review Comment:
   With the current implementation in which `lastCheckpointTriggeringRelativeTime` could be `Long.MIN_VALUE`, `currentTime - lastCheckpointTriggeringRelativeTime` could cause stack overflow. Thus I treat `Long.MIN_VALUE` specially here. This would be resolved after I replace the default value with 0 or -1 according to the previous comment.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] yunfengzhou-hub commented on a diff in pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "yunfengzhou-hub (via GitHub)" <gi...@apache.org>.
yunfengzhou-hub commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1267760838


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -2079,11 +2149,19 @@ private final class ScheduledTrigger implements Runnable {
 
         @Override
         public void run() {
+            long currentTime = clock.relativeTimeMillis();
+            if (lastCheckpointTriggeringRelativeTime != NO_CHECKPOINT
+                    && currentTime - lastCheckpointTriggeringRelativeTime < baseInterval) {

Review Comment:
   With the current implementation in which `lastCheckpointTriggeringRelativeTime` could be `Long.MIN_VALUE`, `currentTime - lastCheckpointTriggeringRelativeTime` could cause stack overflow. Thus I treat `Long.MIN_VALUE` specially here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] yunfengzhou-hub commented on a diff in pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "yunfengzhou-hub (via GitHub)" <gi...@apache.org>.
yunfengzhou-hub commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1267763705


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -291,13 +313,27 @@ public CheckpointCoordinator(
 
         // it does not make sense to schedule checkpoints more often then the desired
         // time between checkpoints
-        long baseInterval = chkConfig.getCheckpointInterval();
-        if (baseInterval < minPauseBetweenCheckpoints) {
-            baseInterval = minPauseBetweenCheckpoints;
+        long checkpointInterval = chkConfig.getCheckpointInterval();
+        if (checkpointInterval < minPauseBetweenCheckpoints) {
+            checkpointInterval = minPauseBetweenCheckpoints;
+        }
+
+        long checkpointIntervalDuringBacklog = chkConfig.getCheckpointIntervalDuringBacklog();
+        if (checkpointIntervalDuringBacklog < minPauseBetweenCheckpoints) {
+            checkpointIntervalDuringBacklog = minPauseBetweenCheckpoints;
+        }
+
+        if (checkpointIntervalDuringBacklog < checkpointInterval) {
+            throw new IllegalArgumentException(
+                    "execution.checkpoint.interval-during-backlog must either be 0, "
+                            + "or be larger than or equal to execution.checkpoint.interval.");
         }
 
         this.job = checkNotNull(job);
-        this.baseInterval = baseInterval;
+        this.checkpointInterval = checkpointInterval;
+        this.checkpointIntervalDuringBacklog = checkpointIntervalDuringBacklog;
+        this.lastCheckpointTriggeringRelativeTime = NO_CHECKPOINT;

Review Comment:
   It might be better to initialize this variable into -1L in accordance with `OperatorCoordinator.NO_CHECKPOINT`. I'll change the default value this way for now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] yunfengzhou-hub commented on a diff in pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "yunfengzhou-hub (via GitHub)" <gi...@apache.org>.
yunfengzhou-hub commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1267876016


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -2079,11 +2149,19 @@ private final class ScheduledTrigger implements Runnable {
 
         @Override
         public void run() {
+            long currentTime = clock.relativeTimeMillis();
+            if (lastCheckpointTriggeringRelativeTime != NO_CHECKPOINT
+                    && currentTime - lastCheckpointTriggeringRelativeTime < baseInterval) {
+                return;
+            }
+            lastCheckpointTriggeringRelativeTime = currentTime;
+
             try {
                 triggerCheckpoint(checkpointProperties, null, true);
             } catch (Exception e) {
                 LOG.error("Exception while triggering checkpoint for job {}.", job, e);
             }
+            timer.schedule(this, baseInterval, TimeUnit.MILLISECONDS);

Review Comment:
   I'm not sure I understand the expected behavior clearly. The current implementation ensures that the next checkpoint will be <= T2+ X.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] lindong28 commented on a diff in pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "lindong28 (via GitHub)" <gi...@apache.org>.
lindong28 commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1270193281


##########
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultOperatorCoordinatorHandler.java:
##########
@@ -74,10 +75,14 @@ private static Map<OperatorID, OperatorCoordinatorHolder> createCoordinatorMap(
     @Override
     public void initializeOperatorCoordinators(
             ComponentMainThreadExecutor mainThreadExecutor,
-            JobManagerJobMetricGroup jobManagerJobMetricGroup) {
+            JobManagerJobMetricGroup jobManagerJobMetricGroup,
+            CheckpointCoordinator checkpointCoordinator) {
         for (OperatorCoordinatorHolder coordinatorHolder : coordinatorMap.values()) {
             coordinatorHolder.lazyInitialize(
-                    globalFailureHandler, mainThreadExecutor, jobManagerJobMetricGroup);
+                    globalFailureHandler,
+                    mainThreadExecutor,
+                    jobManagerJobMetricGroup,
+                    checkpointCoordinator);

Review Comment:
   Would it be simpler to use `this.executionGraph.getCheckpointCoordinator()` without having the caller pass `checkpointCoordinator` as parameter?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/operators/coordination/OperatorCoordinatorHolder.java:
##########
@@ -164,6 +167,13 @@ public void lazyInitialize(
         context.lazyInitialize(
                 globalFailureHandler, mainThreadExecutor, operatorCoordinatorMetricGroup);
 
+        if (coordinator instanceof RecreateOnResetOperatorCoordinator) {

Review Comment:
   Instead of passing checkpoint coordinator like this, would it be simpler to do the following:
   
   - `OperatorCoordinatorHolder#lazyInitialize` can pass checkpoint coordinator to `context.lazyInitialize`.
   - Add method `getCheckpointCoordinator` to `OperatorCoordinator#Context`, which can return the checkpoint coordinator obtained in the above step.
   - `SourceCoordinatorContext#setIsProcessingBacklog` can use the checkpoint coordinator obtained from `operatorCoordinatorContext`.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] lindong28 commented on a diff in pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "lindong28 (via GitHub)" <gi...@apache.org>.
lindong28 commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1281616827


##########
flink-tests/src/test/java/org/apache/flink/test/util/NumberSequenceSourceBlockableByCheckpoint.java:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.test.util;
+
+import org.apache.flink.api.connector.source.ReaderOutput;
+import org.apache.flink.api.connector.source.SourceReader;
+import org.apache.flink.api.connector.source.SourceReaderContext;
+import org.apache.flink.api.connector.source.SplitEnumerator;
+import org.apache.flink.api.connector.source.SplitEnumeratorContext;
+import org.apache.flink.api.connector.source.lib.NumberSequenceSource;
+import org.apache.flink.api.connector.source.lib.util.IteratorSourceEnumerator;
+import org.apache.flink.api.connector.source.lib.util.IteratorSourceReader;
+import org.apache.flink.api.connector.source.lib.util.IteratorSourceSplit;
+import org.apache.flink.core.io.InputStatus;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayDeque;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Queue;
+
+/**
+ * A {@link NumberSequenceSource} that can be blocked by checkpoints. If {@link
+ * #isBlockByCheckpoint} is set to true, this source would guarantee that a checkpoint has been
+ * invoked before the source ends its input.
+ */
+public class NumberSequenceSourceBlockableByCheckpoint extends NumberSequenceSource {

Review Comment:
   Instead of replacing `TestingNumberSequenceSource` with this new class with the extra ability to set isBlockByCheckpoint to false, it seems simpler to keep `TestingNumberSequenceSource` as is and use `NumberSequenceSource` when needed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] yunfengzhou-hub commented on a diff in pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "yunfengzhou-hub (via GitHub)" <gi...@apache.org>.
yunfengzhou-hub commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1271940611


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -426,6 +446,36 @@ public boolean isShutdown() {
         return shutdown;
     }
 
+    /**
+     * Reports whether a source operator is currently processing backlog.
+     *
+     * <p>If any source operator is processing backlog, the checkpoint interval would be decided by
+     * {@code execution.checkpointing.interval-during-backlog} instead of {@code
+     * execution.checkpointing.interval}.
+     *
+     * <p>If a source has not invoked this method, the source is considered to have
+     * isProcessingBacklog=false. If a source operator has invoked this method multiple times, the
+     * last reported value is used.
+     *
+     * @param operatorID the operator ID of the source operator.
+     * @param isProcessingBacklog whether the source operator is processing backlog.
+     */
+    public void setIsProcessingBacklog(OperatorID operatorID, boolean isProcessingBacklog) {
+        if (isProcessingBacklog) {
+            backlogOperators.add(operatorID);
+        } else {
+            backlogOperators.remove(operatorID);
+        }
+
+        if (getCurrentCheckpointInterval() != Long.MAX_VALUE) {
+            long newNextCheckpointTriggeringTime =
+                    clock.absoluteTimeMillis() + getCurrentCheckpointInterval();
+            if (newNextCheckpointTriggeringTime < nextCheckpointTriggeringRelativeTime) {
+                rescheduleTrigger(getCurrentCheckpointInterval());

Review Comment:
   `newNextCheckpointTriggeringTime` is not suitable here, as in that case we need to write `rescheduleTrigger(newNextCheckpointTriggeringTime - clock.relativeTimeMillis());`, while `clock.relativeTimeMillis()` might change.
   
   I'll create variable `long currentCheckpointInterval = getCurrentCheckpointInterval()` to avoid getting timestamp twice.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] yunfengzhou-hub commented on a diff in pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "yunfengzhou-hub (via GitHub)" <gi...@apache.org>.
yunfengzhou-hub commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1267857378


##########
flink-runtime/src/main/java/org/apache/flink/runtime/operators/coordination/OperatorCoordinatorHolder.java:
##########
@@ -164,6 +167,15 @@ public void lazyInitialize(
         context.lazyInitialize(
                 globalFailureHandler, mainThreadExecutor, operatorCoordinatorMetricGroup);
 
+        OperatorCoordinator rootCoordinator = coordinator;
+        if (coordinator instanceof RecreateOnResetOperatorCoordinator) {
+            rootCoordinator =
+                    ((RecreateOnResetOperatorCoordinator) rootCoordinator).getInternalCoordinator();
+        }
+        if (rootCoordinator instanceof SourceCoordinator) {
+            ((SourceCoordinator<?, ?>) rootCoordinator).lazyInitialize(checkpointCoordinator);

Review Comment:
   Below is part of the invocation stack of SourceCoordinatorContext's constructor.
   ```
   <init>:142, SourceCoordinatorContext (org.apache.flink.runtime.source.coordinator)
   <init>:119, SourceCoordinatorContext (org.apache.flink.runtime.source.coordinator)
   getCoordinator:89, SourceCoordinatorProvider (org.apache.flink.runtime.source.coordinator)
   createNewInternalCoordinator:337, RecreateOnResetOperatorCoordinator$DeferrableCoordinator (org.apache.flink.runtime.operators.coordination)
   <init>:60, RecreateOnResetOperatorCoordinator (org.apache.flink.runtime.operators.coordination)
   <init>:43, RecreateOnResetOperatorCoordinator (org.apache.flink.runtime.operators.coordination)
   create:200, RecreateOnResetOperatorCoordinator$Provider (org.apache.flink.runtime.operators.coordination)
   create:194, RecreateOnResetOperatorCoordinator$Provider (org.apache.flink.runtime.operators.coordination)
   create:546, OperatorCoordinatorHolder (org.apache.flink.runtime.operators.coordination)
   create:509, OperatorCoordinatorHolder (org.apache.flink.runtime.operators.coordination)
   createOperatorCoordinatorHolder:286, ExecutionJobVertex (org.apache.flink.runtime.executiongraph)
   initialize:223, ExecutionJobVertex (org.apache.flink.runtime.executiongraph)
   initializeJobVertex:888, DefaultExecutionGraph (org.apache.flink.runtime.executiongraph)
   initializeJobVertex:218, ExecutionGraph (org.apache.flink.runtime.executiongraph)
   initializeJobVertices:870, DefaultExecutionGraph (org.apache.flink.runtime.executiongraph)
   attachJobGraph:826, DefaultExecutionGraph (org.apache.flink.runtime.executiongraph)
   buildGraph:219, DefaultExecutionGraphBuilder (org.apache.flink.runtime.executiongraph)
   createAndRestoreExecutionGraph:163, DefaultExecutionGraphFactory (org.apache.flink.runtime.scheduler)
   createAndRestoreExecutionGraph:368, SchedulerBase (org.apache.flink.runtime.scheduler)
   <init>:210, SchedulerBase (org.apache.flink.runtime.scheduler)
   <init>:140, DefaultScheduler (org.apache.flink.runtime.scheduler)
   ```
   
   In order to pass `checkpointCoordinator` through this stack to SourceCoordinatorContext's constructor, we may need to add a `getCheckpointCoordinator` method to `OperatorCoordinator.Context` class. This is a public API change that has not been agreed in the FLIP discussion, and I also personally think this API will yield too much freedom to OperatorCoordinator's developers so should not be introduced. Thus I did not choose this path.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] yunfengzhou-hub commented on a diff in pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "yunfengzhou-hub (via GitHub)" <gi...@apache.org>.
yunfengzhou-hub commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1267876016


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -2079,11 +2149,19 @@ private final class ScheduledTrigger implements Runnable {
 
         @Override
         public void run() {
+            long currentTime = clock.relativeTimeMillis();
+            if (lastCheckpointTriggeringRelativeTime != NO_CHECKPOINT
+                    && currentTime - lastCheckpointTriggeringRelativeTime < baseInterval) {
+                return;
+            }
+            lastCheckpointTriggeringRelativeTime = currentTime;
+
             try {
                 triggerCheckpoint(checkpointProperties, null, true);
             } catch (Exception e) {
                 LOG.error("Exception while triggering checkpoint for job {}.", job, e);
             }
+            timer.schedule(this, baseInterval, TimeUnit.MILLISECONDS);

Review Comment:
   I'm not sure I understand the expected behavior clearly. The current implementation ensures that the next checkpoint will be <= T2+ X, depending on the random init delay.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] yunfengzhou-hub commented on pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "yunfengzhou-hub (via GitHub)" <gi...@apache.org>.
yunfengzhou-hub commented on PR #22931:
URL: https://github.com/apache/flink/pull/22931#issuecomment-1643288055

   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] yunfengzhou-hub commented on a diff in pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "yunfengzhou-hub (via GitHub)" <gi...@apache.org>.
yunfengzhou-hub commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1277383109


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -183,6 +189,15 @@ public class CheckpointCoordinator {
     /** A handle to the current periodic trigger, to cancel it when necessary. */
     private ScheduledFuture<?> currentPeriodicTrigger;
 
+    /**
+     * The timestamp (via {@link Clock#relativeTimeMillis()}) when the next checkpoint will be
+     * triggered.
+     *
+     * <p>If it's value is {@link Long#MAX_VALUE}, it means there is not a next checkpoint
+     * scheduled.
+     */
+    private volatile long nextCheckpointTriggeringRelativeTime;

Review Comment:
   `nextCheckpointTriggeringRelativeTime` and `currentPeriodicTrigger` should be updated together, so I added `synchronized(lock){}` around code blocks related to these variables instead of adding `@GuardedBy("lock")` to each of them.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] lindong28 commented on a diff in pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "lindong28 (via GitHub)" <gi...@apache.org>.
lindong28 commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1279039487


##########
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/SchedulerBase.java:
##########
@@ -1081,6 +1082,21 @@ public void notifyEndOfData(ExecutionAttemptID executionAttemptID) {
             if (vertexEndOfDataListener.areAllTasksEndOfData()) {
                 triggerCheckpoint(CheckpointType.CONFIGURED);
             }
+            if (vertexEndOfDataListener.areAllTasksOfJobVertexEndOfData(

Review Comment:
   Would it be better to move this before `triggerCheckpoint(CheckpointType.CONFIGURED)` shown above?
   
   Same for `StateWithExecutionGraph#notifyEndOfData`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] yunfengzhou-hub commented on a diff in pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "yunfengzhou-hub (via GitHub)" <gi...@apache.org>.
yunfengzhou-hub commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1267819133


##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/CheckpointConfig.java:
##########
@@ -222,6 +222,25 @@ public void setCheckpointInterval(long checkpointInterval) {
                 Duration.ofMillis(checkpointInterval));
     }
 
+    public long getCheckpointIntervalDuringBacklog() {
+        return configuration
+                .getOptional(ExecutionCheckpointingOptions.CHECKPOINTING_INTERVAL_DURING_BACKLOG)
+                .map(Duration::toMillis)
+                .orElseGet(this::getCheckpointInterval);
+    }
+
+    public void setCheckpointIntervalDuringBacklog(long checkpointInterval) {
+        if (checkpointInterval != -1L && checkpointInterval < MINIMAL_CHECKPOINT_TIME) {

Review Comment:
   When users need to disable checkpoint during backlog, they can configure checkpoint-interval-during-backlog into -1L.
   
   In checkpoint-interval's case, user can disable checkpoint by not setting this configuration. However, according to FLIP-309, not setting checkpoint-interval-during-backlog stands for using the same value as checkpoint-interval, so we need to find an alternative special value for this function.
   
   According to [1] and [2], checkpoint interval uses -1L to represent intervals of infinite length, or disabling checkpoint interval[3]. Thus I adopted -1L as the special value in accordance with the current semantics for checkpoint interval.
   
   [1] https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/CheckpointConfig.java#L201
   [2] https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamingJobGraphGenerator.java#L1918
   [3] https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java#L1946



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] lindong28 commented on a diff in pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "lindong28 (via GitHub)" <gi...@apache.org>.
lindong28 commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1271414814


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -1963,7 +2020,14 @@ public void startCheckpointScheduler() {
             stopCheckpointScheduler();
 
             periodicScheduling = true;
-            currentPeriodicTrigger = scheduleTriggerWithDelay(getRandomInitDelay());
+            if (baseIntervalDuringBacklog != baseInterval) {

Review Comment:
   The code that enforces the checkpoint-interval-during-backlog is way more complicated than it should be. The resulting checkpoint trigger time can be wrong (I can explain this offline). And it is not necessary to have two parallel periodic tasks scheduled at the same time.
   
   It should be much simpler and more readable to do the following:
   
   ```
   - Use a variable to maintain the absolute time of the last successful checkpoint trigger.
   - Let method `getNextCheckpointDelay()` return the time to wait from now until we should trigger the next checkpoint.
   - Let method `scheduleAndMaybeTriggerCheckpoint()` do the following:
     - Invoke `triggerCheckpoint()` if `getNextCheckpointDelay()` <= 0
     - Schedule `scheduleAndMaybeTriggerCheckpoint()` with delay=`getNextCheckpointDelay()`.
   - `startCheckpointScheduler()` should schedule `scheduleAndMaybeTriggerCheckpointI()` with delay=`getNextCheckpointDelay()`
   - Whenever isBacklog changes from true to false, we can cancel the pending scheduled task and re-schedule it based on `getNextCheckpointDelay()`.
   ```
   
   One important advantage of the approach described above is that all the logic of determining checkpoint trigger time can be found in `getNextCheckpointDelay()`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] lindong28 commented on a diff in pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "lindong28 (via GitHub)" <gi...@apache.org>.
lindong28 commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1271414814


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -1963,7 +2020,14 @@ public void startCheckpointScheduler() {
             stopCheckpointScheduler();
 
             periodicScheduling = true;
-            currentPeriodicTrigger = scheduleTriggerWithDelay(getRandomInitDelay());
+            if (baseIntervalDuringBacklog != baseInterval) {

Review Comment:
   The code that enforces the checkpoint-interval-during-backlog is way more complicated than it should be. The interval is incorrect (I can explain this offline). And it is not necessary to have two parallel periodic tasks scheduled at the same time.
   
   It should be much simpler and more readable to do the following:
   
   ```
   - Use a variable to maintain the absolute time of the last successful checkpoint trigger.
   - Let method `getNextCheckpointDelay()` return the time to wait from now until we should trigger the next checkpoint.
   - Let method `scheduleAndMaybeTriggerCheckpoint()` do the following:
     - Invoke `triggerCheckpoint()` if `getNextCheckpointDelay()` <= 0
     - Schedule `scheduleAndMaybeTriggerCheckpoint()` with delay=`getNextCheckpointDelay()`.
   - `startCheckpointScheduler()` should schedule `scheduleAndMaybeTriggerCheckpointI()` with delay=`getNextCheckpointDelay()`
   - Whenever isBacklog status changes, we can cancel the pending scheduled task and re-schedule it based on `getNextCheckpointDelay()`.
   ```
   
   One important advantage of the approach described above is that all the logic of determining checkpoint trigger time can be found in `getNextCheckpointDelay()`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] yunfengzhou-hub commented on pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "yunfengzhou-hub (via GitHub)" <gi...@apache.org>.
yunfengzhou-hub commented on PR #22931:
URL: https://github.com/apache/flink/pull/22931#issuecomment-1658026093

   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] yunfengzhou-hub commented on a diff in pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "yunfengzhou-hub (via GitHub)" <gi...@apache.org>.
yunfengzhou-hub commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1282534739


##########
flink-tests/src/test/java/org/apache/flink/test/util/NumberSequenceSourceBlockableByCheckpoint.java:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.test.util;
+
+import org.apache.flink.api.connector.source.ReaderOutput;
+import org.apache.flink.api.connector.source.SourceReader;
+import org.apache.flink.api.connector.source.SourceReaderContext;
+import org.apache.flink.api.connector.source.SplitEnumerator;
+import org.apache.flink.api.connector.source.SplitEnumeratorContext;
+import org.apache.flink.api.connector.source.lib.NumberSequenceSource;
+import org.apache.flink.api.connector.source.lib.util.IteratorSourceEnumerator;
+import org.apache.flink.api.connector.source.lib.util.IteratorSourceReader;
+import org.apache.flink.api.connector.source.lib.util.IteratorSourceSplit;
+import org.apache.flink.core.io.InputStatus;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayDeque;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Queue;
+
+/**
+ * A {@link NumberSequenceSource} that can be blocked by checkpoints. If {@link
+ * #isBlockByCheckpoint} is set to true, this source would guarantee that a checkpoint has been
+ * invoked before the source ends its input.
+ */
+public class NumberSequenceSourceBlockableByCheckpoint extends NumberSequenceSource {

Review Comment:
   I have kept `TestingNumberSequenceSource`'s behavior as it had been in the latest update. But given that this PR is changing this class from a private class to a public class in test utils package, the name `TestingNumberSequenceSource` might not provide enough information about its function. Thus I named it as `NumberSequenceSourceWithWaitForCheckpoint`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] yunfengzhou-hub commented on pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "yunfengzhou-hub (via GitHub)" <gi...@apache.org>.
yunfengzhou-hub commented on PR #22931:
URL: https://github.com/apache/flink/pull/22931#issuecomment-1617912811

   @lindong28 could you please take a look at this PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] yunfengzhou-hub commented on pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "yunfengzhou-hub (via GitHub)" <gi...@apache.org>.
yunfengzhou-hub commented on PR #22931:
URL: https://github.com/apache/flink/pull/22931#issuecomment-1617912169

   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-32514] Support configuring checkpointing interval during process backlog [flink]

Posted by "XComp (via GitHub)" <gi...@apache.org>.
XComp commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1457044758


##########
flink-runtime/src/main/java/org/apache/flink/runtime/operators/coordination/OperatorCoordinatorHolder.java:
##########
@@ -143,12 +144,14 @@ private OperatorCoordinatorHolder(
 
     public void lazyInitialize(
             GlobalFailureHandler globalFailureHandler,
-            ComponentMainThreadExecutor mainThreadExecutor) {
+            ComponentMainThreadExecutor mainThreadExecutor,
+            @Nullable CheckpointCoordinator checkpointCoordinator) {
 
         this.globalFailureHandler = globalFailureHandler;
         this.mainThreadExecutor = mainThreadExecutor;
+        context.lazyInitialize(globalFailureHandler, mainThreadExecutor, checkpointCoordinator);
 
-        context.lazyInitialize(globalFailureHandler, mainThreadExecutor);
+        context.lazyInitialize(globalFailureHandler, mainThreadExecutor, checkpointCoordinator);

Review Comment:
   Thanks for clarification. Doing it as a hotfix commit in one other PR makes sense because of the reasons you mentioned. :+1: 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] lindong28 commented on a diff in pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "lindong28 (via GitHub)" <gi...@apache.org>.
lindong28 commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1303844187


##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/ExecutionCheckpointingOptions.java:
##########
@@ -126,6 +126,23 @@ public class ExecutionCheckpointingOptions {
                                                                     .key()))
                                             .build());
 
+    public static final ConfigOption<Duration> CHECKPOINTING_INTERVAL_DURING_BACKLOG =
+            ConfigOptions.key("execution.checkpointing.interval-during-backlog")
+                    .durationType()
+                    .noDefaultValue()
+                    .withDescription(
+                            Description.builder()
+                                    .text(
+                                            "Gets the interval in which checkpoints are periodically scheduled "

Review Comment:
   Should we use the same doc as specified in FLIP-309?



##########
docs/layouts/shortcodes/generated/execution_checkpointing_configuration.html:
##########
@@ -30,7 +30,13 @@
             <td><h5>execution.checkpointing.interval</h5></td>
             <td style="word-wrap: break-word;">(none)</td>
             <td>Duration</td>
-            <td>Gets the interval in which checkpoints are periodically scheduled.<br /><br />This setting defines the base interval. Checkpoint triggering may be delayed by the settings <code class="highlighter-rouge">execution.checkpointing.max-concurrent-checkpoints</code> and <code class="highlighter-rouge">execution.checkpointing.min-pause</code></td>
+            <td>Gets the interval in which checkpoints are periodically scheduled.<br /><br />This setting defines the base interval. Checkpoint triggering may be delayed by the settings <code class="highlighter-rouge">execution.checkpointing.max-concurrent-checkpoints</code>, <code class="highlighter-rouge">execution.checkpointing.min-pause</code> and <code class="highlighter-rouge">execution.checkpointing.interval-during-backlog</code></td>
+        </tr>
+        <tr>
+            <td><h5>execution.checkpointing.interval-during-backlog</h5></td>
+            <td style="word-wrap: break-word;">(none)</td>
+            <td>Duration</td>
+            <td>Gets the interval in which checkpoints are periodically scheduled when any source reports isProcessingBacklog=true.<br /><br />Checkpoint triggering may be delayed by the settings <code class="highlighter-rouge">execution.checkpointing.max-concurrent-checkpoints</code> and <code class="highlighter-rouge">execution.checkpointing.min-pause</code></td>

Review Comment:
   Should we use the same doc as specified in FLIP-309?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] lindong28 commented on a diff in pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "lindong28 (via GitHub)" <gi...@apache.org>.
lindong28 commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1279012662


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -426,6 +456,39 @@ public boolean isShutdown() {
         return shutdown;
     }
 
+    /**
+     * Reports whether a source operator is currently processing backlog.
+     *
+     * <p>If any source operator is processing backlog, the checkpoint interval would be decided by
+     * {@code execution.checkpointing.interval-during-backlog} instead of {@code
+     * execution.checkpointing.interval}.
+     *
+     * <p>If a source has not invoked this method, the source is considered to have
+     * isProcessingBacklog=false. If a source operator has invoked this method multiple times, the
+     * last reported value is used.
+     *
+     * @param operatorID the operator ID of the source operator.
+     * @param isProcessingBacklog whether the source operator is processing backlog.
+     */
+    public void setIsProcessingBacklog(OperatorID operatorID, boolean isProcessingBacklog) {
+        synchronized (lock) {
+            if (isProcessingBacklog) {
+                backlogOperators.add(operatorID);
+            } else {
+                backlogOperators.remove(operatorID);
+            }
+
+            long currentCheckpointInterval = getCurrentCheckpointInterval();
+            if (currentCheckpointInterval != Long.MAX_VALUE) {
+                long currentRelativeTime = clock.relativeTimeMillis();
+                if (currentRelativeTime + currentCheckpointInterval
+                        < nextCheckpointTriggeringRelativeTime) {

Review Comment:
   It would be safer to do the following:
   
   long delay = Math.max(0, nextCheckpointTriggeringRelativeTime - clock.relativeTimeMillis())



##########
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/SchedulerBase.java:
##########
@@ -1081,6 +1082,21 @@ public void notifyEndOfData(ExecutionAttemptID executionAttemptID) {
             if (vertexEndOfDataListener.areAllTasksEndOfData()) {
                 triggerCheckpoint(CheckpointType.CONFIGURED);
             }
+            if (vertexEndOfDataListener.areAllTasksOfJobVertexEndOfData(

Review Comment:
   Would it be better to move this before `triggerCheckpoint(CheckpointType.CONFIGURED)` shown above?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/VertexEndOfDataListener.java:
##########
@@ -51,6 +51,12 @@ public void recordTaskEndOfData(ExecutionAttemptID executionAttemptID) {
         subtaskStatus.set(executionAttemptID.getSubtaskIndex());
     }
 
+    public boolean areAllTasksOfJobVertexEndOfData(JobVertexID jobVertexID) {
+        BitSet subtaskStatus = tasksReachedEndOfData.get(jobVertexID);
+        return subtaskStatus.cardinality()

Review Comment:
   This would cause NPE given that we remove `jobVertexID` from `tasksReachedEndOfData` after all its subtasks have finished.
   
   Also, given that `areAllTasksOfJobVertexEndOfData()` is currently invoked after `areAllTasksEndOfData()`, it suggests that this code path is not tested. Can you add a test for this scenario?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/operators/coordination/OperatorCoordinator.java:
##########
@@ -277,6 +278,10 @@ interface Context {
          * concurrent running execution attempts.
          */
         boolean isConcurrentExecutionAttemptsSupported();
+
+        /** Gets the checkpoint coordinator of this job, if snapshot checkpoints are enabled. */

Review Comment:
   Would the following doc be more readable?
   
   Gets the checkpoint coordinator of this job. Return null if checkpoint is disabled.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] lindong28 commented on a diff in pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "lindong28 (via GitHub)" <gi...@apache.org>.
lindong28 commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1268832721


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -291,13 +313,27 @@ public CheckpointCoordinator(
 
         // it does not make sense to schedule checkpoints more often then the desired
         // time between checkpoints
-        long baseInterval = chkConfig.getCheckpointInterval();
-        if (baseInterval < minPauseBetweenCheckpoints) {
-            baseInterval = minPauseBetweenCheckpoints;
+        long checkpointInterval = chkConfig.getCheckpointInterval();
+        if (checkpointInterval < minPauseBetweenCheckpoints) {
+            checkpointInterval = minPauseBetweenCheckpoints;
+        }
+
+        long checkpointIntervalDuringBacklog = chkConfig.getCheckpointIntervalDuringBacklog();
+        if (checkpointIntervalDuringBacklog < minPauseBetweenCheckpoints) {
+            checkpointIntervalDuringBacklog = minPauseBetweenCheckpoints;
+        }
+
+        if (checkpointIntervalDuringBacklog < checkpointInterval) {
+            throw new IllegalArgumentException(
+                    "execution.checkpoint.interval-during-backlog must either be 0, "
+                            + "or be larger than or equal to execution.checkpoint.interval.");
         }
 
         this.job = checkNotNull(job);
-        this.baseInterval = baseInterval;
+        this.checkpointInterval = checkpointInterval;
+        this.checkpointIntervalDuringBacklog = checkpointIntervalDuringBacklog;
+        this.lastCheckpointTriggeringRelativeTime = NO_CHECKPOINT;

Review Comment:
   `OperatorCoordinator.NO_CHECKPOINT` refers to operator ID rather than time. It seems weird to assign an operator ID to a variable representing relative time.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] yunfengzhou-hub commented on pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "yunfengzhou-hub (via GitHub)" <gi...@apache.org>.
yunfengzhou-hub commented on PR #22931:
URL: https://github.com/apache/flink/pull/22931#issuecomment-1643425897

   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] lindong28 commented on a diff in pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "lindong28 (via GitHub)" <gi...@apache.org>.
lindong28 commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1271414814


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -1963,7 +2020,14 @@ public void startCheckpointScheduler() {
             stopCheckpointScheduler();
 
             periodicScheduling = true;
-            currentPeriodicTrigger = scheduleTriggerWithDelay(getRandomInitDelay());
+            if (baseIntervalDuringBacklog != baseInterval) {

Review Comment:
   The code that enforces the checkpoint-interval-during-backlog is way more complicated than it should be. For example, it would not be necessary to have two parallel periodic tasks scheduled at the same time.
   
   It should be much simpler and more readable to do the following:
   
   ```
   - Use a variable to maintain the absolute time of last checkpoint.
   - Let method `getNextCheckpointDelay()` return the time to wait from now until we should trigger the next checkpoint. T
   - Let method `scheduleAndMaybeTriggerCheckpoint()` do the following:
     - Invoke `triggerCheckpoint()` if `getNextCheckpointDelay()` <= 0
     - Schedule `scheduleAndMaybeTriggerCheckpoint()` with delay=`getNextCheckpointDelay()`.
   - `startCheckpointScheduler()` should schedule `scheduleAndMaybeTriggerCheckpointI()` with delay=`getNextCheckpointDelay()`
   - Whenever isBacklog status changes, we should cancel the pending scheduled task and re-schedule it based on `getNextCheckpointDelay()`.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-32514] Support configuring checkpointing interval during process backlog [flink]

Posted by "lindong28 (via GitHub)" <gi...@apache.org>.
lindong28 commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1456726081


##########
flink-runtime/src/main/java/org/apache/flink/runtime/operators/coordination/OperatorCoordinatorHolder.java:
##########
@@ -143,12 +144,14 @@ private OperatorCoordinatorHolder(
 
     public void lazyInitialize(
             GlobalFailureHandler globalFailureHandler,
-            ComponentMainThreadExecutor mainThreadExecutor) {
+            ComponentMainThreadExecutor mainThreadExecutor,
+            @Nullable CheckpointCoordinator checkpointCoordinator) {
 
         this.globalFailureHandler = globalFailureHandler;
         this.mainThreadExecutor = mainThreadExecutor;
+        context.lazyInitialize(globalFailureHandler, mainThreadExecutor, checkpointCoordinator);
 
-        context.lazyInitialize(globalFailureHandler, mainThreadExecutor);
+        context.lazyInitialize(globalFailureHandler, mainThreadExecutor, checkpointCoordinator);

Review Comment:
   @XComp Thanks for catching this. No, we didn't do it intentionally. I believe it is introduced when Yunfeng rebased the. I didn't catch this issue because I didn't go over the entire PR end-to-end very carefully when there is only minor remaining comments in the last 2 rounds of review.
   
   It seems that the extra invocation of `lazyInitialize()` would not introduce any visible performance or correctness issue. Maybe one of us can fix it in our next PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] lindong28 commented on a diff in pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "lindong28 (via GitHub)" <gi...@apache.org>.
lindong28 commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1270207209


##########
flink-runtime/src/main/java/org/apache/flink/runtime/operators/coordination/OperatorCoordinatorHolder.java:
##########
@@ -164,6 +167,13 @@ public void lazyInitialize(
         context.lazyInitialize(
                 globalFailureHandler, mainThreadExecutor, operatorCoordinatorMetricGroup);
 
+        if (coordinator instanceof RecreateOnResetOperatorCoordinator) {

Review Comment:
   Instead of passing `checkpointCoordinator` to `SourceCoordinatorContext` like this, would it be simpler to do the following:
   
   - `OperatorCoordinatorHolder#lazyInitialize` can pass checkpoint coordinator to `context.lazyInitialize`.
   - Add method `getCheckpointCoordinator` to `OperatorCoordinator#Context`, which can return the checkpoint coordinator obtained in the above step.
   - `SourceCoordinatorContext#setIsProcessingBacklog` can use the checkpoint coordinator obtained from `operatorCoordinatorContext`.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] yunfengzhou-hub commented on a diff in pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "yunfengzhou-hub (via GitHub)" <gi...@apache.org>.
yunfengzhou-hub commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1278941839


##########
flink-runtime/src/main/java/org/apache/flink/runtime/source/coordinator/SourceCoordinator.java:
##########
@@ -328,6 +337,7 @@ public void executionAttemptFailed(
 
                     context.unregisterSourceReader(subtaskId, attemptNumber);
                     context.attemptFailed(subtaskId, attemptNumber);
+                    markSubtaskAsNoData(subtaskId);

Review Comment:
   As we are adopting `VertexEndOfDataListener` instead of adding additional logic in source operator/coordinator, this comment is not applicable now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] lindong28 commented on a diff in pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "lindong28 (via GitHub)" <gi...@apache.org>.
lindong28 commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1266284788


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -291,13 +313,27 @@ public CheckpointCoordinator(
 
         // it does not make sense to schedule checkpoints more often then the desired
         // time between checkpoints
-        long baseInterval = chkConfig.getCheckpointInterval();
-        if (baseInterval < minPauseBetweenCheckpoints) {
-            baseInterval = minPauseBetweenCheckpoints;
+        long checkpointInterval = chkConfig.getCheckpointInterval();
+        if (checkpointInterval < minPauseBetweenCheckpoints) {
+            checkpointInterval = minPauseBetweenCheckpoints;
+        }
+
+        long checkpointIntervalDuringBacklog = chkConfig.getCheckpointIntervalDuringBacklog();
+        if (checkpointIntervalDuringBacklog < minPauseBetweenCheckpoints) {
+            checkpointIntervalDuringBacklog = minPauseBetweenCheckpoints;
+        }
+
+        if (checkpointIntervalDuringBacklog < checkpointInterval) {
+            throw new IllegalArgumentException(
+                    "execution.checkpoint.interval-during-backlog must either be 0, "
+                            + "or be larger than or equal to execution.checkpoint.interval.");
         }
 
         this.job = checkNotNull(job);
-        this.baseInterval = baseInterval;
+        this.checkpointInterval = checkpointInterval;
+        this.checkpointIntervalDuringBacklog = checkpointIntervalDuringBacklog;
+        this.lastCheckpointTriggeringRelativeTime = NO_CHECKPOINT;

Review Comment:
   Would it be simpler to just initialize this variable to 0 without using `NO_CHECKPOINT`?



##########
flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/hybrid/HybridSourceSplitEnumerator.java:
##########
@@ -303,6 +303,8 @@ public Object getPreviousEnumerator() {
         }
         LOG.info("Starting enumerator for sourceIndex={}", currentSourceIndex);
         currentEnumerator.start();
+
+        context.setIsProcessingBacklog(currentSourceIndex < sources.size() - 1);

Review Comment:
   Would it be better to invoke `#setIsProcessingBacklog` before starting the currentEnumerate?
   
   Otherwise, there might be a short interval during which the hybridSource has started to run the new stage but the backlog status is still not updated. It might not be a problem in most cases in practice. Still, it might be better to avoid it entirely.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/operators/coordination/OperatorCoordinatorHolder.java:
##########
@@ -164,6 +167,15 @@ public void lazyInitialize(
         context.lazyInitialize(
                 globalFailureHandler, mainThreadExecutor, operatorCoordinatorMetricGroup);
 
+        OperatorCoordinator rootCoordinator = coordinator;
+        if (coordinator instanceof RecreateOnResetOperatorCoordinator) {
+            rootCoordinator =
+                    ((RecreateOnResetOperatorCoordinator) rootCoordinator).getInternalCoordinator();
+        }
+        if (rootCoordinator instanceof SourceCoordinator) {
+            ((SourceCoordinator<?, ?>) rootCoordinator).lazyInitialize(checkpointCoordinator);

Review Comment:
   Could you confirm that it is not possible (or much harder) to pass `checkpointCoordinator` to `SourceCoordinatorContext`'s constructor?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -2079,11 +2149,19 @@ private final class ScheduledTrigger implements Runnable {
 
         @Override
         public void run() {
+            long currentTime = clock.relativeTimeMillis();
+            if (lastCheckpointTriggeringRelativeTime != NO_CHECKPOINT
+                    && currentTime - lastCheckpointTriggeringRelativeTime < baseInterval) {

Review Comment:
   Would it be simpler to do this:
   
   ```
   if (currentTime - lastCheckpointTriggeringRelativeTime < baseInterval) {
       return;
   }
   ```



##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/CheckpointConfig.java:
##########
@@ -222,6 +222,25 @@ public void setCheckpointInterval(long checkpointInterval) {
                 Duration.ofMillis(checkpointInterval));
     }
 
+    public long getCheckpointIntervalDuringBacklog() {
+        return configuration
+                .getOptional(ExecutionCheckpointingOptions.CHECKPOINTING_INTERVAL_DURING_BACKLOG)
+                .map(Duration::toMillis)
+                .orElseGet(this::getCheckpointInterval);
+    }
+
+    public void setCheckpointIntervalDuringBacklog(long checkpointInterval) {
+        if (checkpointInterval != -1L && checkpointInterval < MINIMAL_CHECKPOINT_TIME) {

Review Comment:
   Can you explain when will the input `checkpointInterval` has value -1?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -150,10 +152,22 @@ public class CheckpointCoordinator {
     private final CheckpointIDCounter checkpointIdCounter;
 
     /**
-     * The base checkpoint interval. Actual trigger time may be affected by the max concurrent
-     * checkpoints and minimum-pause values
+     * The checkpoint interval in normal situations. Actual trigger time may be affected by the max
+     * concurrent checkpoints, minimum-pause values and checkpoint interval during backlog.
      */
-    private final long baseInterval;
+    private final long checkpointInterval;
+
+    /**
+     * The checkpoint interval when any source reports isProcessingBacklog=true. Actual trigger time
+     * may be affected by the max concurrent checkpoints and minimum-pause values.
+     */
+    private final long checkpointIntervalDuringBacklog;

Review Comment:
   Would it be better to keep the name consistent with the existing `baseInterval`? We can either rename `baseInterval` as `checkpointingInterval`, or rename `checkpointIntervalDuringBacklog` as `baseIntervalDuringBacklog`.
   



##########
flink-core/src/main/java/org/apache/flink/api/connector/source/SplitEnumeratorContext.java:
##########
@@ -181,4 +181,16 @@ <T> void callAsync(
      * @param runnable a runnable to execute
      */
     void runInCoordinatorThread(Runnable runnable);
+
+    /**
+     * Reports to JM whether this source is currently processing backlog.
+     *
+     * <p>When source is processing backlog, it means the records being emitted by this source is
+     * already stale and there is no processing latency requirement for these records. This allows
+     * downstream operators to optimize throughput instead of reducing latency for intermediate
+     * results.
+     *
+     * <p>If this method is not invoked, the source is considered to have isProcessingBacklog=false.

Review Comment:
   Since we plan to introduce other APIs for source to set its backlog status (e.g. in source reader), it might be better to say the following:
   
   ```
   If no API has been explicitly invoked to specify the backlog status of a source, the source is considered to have isProcessingBacklog=false by default.
   ```



##########
flink-runtime/src/main/java/org/apache/flink/runtime/operators/coordination/OperatorCoordinatorHolder.java:
##########
@@ -164,6 +167,15 @@ public void lazyInitialize(
         context.lazyInitialize(
                 globalFailureHandler, mainThreadExecutor, operatorCoordinatorMetricGroup);
 
+        OperatorCoordinator rootCoordinator = coordinator;
+        if (coordinator instanceof RecreateOnResetOperatorCoordinator) {
+            rootCoordinator =
+                    ((RecreateOnResetOperatorCoordinator) rootCoordinator).getInternalCoordinator();

Review Comment:
   Suppose a `RecreateOnResetOperatorCoordinator` creates a new `OperatorCoordinator` instance internally when it is reset to checkpoint, I suppose this code path will not be invoked again and therefore the newly created `OperatorCoordinator` will not know the reference to the checkpoint coordinator. Is it OK?
   
   



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -2079,11 +2149,19 @@ private final class ScheduledTrigger implements Runnable {
 
         @Override
         public void run() {
+            long currentTime = clock.relativeTimeMillis();
+            if (lastCheckpointTriggeringRelativeTime != NO_CHECKPOINT
+                    && currentTime - lastCheckpointTriggeringRelativeTime < baseInterval) {
+                return;
+            }
+            lastCheckpointTriggeringRelativeTime = currentTime;
+
             try {
                 triggerCheckpoint(checkpointProperties, null, true);
             } catch (Exception e) {
                 LOG.error("Exception while triggering checkpoint for job {}.", job, e);
             }
+            timer.schedule(this, baseInterval, TimeUnit.MILLISECONDS);

Review Comment:
   Does the current implementation ensure the following behavior:
   
   Let's say the last checkpoint is triggered at time T1. And then the checkpoint interval is updated to value X at time T2. The next checkpoint will be triggered at the time `max(T2, T1 + X)`.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -426,6 +462,41 @@ public boolean isShutdown() {
         return shutdown;
     }
 
+    /**
+     * Reports whether a source operator is currently processing backlog.
+     *
+     * <p>If any source operator is processing backlog, the checkpoint interval would be decided by
+     * {@code execution.checkpointing.interval-during-backlog} instead of {@code
+     * execution.checkpointing.interval}.
+     *
+     * <p>If a source has not invoked this method, the source is considered to have
+     * isProcessingBacklog=false. If a source operator has invoked this method multiple times, the
+     * last reported value is used.
+     *
+     * @param operatorID the operator ID of the source operator.
+     * @param isProcessingBacklog whether the source operator is processing backlog.
+     */
+    public void setIsProcessingBacklog(OperatorID operatorID, boolean isProcessingBacklog) {
+        boolean wasBacklog = !backlogOperators.isEmpty();
+        if (isProcessingBacklog) {
+            backlogOperators.add(operatorID);
+        } else {
+            backlogOperators.remove(operatorID);
+        }
+
+        if (wasBacklog && backlogOperators.isEmpty()) {
+            baseInterval = checkpointInterval;

Review Comment:
   Instead of keeping the checkpoint interval in a dedicated variable, would it be simpler to add the following method so that we have less (duplicate) state to maintain?
   
   ```
   private long getCurrentCheckpointInterval() {
       return backlogOperators.isEmpty() ? checkpointInterval : checkpointIntervalDuringBacklog;
   }
   ```
   



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -222,6 +241,9 @@ public class CheckpointCoordinator {
 
     private final CheckpointPlanCalculator checkpointPlanCalculator;
 
+    /** IDs of the source operators that are currently processing backlog. */
+    private final Set<OperatorID> backlogOperators = new HashSet<>();

Review Comment:
   Will this map be accessed by multiple threads concurrently? Do we need to make it thread-safe?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -150,10 +152,22 @@ public class CheckpointCoordinator {
     private final CheckpointIDCounter checkpointIdCounter;
 
     /**
-     * The base checkpoint interval. Actual trigger time may be affected by the max concurrent
-     * checkpoints and minimum-pause values
+     * The checkpoint interval in normal situations. Actual trigger time may be affected by the max
+     * concurrent checkpoints, minimum-pause values and checkpoint interval during backlog.
      */
-    private final long baseInterval;
+    private final long checkpointInterval;
+
+    /**
+     * The checkpoint interval when any source reports isProcessingBacklog=true. Actual trigger time
+     * may be affected by the max concurrent checkpoints and minimum-pause values.
+     */
+    private final long checkpointIntervalDuringBacklog;
+
+    /**
+     * The base checkpoint interval. Its value might be either checkpointInterval or
+     * checkpointIntervalDuringBacklog depending on whether there is processing backlog.
+     */
+    private long baseInterval;

Review Comment:
   Since this value change be same as `checkpointIntervalDuringBacklog` depending on the status of source, it seems more intuitive to name it `currentCheckpointInterval`.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -426,6 +462,41 @@ public boolean isShutdown() {
         return shutdown;
     }
 
+    /**
+     * Reports whether a source operator is currently processing backlog.
+     *
+     * <p>If any source operator is processing backlog, the checkpoint interval would be decided by
+     * {@code execution.checkpointing.interval-during-backlog} instead of {@code
+     * execution.checkpointing.interval}.
+     *
+     * <p>If a source has not invoked this method, the source is considered to have
+     * isProcessingBacklog=false. If a source operator has invoked this method multiple times, the
+     * last reported value is used.
+     *
+     * @param operatorID the operator ID of the source operator.
+     * @param isProcessingBacklog whether the source operator is processing backlog.
+     */
+    public void setIsProcessingBacklog(OperatorID operatorID, boolean isProcessingBacklog) {
+        boolean wasBacklog = !backlogOperators.isEmpty();
+        if (isProcessingBacklog) {
+            backlogOperators.add(operatorID);
+        } else {
+            backlogOperators.remove(operatorID);
+        }
+
+        if (wasBacklog && backlogOperators.isEmpty()) {
+            baseInterval = checkpointInterval;
+        } else if (!wasBacklog && !backlogOperators.isEmpty()) {
+            baseInterval = checkpointIntervalDuringBacklog;
+        } else {
+            return;
+        }
+
+        if (baseInterval != Long.MAX_VALUE) {
+            scheduleTriggerWithDelay(getRandomInitDelay());

Review Comment:
   Would it be better (and more consistent with the existing checkpointing behavior) to introduce "random initial delay" only once instead of introducing it every time the backlog status changes?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] lindong28 commented on a diff in pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "lindong28 (via GitHub)" <gi...@apache.org>.
lindong28 commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1272313910


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -2013,23 +2077,25 @@ private void abortPendingCheckpoints(
 
     private void rescheduleTrigger(long tillNextMillis) {
         cancelPeriodicTrigger();
+        nextCheckpointTriggeringRelativeTime = clock.relativeTimeMillis() + tillNextMillis;
         currentPeriodicTrigger = scheduleTriggerWithDelay(tillNextMillis);
     }
 
     private void cancelPeriodicTrigger() {
         if (currentPeriodicTrigger != null) {
+            nextCheckpointTriggeringRelativeTime = Long.MAX_VALUE;
             currentPeriodicTrigger.cancel(false);
             currentPeriodicTrigger = null;
         }
     }
 
     private long getRandomInitDelay() {
-        return ThreadLocalRandom.current().nextLong(minPauseBetweenCheckpoints, baseInterval + 1L);
+        return ThreadLocalRandom.current()
+                .nextLong(minPauseBetweenCheckpoints, getCurrentCheckpointInterval() + 1L);

Review Comment:
   Given that `backlogOperators` must be empty at the time `getRandomInitDelay()` is invoked, would it be simpler to still use `baseInterval + 1L`?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -1963,7 +2025,9 @@ public void startCheckpointScheduler() {
             stopCheckpointScheduler();
 
             periodicScheduling = true;
-            currentPeriodicTrigger = scheduleTriggerWithDelay(getRandomInitDelay());
+            long initDelay = getRandomInitDelay();
+            nextCheckpointTriggeringRelativeTime = clock.relativeTimeMillis() + initDelay;

Review Comment:
   Would it be simpler to update `nextCheckpointTriggeringRelativeTime` inside `scheduleTriggerWithDelay()`?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -2075,10 +2141,21 @@ public void reportStats(long id, ExecutionAttemptID attemptId, CheckpointMetrics
 
     // ------------------------------------------------------------------------
 
-    private final class ScheduledTrigger implements Runnable {
-
+    final class ScheduledTrigger implements Runnable {
         @Override
         public void run() {
+            synchronized (lock) {
+                long checkpointInterval = getCurrentCheckpointInterval();
+                if (checkpointInterval != Long.MAX_VALUE) {
+                    nextCheckpointTriggeringRelativeTime += checkpointInterval;

Review Comment:
   With the current implementation, the difference between the actual time of the next invocation of `ScheduledTrigger()` and the value of `nextCheckpointTriggeringRelativeTime` will likely increase with the number of times this method is executed.
   
   It should be better to use `nextCheckpointTriggeringRelativeTime = clock.relativeTimeMillis() + tillNextMillis` here to avoid accumulation of this difference.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -183,6 +189,15 @@ public class CheckpointCoordinator {
     /** A handle to the current periodic trigger, to cancel it when necessary. */
     private ScheduledFuture<?> currentPeriodicTrigger;
 
+    /**
+     * The timestamp (via {@link Clock#relativeTimeMillis()}) when the next checkpoint will be
+     * triggered.
+     *
+     * <p>If it's value is {@link Long#MAX_VALUE}, it means there is not a next checkpoint
+     * scheduled.
+     */
+    private volatile long nextCheckpointTriggeringRelativeTime;

Review Comment:
   Would it be useful to mark this variable with `@GuardedBy("lock")`?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -426,6 +446,40 @@ public boolean isShutdown() {
         return shutdown;
     }
 
+    /**
+     * Reports whether a source operator is currently processing backlog.
+     *
+     * <p>If any source operator is processing backlog, the checkpoint interval would be decided by
+     * {@code execution.checkpointing.interval-during-backlog} instead of {@code
+     * execution.checkpointing.interval}.
+     *
+     * <p>If a source has not invoked this method, the source is considered to have
+     * isProcessingBacklog=false. If a source operator has invoked this method multiple times, the
+     * last reported value is used.
+     *
+     * @param operatorID the operator ID of the source operator.
+     * @param isProcessingBacklog whether the source operator is processing backlog.
+     */
+    public void setIsProcessingBacklog(OperatorID operatorID, boolean isProcessingBacklog) {
+        if (isProcessingBacklog) {
+            backlogOperators.add(operatorID);
+        } else {
+            backlogOperators.remove(operatorID);
+        }
+
+        synchronized (lock) {
+            long currentCheckpointInterval = getCurrentCheckpointInterval();
+            if (currentCheckpointInterval != Long.MAX_VALUE) {
+                long newNextCheckpointTriggeringTime =
+                        clock.relativeTimeMillis() + currentCheckpointInterval;
+                if (newNextCheckpointTriggeringTime + minPauseBetweenCheckpoints

Review Comment:
   Should we still re-schedule checkpoint to an earlier time if `newNextCheckpointTriggeringTime < nextCheckpointTriggeringRelativeTime`?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] lindong28 commented on a diff in pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "lindong28 (via GitHub)" <gi...@apache.org>.
lindong28 commented on code in PR #22931:
URL: https://github.com/apache/flink/pull/22931#discussion_r1278817765


##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -180,8 +186,25 @@ public class CheckpointCoordinator {
     /** Actor that receives status updates from the execution graph this coordinator works for. */
     private JobStatusListener jobStatusListener;
 
+    /**
+     * The current periodic trigger. Used to deduplicate concurrently scheduled checkpoints if any.
+     */
+    @GuardedBy("lock")
+    private ScheduledTrigger currentPeriodicTrigger;
+
     /** A handle to the current periodic trigger, to cancel it when necessary. */
-    private ScheduledFuture<?> currentPeriodicTrigger;
+    @GuardedBy("lock")
+    private ScheduledFuture<?> currentPeriodicTriggerFuture;
+
+    /**
+     * The timestamp (via {@link Clock#relativeTimeMillis()}) when the next checkpoint will be
+     * triggered.
+     *
+     * <p>If it's value is {@link Long#MAX_VALUE}, it means there is not a next checkpoint
+     * scheduled.
+     */
+    @GuardedBy("lock")
+    private volatile long nextCheckpointTriggeringRelativeTime;

Review Comment:
   Now that it will always be used inside `synchronized (lock)`, we can remove the `volatile` here.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -2013,23 +2083,32 @@ private void abortPendingCheckpoints(
 
     private void rescheduleTrigger(long tillNextMillis) {
         cancelPeriodicTrigger();
-        currentPeriodicTrigger = scheduleTriggerWithDelay(tillNextMillis);
+        scheduleTriggerWithDelay(clock.relativeTimeMillis(), tillNextMillis);

Review Comment:
   It seems more readable to invoke `rescheduleTrigger(clock.relativeTimeMillis(), tillNextMillis)` here.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/source/coordinator/SourceCoordinator.java:
##########
@@ -328,6 +337,7 @@ public void executionAttemptFailed(
 
                     context.unregisterSourceReader(subtaskId, attemptNumber);
                     context.attemptFailed(subtaskId, attemptNumber);
+                    markSubtaskAsNoData(subtaskId);

Review Comment:
   The logical subtask still has data even if the physical subtask failed. Failed subtask will be restarted later and we will set `subtasksWithData.add(subtaskId)` very soon after it is restarted. Therefore, it seems better to remove this line.



##########
flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/hybrid/HybridSourceSplitEnumerator.java:
##########
@@ -422,6 +423,15 @@ public <T> void callAsync(
         public void runInCoordinatorThread(Runnable runnable) {
             realContext.runInCoordinatorThread(runnable);
         }
+
+        @Override
+        public void setIsProcessingBacklog(boolean isProcessingBacklog) {
+            if (sourceIndex < sourceSize - 1) {

Review Comment:
   Would it be simpler to remove the `if` statement here?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/source/coordinator/SourceCoordinator.java:
##########
@@ -106,6 +108,10 @@ public class SourceCoordinator<SplitT extends SourceSplit, EnumChkT>
     private final SourceCoordinatorContext<SplitT> context;
 
     private final CoordinatorStore coordinatorStore;
+
+    /** Indices of the subtasks that may still has data to emit. Used to set backlog status. */
+    private final Set<Integer> subtasksWithData;

Review Comment:
   Instead of maintaining this state in `SourceCoordinator`, would it be simpler to re-use the existing `VertexEndOfDataListener` and let this class inform `CheckpointCoordinator` whether the all subtasks of an operator has received end-of-data?
   
   Note that you can use `ExecutionJobVertex#getOperatorIDs` to retrieve operator Ids of a job vertex.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -150,11 +150,17 @@ public class CheckpointCoordinator {
     private final CheckpointIDCounter checkpointIdCounter;
 
     /**
-     * The base checkpoint interval. Actual trigger time may be affected by the max concurrent
-     * checkpoints and minimum-pause values
+     * The checkpoint interval in normal situations. Actual trigger time may be affected by the max

Review Comment:
   It is not clear what `normal` means.
   
   How about saying this:
   The checkpoint interval when there is no source reporting isProcessingBacklog=true.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##########
@@ -2075,10 +2154,30 @@ public void reportStats(long id, ExecutionAttemptID attemptId, CheckpointMetrics
 
     // ------------------------------------------------------------------------
 
-    private final class ScheduledTrigger implements Runnable {
+    final class ScheduledTrigger implements Runnable {
 
         @Override
         public void run() {
+            synchronized (lock) {
+                if (currentPeriodicTrigger != this) {
+                    // Another periodic trigger has been scheduled but this one
+                    // has not been force cancelled yet.
+                    return;
+                }
+
+                long checkpointInterval = getCurrentCheckpointInterval();
+                if (checkpointInterval != Long.MAX_VALUE) {
+                    nextCheckpointTriggeringRelativeTime =
+                            clock.relativeTimeMillis() + checkpointInterval;

Review Comment:
   Should we use `nextCheckpointTriggeringRelativeTime += checkpointInterval` for consistency with the semantics of `scheduleAtFixedRate`()?
   
   Then we can schedule runnable with delay = `nextCheckpointTriggeringRelativeTime - clock.relativeTimeMillis()`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] flinkbot commented on pull request #22931: [FLINK-32514] Support configuring checkpointing interval during process backlog

Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #22931:
URL: https://github.com/apache/flink/pull/22931#issuecomment-1617871167

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a6ffc17080ec886adee32c062a25d9f215b1c7ae",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a6ffc17080ec886adee32c062a25d9f215b1c7ae",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a6ffc17080ec886adee32c062a25d9f215b1c7ae UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org