You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "rkhachatryan (via GitHub)" <gi...@apache.org> on 2026/05/22 21:18:42 UTC
[PR] [FLINK-39738] Use pekko ask() instead of tell() to ack the checkpoint [flink]
rkhachatryan opened a new pull request, #28238:
URL: https://github.com/apache/flink/pull/28238
## What is the purpose of the change
This allows to fail checkpoints faster in case of exceeding pekko framesize instead of waiting for checkpoint timeout.
## Verifying this change
Added tests:
- `CheckpointAcknowledgeFailureITCase`
- `AsyncCheckpointRunnableTest.testReportFailureDeclinesWithoutDiscardingState` and `testReportFailureDeclinesWithoutDiscardingState`
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: yes
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
---
##### Was generative AI tooling used to co-author this PR?
`AsyncCheckpointRunnableTest.testReportFailureDeclinesWithoutDiscardingState` and `testReportFailureDeclinesWithoutDiscardingState` written by Claude 4.6, the rest is hand-written.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39738] Use pekko ask() instead of tell() to ack the checkpoint [flink]
Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan commented on PR #28238:
URL: https://github.com/apache/flink/pull/28238#issuecomment-4529374670
Thanks for the reviews!
> As I understand, there is a side effect of failing checkpoints in pekko.ask.timeout instead of execution.checkpointing.timeout -> notifyCheckpointAborted reaches downstream committers in seconds rather than minutes.
Yes, that's true.
Do you have any concerns about that @Savonitar?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39738] Use pekko ask() instead of tell() to ack the checkpoint [flink]
Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan commented on PR #28238:
URL: https://github.com/apache/flink/pull/28238#issuecomment-4525014421
cc @Izeren, @pnowojski
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39738] Use pekko ask() instead of tell() to ack the checkpoint [flink]
Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan commented on code in PR #28238:
URL: https://github.com/apache/flink/pull/28238#discussion_r3293288941
##########
flink-tests/src/test/java/org/apache/flink/test/checkpointing/CheckpointAcknowledgeFailureITCase.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.test.checkpointing;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.MemorySize;
+import org.apache.flink.runtime.checkpoint.CheckpointException;
+import org.apache.flink.runtime.minicluster.MiniCluster;
+import org.apache.flink.runtime.minicluster.RpcServiceSharing;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.streaming.api.functions.sink.v2.DiscardingSink;
+import org.apache.flink.test.junit5.InjectMiniCluster;
+import org.apache.flink.test.junit5.MiniClusterExtension;
+import org.apache.flink.testutils.junit.SharedObjectsExtension;
+import org.apache.flink.testutils.junit.SharedReference;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.SerializedThrowable;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.RegisterExtension;
+
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.concurrent.CompletableFuture;
+
+import static org.apache.flink.configuration.CheckpointingOptions.CHECKPOINTING_TIMEOUT;
+import static org.apache.flink.configuration.CheckpointingOptions.CHECKPOINT_STORAGE;
+import static org.apache.flink.configuration.CheckpointingOptions.FS_SMALL_FILE_THRESHOLD;
+import static org.apache.flink.configuration.RpcOptions.ASK_TIMEOUT_DURATION;
+import static org.apache.flink.configuration.RpcOptions.FRAMESIZE;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+/** CheckpointAcknowledgeFailureITCase. */
+public class CheckpointAcknowledgeFailureITCase extends TestLogger {
+
+ // the minimum allowed by pekko
+ private static final int MIN_PEKKO_FRAME_SIZE = 32000;
+
+ // set frame size high enough to allow checkpoint decline RPC after failing to ACK
+ private static final int PEKKO_FRAME_SIZE = MIN_PEKKO_FRAME_SIZE * 2;
+ // set state size higher than frame size so that checkpoint can not be acked
+ private static final int STATE_SIZE = PEKKO_FRAME_SIZE * 2;
+ // let all the state go via checkpoint ACK RPC to exceed frame size limit
+ private static final MemorySize IN_MEM_STATE_THRESHOLD = new MemorySize(STATE_SIZE * 2);
+
+ // let pekko time out checkpoint ACK
+ private static final Duration ASK_TIMEOUT = Duration.ofMillis(250);
+ // do NOT let flink time out the checkpoint
+ private static final Duration CHECKPOINT_TIMEOUT = ASK_TIMEOUT.multipliedBy(1000);
+
+ @RegisterExtension
+ static final MiniClusterExtension MINI_CLUSTER =
+ new MiniClusterExtension(
+ new MiniClusterResourceConfiguration.Builder()
+ .setConfiguration(getConfiguration())
+ .setNumberTaskManagers(1)
+ .setNumberSlotsPerTaskManager(1)
+ .setRpcServiceSharing(RpcServiceSharing.DEDICATED)
+ .build());
+
+ @RegisterExtension
+ private final SharedObjectsExtension sharedObjects = SharedObjectsExtension.create();
+
+ private static Configuration getConfiguration() {
+ Configuration cfg = new Configuration();
+ cfg.set(FRAMESIZE, new MemorySize(PEKKO_FRAME_SIZE).toString());
+ cfg.set(ASK_TIMEOUT_DURATION, ASK_TIMEOUT);
+ cfg.set(FS_SMALL_FILE_THRESHOLD, IN_MEM_STATE_THRESHOLD);
+ cfg.set(CHECKPOINT_STORAGE, "jobmanager");
+
+ return cfg;
+ }
+
+ /**
+ * Test that if a task is unable to acknowledge a checkpoint then the checkpoint fails
+ * (immediately) with org.apache.pekko.pattern.AskTimeoutException caused. This requires the use
+ * of pekko ask, and not pekko tell under the hood for the acknowledgment. Typical failure
+ * reason is exceeding pekko framesize.
+ */
+ @Test
+ void testCheckpointAckFailure(@InjectMiniCluster MiniCluster cluster) throws Exception {
Review Comment:
Master has snapshot compression which made the test unstable.
Fixed by randomizing snapshot data to make it uncompressable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39738] Use pekko ask() instead of tell() to ack the checkpoint [flink]
Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #28238:
URL: https://github.com/apache/flink/pull/28238#discussion_r3292566991
##########
flink-tests/src/test/java/org/apache/flink/test/checkpointing/CheckpointAcknowledgeFailureITCase.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.test.checkpointing;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.MemorySize;
+import org.apache.flink.runtime.checkpoint.CheckpointException;
+import org.apache.flink.runtime.minicluster.MiniCluster;
+import org.apache.flink.runtime.minicluster.RpcServiceSharing;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
+import org.apache.flink.streaming.api.functions.sink.v2.DiscardingSink;
+import org.apache.flink.test.junit5.InjectMiniCluster;
+import org.apache.flink.test.junit5.MiniClusterExtension;
+import org.apache.flink.testutils.junit.SharedObjectsExtension;
+import org.apache.flink.testutils.junit.SharedReference;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.SerializedThrowable;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.RegisterExtension;
+
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.concurrent.CompletableFuture;
+
+import static org.apache.flink.configuration.CheckpointingOptions.CHECKPOINTING_TIMEOUT;
+import static org.apache.flink.configuration.CheckpointingOptions.CHECKPOINT_STORAGE;
+import static org.apache.flink.configuration.CheckpointingOptions.FS_SMALL_FILE_THRESHOLD;
+import static org.apache.flink.configuration.RpcOptions.ASK_TIMEOUT_DURATION;
+import static org.apache.flink.configuration.RpcOptions.FRAMESIZE;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+/** CheckpointAcknowledgeFailureITCase. */
+public class CheckpointAcknowledgeFailureITCase extends TestLogger {
+
+ // the minimum allowed by pekko
+ private static final int MIN_PEKKO_FRAME_SIZE = 32000;
+
+ // set frame size high enough to allow checkpoint decline RPC after failing to ACK
+ private static final int PEKKO_FRAME_SIZE = MIN_PEKKO_FRAME_SIZE * 2;
+ // set state size higher than frame size so that checkpoint can not be acked
+ private static final int STATE_SIZE = PEKKO_FRAME_SIZE * 2;
+ // let all the state go via checkpoint ACK RPC to exceed frame size limit
+ private static final MemorySize IN_MEM_STATE_THRESHOLD = new MemorySize(STATE_SIZE * 2);
+
+ // let pekko time out checkpoint ACK
+ private static final Duration ASK_TIMEOUT = Duration.ofMillis(250);
+ // do NOT let flink time out the checkpoint
+ private static final Duration CHECKPOINT_TIMEOUT = ASK_TIMEOUT.multipliedBy(1000);
+
+ @RegisterExtension
+ static final MiniClusterExtension MINI_CLUSTER =
+ new MiniClusterExtension(
+ new MiniClusterResourceConfiguration.Builder()
+ .setConfiguration(getConfiguration())
+ .setNumberTaskManagers(1)
+ .setNumberSlotsPerTaskManager(1)
+ .setRpcServiceSharing(RpcServiceSharing.DEDICATED)
+ .build());
+
+ @RegisterExtension
+ private final SharedObjectsExtension sharedObjects = SharedObjectsExtension.create();
+
+ private static Configuration getConfiguration() {
+ Configuration cfg = new Configuration();
+ cfg.set(FRAMESIZE, new MemorySize(PEKKO_FRAME_SIZE).toString());
+ cfg.set(ASK_TIMEOUT_DURATION, ASK_TIMEOUT);
+ cfg.set(FS_SMALL_FILE_THRESHOLD, IN_MEM_STATE_THRESHOLD);
+ cfg.set(CHECKPOINT_STORAGE, "jobmanager");
+
+ return cfg;
+ }
+
+ /**
+ * Test that if a task is unable to acknowledge a checkpoint then the checkpoint fails
+ * (immediately) with org.apache.pekko.pattern.AskTimeoutException caused. This requires the use
+ * of pekko ask, and not pekko tell under the hood for the acknowledgment. Typical failure
+ * reason is exceeding pekko framesize.
+ */
+ @Test
+ void testCheckpointAckFailure(@InjectMiniCluster MiniCluster cluster) throws Exception {
Review Comment:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=75330&view=logs&j=5c8e7682-d68f-54d1-16a2-a09310218a49&t=9d734c8c-6253-55e6-3bce-47e7cdf68ac4&l=7343
Ci fails
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39738] Use pekko ask() instead of tell() to ack the checkpoint [flink]
Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan commented on PR #28238:
URL: https://github.com/apache/flink/pull/28238#issuecomment-4525367670
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39738] Use pekko ask() instead of tell() to ack the checkpoint [flink]
Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan merged PR #28238:
URL: https://github.com/apache/flink/pull/28238
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39738] Use pekko ask() instead of tell() to ack the checkpoint [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #28238:
URL: https://github.com/apache/flink/pull/28238#issuecomment-4522818948
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "612fb0d5959f5638fbd43dff14bd3f01550151e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "612fb0d5959f5638fbd43dff14bd3f01550151e2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 612fb0d5959f5638fbd43dff14bd3f01550151e2 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39738] Use pekko ask() instead of tell() to ack the checkpoint [flink]
Posted by "Savonitar (via GitHub)" <gi...@apache.org>.
Savonitar commented on PR #28238:
URL: https://github.com/apache/flink/pull/28238#issuecomment-4529834306
> > As I understand, there is a side effect of failing checkpoints in pekko.ask.timeout instead of execution.checkpointing.timeout -> notifyCheckpointAborted reaches downstream committers in seconds rather than minutes.
> Yes, that's true.
Do you have any concerns about that @Savonitar?
No, I think it has good impact.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org