You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Yuepeng Pan (Jira)" <ji...@apache.org> on 2026/06/13 17:28:00 UTC

[jira] [Commented] (FLINK-39858) RestClient.close() can leave in-flight request futures uncompleted, hanging the caller

    [ https://issues.apache.org/jira/browse/FLINK-39858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18088775#comment-18088775 ] 

Yuepeng Pan commented on FLINK-39858:
-------------------------------------

BP-2.3 was merged via: e2a66296d23625aa5574b6ac4650ce0a0ca9407a

> RestClient.close() can leave in-flight request futures uncompleted, hanging the caller
> --------------------------------------------------------------------------------------
>
>                 Key: FLINK-39858
>                 URL: https://issues.apache.org/jira/browse/FLINK-39858
>             Project: Flink
>          Issue Type: Bug
>          Components: Runtime / REST, Tests
>            Reporter: Martijn Visser
>            Assignee: Martijn Visser
>            Priority: Major
>              Labels: pull-request-available, test-stability
>             Fix For: 2.3.0, 2.4.0
>
>
> {{RestClientTest.testRestClientClosedHandling}} hung intermittently in the {{test_cron_hadoop313}} leg on master, where the surefire JVM produced no output for 900s and was watchdog-killed.
> Unlike a deterministic failure it only reproduces under load: the preceding {{ForwardEdgesAdapterTest}} (100k invocations, ~531s) saturated the agent and widened the race window.
> The thread dump taken at the watchdog kill shows the test worker parked forever on the request future:
> {code:java}
> "ForkJoinPool-295-worker-1"
>    java.lang.Thread.State: WAITING
>         at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2072)
>         at org.apache.flink.core.testutils.FlinkCompletableFutureAssert.assertEventuallyFails(FlinkCompletableFutureAssert.java:161)
>         at org.apache.flink.core.testutils.FlinkCompletableFutureAssert.eventuallyFailsWith(FlinkCompletableFutureAssert.java:135)
>         at org.apache.flink.runtime.rest.RestClientTest.testRestClientClosedHandling(RestClientTest.java:257)
> {code}
> Root cause: {{RestClient}} tracks in-flight requests only via {{responseChannelFutures}}, which holds each request's connect-phase {{CompletableFuture}}. The connect listener removes that future the moment the TCP connection is established, before the request enters its in-flight (response) phase, so from then on the request is tracked by nothing. On {{close()}}, {{notifyResponseFuturesOfShutdown()}} only fails the futures still in {{responseChannelFutures}}. When {{close()}} races with a request that has just passed the connect phase, the terminal response future is never completed (the channel's {{channelInactive}} callback may not be dispatched once the event-loop group is being torn down), so a caller blocking on it hangs indefinitely.
> FLINK-39180 previously treated the same test failure as a benign assertion-type mismatch and assumed the future is always completed on close; that holds only for the connect phase, not the in-flight phase, so the underlying defect remained.
> Solution: track the terminal per-request response future for its whole lifetime in a dedicated set, fail those futures on close, and re-check {{isRunning}} after registration (failing only a
> future still atomically registered) to close the check-then-act race.
> Failed CI build (Azure DevOps {{flink-ci.flink-master-mirror}}, 20260604.1): https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=75618



--
This message was sent by Atlassian Jira
(v8.20.10#820010)