[jira] [Updated] (FLINK-40132) MixedKubernetesServerExtension teardown flaky with RejectedExecutionException from Fabric8 Vert.x mock server shutdown

["Martijn Visser (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-40132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Martijn Visser updated FLINK-40132:
-----------------------------------
    Affects Version/s: 2.3.1

> MixedKubernetesServerExtension teardown flaky with RejectedExecutionException from Fabric8 Vert.x mock server shutdown
> ----------------------------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-40132
>                 URL: https://issues.apache.org/jira/browse/FLINK-40132
>             Project: Flink
>          Issue Type: Bug
>          Components: Deployment / Kubernetes
>    Affects Versions: 2.3.1, 2.4.0
>            Reporter: Martijn Visser
>            Assignee: Pedro Mázala
>            Priority: Major
>              Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=77011&view=results (leg: test_cron_azure misc)
> {code} 
> 05:07:40.216 [ERROR] ...InitTaskManagerDecoratorWithPodTemplateTest.testNodeSelectorMerging <<< ERROR! java.util.concurrent.RejectedExecutionException: event executor terminated at io.netty.util.concurrent.SingleThreadEventExecutor.reject(...) at io.vertx.core.impl.future.FutureImpl.onComplete(...) at io.fabric8.mockwebserver.MockWebServer.await(MockWebServer.java:257) at io.fabric8.mockwebserver.MockWebServer.shutdown(MockWebServer.java:169) at io.fabric8.kubernetes.client.server.mock.KubernetesMockServer.destroy(...) at org.apache.flink.kubernetes.MixedKubernetesServerExtension.afterEach(MixedKubernetesServerExtension.java:76) {code}
> Tests using the Fabric8 mock server intermittently fail in {{MixedKubernetesServerExtension#afterEach}} -> {{KubernetesMockServer.destroy()}} -> {{MockWebServer#shutdown()}} with {{RejectedExecutionException: event executor terminated}}. The test body passes; the exception is thrown during teardown (surefire attributes it to the last-run method, here {{testNodeSelectorMerging}}).
> This is an upstream Fabric8 bug: a timing race (+ non-idempotent shutdown) in the Vert.x-based {{MockWebServer#shutdown()}} — fabric8io/kubernetes-client#7747, fixed in Fabric8 7.7.0. It was introduced into Flink by FLINK-38093, which moved the mock server onto the Vert.x implementation when bumping Fabric8 to 7.3.1 (Flink master pins 7.3.1).
> We could either tolerate the RejectedExecutionException on teardown, or choose upgrading Fabric8 to >= 7.7.0. That is the eventual clean fix but drags OkHttp 3->5 and an unrelocated Kotlin runtime into the core distribution by default.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)