You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Xintong Song (Jira)" <ji...@apache.org> on 2023/02/22 03:00:00 UTC
[jira] [Commented] (FLINK-31169) KubernetesResourceManagerDriverTest.testOnPodDeleted fails fatally due to 239 exit code
[ https://issues.apache.org/jira/browse/FLINK-31169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17691892#comment-17691892 ]
Xintong Song commented on FLINK-31169:
--------------------------------------
This is indeed caused by FLINK-30908. The "pod termination before being scheduled" is mis-treated as fatal error. I have provided a fix. Without the fix, I can reproduce the problem locally in about 200~300 runs. After the fix, I cannot reproduce the problem in 100k run.
> KubernetesResourceManagerDriverTest.testOnPodDeleted fails fatally due to 239 exit code
> ---------------------------------------------------------------------------------------
>
> Key: FLINK-31169
> URL: https://issues.apache.org/jira/browse/FLINK-31169
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
> Affects Versions: 1.17.0
> Reporter: Matthias Pohl
> Assignee: Xintong Song
> Priority: Blocker
> Labels: test-stability
>
> master: https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46341&view=logs&j=5cae8624-c7eb-5c51-92d3-4d2dacedd221&t=5acec1b4-945b-59ca-34f8-168928ce5199&l=27329
> {code}
> [...]
> Feb 21 04:44:11 [ERROR] Process Exit Code: 239
> Feb 21 04:44:11 [ERROR] Crashed tests:
> Feb 21 04:44:11 [ERROR] org.apache.flink.kubernetes.KubernetesResourceManagerDriverTest
> Feb 21 04:44:11 [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:748)
> [...]
> {code}
> {code}
> [...]
> Test org.apache.flink.kubernetes.KubernetesResourceManagerDriverTest.testOnPodDeleted[testOnPodDeleted()] is running.
> --------------------------------------------------------------------------------
> 04:43:57,681 [ForkJoinPool-4-worker-1] INFO org.apache.flink.kubernetes.KubernetesResourceManagerDriver [] - Recovered 0 pods from previous attempts, current attempt id is 1.
> 04:43:57,701 [testing-rpc-main-thread] INFO org.apache.flink.runtime.externalresource.ExternalResourceUtils [] - Enabled external resources: []
> 04:43:57,705 [testing-rpc-main-thread] INFO org.apache.flink.kubernetes.KubernetesResourceManagerDriver [] - Creating new TaskManager pod with name testing-flink-cluster-taskmanager-1-1 and resource <704,0.0>.
> 04:43:57,708 [testing-rpc-main-thread] INFO org.apache.flink.kubernetes.KubernetesResourceManagerDriver [] - Received new TaskManager pod: testing-flink-cluster-taskmanager-1-1
> 04:43:57,708 [testing-rpc-main-thread] INFO org.apache.flink.kubernetes.KubernetesResourceManagerDriver [] - Pod testing-flink-cluster-taskmanager-1-1 is created.
> 04:43:57,708 [testing-rpc-main-thread] WARN org.apache.flink.kubernetes.KubernetesResourceManagerDriver [] - Pod testing-flink-cluster-taskmanager-1-1 is terminated before being scheduled.
> 04:43:57,709 [testing-rpc-main-thread] ERROR org.apache.flink.kubernetes.KubernetesResourceManagerDriver [] - Error completing resource request.
> org.apache.flink.util.FlinkException: Pod is terminated.
> at org.apache.flink.kubernetes.KubernetesResourceManagerDriver.onPodTerminated(KubernetesResourceManagerDriver.java:379) ~[classes/:?]
> at org.apache.flink.kubernetes.KubernetesResourceManagerDriver.lambda$handlePodEventsInMainThread$2(KubernetesResourceManagerDriver.java:347) ~[classes/:?]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_292]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_292]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_292]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_292]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_292]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_292]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
> 04:43:57,724 [testing-rpc-main-thread] ERROR org.apache.flink.util.FatalExitExceptionHandler [] - FATAL: Thread 'testing-rpc-main-thread' produced an uncaught exception. Stopping the process...
> java.util.concurrent.CompletionException: java.lang.RuntimeException: org.apache.flink.util.FlinkException: Pod is terminated.
> at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273) ~[?:1.8.0_292]
> at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280) ~[?:1.8.0_292]
> at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:838) ~[?:1.8.0_292]
> at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811) ~[?:1.8.0_292]
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_292]
> at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[?:1.8.0_292]
> at org.apache.flink.kubernetes.KubernetesResourceManagerDriver.onPodTerminated(KubernetesResourceManagerDriver.java:379) ~[classes/:?]
> at org.apache.flink.kubernetes.KubernetesResourceManagerDriver.lambda$handlePodEventsInMainThread$2(KubernetesResourceManagerDriver.java:347) ~[classes/:?]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_292]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_292]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_292]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_292]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_292]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_292]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
> Caused by: java.lang.RuntimeException: org.apache.flink.util.FlinkException: Pod is terminated.
> at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321) ~[flink-core-1.18-SNAPSHOT.jar:1.18-SNAPSHOT]
> at org.apache.flink.kubernetes.KubernetesResourceManagerDriver.lambda$requestResource$1(KubernetesResourceManagerDriver.java:233) ~[classes/:?]
> at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836) ~[?:1.8.0_292]
> ... 12 more
> Caused by: org.apache.flink.util.FlinkException: Pod is terminated.
> ... 9 more
> [...]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)