You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by "Guowei Ma (Jira)" <ji...@apache.org> on 2021/03/04 07:57:00 UTC
[jira] [Created] (FLINK-21600) Resuming Savepoint (rocks, no
parallelism change, heap timers) end-to-end test failed
Guowei Ma created FLINK-21600:
---------------------------------
Summary: Resuming Savepoint (rocks, no parallelism change, heap timers) end-to-end test failed
Key: FLINK-21600
URL: https://issues.apache.org/jira/browse/FLINK-21600
Project: Flink
Issue Type: Bug
Components: Runtime / Checkpointing
Affects Versions: 1.12.2
Reporter: Guowei Ma
{code:java}
2021-03-03T23:23:22.9286204Z The program finished with the following exception:
2021-03-03T23:23:22.9286551Z
2021-03-03T23:23:22.9287394Z org.apache.flink.util.FlinkException: Could not stop with a savepoint job "6c88aa43b703192e88987473e722c22c".
2021-03-03T23:23:22.9288135Z at org.apache.flink.client.cli.CliFrontend.lambda$stop$5(CliFrontend.java:581)
2021-03-03T23:23:22.9288846Z at org.apache.flink.client.cli.CliFrontend.runClusterAction(CliFrontend.java:1002)
2021-03-03T23:23:22.9289787Z at org.apache.flink.client.cli.CliFrontend.stop(CliFrontend.java:569)
2021-03-03T23:23:22.9290721Z at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1069)
2021-03-03T23:23:22.9291800Z at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
2021-03-03T23:23:22.9292565Z at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
2021-03-03T23:23:22.9293368Z at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
2021-03-03T23:23:22.9294111Z Caused by: java.util.concurrent.TimeoutException
2021-03-03T23:23:22.9294842Z at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886)
2021-03-03T23:23:22.9295735Z at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021)
2021-03-03T23:23:22.9296418Z at org.apache.flink.client.cli.CliFrontend.lambda$stop$5(CliFrontend.java:579)
2021-03-03T23:23:22.9297527Z ... 6 more
2021-03-03T23:23:22.9622887Z Mar 03 23:23:22 Waiting for job (6c88aa43b703192e88987473e722c22c) to reach terminal state FINISHED ...
2021-03-03T23:36:57.3094695Z Mar 03 23:36:57 Test (pid: 3517) did not finish after 900 seconds.
{code}
killing logs
{code:java}
2021-03-03T23:36:57.3232128Z Mar 03 23:36:57 java.util.concurrent.RejectedExecutionException: event executor terminated
2021-03-03T23:36:57.3233348Z Mar 03 23:36:57 at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:926) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3234766Z Mar 03 23:36:57 at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:353) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3236170Z Mar 03 23:36:57 at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:346) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3237567Z Mar 03 23:36:57 at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:828) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3265526Z Mar 03 23:36:57 at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:818) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3267472Z Mar 03 23:36:57 at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:471) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3268806Z Mar 03 23:36:57 at org.apache.flink.shaded.netty4.io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:87) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3275491Z Mar 03 23:36:57 at org.apache.flink.shaded.netty4.io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:81) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3277092Z Mar 03 23:36:57 at org.apache.flink.shaded.netty4.io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:86) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3279030Z Mar 03 23:36:57 at org.apache.flink.shaded.netty4.io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:323) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3280277Z Mar 03 23:36:57 at org.apache.flink.shaded.netty4.io.netty.bootstrap.Bootstrap.doResolveAndConnect(Bootstrap.java:155) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3281591Z Mar 03 23:36:57 at org.apache.flink.shaded.netty4.io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:139) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3282748Z Mar 03 23:36:57 at org.apache.flink.shaded.netty4.io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:123) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3283868Z Mar 03 23:36:57 at org.apache.flink.runtime.rest.RestClient.submitRequest(RestClient.java:421) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3284969Z Mar 03 23:36:57 at org.apache.flink.runtime.rest.RestClient.sendRequest(RestClient.java:344) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3286053Z Mar 03 23:36:57 at org.apache.flink.runtime.rest.RestClient.sendRequest(RestClient.java:258) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3287244Z Mar 03 23:36:57 at org.apache.flink.client.program.rest.RestClusterClient.lambda$sendRetriableRequest$23(RestClusterClient.java:777) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3288091Z Mar 03 23:36:57 at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1072) [?:?]
2021-03-03T23:36:57.3288763Z Mar 03 23:36:57 at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) [?:?]
2021-03-03T23:36:57.3289660Z Mar 03 23:36:57 at java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:610) [?:?]
2021-03-03T23:36:57.3290339Z Mar 03 23:36:57 at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:649) [?:?]
2021-03-03T23:36:57.3291011Z Mar 03 23:36:57 at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478) [?:?]
2021-03-03T23:36:57.3291678Z Mar 03 23:36:57 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
2021-03-03T23:36:57.3292340Z Mar 03 23:36:57 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
2021-03-03T23:36:57.3292927Z Mar 03 23:36:57 at java.lang.Thread.run(Thread.java:834) [?:?]
2021-03-03T23:36:57.3294023Z Mar 03 23:36:57 2021-03-03 23:23:22,926 ERROR org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.rejectedExecution [] - Failed to submit a listener notification task. Event loop shut down?
2021-03-03T23:36:57.3294797Z Mar 03 23:36:57 java.util.concurrent.RejectedExecutionException: event executor terminated
2021-03-03T23:36:57.3330282Z Mar 03 23:36:57 at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:926) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3332091Z Mar 03 23:36:57 at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:353) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3333454Z Mar 03 23:36:57 at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:346) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3334983Z Mar 03 23:36:57 at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:828) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3336322Z Mar 03 23:36:57 at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:818) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3337809Z Mar 03 23:36:57 at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:841) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3339118Z Mar 03 23:36:57 at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:498) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3368256Z Mar 03 23:36:57 at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:183) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3369794Z Mar 03 23:36:57 at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:95) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3371107Z Mar 03 23:36:57 at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:30) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3372302Z Mar 03 23:36:57 at org.apache.flink.runtime.rest.RestClient.submitRequest(RestClient.java:425) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3373399Z Mar 03 23:36:57 at org.apache.flink.runtime.rest.RestClient.sendRequest(RestClient.java:344) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3374497Z Mar 03 23:36:57 at org.apache.flink.runtime.rest.RestClient.sendRequest(RestClient.java:258) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3375694Z Mar 03 23:36:57 at org.apache.flink.client.program.rest.RestClusterClient.lambda$sendRetriableRequest$23(RestClusterClient.java:777) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3376535Z Mar 03 23:36:57 at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1072) [?:?]
2021-03-03T23:36:57.3377218Z Mar 03 23:36:57 at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) [?:?]
2021-03-03T23:36:57.3377874Z Mar 03 23:36:57 at java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:610) [?:?]
2021-03-03T23:36:57.3378745Z Mar 03 23:36:57 at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:649) [?:?]
2021-03-03T23:36:57.3379443Z Mar 03 23:36:57 at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478) [?:?]
2021-03-03T23:36:57.3380305Z Mar 03 23:36:57 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
2021-03-03T23:36:57.3381397Z Mar 03 23:36:57 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
2021-03-03T23:36:57.3382019Z Mar 03 23:36:57 at java.lang.Thread.run(Thread.java:834) [?:?]
2021-03-03T23:36:57.3382989Z Mar 03 23:36:57 2021-03-03 23:23:22,926 ERROR org.apache.flink.client.cli.CliFrontend [] - Error while running the command.
2021-03-03T23:36:57.3383776Z Mar 03 23:36:57 org.apache.flink.util.FlinkException: Could not stop with a savepoint job "6c88aa43b703192e88987473e722c22c".
2021-03-03T23:36:57.3385129Z Mar 03 23:36:57 at org.apache.flink.client.cli.CliFrontend.lambda$stop$5(CliFrontend.java:581) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3422634Z Mar 03 23:36:57 at org.apache.flink.client.cli.CliFrontend.runClusterAction(CliFrontend.java:1002) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3423852Z Mar 03 23:36:57 at org.apache.flink.client.cli.CliFrontend.stop(CliFrontend.java:569) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2021-03-03T23:36:57.3425311Z Mar 03 23:36:57 at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1069) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)