You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Congxian Qiu(klion26) (Jira)" <ji...@apache.org> on 2019/10/28 02:07:00 UTC
[jira] [Commented] (FLINK-13969) Resuming Externalized Checkpoint
(rocks, incremental, scale down) end-to-end test fails on Travis
[ https://issues.apache.org/jira/browse/FLINK-13969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16960720#comment-16960720 ]
Congxian Qiu(klion26) commented on FLINK-13969:
-----------------------------------------------
From the log, I found that the restore is successful (from checkpoint 15, checkpoint 1 to checkpoint 15 are all successful), and the second job can do checkpoint successfully, but we have a failed checkpoint(checkpoint 16) in the first job, because the first job has been canceled when doing the snapshot. when we check whethe there is any exception in the log, then found the snapshot exception there, and the test failed.
I think for this test, maybe modify the check logic is ok.
{code:java}
2019-09-05 02:26:19,154 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 15 for job 5614f2d16ac6f58966f738f0c93a2f63 (628221 bytes in 1824 ms).
2019-09-05 02:26:19,189 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 16 @ 1567650379153 for job 5614f2d16ac6f58966f738f0c93a2f63.
2019-09-05 02:26:19,788 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Discarding checkpoint 16 of job 5614f2d16ac6f58966f738f0c93a2f63.
org.apache.flink.runtime.checkpoint.CheckpointException: Could not complete snapshot 16 for operator ArtificalKeyedStateMapper_Avro -> ArtificalOperatorStateMapper (2/4). Failure reason: Checkpoint was declined.
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:431)
at org.apache.flink.streaming.runtime.tasks.StreamTask$CheckpointingOperation.checkpointStreamOperator(StreamTask.java:1302)
at org.apache.flink.streaming.runtime.tasks.StreamTask$CheckpointingOperation.executeCheckpointing(StreamTask.java:1236)
at org.apache.flink.streaming.runtime.tasks.StreamTask.checkpointState(StreamTask.java:892)
at org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:797)
at org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:728)
at org.apache.flink.streaming.runtime.io.CheckpointBarrierHandler.notifyCheckpoint(CheckpointBarrierHandler.java:88)
at org.apache.flink.streaming.runtime.io.CheckpointBarrierAligner.processBarrier(CheckpointBarrierAligner.java:177)
at org.apache.flink.streaming.runtime.io.CheckpointedInputGate.pollNext(CheckpointedInputGate.java:155)
at org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.pollNextNullable(StreamTaskNetworkInput.java:118)
at org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.pollNextNullable(StreamTaskNetworkInput.java:48)
at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:144)
at org.apache.flink.streaming.runtime.tasks.StreamTask.performDefaultAction(StreamTask.java:277)
2019-09-05 02:26:25,862 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Starting job 74cca894c6d50cad64d5377be54cf14b from savepoint file:///home/travis/build/apache/flink/flink-end-to-end-tests/test-scripts/temp-test-directory-44167013242/externalized-chckpt-e2e-backend-dir/5614f2d16ac6f58966f738f0c93a2f63/chk-15 ()
2019-09-05 02:26:25,916 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Reset the checkpoint ID of job 74cca894c6d50cad64d5377be54cf14b to 16.
2019-09-05 02:26:25,917 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Restoring job 74cca894c6d50cad64d5377be54cf14b from latest valid checkpoint: Checkpoint 15 @ 0 for 74cca894c6d50cad64d5377be54cf14b.
2019-09-05 02:26:25,928 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - No master state to restore
2019-09-05 02:26:25,928 INFO org.apache.flink.runtime.jobmaster.JobManagerRunner - JobManager runner for job General purpose test job (74cca894c6d50cad64d5377be54cf14b) was granted leadership with session id 00000000-0000-0000-0000-000000000000 at akka.tcp://flink@localhost:6123/user/jobmanager_1.
2019-09-05 02:26:25,929 INFO org.apache.flink.runtime.jobmaster.JobMaster - Starting execution of job General purpose test job (74cca894c6d50cad64d5377be54cf14b) under job master id 000000000000
2019-09-05 02:26:27,402 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 16 @ 1567650387401 for job 74cca894c6d50cad64d5377be54cf14b.
2019-09-05 02:26:27,979 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 16 for job 74cca894c6d50cad64d5377be54cf14b (482657 bytes in 577 ms).
2019-09-05 02:26:28,402 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 17 @ 1567650388402 for job 74cca894c6d50cad64d5377be54cf14b.
{code}
> Resuming Externalized Checkpoint (rocks, incremental, scale down) end-to-end test fails on Travis
> -------------------------------------------------------------------------------------------------
>
> Key: FLINK-13969
> URL: https://issues.apache.org/jira/browse/FLINK-13969
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
> Affects Versions: 1.10.0
> Reporter: Till Rohrmann
> Priority: Critical
> Labels: test-stability
> Fix For: 1.10.0
>
>
> The {{Resuming Externalized Checkpoint (rocks, incremental, scale down)}} end-to-end test fails on Travis because its log contains an exception
> {code}
> org.apache.flink.runtime.checkpoint.CheckpointException: Could not complete snapshot 16 for operator ArtificalKeyedStateMapper_Avro -> ArtificalOperatorStateMapper (2/4). Failure reason: Checkpoint was declined.
> at org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:431)
> at org.apache.flink.streaming.runtime.tasks.StreamTask$CheckpointingOperation.checkpointStreamOperator(StreamTask.java:1302)
> at org.apache.flink.streaming.runtime.tasks.StreamTask$CheckpointingOperation.executeCheckpointing(StreamTask.java:1236)
> at org.apache.flink.streaming.runtime.tasks.StreamTask.checkpointState(StreamTask.java:892)
> at org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:797)
> at org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:728)
> at org.apache.flink.streaming.runtime.io.CheckpointBarrierHandler.notifyCheckpoint(CheckpointBarrierHandler.java:88)
> at org.apache.flink.streaming.runtime.io.CheckpointBarrierAligner.processBarrier(CheckpointBarrierAligner.java:177)
> at org.apache.flink.streaming.runtime.io.CheckpointedInputGate.pollNext(CheckpointedInputGate.java:155)
> at org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.pollNextNullable(StreamTaskNetworkInput.java:118)
> at org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.pollNextNullable(StreamTaskNetworkInput.java:48)
> at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:144)
> at org.apache.flink.streaming.runtime.tasks.StreamTask.performDefaultAction(StreamTask.java:277)
> at org.apache.flink.streaming.runtime.tasks.mailbox.execution.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:147)
> at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:404)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException: Cannot register Closeable, registry is already closed. Closing argument.
> at org.apache.flink.util.AbstractCloseableRegistry.registerCloseable(AbstractCloseableRegistry.java:85)
> at org.apache.flink.runtime.state.AsyncSnapshotCallable$AsyncSnapshotTask.<init>(AsyncSnapshotCallable.java:122)
> at org.apache.flink.runtime.state.AsyncSnapshotCallable$AsyncSnapshotTask.<init>(AsyncSnapshotCallable.java:110)
> at org.apache.flink.runtime.state.AsyncSnapshotCallable.toAsyncSnapshotFutureTask(AsyncSnapshotCallable.java:104)
> at org.apache.flink.contrib.streaming.state.snapshot.RocksIncrementalSnapshotStrategy.doSnapshot(RocksIncrementalSnapshotStrategy.java:170)
> at org.apache.flink.contrib.streaming.state.snapshot.RocksDBSnapshotStrategyBase.snapshot(RocksDBSnapshotStrategyBase.java:126)
> at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.snapshot(RocksDBKeyedStateBackend.java:439)
> at org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:411)
> ... 17 more
> {code}
> https://api.travis-ci.org/v3/job/580915660/log.txt
--
This message was sent by Atlassian Jira
(v8.3.4#803005)