[jira] [Created] (FLINK-23874) JM did not store latest checkpiont id into Zookeeper, silently

["Youjun Yuan (Jira)" <ji...@apache.org>]

Youjun Yuan created FLINK-23874:
-----------------------------------

             Summary: JM did not store latest checkpiont id into Zookeeper, silently
                 Key: FLINK-23874
                 URL: https://issues.apache.org/jira/browse/FLINK-23874
             Project: Flink
          Issue Type: Bug
          Components: Runtime / Checkpointing
    Affects Versions: 1.12.1
            Reporter: Youjun Yuan


Job manager did not update the latest successful checkpoint id into zookeeper (with ZK HA setup), at path /flink/\{app_id}/checkpoints/, when JM restart, the job resumed from a very old position.

 

We had a job which was resumed from save point 258, after running for a few days, the latest successful checkpoint was about chk 686. When something trigged the JM to restart, it restored state to save point 258, instead of chk 686.

We checked zookeeper, indeed the stored checkpoint was still 258, which means JM hasn't stored checkpoint id into zookeeper for few days, and without any error message.

 

below are the relevant logs around the restart:
{quote}
{{2021-08-18 11:09:16,505 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed checkpoint 686 for job 00000000000000000000000000000000 (228296 bytes in 827 ms).}}
{quote}
 
{quote}2021-08-18 11:10:13,066 INFO org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation [] - Finished restoring from state handle: IncrementalRemoteKeyedStateHandle\{backendIdentifier=c11d290c-617b-4ea5-b7ed-4853272f32a3, keyGroupRange=KeyGroupRange{startKeyGroup=47, endKeyGroup=48}, checkpointId=258, sharedState={}, privateState=\{OPTIONS-000016=ByteStreamStateHandle{handleName='s3://dp-flink/prd/checkpoints/3beb4dd5-9008-4fd0-9910-f564759b466a/1628912020683/00000000000000000000000000000000/shared/20f35556-5c60-4fca-908c-d05d641c2614', dataBytes=15818}, MANIFEST-000006=ByteStreamStateHandle\{handleName='s3://dp-flink/prd/checkpoints/3beb4dd5-9008-4fd0-9910-f564759b466a/1628912020683/00000000000000000000000000000000/shared/3c8e1c2f-616d-4f18-8b07-4a818e3ca110', dataBytes=336}, CURRENT=ByteStreamStateHandle\{handleName='s3://dp-flink/prd/checkpoints/3beb4dd5-9008-4fd0-9910-f564759b466a/1628912020683/00000000000000000000000000000000/shared/1dc5f341-8a73-4e69-96fb-4b026653da6d', dataBytes=16}}, metaStateHandle=ByteStreamStateHandle\{handleName='s3://dp-flink/prd/checkpoints/3beb4dd5-9008-4fd0-9910-f564759b466a/1628912020683/00000000000000000000000000000000/chk-258/0ef57eb3-0f38-45f5-8f3d-3e7b87f5fd15', dataBytes=1704}, registered=false} without rescaling.
{quote}
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)