You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Peng Zhang (Jira)" <ji...@apache.org> on 2021/03/05 09:09:00 UTC
[jira] [Comment Edited] (FLINK-21472) FencingTokenException:
Fencing token mismatch
[ https://issues.apache.org/jira/browse/FLINK-21472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17295887#comment-17295887 ]
Peng Zhang edited comment on FLINK-21472 at 3/5/21, 9:08 AM:
-------------------------------------------------------------
[~fly_in_gis] Thanks! I will try Flink 1.12.2 once it is available in docker. For more information, in our case the FencingTokenException happened when a JobManager is redeployed to another node by K8S. And the new JobManager cannot start the jobs from checkpoints
{{2021-03-04 17:04:44,928 INFO org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] - Recovering checkpoints from KubernetesStateHandleStore\{configMapName='stellar-flink-cluster-8ea8bb860bdefc3884cd586f4473295a-jobmanager-leader'}. }}
{{2021-03-04 17:04:44,928 INFO org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] - Recovering checkpoints from KubernetesStateHandleStore\{configMapName='stellar-flink-cluster-8ea8bb860bdefc3884cd586f4473295a-jobmanager-leader'}. }}
{{2021-03-04 17:04:44,933 INFO org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] - Found 1 checkpoints in KubernetesStateHandleStore\{configMapName='stellar-flink-cluster-8ea8bb860bdefc3884cd586f4473295a-jobmanager-leader'}. 2021-03-04 17:04:44,933 INFO org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] - Trying to fetch 1 checkpoints from storage. }}
{{2021-03-04 17:04:44,933 INFO org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] - Trying to retrieve checkpoint 18. 2021-03-04 17:04:44,963 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Restoring job 8ea8bb860bdefc3884cd586f4473295a from Checkpoint 18 @ 1614877356663 for 8ea8bb860bdefc3884cd586f4473295a located at s3a://zalando-stellar-flink-state-eu-central-1-staging/checkpoints/8ea8bb860bdefc3884cd586f4473295a/chk-18. }}
{{2021-03-04 17:04:44,964 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - No master state to restore 2021-03-04 17:04:44,965 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Using failover strategy org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy@530feb4d for BrandCollectionTrackingJob (8ea8bb860bdefc3884cd586f4473295a). }}
{{2021-03-04 17:04:44,970 INFO org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl [] - JobManager runner for job BrandCollectionTrackingJob (8ea8bb860bdefc3884cd586f4473295a) was granted leadership with session id ecb717f4-089f-48af-8d82-63333f7d4b17 at akka.tcp://flink@stellar-flink-jobmanager:6123/user/rpc/jobmanager_4. 2021-03-04 17:05:09,618 WARN io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager [] - Exec Failure java.net.SocketTimeoutException: sent ping but didn't receive pong within 30000ms (after 1 successful ping/pongs) }}
{{2021-03-04 17:05:14,990 ERROR org.apache.flink.runtime.rest.handler.job.JobsOverviewHandler [] - Unhandled exception. org.apache.flink.runtime.rpc.exceptions.FencingTokenException: Fencing token mismatch: Ignoring message LocalFencedMessage(9c31a87cf2ff475d049819f3fb9e4cd7, LocalRpcInvocation(requestMultipleJobDetails(Time))) because the fencing token 9c31a87cf2ff475d049819f3fb9e4cd7 did not match the expected fencing token bbc60d6ee1cc9717561f755149454d94.}}
was (Author: petrizhang):
[~fly_in_gis] Thanks! I will try Flink 1.12.2 once it is available in docker. For more information, in our case the FencingTokenException happened when a JobManager is redeployed to another node by K8S. And the new JobManager cannot start the jobs from checkpoints
{{2021-03-04 17:04:44,928 INFO org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] - Recovering checkpoints from KubernetesStateHandleStore\{configMapName='stellar-flink-cluster-8ea8bb860bdefc3884cd586f4473295a-jobmanager-leader'}. 2021-03-04 17:04:44,928 INFO org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] - Recovering checkpoints from KubernetesStateHandleStore\{configMapName='stellar-flink-cluster-8ea8bb860bdefc3884cd586f4473295a-jobmanager-leader'}. 2021-03-04 17:04:44,933 INFO org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] - Found 1 checkpoints in KubernetesStateHandleStore\{configMapName='stellar-flink-cluster-8ea8bb860bdefc3884cd586f4473295a-jobmanager-leader'}. 2021-03-04 17:04:44,933 INFO org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] - Trying to fetch 1 checkpoints from storage. 2021-03-04 17:04:44,933 INFO org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] - Trying to retrieve checkpoint 18. 2021-03-04 17:04:44,963 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Restoring job 8ea8bb860bdefc3884cd586f4473295a from Checkpoint 18 @ 1614877356663 for 8ea8bb860bdefc3884cd586f4473295a located at s3a://zalando-stellar-flink-state-eu-central-1-staging/checkpoints/8ea8bb860bdefc3884cd586f4473295a/chk-18. 2021-03-04 17:04:44,964 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - No master state to restore 2021-03-04 17:04:44,965 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Using failover strategy org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy@530feb4d for BrandCollectionTrackingJob (8ea8bb860bdefc3884cd586f4473295a). 2021-03-04 17:04:44,970 INFO org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl [] - JobManager runner for job BrandCollectionTrackingJob (8ea8bb860bdefc3884cd586f4473295a) was granted leadership with session id ecb717f4-089f-48af-8d82-63333f7d4b17 at akka.tcp://flink@stellar-flink-jobmanager:6123/user/rpc/jobmanager_4. 2021-03-04 17:05:09,618 WARN io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager [] - Exec Failure java.net.SocketTimeoutException: sent ping but didn't receive pong within 30000ms (after 1 successful ping/pongs) 2021-03-04 17:05:14,990 ERROR org.apache.flink.runtime.rest.handler.job.JobsOverviewHandler [] - Unhandled exception. org.apache.flink.runtime.rpc.exceptions.FencingTokenException: Fencing token mismatch: Ignoring message LocalFencedMessage(9c31a87cf2ff475d049819f3fb9e4cd7, LocalRpcInvocation(requestMultipleJobDetails(Time))) because the fencing token 9c31a87cf2ff475d049819f3fb9e4cd7 did not match the expected fencing token bbc60d6ee1cc9717561f755149454d94.}}
> FencingTokenException: Fencing token mismatch
> ---------------------------------------------
>
> Key: FLINK-21472
> URL: https://issues.apache.org/jira/browse/FLINK-21472
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes
> Affects Versions: 1.12.1
> Reporter: hayden zhou
> Priority: Major
> Attachments: flink--standalonesession-0-mta-flink-jobmanager-864d6c8cbb-rmsxw.log
>
>
> org.apache.flink.runtime.rest.handler.job.JobsOverviewHandler [] - Unhandled exception.
> org.apache.flink.runtime.rpc.exceptions.FencingTokenException: Fencing token mismatch: Ignoring message LocalFencedMessage(8fac01d8e3e3988223a2e5c6e3f04f1e, LocalRpcInvocation(requestMultipleJobDetails(Time))) because the fencing token 8fac01d8e3e3988223a2e5c6e3f04f1e did not match the expected fencing token 8c37414f464bca76144e6cabc946474b.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)