You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "jackray wang (Jira)" <ji...@apache.org> on 2025/08/11 11:58:00 UTC
[jira] (FLINK-37802) files in directory job-result-store was incorrect
[ https://issues.apache.org/jira/browse/FLINK-37802 ]
jackray wang deleted comment on FLINK-37802:
--------------------------------------
was (Author: jackray):
!image-2025-08-11-19-56-32-416.png!
> files in directory job-result-store was incorrect
> -------------------------------------------------
>
> Key: FLINK-37802
> URL: https://issues.apache.org/jira/browse/FLINK-37802
> Project: Flink
> Issue Type: Bug
> Affects Versions: 1.17.1
> Reporter: yanggang
> Priority: Major
> Attachments: image-2025-05-14-16-32-05-505.png, image-2025-05-14-17-18-20-701.png, image-2025-05-14-17-18-32-785.png
>
>
> there parse json file , throw a exception when json is not correct
> !image-2025-05-14-17-18-32-785.png!
> file size about 3.3MB
> !image-2025-05-14-16-32-05-505.png!
> this is file content. business data and some trace was hidden:
> {
> "result": {
> "id": "fcae72c8faccbab418489585445c1969",
> "application-status": "FAILED",
> "accumulator-results": {},
> "net-runtime": 169179245,
> "failure-cause":
> { "class": "org.apache.flink.runtime.JobException", "stack-trace": "org.apache.flink.runtime.JobException: Recovery is suppressed by FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=3, backoffTimeMS=30000)\n\tat org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:139)\n\tat org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:83)\n\tat org.apache.flink.runtime.scheduler.DefaultScheduler.recordTaskFailure(DefaultScheduler.java:258)\n\tat org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure --- hidden --- }
> \n\tat org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:839)\n\tat org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:788)\n\tat org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:952)\n\tat org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:931)\n\tat org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:745)\n\tat org.apache.flink.runtime.taskmanager.Task.run(Task.java:562)\n\tat java.lang.Thread.run(Thread.java:750)\nCaused by: java.util.concurrent.TimeoutException: Async function call has timed out.\n\t... 19 more\n"
>
>
> The above is the json content of the file, mybe missed }}}
>
>
> flink error log:
> 2025-05-13 22:30:19,082 ERROR [cluster-io-thread-1] [org.apache.flink.runtime.entrypoint.ClusterEntrypoint :548] - Fatal error occurred in the cluster entrypoint.
> java.util.concurrent.CompletionException: org.apache.flink.util.FlinkRuntimeException: Could not retrieve JobResults of globally-terminated jobs from JobResultStore
> at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273) ~[?:1.8.0_432]
> at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280) [?:1.8.0_432]
> at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1606) [?:1.8.0_432]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_432]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_432]
> at java.lang.Thread.run(Thread.java:750) [?:1.8.0_432]
> Caused by: org.apache.flink.util.FlinkRuntimeException: Could not retrieve JobResults of globally-terminated jobs from JobResultStore
> at org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess.getDirtyJobResults(SessionDispatcherLeaderProcess.java:196) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.runtime.dispatcher.runner.AbstractDispatcherLeaderProcess.supplyUnsynchronizedIfRunning(AbstractDispatcherLeaderProcess.java:198) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess.getDirtyJobResultsIfRunning(SessionDispatcherLeaderProcess.java:188) ~[performance-flink-job-24.0.2118.jar:?]
> at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604) ~[?:1.8.0_432]
> ... 3 more
> Caused by: org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (org.apache.flink.core.fs.local.LocalDataInputStream); line: 1, column: 147])
> at [Source: (org.apache.flink.core.fs.local.LocalDataInputStream); line: 1, column: 3223450] (through reference chain: org.apache.flink.runtime.highavailability.FileSystemJobResultStore$JsonJobResultEntry["result"])
> at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:392) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:351) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializerBase.wrapAndThrow(BeanDeserializerBase.java:1821) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:566) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:439) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1405) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:352) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:185) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4674) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3666) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.runtime.highavailability.FileSystemJobResultStore.getDirtyResultsInternal(FileSystemJobResultStore.java:208) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.runtime.highavailability.AbstractThreadsafeJobResultStore.withReadLock(AbstractThreadsafeJobResultStore.java:118) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.runtime.highavailability.AbstractThreadsafeJobResultStore.getDirtyResults(AbstractThreadsafeJobResultStore.java:100) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess.getDirtyJobResults(SessionDispatcherLeaderProcess.java:194) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.runtime.dispatcher.runner.AbstractDispatcherLeaderProcess.supplyUnsynchronizedIfRunning(AbstractDispatcherLeaderProcess.java:198) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess.getDirtyJobResultsIfRunning(SessionDispatcherLeaderProcess.java:188) ~[performance-flink-job-24.0.2118.jar:?]
> at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604) ~[?:1.8.0_432]
> ... 3 more
> Caused by: org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (org.apache.flink.core.fs.local.LocalDataInputStream); line: 1, column: 147])
> at [Source: (org.apache.flink.core.fs.local.LocalDataInputStream); line: 1, column: 3223450]
> at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.base.ParserMinimalBase._reportInvalidEOF(ParserMinimalBase.java:682) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.base.ParserBase._handleEOF(ParserBase.java:494) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.base.ParserBase._eofAsNextChar(ParserBase.java:511) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.json.UTF8StreamJsonParser._skipWSOrEnd(UTF8StreamJsonParser.java:3039) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextFieldName(UTF8StreamJsonParser.java:1034) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer._deserializeContainerNoRecursion(JsonNodeDeserializer.java:440) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:84) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:20) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:4650) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2941) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParser.readValueAsTree(JsonParser.java:2365) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.runtime.rest.messages.json.SerializedThrowableDeserializer.deserialize(SerializedThrowableDeserializer.java:45) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.runtime.rest.messages.json.JobResultDeserializer.deserialize(JobResultDeserializer.java:106) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.runtime.rest.messages.json.JobResultDeserializer.deserialize(JobResultDeserializer.java:50) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:542) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:564) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:439) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1405) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:352) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:185) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4674) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3666) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.runtime.highavailability.FileSystemJobResultStore.getDirtyResultsInternal(FileSystemJobResultStore.java:208) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.runtime.highavailability.AbstractThreadsafeJobResultStore.withReadLock(AbstractThreadsafeJobResultStore.java:118) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.runtime.highavailability.AbstractThreadsafeJobResultStore.getDirtyResults(AbstractThreadsafeJobResultStore.java:100) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess.getDirtyJobResults(SessionDispatcherLeaderProcess.java:194) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.runtime.dispatcher.runner.AbstractDispatcherLeaderProcess.supplyUnsynchronizedIfRunning(AbstractDispatcherLeaderProcess.java:198) ~[performance-flink-job-24.0.2118.jar:?]
> at org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess.getDirtyJobResultsIfRunning(SessionDispatcherLeaderProcess.java:188) ~[performance-flink-job-24.0.2118.jar:?]
> at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604) ~[?:1.8.0_432]
> ... 3 more
--
This message was sent by Atlassian Jira
(v8.20.10#820010)