You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@storm.apache.org by "Aaron Gresch (Jira)" <ji...@apache.org> on 2021/04/29 19:42:00 UTC
[jira] [Commented] (STORM-3769) Failed adding references to blobs:
FileNotFoundException
[ https://issues.apache.org/jira/browse/STORM-3769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17335791#comment-17335791 ]
Aaron Gresch commented on STORM-3769:
-------------------------------------
In this case, a worker for topology A was no longer assigned on a supervisor. At some point the blob cache exceeded the size limit. When cleanup was called, 1 or 2 of the topology A's blobs were deleted, and this was enough to get under the cache size limit. Because some of the topology blobs remain, the code currently assumes the blobs all remain downloaded. Then when a worker for topology A gets reassigned back to the node, this exception occurs and workers will be unable to start.
> Failed adding references to blobs: FileNotFoundException
> --------------------------------------------------------
>
> Key: STORM-3769
> URL: https://issues.apache.org/jira/browse/STORM-3769
> Project: Apache Storm
> Issue Type: Bug
> Reporter: Aaron Gresch
> Assignee: Aaron Gresch
> Priority: Major
>
> We hit a file not found exception with AsyncLocalizer:
> {code:java}
> 2021-04-23 17:39:13.380 o.a.s.l.AsyncLocalizer ForkJoinPool.commonPool-worker-23 [ERROR] Failed adding references to blobs for TimePortAndAssignment{xxx-1-15-1616201755 on 6708}
> java.io.FileNotFoundException: File '/home/y/var/storm/supervisor/stormdist/xxx-1-15-1616201755/stormconf.ser' does not exist
> at org.apache.storm.shade.org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:297) ~[storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
> at org.apache.storm.shade.org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1851) ~[storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
> at org.apache.storm.utils.ConfigUtils.readSupervisorStormConfGivenPath(ConfigUtils.java:311) ~[storm-client-2.3.0.y.jar:2.3.0.y]
> at org.apache.storm.utils.ConfigUtils.readSupervisorStormConfImpl(ConfigUtils.java:472) ~[storm-client-2.3.0.y.jar:2.3.0.y]
> at org.apache.storm.utils.ConfigUtils.readSupervisorStormConf(ConfigUtils.java:306) ~[storm-client-2.3.0.y.jar:2.3.0.y]
> at org.apache.storm.localizer.AsyncLocalizer.getLocalResources(AsyncLocalizer.java:368) ~[storm-server-2.3.0.y.jar:2.3.0.y]
> at org.apache.storm.localizer.AsyncLocalizer.addReferencesToBlobs(AsyncLocalizer.java:398) ~[storm-server-2.3.0.y.jar:2.3.0.y]
> at org.apache.storm.localizer.AsyncLocalizer.lambda$null$7(AsyncLocalizer.java:235) ~[storm-server-2.3.0.y.jar:2.3.0.y]
> at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1877) ~[?:1.8.0_262]
> at org.apache.storm.localizer.AsyncLocalizer.lambda$requestDownloadTopologyBlobs$8(AsyncLocalizer.java:229) ~[storm-server-2.3.0.y.jar:2.3.0.y]
> at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:966) [?:1.8.0_262]
> at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:940) [?:1.8.0_262]
> at java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:457) [?:1.8.0_262]
> at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) [?:1.8.0_262]
> at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) [?:1.8.0_262]
> at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) [?:1.8.0_262]
> at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:163) [?:1.8.0_262]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)