strange nimbus warnings

["Lieberman, Yuval" <Yu...@2bprecisehealth.com>]

Since migrating from Storm 1.06 to 2.1.0, I am seeing some strange warnings in nimbus log.

I am getting a lot of warnings such as:
2020-04-25 20:22:05.604 o.a.s.d.n.Nimbus timer [WARN] Negative fragmented CPU on 9971b56c-3549-445a-9132-c5f2185cb2f8-10.4.0.5
2020-04-25 20:22:05.605 o.a.s.d.n.Nimbus timer [WARN] Negative fragmented Mem on 9971b56c-3549-445a-9132-c5f2185cb2f8-10.4.0.5
I understand if the message was not enough memory, but what does negative memory mean??

Also, during deployment of topologies (after undeploying the previous versions of these topologies) I am getting ~10 warnings per topology saying:
2020-04-25 20:20:01.653 o.a.s.d.n.Nimbus pool-29-thread-50 [WARN] get blob meta exception.
org.apache.storm.utils.WrappedKeyNotFoundException: summary-generator-17-1587642602-stormcode.ser
                at org.apache.storm.blobstore.LocalFsBlobStore.getStoredBlobMeta(LocalFsBlobStore.java:257) ~[storm-server-2.1.0.jar:2.1.0]
                at org.apache.storm.blobstore.LocalFsBlobStore.getBlobMeta(LocalFsBlobStore.java:287) ~[storm-server-2.1.0.jar:2.1.0]
                at org.apache.storm.daemon.nimbus.Nimbus.getBlobMeta(Nimbus.java:3651) [storm-server-2.1.0.jar:2.1.0]
                at org.apache.storm.generated.Nimbus$Processor$getBlobMeta.getResult(Nimbus.java:3951) [storm-client-2.1.0.jar:2.1.0]
                at org.apache.storm.generated.Nimbus$Processor$getBlobMeta.getResult(Nimbus.java:3930) [storm-client-2.1.0.jar:2.1.0]
                at org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:38) [storm-shaded-deps-2.1.0.jar:2.1.0]
                at org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:39) [storm-shaded-deps-2.1.0.jar:2.1.0]
                at org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:174) [storm-client-2.1.0.jar:2.1.0]
                at org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518) [storm-shaded-deps-2.1.0.jar:2.1.0]
                at org.apache.storm.thrift.server.Invocation.run(Invocation.java:18) [storm-shaded-deps-2.1.0.jar:2.1.0]
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_252]
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_252]
                at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
The strange thing is that the reported version “summary-generator-17-1587642602” is the instance that was just undeployed, so it really should not be in the blob store.

Not sure at this point if this is some misconfiguration at my side, or something else.

Yuval Lieberman
Team Leader, 2bPrecise
IL: +972.52.5311.790 | M
2bPreciseHealth.com

It’s Time 2bPrecise