You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@storm.apache.org by "mwkang (via GitHub)" <gi...@apache.org> on 2026/06/10 04:47:14 UTC
[I] Supervisor keeps reporting stale heartbeats for orphaned worker directories (storm)
mwkang opened a new issue, #8786:
URL: https://github.com/apache/storm/issues/8786
If a local worker directory is ever left behind (not removed by `Container.cleanUpForRestart()`), the supervisor keeps reporting its heartbeat to Nimbus forever — for a topology that no longer exists.
`ReportWorkerHeartbeats` builds the batch from `SupervisorUtils.readWorkerHeartbeats()`, which just lists every directory under the worker root (`SupervisorUtils.supervisorWorkerIds()`) and reads each `LSWorkerHeartbeat`. There's no filtering by current assignment or by staleness, so a leftover directory is reported as if it were a live worker.
Orphaned worker directories are only cleaned up once, in the `ReadClusterState` constructor (supervisor startup). The periodic sync loop (`run()`) never re-checks for detached workers. So once a directory is orphaned at runtime, it survives — and keeps being reported — until the next supervisor restart.
On the Nimbus side this shows up as repeated `getTopologyHeartbeatTimeoutSecs` -> `tryReadTopoConf` -> `NotAliveException` for the dead topology (logged as `Exception when getting heartbeat timeout` before STORM-4022). It's also a slow disk leak, since the directory is never reclaimed.
We hit this in production: a worker that shut down cleanly left its `<worker-id>/heartbeats` localstate behind, and the supervisor kept reporting it long after the topology was gone. STORM-4022 only silences the Nimbus log — it doesn't stop the bad heartbeats or reclaim the directory.
### Suggested fix
- Run the detached-worker cleanup (currently only in the `ReadClusterState` constructor) periodically from the sync loop, so orphaned directories are reclaimed at runtime; and/or
- In `ReportWorkerHeartbeats`, skip heartbeats whose topology isn't currently assigned to this supervisor, or whose `time_secs` is older than the timeout.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@storm.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Supervisor keeps reporting stale heartbeats for orphaned worker directories (storm)
Posted by "mwkang (via GitHub)" <gi...@apache.org>.
mwkang commented on issue #8786:
URL: https://github.com/apache/storm/issues/8786#issuecomment-4677345510
Thanks @reiabreu! Yes, I'd like to take this one — I'll open a PR shortly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@storm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Supervisor keeps reporting stale heartbeats for orphaned worker directories (storm)
Posted by "reiabreu (via GitHub)" <gi...@apache.org>.
reiabreu commented on issue #8786:
URL: https://github.com/apache/storm/issues/8786#issuecomment-4673197312
Hey @mwkang ! Thanks for the report.
Do you want to submit a PR fkr this? If not, I can have a look. Thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@storm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Supervisor keeps reporting stale heartbeats for orphaned worker directories (storm)
Posted by "reiabreu (via GitHub)" <gi...@apache.org>.
reiabreu closed issue #8786: Supervisor keeps reporting stale heartbeats for orphaned worker directories
URL: https://github.com/apache/storm/issues/8786
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@storm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org