You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "smengcl (via GitHub)" <gi...@apache.org> on 2026/10/22 10:33:47 UTC
[PR] HDDS-17314. Make hdds.heartbeat.recon.interval control the Datanode to Recon heartbeat interval [ozone]
smengcl opened a new pull request, #11695:
URL: https://github.com/apache/ozone/pull/11695
Generated-by: Claude Code (Opus 5.5)
## What changes were proposed in this pull request?
`hdds.heartbeat.recon.interval` (default 60s) does not control how often a datanode heartbeats Recon. `RunningDatanodeState` sends a heartbeat to every endpoint, Recon included, on each SCM heartbeat cycle. The Recon interval is read into `StateContext` (HDDS-5862), but it has been unused since HDDS-12639 removed it from the task wait.
This change:
* Skips the Recon heartbeat in `RunningDatanodeState` until the Recon interval has passed since the last one. Version and register calls to Recon are not delayed. Half of the SCM heartbeat interval is allowed as slack, so small timing differences do not push the Recon heartbeat a whole cycle later.
* Bounds the interval in `HddsServerUtil.getReconHeartbeatInterval`: at least `hdds.heartbeat.interval`, and at most a third of `ozone.scm.stale.node.interval`, so Recon (which uses the SCM stale and dead node intervals) does not mark datanodes stale between heartbeats.
* Sets a short Recon interval in the `ozone`, `ozonesecure`, `ozone-topology` and `compatibility` compose clusters (which run Recon tests) and in `MiniOzoneCluster`, so those tests do not wait longer.
* Updates the description in `ozone-default.xml`.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-17314
## How was this patch tested?
Added `TestRunningDatanodeState#testReconHeartbeatInterval` and `TestHddsServerUtils#testGetReconHeartbeatInterval`. Both fail without the fix. Existing tests in both classes pass.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-17314. Make hdds.heartbeat.recon.interval control the Datanode to Recon heartbeat interval [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on PR #11695:
URL: https://github.com/apache/ozone/pull/11695#issuecomment-6433042878
Thanks @smengcl for the reply.
> Yes, that is intended. By default, hdds.heartbeat.recon.interval (60s) is longer than hdds.heartbeat.interval (30s), so the default case works. A shorter Recon interval was never possible. Before this patch, every cycle heartbeated Recon at the SCM interval, whatever the key said. The patch keeps that limit explicit.
Yes, I mistakenly thought recon interval is by default lower than heartbeat interval.
LGTM +1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-17314. Make hdds.heartbeat.recon.interval control the Datanode to Recon heartbeat interval [ozone]
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on PR #11695:
URL: https://github.com/apache/ozone/pull/11695#issuecomment-6421066985
Thanks @ivandika3 for the review.
> The current solution seems to only work if recon heartbeat interval is higher than the datanode heartbeat interval.
Yes, that is intended. By default, `hdds.heartbeat.recon.interval` (60s) is longer than `hdds.heartbeat.interval` (30s), so the default case works. A shorter Recon interval was never possible. Before this patch, every cycle heartbeated Recon at the SCM interval, whatever the key said. The patch keeps that limit explicit.
Is there a use case for heartbeating Recon more often than SCM? Achieving that would need a heartbeat thread per endpoint, as you suggested. Recon only needs heartbeats often enough not to mark the datanode stale.
> This also creates another concern where a single stuck SCM RPC can cause heartbeats to the other unstuck SCMs to be delayed.
The regular heartbeats to the other SCMs and Recon are still sent every `hdds.heartbeat.interval`. Each endpoint runs its tasks on its own single thread executor, `RunningDatanodeState` waits for each endpoint at most one interval, and `DatanodeStateMachine` sets the next cycle time before it runs the cycle (existing TestRunningDatanodeState#testAwait` covers a slow endpoint).
The exception is the shared `DatanodeStateMachine` thread pool (one thread per endpoint). Each wait on the stuck SCM holds a thread for up to one interval, and each `triggerHeartbeat()` cycle adds another. Once they hold all threads, tasks for the other SCMs are queued and cycles are skipped, so their heartbeats can be up to about one interval late. Let's fix this in another Jira HDDS-17488.
The stuck SCM's own endpoint also builds up a queue of heartbeat tasks. With dropped packets as in HDDS-16376, each new connection blocks for about 15 minutes on connect timeout retries. That is HDDS-17322, and HDDS-17269 covers the queued tasks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-17314. Make hdds.heartbeat.recon.interval control the Datanode to Recon heartbeat interval [ozone]
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl merged PR #11695:
URL: https://github.com/apache/ozone/pull/11695
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-17314. Make hdds.heartbeat.recon.interval control the Datanode to Recon heartbeat interval [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #11695:
URL: https://github.com/apache/ozone/pull/11695#issuecomment-6461562964
Before merging another PR to `master`, please wait until most CI jobs for the previous merge have completed (typically about one hour), leaving CI capacity for other PRs. [source](https://github.com/apache/ozone/blob/master/CONTRIBUTING.md#committer-guidelines)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-17314. Make hdds.heartbeat.recon.interval control the Datanode to Recon heartbeat interval [ozone]
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on PR #11695:
URL: https://github.com/apache/ozone/pull/11695#issuecomment-6460750786
Thanks @ivandika3 for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org