You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "devmadhuu (via GitHub)" <gi...@apache.org> on 2025/08/14 14:05:03 UTC
[PR] HDDS-13573. Intermittent failure in TestNSSummaryUnifiedControl.testMultipleConcurrentAttempts. [ozone]
devmadhuu opened a new pull request, #8945:
URL: https://github.com/apache/ozone/pull/8945
## What changes were proposed in this pull request?
This PR change is to fix the concurrent issues related to rebuild of NSSummary tree, So that exactly one thread can acquire the rebuild lock, eliminating the race condition that was causing the intermittent test failure.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-13573
## How was this patch tested?
Tested using existing test case failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13573. Intermittent failure in TestNSSummaryUnifiedControl.testMultipleConcurrentAttempts. [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #8945: HDDS-13573. Intermittent failure in TestNSSummaryUnifiedControl.testMultipleConcurrentAttempts.
URL: https://github.com/apache/ozone/pull/8945
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13573. Intermittent failure in TestNSSummaryUnifiedControl.testMultipleConcurrentAttempts. [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #8945:
URL: https://github.com/apache/ozone/pull/8945#discussion_r2277053633
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/NSSummaryTask.java:
##########
@@ -189,14 +189,13 @@ public TaskResult process(
@Override
public TaskResult reprocess(OMMetadataManager omMetadataManager) {
// Unified control for all NSS tree rebuild operations
- RebuildState currentState = REBUILD_STATE.get();
- if (currentState == RebuildState.RUNNING) {
- LOG.info("NSSummary tree rebuild is already in progress, skipping duplicate request.");
- return buildTaskResult(false);
- }
+ // Use a single atomic operation to prevent race conditions
+ RebuildState previousState = REBUILD_STATE.getAndSet(RebuildState.RUNNING);
- if (!REBUILD_STATE.compareAndSet(currentState, RebuildState.RUNNING)) {
- LOG.info("Failed to acquire rebuild lock, another thread may have started rebuild.");
+ if (previousState == RebuildState.RUNNING) {
+ LOG.info("NSSummary tree rebuild is already in progress, skipping duplicate request.");
+ // Restore the previous state since we didn't actually start
+ REBUILD_STATE.set(RebuildState.RUNNING);
Review Comment:
```java
1. RebuildState previousState = REBUILD_STATE.getAndSet(RebuildState.RUNNING);
2. if (previousState == RebuildState.RUNNING) {
3. REBUILD_STATE.set(RebuildState.RUNNING);
```
This doesn't look right: line (1) sets state to `RUNNING`, so line (3) is no-op.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13573. Intermittent failure in TestNSSummaryUnifiedControl.testMultipleConcurrentAttempts. [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on code in PR #8945:
URL: https://github.com/apache/ozone/pull/8945#discussion_r2280772305
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/NSSummaryTask.java:
##########
@@ -189,14 +189,13 @@ public TaskResult process(
@Override
public TaskResult reprocess(OMMetadataManager omMetadataManager) {
// Unified control for all NSS tree rebuild operations
- RebuildState currentState = REBUILD_STATE.get();
- if (currentState == RebuildState.RUNNING) {
- LOG.info("NSSummary tree rebuild is already in progress, skipping duplicate request.");
- return buildTaskResult(false);
- }
+ // Use a single atomic operation to prevent race conditions
+ RebuildState previousState = REBUILD_STATE.getAndSet(RebuildState.RUNNING);
- if (!REBUILD_STATE.compareAndSet(currentState, RebuildState.RUNNING)) {
- LOG.info("Failed to acquire rebuild lock, another thread may have started rebuild.");
+ if (previousState == RebuildState.RUNNING) {
+ LOG.info("NSSummary tree rebuild is already in progress, skipping duplicate request.");
+ // Restore the previous state since we didn't actually start
+ REBUILD_STATE.set(RebuildState.RUNNING);
Review Comment:
Thanks @adoroszlai for reviewing the code. Here are the flaky test run results with master branch and with fixed code in HDDS-13573 branch.
master branch flaky test workflow run results: https://github.com/devmadhuu/ozone/actions/runs/16983139469
HDDS-13573 branch flaky test workflow run results: https://github.com/devmadhuu/ozone/actions/runs/17018339113
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13573. Intermittent failure in TestNSSummaryUnifiedControl.testMultipleConcurrentAttempts. [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8945:
URL: https://github.com/apache/ozone/pull/8945#issuecomment-3573218673
Thank you for your contribution. This PR is being closed due to inactivity. If needed, feel free to reopen it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13573. Intermittent failure in TestNSSummaryUnifiedControl.testMultipleConcurrentAttempts. [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8945:
URL: https://github.com/apache/ozone/pull/8945#issuecomment-3514417634
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13573. Intermittent failure in TestNSSummaryUnifiedControl.testMultipleConcurrentAttempts. [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #8945:
URL: https://github.com/apache/ozone/pull/8945#discussion_r2277053633
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/NSSummaryTask.java:
##########
@@ -189,14 +189,13 @@ public TaskResult process(
@Override
public TaskResult reprocess(OMMetadataManager omMetadataManager) {
// Unified control for all NSS tree rebuild operations
- RebuildState currentState = REBUILD_STATE.get();
- if (currentState == RebuildState.RUNNING) {
- LOG.info("NSSummary tree rebuild is already in progress, skipping duplicate request.");
- return buildTaskResult(false);
- }
+ // Use a single atomic operation to prevent race conditions
+ RebuildState previousState = REBUILD_STATE.getAndSet(RebuildState.RUNNING);
- if (!REBUILD_STATE.compareAndSet(currentState, RebuildState.RUNNING)) {
- LOG.info("Failed to acquire rebuild lock, another thread may have started rebuild.");
+ if (previousState == RebuildState.RUNNING) {
+ LOG.info("NSSummary tree rebuild is already in progress, skipping duplicate request.");
+ // Restore the previous state since we didn't actually start
+ REBUILD_STATE.set(RebuildState.RUNNING);
Review Comment:
```java
1. RebuildState previousState = REBUILD_STATE.getAndSet(RebuildState.RUNNING);
2. if (previousState == RebuildState.RUNNING) {
3. REBUILD_STATE.set(RebuildState.RUNNING);
```
This doesn't look right :line (1) sets state to `RUNNING`, so line (3) is no-op.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13573. Intermittent failure in TestNSSummaryUnifiedControl.testMultipleConcurrentAttempts. [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #8945:
URL: https://github.com/apache/ozone/pull/8945#discussion_r2338870407
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/NSSummaryTask.java:
##########
@@ -295,7 +302,10 @@ public TaskResult buildTaskResult(boolean success) {
*/
@VisibleForTesting
public static void resetRebuildState() {
+ // Only reset to IDLE if currently FAILED - never interrupt a RUNNING operation
REBUILD_STATE.set(RebuildState.IDLE);
+ // If state is RUNNING, leave it alone to prevent race conditions
+ // If state is already IDLE, no change needed
Review Comment:
Seems like mismatch between comment and code (code does not look at current state).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org