You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Gargi-jais11 (via GitHub)" <gi...@apache.org> on 2026/10/22 04:38:16 UTC
[PR] HDDS-16807. DiskBalancer should skip containers which are being reconciled [ozone]
Gargi-jais11 opened a new pull request, #11676:
URL: https://github.com/apache/ozone/pull/11676
## What changes were proposed in this pull request?
DiskBalancer and `ozone admin container reconcile` can run on the same container replica on one datanode at the same time. Reconciliation keeps a reference to the container object it looked up at the start and repairs that object without holding the container lock. DiskBalancer holds only the read lock, copies the replica to another volume, swaps the live entry in **ContianerSet** and marks the old replica **DELETED**. Neither side knows about the other.
*What goes wrong*
* Repairs written after the copy was taken exist only on the old replica, which is then deleted — the admin sees a successful reconcile log, but the replica that remains was not repaired.
* The closing incremental container report (ICR) can be built from the retired replica (wrong BCSID, checksum, or **DELETED** state) while the live replica on the new disk is different.
* SCM can keep a stale view until the next full container report (default 60m), or briefly treat the container as under-replicated if the ICR says **DELETED**.
As a result : No data loss relative to the state before reconcile, but the repair is lost and is not retried automatically. Operators may see false success in logs and wrong replica/mismatch signals in SCM/Recon until the next report or scan.
**Proposed Fix**
Mark the replica as "being reconciled" under `ContainerSet.getContainerWithWriteLock` at reconcile start; DiskBalancer skips containers with that mark under its read lock. Reconcile waits for an in-progress move (read lock) and then works on the live replica.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16807
## How was this patch tested?
Added unit tests.
Generated with Specula (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16807. DiskBalancer should skip containers which are being reconciled [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on PR #11676:
URL: https://github.com/apache/ozone/pull/11676#issuecomment-6370295305
The issues mentioned here are valid. Since disk balancer does a delete of the original replica, these issues are a subset of what HDDS-10929 intended to cover.
Reconciliation is designed as an eventually consistent conflict-free and lock-free process backed by a CRDT. The only requirement when reconciliation finishes is that the system is moved to an equal or safer state. A partial or no-op reconcile alone does not violate this, but the inaccurate state reported to users and SCM described here does.
With this in mind, I believe we can solve all the issues mentioned here with minor changes to disk balancer and reconciler instead of the blocking approach:
- Reconciliation needs to check for a container's DELETED state before reporting progress in logs or ICR.
- If a container is moved to DELETED during the reconciliation process, reconciler should abort and log a message indicating this instead of a summary of the reconciliation work. This is a similar pattern to what the container data scanner does already.
- Reconciler should not send an ICR if the container it is working on is deleted, since it did not end up modifying the live replica. The process that issued the delete is responsible for the ICR. This also mirrors the scanner.
- Disk balancer needs to clear the container's last scanned timestamp and trigger an on-demand data scan. This on-demand scan will send an ICR when it completes.
- A copy is a new container, so information about the original is no longer valid.
- This is similar to what container import already does.
Let me know if there are more corner cases, but I believe we can re-use these existing patterns from the scanner and import processes to solve the issues here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16807. DiskBalancer should skip containers which are being reconciled [ozone]
Posted by "Gargi-jais11 (via GitHub)" <gi...@apache.org>.
Gargi-jais11 commented on PR #11676:
URL: https://github.com/apache/ozone/pull/11676#issuecomment-6460765693
>
- Reconciliation needs to check for a container's DELETED state before reporting progress in logs or ICR.
- If a container is moved to DELETED during the reconciliation process, reconciler should abort and log a message indicating this instead of a summary of the reconciliation work. This is a similar pattern to what the container data scanner does already.
- Reconciler should not send an ICR if the container it is working on is deleted, since it did not end up modifying the live replica. The process that issued the delete is responsible for the ICR. This also mirrors the scanner.
- Disk balancer needs to clear the container's last scanned timestamp and trigger an on-demand data scan. This on-demand scan will send an ICR when it completes.
- A copy is a new container, so information about the original is no longer valid.
- This is similar to what container import already does.
Thanks @errose28 and @ChenSammi. I'll rework the PR with new approach mentioned above.
Few things to note is:
The new replica's last scan timestamp is already empty after a DiskBalancer import, because the new ContainerData is built with the copy constructor, which doesn't carry the timestamp. So as per Ethan's approach only the scan trigger is needed.
**One corner case which will still be left after the above approach is:**
if reconciliation finishes after the copy but before the swap (or before the old replica is marked DELETED), the reconcile ICR reports the repaired replica. The post-move scan won't correct SCM, because the copy's stored checksum matches its data. Should DiskBalancer send one ICR for the new replica after the move, as container import does?
@ChenSammi "ICR is not needed": it isn't needed for the move itself, but it is when a reconcile report landed during the move. So on seeing the cases should we send ICR?
Also, the on-demand scanner is single-threaded at 5 MB/s per datanode, while DiskBalancer moves up to 10 MB/s by default, so long balancing runs will build a scan backlog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16807. DiskBalancer should skip containers which are being reconciled [ozone]
Posted by "Gargi-jais11 (via GitHub)" <gi...@apache.org>.
Gargi-jais11 commented on PR #11676:
URL: https://github.com/apache/ozone/pull/11676#issuecomment-6336941018
> The thing is there was an effort to not pick the container when it's being replicated by SCM replication manager, like this case, but Stephen thought we should not introduce the complexity to do that, so the patch was finally abandoned.
@ChenSammi I do remember this discussion and somewhat it was there at back of my mind, so I thought it's better if we could discuss on the PR and gather informations from @errose28 about whether we want to handle that disk balancer and reconciler cannot work together on the same container or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16807. DiskBalancer should skip containers which are being reconciled [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #11676:
URL: https://github.com/apache/ozone/pull/11676#issuecomment-6433352084
> Let me know if there are more corner cases, but I believe we can re-use these existing patterns from the scanner and import processes to solve the issues here.
@errose28 , I agree with the overall approach.
For disk balancer, since the replica is still on this datanode, ICR is not needed as SCM currently doesn't aware which disk volume of each replica exists. I'm also not sure "clear the container's last scanned timestamp and trigger an on-demand data scan" is needed for disk balancer, as the moved replica is scanned by the same data scanner of Datanode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16807. DiskBalancer should skip containers which are being reconciled [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #11676:
URL: https://github.com/apache/ozone/pull/11676#issuecomment-6336419054
Thanks @smengcl for reporting this, and @Gargi-jais11 working on the patch.
Current reconcileContainer doesn't hold any container LOCK during the operation, just like the data scanner. So there is chance that replica can be deleted during this phase. I tend to believe we have considered this situation during the container reconciler development. It would be very helpful if @errose28 can share more info and the expectation of this case.
The thing is there was an effort to not pick the container when it's being replicated by SCM replication manager, like this case, but Stephen thought we should not introduce the complexity to do that, so the patch was finally abandoned.
If we want to handle that disk balancer and reconciler cannot work together on the same container, we should also consider other cases too, such as delete cannot happen on the same container as reconciler, or container replication cannot happen on the same container as reconciler.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16807. DiskBalancer should skip containers which are being reconciled [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on PR #11676:
URL: https://github.com/apache/ozone/pull/11676#issuecomment-6445755626
Thanks @ChenSammi for checking on this. Let me clarify part of my original reply:
> Disk balancer needs to clear the container's last scanned timestamp and trigger an on-demand data scan. This on-demand scan will send an ICR when it completes.
The bytes were copied from one medium to another which is a chance to introduce corruption, so an on-demand scan is definitely necessary. However, the scanner will only trigger an ICR if it finds corruption or updates the container's data checksum/merkle tree from its previously stored value, so no redundant ICR is triggered. All the disk balancer has to do is trigger the scan, then the scanner handles the ICR if necessary.
> Anyway, based on this discussion, I think we can close this one, and count on [HDDS-10929](https://issues.apache.org/jira/browse/HDDS-10929) to do the Reconciler part work.
Unfortunately we don't have any bandwidth assigned to reconciliation right now, although I do want to revisit it once the remaining ZDU work is complete. While this is fresh in our minds I think it would be good to update the disk balancer and reconciler in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org