You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Attila Doroszlai (Jira)" <ji...@apache.org> on 2023/04/28 09:46:00 UTC
[jira] [Updated] (HDDS-8471) Ensure replication processors use a single queue for each iteration
[ https://issues.apache.org/jira/browse/HDDS-8471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Attila Doroszlai updated HDDS-8471:
-----------------------------------
Summary: Ensure replication processors use a single queue for each iteration (was: ReplicationManager: Avoid re-queuing duplicate message for under / over replicated containers)
> Ensure replication processors use a single queue for each iteration
> -------------------------------------------------------------------
>
> Key: HDDS-8471
> URL: https://issues.apache.org/jira/browse/HDDS-8471
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Stephen O'Donnell
> Assignee: Attila Doroszlai
> Priority: Major
> Labels: pull-request-available
>
> The under and over replication queues in ReplicationManager are created when replicationManager checks the health of all containers in the system. When it does that, it forms a new "ReplicationQueue" object wrapping the under and over replicated queues.
> The OverReplicatedProcessor and UnderReplicatedProcessor both extend UnhealthyReplicationProcessor. Within it, it dequeues messages and processes them. If there is an exception, it saves the message in a list, ready to enqueue it again later. It saves the message, rather than enqueuing it immediately, to avoid the queue entering an infinite loop when a container fails repeatedly.
> The issue is that while the Under / Over process is running, it could be saving up containers to requeue, but then ReplicationManager could process all the containers and replace the queue. Then the bad containers are requeued onto the "new" queue, possibly creating duplicates.
> While the duplicates should not cause any problem, it would be better if this was handled more gracefully.
> For example, if the queue has been replaced, drop the failed containers - but how to check if the queue has been replaced?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org