You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "devmadhuu (via GitHub)" <gi...@apache.org> on 2025/07/10 10:36:22 UTC
[PR] HDDS-8633. Separate Recon OM Synchronization and Tasks Processing. [ozone]
devmadhuu opened a new pull request, #8777:
URL: https://github.com/apache/ozone/pull/8777
## What changes were proposed in this pull request?
This PR change is to keep Recon OM DB sync and events processing by OM tasks separate and async fashion.
**Details of changes:**
```
1. Asynchronous Task Processing
- Events are queued in LinkedBlockingQueue<OMUpdateEventBatch> eventQueue
- Processing runs asynchronously using taskProcessingExecutor
- taskProcessingInProgress atomic boolean prevents duplicate processing
- OM sync can continue updating sequence numbers while tasks process
2. Event Queue Management
- Processes all queued events until queue is empty
- Ensures no events are lost during async processing
- Continues processing other batches even if one fails
3. Separate Executor for Task Processing
- taskProcessingExecutor: Single-threaded executor for async task processing
- executorService: Multi-threaded executor for parallel task execution
- Prevents blocking OM sync operations
4. Reinitialization Without Blocking
- Submitted to task processing executor to avoid blocking OM sync
- Clears event queue during reinitialization
- Runs asynchronously to maintain OM sync performance
5. Robust Error Handling
- Exceptions in task processing don't block OM sync
- Failed tasks are retried and eventually ignored if they fail repeatedly
- Task status is properly tracked and updated
Test Coverage
The test file TestReconTaskControllerImpl.java shows comprehensive testing of:
- Task registration and execution
- Event consumption and processing
- Failure handling and retry logic
- Status tracking and recording
```
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-8633
## How was this patch tested?
This patch is tested with existing integration tests of recon and `TestReconWithOzoneManager`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8633. Separate Recon OM Synchronization and Tasks Processing. [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on PR #8777:
URL: https://github.com/apache/ozone/pull/8777#issuecomment-3197502102
> @devmadhuu I think we need to use a persistent queue (e.g. Chronicle queue) instead. From my experiments last time, in-memory queue would run out of space quite quickly for busy OM. However, the main issue I encountered last time was the serde of the OMUpdateEvent to the disk.
Thanks @ivandika3 for your review. Actually its a trade-off in one sense. Persistent queue can keep increasing disk space. So we should better go for reprocess to maintain simplicity. Also we are working on a staging DB solution with multiple follow up PRs which makes overall solution more reliable. Here is the design: for staging DB approach. https://github.com/apache/ozone/pull/8872
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8633. Separate Recon OM Synchronization and Tasks Processing. [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on PR #8777:
URL: https://github.com/apache/ozone/pull/8777#issuecomment-3056949841
@devmadhuu Thanks for the patch, I'll take a look when I'm available.
Can I check whether the patch is generated using AI? If so, please help to indicate the AI model use and the prompts as per Apache regulation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8633. Separate Recon OM Synchronization and Tasks Processing. [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on PR #8777:
URL: https://github.com/apache/ozone/pull/8777#issuecomment-3197163864
@devmadhuu I think we need to use a persistent queue instead. From my experiments last time, in-memory queue would run out of space quite quickly. However, the main issue I encountered last time was the serde of the OMUpdateEvent to the disk.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8633. Separate Recon OM Synchronization and Tasks Processing. [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on PR #8777:
URL: https://github.com/apache/ozone/pull/8777#issuecomment-3057009534
> @devmadhuu Thanks for the patch, I'll take a look when I'm available.
>
> Can I check whether the patch is generated using AI? If so, please help to indicate the AI model use and the prompts as per Apache regulation.
Part of the code snippets were completed by github co-pilot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8633. Separate Recon OM Synchronization and Tasks Processing. [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on PR #8777:
URL: https://github.com/apache/ozone/pull/8777#issuecomment-3200605777
> > @devmadhuu I think we need to use a persistent queue (e.g. Chronicle queue) instead. From my experiments last time, in-memory queue would run out of space quite quickly for busy OM. However, the main issue I encountered last time was the serde of the OMUpdateEvent to the disk.
>
> Thanks @ivandika3 for your review. Actually its a trade-off in one sense. Persistent queue can keep increasing disk space. So we should better go for reprocess to maintain simplicity. Also we are working on a staging DB solution with multiple follow up PRs which makes overall solution more reliable. Here is the design: for staging DB approach. #8872
@ivandika3
Having Persistent queue helps increasing queue limit to higher value, but it still can not be too high, as will consume disk and may have further IO performance impact. And may never able to meet incoming delta speed expectation. Generally incoming deltas an processing should match in idle system.
We are following below strategy,
1. If memory queue is full, we are triggering reprocess, which will work in staging env
-- This will avoid problem of partial data shown over UI
-- data is always available every stage once reprocess is done in case of OM too loaded
3. Also plan to improve reprocess with parallel iteration of db
4. This reprocess will also be async to avoid blocking of deltas from OM, so to avoid full download from OM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8633. Separate Recon OM Synchronization and Tasks Processing. [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl merged PR #8777:
URL: https://github.com/apache/ozone/pull/8777
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org