You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "ArafatKhan2198 (via GitHub)" <gi...@apache.org> on 2025/02/03 08:17:35 UTC
[PR] HDDS-12184. Enhance NSSummaryTask process() for Faster Processing. [ozone]
ArafatKhan2198 opened a new pull request, #7795:
URL: https://github.com/apache/ozone/pull/7795
## What changes were proposed in this pull request?
We can parallelize the `process()` method in `NSSummaryTask` similarly to how reprocess() was parallelized. Instead of running `processWithFSO()`, `processWithLegacy()`, and `processWithOBS()` sequentially, we can execute them concurrently using an ExecutorService.
We will :-
1. Use a thread pool to execute processWithFSO(), processWithLegacy(), and processWithOBS() concurrently.
2. Use Future<Boolean> objects to collect the results from each task.
3. Wait for all tasks to complete and check their success/failure.
4. Shutdown the executor service properly.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-12184
## How was this patch tested?
Manually Tested
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12184. Enhance NSSummaryTask process() for Faster Processing. [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7795:
URL: https://github.com/apache/ozone/pull/7795#issuecomment-2829116926
Thank you very much for the patch. I am closing this PR __temporarily__ as there was no activity recently and it is waiting for response from its author.
It doesn't mean that this PR is not important or ignored: feel free to reopen the PR at any time.
It only means that attention of committers is not required. We prefer to keep the review queue clean. This ensures PRs in need of review are more visible, which results in faster feedback for all PRs.
If you need ANY help to finish this PR, please [contact the community](https://github.com/apache/hadoop-ozone#contact) on the mailing list or the slack channel."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12184. Enhance NSSummaryTask process() for Faster Processing. [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #7795:
URL: https://github.com/apache/ozone/pull/7795#issuecomment-2775335878
/pending
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12184. Enhance NSSummaryTask process() for Faster Processing. [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #7795:
URL: https://github.com/apache/ozone/pull/7795#discussion_r1938998104
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/NSSummaryTask.java:
##########
@@ -101,21 +101,37 @@ public String getTaskName() {
@Override
public Pair<String, Boolean> process(OMUpdateEventBatch events) {
long startTime = System.currentTimeMillis();
- boolean success = nsSummaryTaskWithFSO.processWithFSO(events);
- if (!success) {
- LOG.error("processWithFSO failed.");
- }
- success = nsSummaryTaskWithLegacy.processWithLegacy(events);
- if (!success) {
- LOG.error("processWithLegacy failed.");
- }
- success = nsSummaryTaskWithOBS.processWithOBS(events);
- if (!success) {
- LOG.error("processWithOBS failed.");
+
+ // Thread pool to execute tasks concurrently
+ ExecutorService executorService = Executors.newFixedThreadPool(3);
Review Comment:
Creating a thread pool is heavy-weight, it should be reused.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/NSSummaryTask.java:
##########
@@ -101,21 +101,37 @@ public String getTaskName() {
@Override
public Pair<String, Boolean> process(OMUpdateEventBatch events) {
long startTime = System.currentTimeMillis();
- boolean success = nsSummaryTaskWithFSO.processWithFSO(events);
- if (!success) {
- LOG.error("processWithFSO failed.");
- }
- success = nsSummaryTaskWithLegacy.processWithLegacy(events);
- if (!success) {
- LOG.error("processWithLegacy failed.");
- }
- success = nsSummaryTaskWithOBS.processWithOBS(events);
- if (!success) {
- LOG.error("processWithOBS failed.");
+
+ // Thread pool to execute tasks concurrently
+ ExecutorService executorService = Executors.newFixedThreadPool(3);
+
+ // Create a list of tasks
+ List<Callable<Boolean>> tasks = new ArrayList<>();
+ tasks.add(() -> nsSummaryTaskWithFSO.processWithFSO(events));
+ tasks.add(() -> nsSummaryTaskWithLegacy.processWithLegacy(events));
+ tasks.add(() -> nsSummaryTaskWithOBS.processWithOBS(events));
+
+ try {
+ // Execute all tasks in parallel
+ List<Future<Boolean>> results = executorService.invokeAll(tasks);
+
+ // Check results and return failure if any task failed
+ for (Future<Boolean> result : results) {
+ if (!result.get()) {
+ LOG.error("One or more process tasks failed.");
+ return new ImmutablePair<>(getTaskName(), false);
Review Comment:
Wait for all results before returning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12184. Enhance NSSummaryTask process() for Faster Processing. [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #7795: HDDS-12184. Enhance NSSummaryTask process() for Faster Processing.
URL: https://github.com/apache/ozone/pull/7795
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org