You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "wuzhoupei (via GitHub)" <gi...@apache.org> on 2025/01/15 09:20:48 UTC
[PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
wuzhoupei opened a new pull request, #45268:
URL: https://github.com/apache/arrow/pull/45268
<!--
Thanks for opening a pull request!
If this is your first pull request you can find detailed information on how
to contribute here:
* [New Contributor's Guide](https://arrow.apache.org/docs/dev/developers/guide/step_by_step/pr_lifecycle.html#reviews-and-merge-of-the-pull-request)
* [Contributing Overview](https://arrow.apache.org/docs/dev/developers/overview.html)
If this is not a [minor PR](https://github.com/apache/arrow/blob/main/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
or
MINOR: [${COMPONENT}] ${SUMMARY}
-->
### Rationale for this change
When the TaskGroup should be canceled, it will move the number which not-start to finished to avoid do them(in `TaskSchedulerImpl::Abort`). But this is one operation that happens in multi-threads. At the same time, maybe some task start to running and happen some error. Then they will return the bad status.
But the tasks are running for Scheduler, they will just return bad status and not change the running_task count. Because the code uses `RETURN_NOT_OK`.
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
### What changes are included in this PR?
For any task, what status weather it returns, it will change the running_count before return.
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
### Are these changes tested?
No. It is too hard to build ut.
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
### Are there any user-facing changes?
No. But I am very shocked at hasn't this happened to anyone?
- GitHub Issue: https://github.com/apache/arrow/issues/45266
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please uncomment the line below and explain which changes are breaking.
-->
<!-- **This PR includes breaking changes to public APIs.** -->
<!--
Please uncomment the line below (and provide explanation) if the changes fix either (a) a security vulnerability, (b) a bug that caused incorrect or invalid data to be produced, or (c) a bug that causes a crash (even when the API contract is upheld). We use this to highlight fixes to issues that may affect users without their knowledge. For this reason, fixing bugs that cause errors don't count, since those are usually obvious.
-->
<!-- **This PR contains a "Critical Fix".** -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #45268:
URL: https://github.com/apache/arrow/pull/45268#issuecomment-2648629186
Revision: 894dd6830fd96406a8c5aa54d710dda6f69a6e92
Submitted crossbow builds: [ursacomputing/crossbow @ actions-c374bc6955](https://github.com/ursacomputing/crossbow/branches/all?query=actions-c374bc6955)
|Task|Status|
|----|------|
|example-cpp-minimal-build-static|[](https://github.com/ursacomputing/crossbow/actions/runs/13245859388/job/36972079914)|
|example-cpp-minimal-build-static-system-dependency|[](https://github.com/ursacomputing/crossbow/actions/runs/13245860264/job/36972085252)|
|example-cpp-tutorial|[](https://github.com/ursacomputing/crossbow/actions/runs/13245859829/job/36972083813)|
|test-alpine-linux-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/13245859749/job/36972082854)|
|test-build-cpp-fuzz|[](https://github.com/ursacomputing/crossbow/actions/runs/13245859574/job/36972081084)|
|test-conda-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/13245859836/job/36972083207)|
|test-conda-cpp-valgrind|[](https://github.com/ursacomputing/crossbow/actions/runs/13245859830/job/36972083802)|
|test-cuda-cpp-ubuntu-20.04-cuda-11.2.2|[](https://github.com/ursacomputing/crossbow/actions/runs/13245859729/job/36972083445)|
|test-cuda-cpp-ubuntu-22.04-cuda-11.7.1|[](https://github.com/ursacomputing/crossbow/actions/runs/13245860060/job/36972084666)|
|test-debian-12-cpp-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/13245859575/job/36972081762)|
|test-debian-12-cpp-i386|[](https://github.com/ursacomputing/crossbow/actions/runs/13245859534/job/36972081082)|
|test-fedora-39-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/13245859363/job/36972079922)|
|test-ubuntu-20.04-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/13245860025/job/36972084651)|
|test-ubuntu-20.04-cpp-bundled|[](https://github.com/ursacomputing/crossbow/actions/runs/13245859367/job/36972079963)|
|test-ubuntu-22.04-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/13245859873/job/36972082861)|
|test-ubuntu-22.04-cpp-20|[](https://github.com/ursacomputing/crossbow/actions/runs/13245859825/job/36972083493)|
|test-ubuntu-22.04-cpp-emscripten|[](https://github.com/ursacomputing/crossbow/actions/runs/13245859686/job/36972082407)|
|test-ubuntu-22.04-cpp-no-threading|[](https://github.com/ursacomputing/crossbow/actions/runs/13245860098/job/36972085545)|
|test-ubuntu-24.04-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/13245859783/job/36972083143)|
|test-ubuntu-24.04-cpp-bundled-offline|[](https://github.com/ursacomputing/crossbow/actions/runs/13245859822/job/36972083219)|
|test-ubuntu-24.04-cpp-gcc-13-bundled|[](https://github.com/ursacomputing/crossbow/actions/runs/13245859722/job/36972082823)|
|test-ubuntu-24.04-cpp-gcc-14|[](https://github.com/ursacomputing/crossbow/actions/runs/13245859887/job/36972084047)|
|test-ubuntu-24.04-cpp-minimal-with-formats|[](https://github.com/ursacomputing/crossbow/actions/runs/13245859392/job/36972080006)|
|test-ubuntu-24.04-cpp-thread-sanitizer|[](https://github.com/ursacomputing/crossbow/actions/runs/13245859414/job/36972080323)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #45268:
URL: https://github.com/apache/arrow/pull/45268#issuecomment-2647592797
Revision: 4de515e7c5978746ce067ec9c4c0a5b6ae901abd
Submitted crossbow builds: [ursacomputing/crossbow @ actions-5c7ec2edf2](https://github.com/ursacomputing/crossbow/branches/all?query=actions-5c7ec2edf2)
|Task|Status|
|----|------|
|example-cpp-minimal-build-static|[](https://github.com/ursacomputing/crossbow/actions/runs/13238909058/job/36949691180)|
|example-cpp-minimal-build-static-system-dependency|[](https://github.com/ursacomputing/crossbow/actions/runs/13238909618/job/36949694442)|
|example-cpp-tutorial|[](https://github.com/ursacomputing/crossbow/actions/runs/13238909268/job/36949692449)|
|test-alpine-linux-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/13238909388/job/36949693159)|
|test-build-cpp-fuzz|[](https://github.com/ursacomputing/crossbow/actions/runs/13238909341/job/36949693206)|
|test-conda-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/13238909192/job/36949692142)|
|test-conda-cpp-valgrind|[](https://github.com/ursacomputing/crossbow/actions/runs/13238909094/job/36949690941)|
|test-cuda-cpp-ubuntu-20.04-cuda-11.2.2|[](https://github.com/ursacomputing/crossbow/actions/runs/13238909960/job/36949695641)|
|test-cuda-cpp-ubuntu-22.04-cuda-11.7.1|[](https://github.com/ursacomputing/crossbow/actions/runs/13238909035/job/36949691192)|
|test-debian-12-cpp-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/13238909068/job/36949690828)|
|test-debian-12-cpp-i386|[](https://github.com/ursacomputing/crossbow/actions/runs/13238909142/job/36949690781)|
|test-fedora-39-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/13238910114/job/36949696508)|
|test-ubuntu-20.04-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/13238909078/job/36949690811)|
|test-ubuntu-20.04-cpp-bundled|[](https://github.com/ursacomputing/crossbow/actions/runs/13238909613/job/36949694378)|
|test-ubuntu-22.04-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/13238909625/job/36949694420)|
|test-ubuntu-22.04-cpp-20|[](https://github.com/ursacomputing/crossbow/actions/runs/13238909442/job/36949693165)|
|test-ubuntu-22.04-cpp-emscripten|[](https://github.com/ursacomputing/crossbow/actions/runs/13238909305/job/36949693157)|
|test-ubuntu-22.04-cpp-no-threading|[](https://github.com/ursacomputing/crossbow/actions/runs/13238909775/job/36949695663)|
|test-ubuntu-24.04-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/13238909893/job/36949695675)|
|test-ubuntu-24.04-cpp-bundled-offline|[](https://github.com/ursacomputing/crossbow/actions/runs/13238909635/job/36949694377)|
|test-ubuntu-24.04-cpp-gcc-13-bundled|[](https://github.com/ursacomputing/crossbow/actions/runs/13238910094/job/36949696527)|
|test-ubuntu-24.04-cpp-gcc-14|[](https://github.com/ursacomputing/crossbow/actions/runs/13238909640/job/36949694803)|
|test-ubuntu-24.04-cpp-minimal-with-formats|[](https://github.com/ursacomputing/crossbow/actions/runs/13238909478/job/36949693548)|
|test-ubuntu-24.04-cpp-thread-sanitizer|[](https://github.com/ursacomputing/crossbow/actions/runs/13238909508/job/36949693538)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #45268:
URL: https://github.com/apache/arrow/pull/45268#issuecomment-2651159426
Fair enough. Thank you for doing this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on PR #45268:
URL: https://github.com/apache/arrow/pull/45268#issuecomment-2650341964
> It does, but I was wondering if the overall design is still correct now that we've made another variable atomic, with updates being done without the mutex.
I totally understand the concern here. But I think at this point only the original author can really answer this.
I'm making this change based on the following reasons:
1) In the original code there already exist several accesses to `aborted_` without locking. I assume we didn't have a clear definition of how this `aborted_` should be synchronized at the beginning so it has been half locking-based half lockless. (Acero has been fine using it because Acero establishes explicit synchronization from the outside by using future/continuation semantics).
2) AFAICT, The access pattern of `aborted_` is fairly straightforward - there isn't any other state that's relying on it.
3) The test is harsh enough. It triggers the `Abort()` in very fragile timing. Green TSAN gives me a little more confidence.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #45268:
URL: https://github.com/apache/arrow/pull/45268#issuecomment-2648696921
Is it important to avoid taking the mutex in `ScheduleMore`? I see a bunch of atomic variable accesses, which technically pass under TSan, but there is no synchronization otherwise, and it's not ok that the overall state remains consistent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #45268:
URL: https://github.com/apache/arrow/pull/45268#issuecomment-2592092830
:warning: GitHub issue #45266 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #45268:
URL: https://github.com/apache/arrow/pull/45268#issuecomment-2647586690
@github-actions crossbow submit -g cpp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on code in PR #45268:
URL: https://github.com/apache/arrow/pull/45268#discussion_r1924588082
##########
cpp/src/arrow/acero/task_util.cc:
##########
@@ -369,17 +369,25 @@ Status TaskSchedulerImpl::ScheduleMore(size_t thread_id, int num_tasks_finished)
int group_id = tasks[i].first;
int64_t task_id = tasks[i].second;
RETURN_NOT_OK(schedule_impl_([this, group_id, task_id](size_t thread_id) -> Status {
- RETURN_NOT_OK(ScheduleMore(thread_id, 1));
-
bool task_group_finished = false;
- RETURN_NOT_OK(ExecuteTask(thread_id, group_id, task_id, &task_group_finished));
+ // PostExecuteTask must be called later if any error ocurres during task execution
+ // (including ScheduleMore), so we preserve the status.
+ auto status = [&]() {
+ RETURN_NOT_OK(ScheduleMore(thread_id, 1));
+ return ExecuteTask(thread_id, group_id, task_id, &task_group_finished);
+ }();
+
+ if (!status.ok()) {
+ task_group_finished = PostExecuteTask(thread_id, group_id);
+ }
if (task_group_finished) {
bool all_task_groups_finished = false;
- return OnTaskGroupFinished(thread_id, group_id, &all_task_groups_finished);
+ RETURN_NOT_OK(
+ OnTaskGroupFinished(thread_id, group_id, &all_task_groups_finished));
Review Comment:
I made some refinement on top of your original change, PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "wuzhoupei (via GitHub)" <gi...@apache.org>.
wuzhoupei commented on PR #45268:
URL: https://github.com/apache/arrow/pull/45268#issuecomment-2601478925
Thanks the review from @zanmato1984.
I will be very grateful if you have a good idea to test it and can add the test for this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on PR #45268:
URL: https://github.com/apache/arrow/pull/45268#issuecomment-2648653069
The race is that we didn't create synchronization for `aborted_` flag between `Abort` and `ExecuteTask`. I'm changing it to atomic variable to avoid the race (and moving it out from the critical sections protected by the `mutext_`). I assume the original design decision didn't forbid calling `Abort` during task execution (you can see that in the original code the `Abort` function checks if the task group state is `TaskGroupState::READY` which implies it already assumes a particular task groups is already started but not ran yet - but it could also be the case that it is running already).
cc @pitrou @westonpace
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on PR #45268:
URL: https://github.com/apache/arrow/pull/45268#issuecomment-2611448523
Since this includes some change of mine, I'd request @pitrou to take a look too. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #45268:
URL: https://github.com/apache/arrow/pull/45268#discussion_r1924645242
##########
cpp/src/arrow/acero/task_util.cc:
##########
@@ -278,8 +278,8 @@ Status TaskSchedulerImpl::ExecuteMore(size_t thread_id, int num_tasks_to_execute
bool task_group_finished = false;
Status status = ExecuteTask(thread_id, group_id, task_id, &task_group_finished);
if (!status.ok()) {
- // Mark the remaining picked tasks as finished
- for (size_t j = i + 1; j < tasks.size(); ++j) {
+ // Mark the current and remaining picked tasks as finished
+ for (size_t j = i; j < tasks.size(); ++j) {
Review Comment:
So PostExecuteTask need execute the task even if failed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on code in PR #45268:
URL: https://github.com/apache/arrow/pull/45268#discussion_r1948436541
##########
cpp/src/arrow/acero/task_util_test.cc:
##########
@@ -231,5 +231,96 @@ TEST(TaskScheduler, StressTwo) {
}
}
+TEST(TaskScheduler, AbortContOnTaskErrorSerial) {
+ constexpr int kNumTasks = 16;
+
+ auto scheduler = TaskScheduler::Make();
+ auto task = [&](std::size_t, int64_t task_id) {
+ if (task_id == kNumTasks / 2) {
+ return Status::Invalid("Task failed");
+ }
+ return Status::OK();
+ };
+
+ int task_group =
+ scheduler->RegisterTaskGroup(task, [](std::size_t) { return Status::OK(); });
+ scheduler->RegisterEnd();
+
+ ASSERT_OK(scheduler->StartScheduling(
+ 0, [](TaskScheduler::TaskGroupContinuationImpl) { return Status::OK(); }, 1, true));
+ ASSERT_RAISES_WITH_MESSAGE(Invalid, "Invalid: Task failed",
+ scheduler->StartTaskGroup(0, task_group, kNumTasks));
+
+ bool abort_cont_called = false;
+ auto abort_cont = [&]() {
+ ASSERT_FALSE(abort_cont_called);
+ abort_cont_called = true;
+ };
+
+ scheduler->Abort(abort_cont);
+
+ ASSERT_TRUE(abort_cont_called);
+}
+
+TEST(TaskScheduler, AbortContOnTaskErrorParallel) {
+#ifndef ARROW_ENABLE_THREADING
+ GTEST_SKIP() << "Test requires threading support";
+#endif
+ constexpr int kNumThreads = 16;
+
+ ThreadIndexer thread_indexer;
+ int num_threads = std::min(static_cast<int>(thread_indexer.Capacity()), kNumThreads);
+ ASSERT_OK_AND_ASSIGN(std::shared_ptr<ThreadPool> thread_pool,
+ MakePrimedThreadPool(num_threads));
+ TaskScheduler::ScheduleImpl schedule =
+ [&](TaskScheduler::TaskGroupContinuationImpl task) {
+ return thread_pool->Spawn([&, task] {
+ std::size_t thread_id = thread_indexer();
+ auto status = task(thread_id);
+ ASSERT_TRUE(status.ok() || status.IsInvalid() || status.IsCancelled());
+ });
+ };
+
+ for (int num_tasks :
+ {2, num_threads - 1, num_threads, num_threads + 1, 2 * num_threads}) {
+ ARROW_SCOPED_TRACE("num_tasks = ", num_tasks);
+ for (int num_concurrent_tasks :
+ {1, num_tasks - 1, num_tasks, num_tasks + 1, 2 * num_tasks}) {
+ ARROW_SCOPED_TRACE("num_concurrent_tasks = ", num_concurrent_tasks);
+ for (int aborting_task_id = 0; aborting_task_id < num_tasks; ++aborting_task_id) {
+ ARROW_SCOPED_TRACE("aborting_task_id = ", aborting_task_id);
+ auto scheduler = TaskScheduler::Make();
+
+ bool abort_cont_called = false;
+ auto abort_cont = [&]() {
+ ASSERT_FALSE(abort_cont_called);
+ abort_cont_called = true;
+ };
+
+ auto task = [&](std::size_t, int64_t task_id) {
+ if (task_id == aborting_task_id) {
+ scheduler->Abort(abort_cont);
+ }
+ if (task_id % 2 == 0) {
+ return Status::Invalid("Task failed");
+ }
+ return Status::OK();
+ };
+
+ int task_group =
+ scheduler->RegisterTaskGroup(task, [](std::size_t) { return Status::OK(); });
+ scheduler->RegisterEnd();
+
+ ASSERT_OK(scheduler->StartScheduling(0, schedule, num_concurrent_tasks, false));
+ ASSERT_OK(scheduler->StartTaskGroup(0, task_group, num_tasks));
Review Comment:
At least not here. This `StartTaskGroup` merely schedules the parallel tasks into the underlying executor (the raw thread pool in this specific case, and the more sophisticated `AsyncScheduler` in official acero code path). The status returned here is merely if the tasks are successfully scheduled. The error of a specific task will be propagated thru the abort continuation of this `TaskScheduler`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on code in PR #45268:
URL: https://github.com/apache/arrow/pull/45268#discussion_r1948445001
##########
cpp/src/arrow/acero/task_util.cc:
##########
@@ -413,6 +421,8 @@ void TaskSchedulerImpl::Abort(AbortContinuationImpl impl) {
all_finished = false;
task_group.state_ = TaskGroupState::ALL_TASKS_STARTED;
}
+ } else if (task_group.state_ == TaskGroupState::ALL_TASKS_STARTED) {
Review Comment:
OK, done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #45268:
URL: https://github.com/apache/arrow/pull/45268#issuecomment-2647103099
> and the problem lies in the recursive call to `ScheduleMore`
Ahh got this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on PR #45268:
URL: https://github.com/apache/arrow/pull/45268#issuecomment-2605300874
Hi @wuzhoupei , I've committed two test cases that can stably reproduce the issue of abort continuation not being invoked after aborting. Along with them are some more fixes necessary. Will you take a look? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on code in PR #45268:
URL: https://github.com/apache/arrow/pull/45268#discussion_r1928372488
##########
cpp/src/arrow/acero/task_util.cc:
##########
@@ -369,17 +369,25 @@ Status TaskSchedulerImpl::ScheduleMore(size_t thread_id, int num_tasks_finished)
int group_id = tasks[i].first;
int64_t task_id = tasks[i].second;
RETURN_NOT_OK(schedule_impl_([this, group_id, task_id](size_t thread_id) -> Status {
- RETURN_NOT_OK(ScheduleMore(thread_id, 1));
-
bool task_group_finished = false;
- RETURN_NOT_OK(ExecuteTask(thread_id, group_id, task_id, &task_group_finished));
+ // PostExecuteTask must be called later if any error ocurres during task execution
+ // (including ScheduleMore), so we preserve the status.
+ auto status = [&]() {
+ RETURN_NOT_OK(ScheduleMore(thread_id, 1));
+ return ExecuteTask(thread_id, group_id, task_id, &task_group_finished);
+ }();
+
+ if (!status.ok()) {
+ task_group_finished = PostExecuteTask(thread_id, group_id);
+ }
if (task_group_finished) {
bool all_task_groups_finished = false;
- return OnTaskGroupFinished(thread_id, group_id, &all_task_groups_finished);
+ RETURN_NOT_OK(
+ OnTaskGroupFinished(thread_id, group_id, &all_task_groups_finished));
Review Comment:
Yeah, I don't think it really matters which error to return. So I chose the one can most simplify the code :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "wuzhoupei (via GitHub)" <gi...@apache.org>.
wuzhoupei commented on PR #45268:
URL: https://github.com/apache/arrow/pull/45268#issuecomment-2597339515
I agree some of you. @zanmato1984
But I still think we need to change the task count when task get an error status. Maybe my word in old comment is not exact. The count is the num of running-tasks which I use the 'finished-tasks' in old comment.
So this case can be said like that:
1. when the scheduler calling the Abort, if some tasks have started and not finish, this TaskGroup's status will be set as `TaskGroupState::ALL_TASKS_STARTED`.
2. if the running-tasks at above get error, they will not change the num of running-tasks.
3. but the must condition for scheduler calls the AbortFunc is the num of running-tasks is 0.
4. so this case will let the AbortFunc can not be called, and then the ExecPlan will can not call the StopProducingImpl forever(StopProducingImpl will be called in AbortFunc).
So I think we should change the count which belongs to scheduler whatever status the task finished with.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #45268:
URL: https://github.com/apache/arrow/pull/45268#issuecomment-2653466779
After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit e9ec38396ed66cd34fea2884fc7ebbe0281a05ef.
There were no benchmark performance regressions. 🎉
The [full Conbench report](https://github.com/apache/arrow/runs/37090108774) has more details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #45268:
URL: https://github.com/apache/arrow/pull/45268#issuecomment-2647587085
Let's wait for the Thread Sanitizer results on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #45268:
URL: https://github.com/apache/arrow/pull/45268#issuecomment-2650095844
> Does this answer your question? Thanks.
It does, but I was wondering if the overall design is still correct now that we've made another variable atomic, with updates being done without the mutex.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "wuzhoupei (via GitHub)" <gi...@apache.org>.
wuzhoupei commented on code in PR #45268:
URL: https://github.com/apache/arrow/pull/45268#discussion_r1944269867
##########
cpp/src/arrow/acero/task_util.cc:
##########
@@ -369,17 +369,25 @@ Status TaskSchedulerImpl::ScheduleMore(size_t thread_id, int num_tasks_finished)
int group_id = tasks[i].first;
int64_t task_id = tasks[i].second;
RETURN_NOT_OK(schedule_impl_([this, group_id, task_id](size_t thread_id) -> Status {
- RETURN_NOT_OK(ScheduleMore(thread_id, 1));
-
bool task_group_finished = false;
- RETURN_NOT_OK(ExecuteTask(thread_id, group_id, task_id, &task_group_finished));
+ // PostExecuteTask must be called later if any error ocurres during task execution
+ // (including ScheduleMore), so we preserve the status.
+ auto status = [&]() {
Review Comment:
To avoid too many `if` in code. If we don't do some special handling, we must use 2 `if` to check the return status is OK or NOT-OK. So I use a lambda(my special handling) to keep the clear code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on code in PR #45268:
URL: https://github.com/apache/arrow/pull/45268#discussion_r1948422554
##########
cpp/src/arrow/acero/task_util_test.cc:
##########
@@ -231,5 +231,96 @@ TEST(TaskScheduler, StressTwo) {
}
}
+TEST(TaskScheduler, AbortContOnTaskErrorSerial) {
+ constexpr int kNumTasks = 16;
+
+ auto scheduler = TaskScheduler::Make();
+ auto task = [&](std::size_t, int64_t task_id) {
+ if (task_id == kNumTasks / 2) {
+ return Status::Invalid("Task failed");
+ }
+ return Status::OK();
+ };
+
+ int task_group =
+ scheduler->RegisterTaskGroup(task, [](std::size_t) { return Status::OK(); });
+ scheduler->RegisterEnd();
+
+ ASSERT_OK(scheduler->StartScheduling(
+ 0, [](TaskScheduler::TaskGroupContinuationImpl) { return Status::OK(); }, 1, true));
Review Comment:
Thank you. Done.
##########
cpp/src/arrow/acero/task_util_test.cc:
##########
@@ -231,5 +231,96 @@ TEST(TaskScheduler, StressTwo) {
}
}
+TEST(TaskScheduler, AbortContOnTaskErrorSerial) {
+ constexpr int kNumTasks = 16;
+
+ auto scheduler = TaskScheduler::Make();
+ auto task = [&](std::size_t, int64_t task_id) {
+ if (task_id == kNumTasks / 2) {
+ return Status::Invalid("Task failed");
+ }
+ return Status::OK();
+ };
+
+ int task_group =
+ scheduler->RegisterTaskGroup(task, [](std::size_t) { return Status::OK(); });
+ scheduler->RegisterEnd();
+
+ ASSERT_OK(scheduler->StartScheduling(
+ 0, [](TaskScheduler::TaskGroupContinuationImpl) { return Status::OK(); }, 1, true));
+ ASSERT_RAISES_WITH_MESSAGE(Invalid, "Invalid: Task failed",
+ scheduler->StartTaskGroup(0, task_group, kNumTasks));
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on code in PR #45268:
URL: https://github.com/apache/arrow/pull/45268#discussion_r1948422879
##########
cpp/src/arrow/acero/task_util_test.cc:
##########
@@ -231,5 +231,96 @@ TEST(TaskScheduler, StressTwo) {
}
}
+TEST(TaskScheduler, AbortContOnTaskErrorSerial) {
+ constexpr int kNumTasks = 16;
+
+ auto scheduler = TaskScheduler::Make();
+ auto task = [&](std::size_t, int64_t task_id) {
+ if (task_id == kNumTasks / 2) {
+ return Status::Invalid("Task failed");
+ }
+ return Status::OK();
+ };
+
+ int task_group =
+ scheduler->RegisterTaskGroup(task, [](std::size_t) { return Status::OK(); });
+ scheduler->RegisterEnd();
+
+ ASSERT_OK(scheduler->StartScheduling(
+ 0, [](TaskScheduler::TaskGroupContinuationImpl) { return Status::OK(); }, 1, true));
+ ASSERT_RAISES_WITH_MESSAGE(Invalid, "Invalid: Task failed",
+ scheduler->StartTaskGroup(0, task_group, kNumTasks));
+
+ bool abort_cont_called = false;
+ auto abort_cont = [&]() {
+ ASSERT_FALSE(abort_cont_called);
+ abort_cont_called = true;
+ };
+
+ scheduler->Abort(abort_cont);
+
+ ASSERT_TRUE(abort_cont_called);
Review Comment:
Good idea, done! Thank you.
##########
cpp/src/arrow/acero/task_util_test.cc:
##########
@@ -231,5 +231,96 @@ TEST(TaskScheduler, StressTwo) {
}
}
+TEST(TaskScheduler, AbortContOnTaskErrorSerial) {
+ constexpr int kNumTasks = 16;
+
+ auto scheduler = TaskScheduler::Make();
+ auto task = [&](std::size_t, int64_t task_id) {
+ if (task_id == kNumTasks / 2) {
+ return Status::Invalid("Task failed");
+ }
+ return Status::OK();
+ };
+
+ int task_group =
+ scheduler->RegisterTaskGroup(task, [](std::size_t) { return Status::OK(); });
+ scheduler->RegisterEnd();
+
+ ASSERT_OK(scheduler->StartScheduling(
+ 0, [](TaskScheduler::TaskGroupContinuationImpl) { return Status::OK(); }, 1, true));
+ ASSERT_RAISES_WITH_MESSAGE(Invalid, "Invalid: Task failed",
+ scheduler->StartTaskGroup(0, task_group, kNumTasks));
+
+ bool abort_cont_called = false;
+ auto abort_cont = [&]() {
+ ASSERT_FALSE(abort_cont_called);
+ abort_cont_called = true;
+ };
+
+ scheduler->Abort(abort_cont);
+
+ ASSERT_TRUE(abort_cont_called);
+}
+
+TEST(TaskScheduler, AbortContOnTaskErrorParallel) {
+#ifndef ARROW_ENABLE_THREADING
+ GTEST_SKIP() << "Test requires threading support";
+#endif
+ constexpr int kNumThreads = 16;
+
+ ThreadIndexer thread_indexer;
+ int num_threads = std::min(static_cast<int>(thread_indexer.Capacity()), kNumThreads);
+ ASSERT_OK_AND_ASSIGN(std::shared_ptr<ThreadPool> thread_pool,
+ MakePrimedThreadPool(num_threads));
+ TaskScheduler::ScheduleImpl schedule =
+ [&](TaskScheduler::TaskGroupContinuationImpl task) {
+ return thread_pool->Spawn([&, task] {
+ std::size_t thread_id = thread_indexer();
+ auto status = task(thread_id);
+ ASSERT_TRUE(status.ok() || status.IsInvalid() || status.IsCancelled());
+ });
+ };
+
+ for (int num_tasks :
+ {2, num_threads - 1, num_threads, num_threads + 1, 2 * num_threads}) {
+ ARROW_SCOPED_TRACE("num_tasks = ", num_tasks);
+ for (int num_concurrent_tasks :
+ {1, num_tasks - 1, num_tasks, num_tasks + 1, 2 * num_tasks}) {
+ ARROW_SCOPED_TRACE("num_concurrent_tasks = ", num_concurrent_tasks);
+ for (int aborting_task_id = 0; aborting_task_id < num_tasks; ++aborting_task_id) {
+ ARROW_SCOPED_TRACE("aborting_task_id = ", aborting_task_id);
+ auto scheduler = TaskScheduler::Make();
+
+ bool abort_cont_called = false;
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #45268:
URL: https://github.com/apache/arrow/pull/45268#discussion_r1944293026
##########
cpp/src/arrow/acero/task_util.cc:
##########
@@ -413,6 +421,8 @@ void TaskSchedulerImpl::Abort(AbortContinuationImpl impl) {
all_finished = false;
task_group.state_ = TaskGroupState::ALL_TASKS_STARTED;
}
+ } else if (task_group.state_ == TaskGroupState::ALL_TASKS_STARTED) {
Review Comment:
Minor nit: can we change this to a switch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "wuzhoupei (via GitHub)" <gi...@apache.org>.
wuzhoupei commented on code in PR #45268:
URL: https://github.com/apache/arrow/pull/45268#discussion_r1928364435
##########
cpp/src/arrow/acero/task_util.cc:
##########
@@ -369,17 +369,25 @@ Status TaskSchedulerImpl::ScheduleMore(size_t thread_id, int num_tasks_finished)
int group_id = tasks[i].first;
int64_t task_id = tasks[i].second;
RETURN_NOT_OK(schedule_impl_([this, group_id, task_id](size_t thread_id) -> Status {
- RETURN_NOT_OK(ScheduleMore(thread_id, 1));
-
bool task_group_finished = false;
- RETURN_NOT_OK(ExecuteTask(thread_id, group_id, task_id, &task_group_finished));
+ // PostExecuteTask must be called later if any error ocurres during task execution
+ // (including ScheduleMore), so we preserve the status.
+ auto status = [&]() {
+ RETURN_NOT_OK(ScheduleMore(thread_id, 1));
+ return ExecuteTask(thread_id, group_id, task_id, &task_group_finished);
+ }();
+
+ if (!status.ok()) {
+ task_group_finished = PostExecuteTask(thread_id, group_id);
+ }
if (task_group_finished) {
bool all_task_groups_finished = false;
- return OnTaskGroupFinished(thread_id, group_id, &all_task_groups_finished);
+ RETURN_NOT_OK(
+ OnTaskGroupFinished(thread_id, group_id, &all_task_groups_finished));
Review Comment:
Actually, I am not sure which one should be returned when we get two error status: CurrentTask's and OnTaskGroupFinished's. I think maybe we can just return the first error status(CurrentTask always error before OnTaskGroupFinished).
Anyway, it is ok for each one be returned :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #45268:
URL: https://github.com/apache/arrow/pull/45268#issuecomment-2592112818
Great! Also would it easy to add a test for this case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on code in PR #45268:
URL: https://github.com/apache/arrow/pull/45268#discussion_r1924100107
##########
cpp/src/arrow/acero/task_util.cc:
##########
@@ -278,8 +278,8 @@ Status TaskSchedulerImpl::ExecuteMore(size_t thread_id, int num_tasks_to_execute
bool task_group_finished = false;
Status status = ExecuteTask(thread_id, group_id, task_id, &task_group_finished);
if (!status.ok()) {
- // Mark the remaining picked tasks as finished
- for (size_t j = i + 1; j < tasks.size(); ++j) {
+ // Mark the current and remaining picked tasks as finished
+ for (size_t j = i; j < tasks.size(); ++j) {
Review Comment:
This is needed for the serial execution path to correctly set the task count.
##########
cpp/src/arrow/acero/task_util.cc:
##########
@@ -413,6 +421,8 @@ void TaskSchedulerImpl::Abort(AbortContinuationImpl impl) {
all_finished = false;
task_group.state_ = TaskGroupState::ALL_TASKS_STARTED;
}
+ } else if (task_group.state_ == TaskGroupState::ALL_TASKS_STARTED) {
Review Comment:
This is needed for allowing the `Abort` to be called in a task itself, otherwise the abort continuation will be called twice.
I suppose the `Abort` function is not necessarily to be called inside a task body to trigger the issue. In other words, it could happen when `Abort` is called in a timing that is subtle enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on PR #45268:
URL: https://github.com/apache/arrow/pull/45268#issuecomment-2595171788
I'm thinking this might not be very necessary for the following two reasons:
1. I think in the current acero design, once an error occurs in a task, this error will be propagated to the upmost `ExecPlan`, which will in turn call the `Abort` method of task scheduler, which will finish all the running tasks regardless of an individual task group's finished count being arguably "wrong".
2. Even if we think the task group's finished count as "wrong" (once errors occur in a task group), and we want to make it "right" so that the task group will finish in a graceful way (it doesn't depend on the `Abort`), we have a problem of invoking the continuation. Generally a continuation of a task group shouldn't be called if some tasks of the group meet error - this is the case for current implementation because the task group is not considered "finished" (thanks to the "wrong" finished count). But this will change if we make the finished count "right" - the continuation will be (undesirably) invoked.
What do you think? @wuzhoupei
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on code in PR #45268:
URL: https://github.com/apache/arrow/pull/45268#discussion_r1924653432
##########
cpp/src/arrow/acero/task_util.cc:
##########
@@ -278,8 +278,8 @@ Status TaskSchedulerImpl::ExecuteMore(size_t thread_id, int num_tasks_to_execute
bool task_group_finished = false;
Status status = ExecuteTask(thread_id, group_id, task_id, &task_group_finished);
if (!status.ok()) {
- // Mark the remaining picked tasks as finished
- for (size_t j = i + 1; j < tasks.size(); ++j) {
+ // Mark the current and remaining picked tasks as finished
+ for (size_t j = i; j < tasks.size(); ++j) {
Review Comment:
Yeah this is pretty much the whole point of this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 merged PR #45268:
URL: https://github.com/apache/arrow/pull/45268
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on PR #45268:
URL: https://github.com/apache/arrow/pull/45268#issuecomment-2648617186
@github-actions crossbow submit -g cpp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on PR #45268:
URL: https://github.com/apache/arrow/pull/45268#issuecomment-2649597774
> Is it important to avoid taking the mutex in `ScheduleMore`? I see a bunch of atomic variable accesses, which technically pass under TSan, but there is no synchronization otherwise, and it's not ok that the overall state remains consistent.
Sorry I'm not sure if I fully understand your question. If your concern is about some variables not being protected by the mutex, the original design, IIUC seems to be a hybrid scheduling of both locking-based and lockless. In particular, the metadata, namely the "task groups", (before this change the `aborted_` flag too) is protected by the mutex, whereas the rest are lockless. Does this answer your question? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45268:
URL: https://github.com/apache/arrow/pull/45268#discussion_r1944329079
##########
cpp/src/arrow/acero/task_util.cc:
##########
@@ -369,17 +369,25 @@ Status TaskSchedulerImpl::ScheduleMore(size_t thread_id, int num_tasks_finished)
int group_id = tasks[i].first;
int64_t task_id = tasks[i].second;
RETURN_NOT_OK(schedule_impl_([this, group_id, task_id](size_t thread_id) -> Status {
- RETURN_NOT_OK(ScheduleMore(thread_id, 1));
-
bool task_group_finished = false;
- RETURN_NOT_OK(ExecuteTask(thread_id, group_id, task_id, &task_group_finished));
+ // PostExecuteTask must be called later if any error ocurres during task execution
+ // (including ScheduleMore), so we preserve the status.
+ auto status = [&]() {
Review Comment:
Fair enough, thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "wuzhoupei (via GitHub)" <gi...@apache.org>.
wuzhoupei commented on PR #45268:
URL: https://github.com/apache/arrow/pull/45268#issuecomment-2592129013
> Great! Also would it easy to add a test for this case?
Actually, I think it is hard to add some tests for this case. Because It is a multi-threads case. If we wanna add a test about it, we need to make sure the Abort-operation's happen time and the task which in this time should be return error-status.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on PR #45268:
URL: https://github.com/apache/arrow/pull/45268#issuecomment-2601431008
Hi @wuzhoupei , thank you for the further explanation. Yes you are right, the problem of abort continuation not being called exists if the task count is not correct. And by looking at the code, the task group continuation being called after task meets error seems to be by design allowed - it will first examine the somewhat internal error state to exist early.
I think we can proceed with this PR, I'll put my review comment on the code. Meanwhile, it will be very helpful to have a test cast about this. I can help if you have trouble on that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #45268:
URL: https://github.com/apache/arrow/pull/45268#issuecomment-2648624481
Revision: 7598af62326a3b4cf39cc33de9e7ecac1ad31840
Submitted crossbow builds: [ursacomputing/crossbow @ actions-65d9b3450c](https://github.com/ursacomputing/crossbow/branches/all?query=actions-65d9b3450c)
|Task|Status|
|----|------|
|example-cpp-minimal-build-static|[](https://github.com/ursacomputing/crossbow/actions/runs/13245823023/job/36971967227)|
|example-cpp-minimal-build-static-system-dependency|[](https://github.com/ursacomputing/crossbow/actions/runs/13245822502/job/36971964986)|
|example-cpp-tutorial|[](https://github.com/ursacomputing/crossbow/actions/runs/13245823115/job/36971968274)|
|test-alpine-linux-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/13245823182/job/36971969468)|
|test-build-cpp-fuzz|[](https://github.com/ursacomputing/crossbow/actions/runs/13245822615/job/36971965936)|
|test-conda-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/13245822965/job/36971966893)|
|test-conda-cpp-valgrind|[](https://github.com/ursacomputing/crossbow/actions/runs/13245823561/job/36971970360)|
|test-cuda-cpp-ubuntu-20.04-cuda-11.2.2|[](https://github.com/ursacomputing/crossbow/actions/runs/13245822926/job/36971966879)|
|test-cuda-cpp-ubuntu-22.04-cuda-11.7.1|[](https://github.com/ursacomputing/crossbow/actions/runs/13245823520/job/36971970361)|
|test-debian-12-cpp-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/13245822327/job/36971964599)|
|test-debian-12-cpp-i386|[](https://github.com/ursacomputing/crossbow/actions/runs/13245822622/job/36971965969)|
|test-fedora-39-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/13245823080/job/36971967531)|
|test-ubuntu-20.04-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/13245823439/job/36971969593)|
|test-ubuntu-20.04-cpp-bundled|[](https://github.com/ursacomputing/crossbow/actions/runs/13245823081/job/36971968285)|
|test-ubuntu-22.04-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/13245823590/job/36971970374)|
|test-ubuntu-22.04-cpp-20|[](https://github.com/ursacomputing/crossbow/actions/runs/13245822308/job/36971964600)|
|test-ubuntu-22.04-cpp-emscripten|[](https://github.com/ursacomputing/crossbow/actions/runs/13245822763/job/36971966799)|
|test-ubuntu-22.04-cpp-no-threading|[](https://github.com/ursacomputing/crossbow/actions/runs/13245822483/job/36971964667)|
|test-ubuntu-24.04-cpp|[](https://github.com/ursacomputing/crossbow/actions/runs/13245822757/job/36971966815)|
|test-ubuntu-24.04-cpp-bundled-offline|[](https://github.com/ursacomputing/crossbow/actions/runs/13245822485/job/36971964624)|
|test-ubuntu-24.04-cpp-gcc-13-bundled|[](https://github.com/ursacomputing/crossbow/actions/runs/13245822993/job/36971967224)|
|test-ubuntu-24.04-cpp-gcc-14|[](https://github.com/ursacomputing/crossbow/actions/runs/13245823231/job/36971969335)|
|test-ubuntu-24.04-cpp-minimal-with-formats|[](https://github.com/ursacomputing/crossbow/actions/runs/13245823195/job/36971968796)|
|test-ubuntu-24.04-cpp-thread-sanitizer|[](https://github.com/ursacomputing/crossbow/actions/runs/13245823628/job/36971970357)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on PR #45268:
URL: https://github.com/apache/arrow/pull/45268#issuecomment-2648622206
@github-actions crossbow submit -g cpp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on PR #45268:
URL: https://github.com/apache/arrow/pull/45268#issuecomment-2652645699
Thank you @pitrou for reviewing. And I take this as an approval to merge so I'm doing it.
Also thank you @wuzhoupei for spotting and fixing this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on PR #45268:
URL: https://github.com/apache/arrow/pull/45268#issuecomment-2647678958
Oops, seems there is a race. Looking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #45268:
URL: https://github.com/apache/arrow/pull/45268#discussion_r1942803312
##########
cpp/src/arrow/acero/task_util.cc:
##########
@@ -369,17 +369,25 @@ Status TaskSchedulerImpl::ScheduleMore(size_t thread_id, int num_tasks_finished)
int group_id = tasks[i].first;
int64_t task_id = tasks[i].second;
RETURN_NOT_OK(schedule_impl_([this, group_id, task_id](size_t thread_id) -> Status {
- RETURN_NOT_OK(ScheduleMore(thread_id, 1));
-
bool task_group_finished = false;
- RETURN_NOT_OK(ExecuteTask(thread_id, group_id, task_id, &task_group_finished));
+ // PostExecuteTask must be called later if any error ocurres during task execution
+ // (including ScheduleMore), so we preserve the status.
+ auto status = [&]() {
Review Comment:
Why the lambda? This does not seem to be needed.
##########
cpp/src/arrow/acero/task_util_test.cc:
##########
@@ -231,5 +231,96 @@ TEST(TaskScheduler, StressTwo) {
}
}
+TEST(TaskScheduler, AbortContOnTaskErrorSerial) {
+ constexpr int kNumTasks = 16;
+
+ auto scheduler = TaskScheduler::Make();
+ auto task = [&](std::size_t, int64_t task_id) {
+ if (task_id == kNumTasks / 2) {
+ return Status::Invalid("Task failed");
+ }
+ return Status::OK();
+ };
+
+ int task_group =
+ scheduler->RegisterTaskGroup(task, [](std::size_t) { return Status::OK(); });
+ scheduler->RegisterEnd();
+
+ ASSERT_OK(scheduler->StartScheduling(
+ 0, [](TaskScheduler::TaskGroupContinuationImpl) { return Status::OK(); }, 1, true));
Review Comment:
/Can we annotate non-obvious parameters with their names?
```suggestion
ASSERT_OK(scheduler->StartScheduling(
/*xxx=*/0, [](TaskScheduler::TaskGroupContinuationImpl) { return Status::OK(); },
/*xxx=*/1, /*xxx=*/true));
```
##########
cpp/src/arrow/acero/task_util_test.cc:
##########
@@ -231,5 +231,96 @@ TEST(TaskScheduler, StressTwo) {
}
}
+TEST(TaskScheduler, AbortContOnTaskErrorSerial) {
+ constexpr int kNumTasks = 16;
+
+ auto scheduler = TaskScheduler::Make();
+ auto task = [&](std::size_t, int64_t task_id) {
+ if (task_id == kNumTasks / 2) {
+ return Status::Invalid("Task failed");
+ }
+ return Status::OK();
+ };
+
+ int task_group =
+ scheduler->RegisterTaskGroup(task, [](std::size_t) { return Status::OK(); });
+ scheduler->RegisterEnd();
+
+ ASSERT_OK(scheduler->StartScheduling(
+ 0, [](TaskScheduler::TaskGroupContinuationImpl) { return Status::OK(); }, 1, true));
+ ASSERT_RAISES_WITH_MESSAGE(Invalid, "Invalid: Task failed",
+ scheduler->StartTaskGroup(0, task_group, kNumTasks));
+
+ bool abort_cont_called = false;
+ auto abort_cont = [&]() {
+ ASSERT_FALSE(abort_cont_called);
+ abort_cont_called = true;
+ };
+
+ scheduler->Abort(abort_cont);
+
+ ASSERT_TRUE(abort_cont_called);
+}
+
+TEST(TaskScheduler, AbortContOnTaskErrorParallel) {
+#ifndef ARROW_ENABLE_THREADING
+ GTEST_SKIP() << "Test requires threading support";
+#endif
+ constexpr int kNumThreads = 16;
+
+ ThreadIndexer thread_indexer;
+ int num_threads = std::min(static_cast<int>(thread_indexer.Capacity()), kNumThreads);
+ ASSERT_OK_AND_ASSIGN(std::shared_ptr<ThreadPool> thread_pool,
+ MakePrimedThreadPool(num_threads));
+ TaskScheduler::ScheduleImpl schedule =
+ [&](TaskScheduler::TaskGroupContinuationImpl task) {
+ return thread_pool->Spawn([&, task] {
+ std::size_t thread_id = thread_indexer();
+ auto status = task(thread_id);
+ ASSERT_TRUE(status.ok() || status.IsInvalid() || status.IsCancelled());
+ });
+ };
+
+ for (int num_tasks :
+ {2, num_threads - 1, num_threads, num_threads + 1, 2 * num_threads}) {
+ ARROW_SCOPED_TRACE("num_tasks = ", num_tasks);
+ for (int num_concurrent_tasks :
+ {1, num_tasks - 1, num_tasks, num_tasks + 1, 2 * num_tasks}) {
+ ARROW_SCOPED_TRACE("num_concurrent_tasks = ", num_concurrent_tasks);
+ for (int aborting_task_id = 0; aborting_task_id < num_tasks; ++aborting_task_id) {
+ ARROW_SCOPED_TRACE("aborting_task_id = ", aborting_task_id);
+ auto scheduler = TaskScheduler::Make();
+
+ bool abort_cont_called = false;
+ auto abort_cont = [&]() {
+ ASSERT_FALSE(abort_cont_called);
+ abort_cont_called = true;
+ };
+
+ auto task = [&](std::size_t, int64_t task_id) {
+ if (task_id == aborting_task_id) {
+ scheduler->Abort(abort_cont);
+ }
+ if (task_id % 2 == 0) {
+ return Status::Invalid("Task failed");
+ }
+ return Status::OK();
+ };
+
+ int task_group =
+ scheduler->RegisterTaskGroup(task, [](std::size_t) { return Status::OK(); });
+ scheduler->RegisterEnd();
+
+ ASSERT_OK(scheduler->StartScheduling(0, schedule, num_concurrent_tasks, false));
+ ASSERT_OK(scheduler->StartTaskGroup(0, task_group, num_tasks));
Review Comment:
Hmm, so the `Status::Invalid` that's returned above is never reported?
##########
cpp/src/arrow/acero/task_util_test.cc:
##########
@@ -231,5 +231,96 @@ TEST(TaskScheduler, StressTwo) {
}
}
+TEST(TaskScheduler, AbortContOnTaskErrorSerial) {
+ constexpr int kNumTasks = 16;
+
+ auto scheduler = TaskScheduler::Make();
+ auto task = [&](std::size_t, int64_t task_id) {
+ if (task_id == kNumTasks / 2) {
+ return Status::Invalid("Task failed");
+ }
+ return Status::OK();
+ };
+
+ int task_group =
+ scheduler->RegisterTaskGroup(task, [](std::size_t) { return Status::OK(); });
+ scheduler->RegisterEnd();
+
+ ASSERT_OK(scheduler->StartScheduling(
+ 0, [](TaskScheduler::TaskGroupContinuationImpl) { return Status::OK(); }, 1, true));
+ ASSERT_RAISES_WITH_MESSAGE(Invalid, "Invalid: Task failed",
+ scheduler->StartTaskGroup(0, task_group, kNumTasks));
+
+ bool abort_cont_called = false;
+ auto abort_cont = [&]() {
+ ASSERT_FALSE(abort_cont_called);
+ abort_cont_called = true;
+ };
+
+ scheduler->Abort(abort_cont);
+
+ ASSERT_TRUE(abort_cont_called);
Review Comment:
```suggestion
int num_abort_calls = 0;
auto abort_cont = [&]() {
++num_abort_calls;
};
scheduler->Abort(abort_cont);
ASSERT_EQ(num_abort_calls, 1);
```
##########
cpp/src/arrow/acero/task_util_test.cc:
##########
@@ -231,5 +231,96 @@ TEST(TaskScheduler, StressTwo) {
}
}
+TEST(TaskScheduler, AbortContOnTaskErrorSerial) {
+ constexpr int kNumTasks = 16;
+
+ auto scheduler = TaskScheduler::Make();
+ auto task = [&](std::size_t, int64_t task_id) {
+ if (task_id == kNumTasks / 2) {
+ return Status::Invalid("Task failed");
+ }
+ return Status::OK();
+ };
+
+ int task_group =
+ scheduler->RegisterTaskGroup(task, [](std::size_t) { return Status::OK(); });
+ scheduler->RegisterEnd();
+
+ ASSERT_OK(scheduler->StartScheduling(
+ 0, [](TaskScheduler::TaskGroupContinuationImpl) { return Status::OK(); }, 1, true));
+ ASSERT_RAISES_WITH_MESSAGE(Invalid, "Invalid: Task failed",
+ scheduler->StartTaskGroup(0, task_group, kNumTasks));
+
+ bool abort_cont_called = false;
+ auto abort_cont = [&]() {
+ ASSERT_FALSE(abort_cont_called);
+ abort_cont_called = true;
+ };
+
+ scheduler->Abort(abort_cont);
+
+ ASSERT_TRUE(abort_cont_called);
+}
+
+TEST(TaskScheduler, AbortContOnTaskErrorParallel) {
+#ifndef ARROW_ENABLE_THREADING
+ GTEST_SKIP() << "Test requires threading support";
+#endif
+ constexpr int kNumThreads = 16;
+
+ ThreadIndexer thread_indexer;
+ int num_threads = std::min(static_cast<int>(thread_indexer.Capacity()), kNumThreads);
+ ASSERT_OK_AND_ASSIGN(std::shared_ptr<ThreadPool> thread_pool,
+ MakePrimedThreadPool(num_threads));
+ TaskScheduler::ScheduleImpl schedule =
+ [&](TaskScheduler::TaskGroupContinuationImpl task) {
+ return thread_pool->Spawn([&, task] {
+ std::size_t thread_id = thread_indexer();
+ auto status = task(thread_id);
+ ASSERT_TRUE(status.ok() || status.IsInvalid() || status.IsCancelled());
+ });
+ };
+
+ for (int num_tasks :
+ {2, num_threads - 1, num_threads, num_threads + 1, 2 * num_threads}) {
+ ARROW_SCOPED_TRACE("num_tasks = ", num_tasks);
+ for (int num_concurrent_tasks :
+ {1, num_tasks - 1, num_tasks, num_tasks + 1, 2 * num_tasks}) {
+ ARROW_SCOPED_TRACE("num_concurrent_tasks = ", num_concurrent_tasks);
+ for (int aborting_task_id = 0; aborting_task_id < num_tasks; ++aborting_task_id) {
+ ARROW_SCOPED_TRACE("aborting_task_id = ", aborting_task_id);
+ auto scheduler = TaskScheduler::Make();
+
+ bool abort_cont_called = false;
Review Comment:
Perhaps use a counter as suggested above as well?
##########
cpp/src/arrow/acero/task_util_test.cc:
##########
@@ -231,5 +231,96 @@ TEST(TaskScheduler, StressTwo) {
}
}
+TEST(TaskScheduler, AbortContOnTaskErrorSerial) {
+ constexpr int kNumTasks = 16;
+
+ auto scheduler = TaskScheduler::Make();
+ auto task = [&](std::size_t, int64_t task_id) {
+ if (task_id == kNumTasks / 2) {
+ return Status::Invalid("Task failed");
+ }
+ return Status::OK();
+ };
+
+ int task_group =
+ scheduler->RegisterTaskGroup(task, [](std::size_t) { return Status::OK(); });
+ scheduler->RegisterEnd();
+
+ ASSERT_OK(scheduler->StartScheduling(
+ 0, [](TaskScheduler::TaskGroupContinuationImpl) { return Status::OK(); }, 1, true));
+ ASSERT_RAISES_WITH_MESSAGE(Invalid, "Invalid: Task failed",
+ scheduler->StartTaskGroup(0, task_group, kNumTasks));
Review Comment:
```suggestion
ASSERT_RAISES_WITH_MESSAGE(Invalid, "Invalid: Task failed",
scheduler->StartTaskGroup(/*xxx=*/0, task_group, kNumTasks));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on PR #45268:
URL: https://github.com/apache/arrow/pull/45268#issuecomment-2647086749
> Also, so If i'm right, can we also encapsulate failed task `PostExecuteTask` also in `TaskSchedulerImpl::ExecuteTask`? Or this should handled by outside caller
This is a great question! Actually this is what I intended to do in the very beginning of my attempt to enhance this PR ([`1db3f65` (#45268)](https://github.com/apache/arrow/pull/45268/commits/1db3f653b4656c2d64b93fdff4fccb9047231fa2)). Unfortunately there is a caveat:
Note the ultimate task invoking is https://github.com/apache/arrow/pull/45268/files#diff-0fd20b6ddfd91e0e5417e0c841ab3becd4ac82612647b736be3cf404aa4e2b86R371-R378, and the problem lies in the recursive call to `ScheduleMore`, which will return `Cancelled` if the scheduler has been aborted and `RETURN_NOT_OK` will shortcut all the subsequent code (including the next `ExecuteTask`). In this case, the `PostExecuteTask` inside the `ExecuteTask` will lose the chance to execute and so does the `OnTaskGroupFinished`.
If you are interested, you can change the code this way and try my UT - it will fail :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45266: [C++][Acero] Fix the running tasks count of Scheduler when get error tasks in multi-threads [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on PR #45268:
URL: https://github.com/apache/arrow/pull/45268#issuecomment-2647007746
Good for review again. @pitrou @mapleFU , mind to take a look? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org