You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "mrdrivingduck (via GitHub)" <gi...@apache.org> on 2026/08/21 04:34:02 UTC

[PR] fix(executor): support destruction from worker threads [paimon-cpp]

mrdrivingduck opened a new pull request, #233:
URL: https://github.com/apache/paimon-cpp/pull/233

   ## What does this PR do?
   
   While testing the OSS filesystem asynchronous read path, we found a lifecycle issue in `DefaultExecutor`. The issue is not OSS-specific: S3 asynchronous reads use the same ownership pattern and can trigger it as well.
   
   The sequence is:
   
   1. An asynchronous object-store request submits a task that captures a `shared_ptr` to its client.
   2. The caller releases its client reference.
   3. A worker completes the task and releases the final client reference.
   4. The client and its executor are then destroyed on that worker thread.
   5. Executor shutdown tries to `join()` the current worker thread.
   
   Without this fix, joining the current thread throws `std::system_error` with `Resource deadlock avoided`. Since this happens during destruction, it can terminate the process.
   
   This PR separates the executor's shared scheduling state from the executor object. Workers retain the shared state rather than accessing the executor through `this`. When destruction happens on a worker, shutdown stops the shared state, joins the other workers, and detaches the current one. The current worker then finishes its task and exits normally.
   
   The PR adds a generic executor regression test and an S3 asynchronous range-read test covering this lifecycle.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(executor): support destruction from worker threads [paimon-cpp]

Posted by "lucasfang (via GitHub)" <gi...@apache.org>.
lucasfang commented on code in PR #233:
URL: https://github.com/apache/paimon-cpp/pull/233#discussion_r3860931749


##########
src/paimon/common/executor/executor.cpp:
##########
@@ -66,21 +69,22 @@ uint32_t DefaultExecutor::GetThreadNum() const {
 
 void DefaultExecutor::ShutdownInternal(bool wait_for_pending_tasks) {
     {
-        std::unique_lock<std::mutex> lock(queue_mutex_);
-        if (stop_) {
-            return;
-        }
-        stop_ = true;
+        std::unique_lock<std::mutex> lock(state_->mutex);
+        state_->stop = true;

Review Comment:
   And please add an extra test



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(executor): support destruction from worker threads [paimon-cpp]

Posted by "lucasfang (via GitHub)" <gi...@apache.org>.
lucasfang commented on PR #233:
URL: https://github.com/apache/paimon-cpp/pull/233#issuecomment-5449898316

   +1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(executor): support destruction from worker threads [paimon-cpp]

Posted by "lucasfang (via GitHub)" <gi...@apache.org>.
lucasfang commented on code in PR #233:
URL: https://github.com/apache/paimon-cpp/pull/233#discussion_r3860925650


##########
src/paimon/common/executor/executor.cpp:
##########
@@ -66,21 +69,22 @@ uint32_t DefaultExecutor::GetThreadNum() const {
 
 void DefaultExecutor::ShutdownInternal(bool wait_for_pending_tasks) {
     {
-        std::unique_lock<std::mutex> lock(queue_mutex_);
-        if (stop_) {
-            return;
-        }
-        stop_ = true;
+        std::unique_lock<std::mutex> lock(state_->mutex);
+        state_->stop = true;

Review Comment:
   ShutdownInternal() dropped the early return that previously made shutdown idempotent:
   if (stop_) {
       return;
   }
   Without it, two threads can both observe worker.joinable() == true and call join() on the same std::thread. That is a data race on the thread object, and in practice the second pthread_join() blocks forever on an already-reaped thread id.I reproduced this against the real build by adding a temporary probe test (4-thread executor, two threads calling ShutdownNow() concurrently, 50 iterations).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(executor): support destruction from worker threads [paimon-cpp]

Posted by "lucasfang (via GitHub)" <gi...@apache.org>.
lucasfang commented on PR #233:
URL: https://github.com/apache/paimon-cpp/pull/233#issuecomment-5422616399

   Thanks for your contribution. The core fix is correct and effective. Moving the scheduling state into a shared_ptr<State> so that workers never touch this, combined with detach() when the thread being shut down is the current one, is the right approach — both parts are necessary, since detaching without relocating the state would leave the surviving worker touching freed members.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(executor): support destruction from worker threads [paimon-cpp]

Posted by "mrdrivingduck (via GitHub)" <gi...@apache.org>.
mrdrivingduck commented on code in PR #233:
URL: https://github.com/apache/paimon-cpp/pull/233#discussion_r3878183113


##########
src/paimon/common/executor/executor.cpp:
##########
@@ -66,21 +69,22 @@ uint32_t DefaultExecutor::GetThreadNum() const {
 
 void DefaultExecutor::ShutdownInternal(bool wait_for_pending_tasks) {
     {
-        std::unique_lock<std::mutex> lock(queue_mutex_);
-        if (stop_) {
-            return;
-        }
-        stop_ = true;
+        std::unique_lock<std::mutex> lock(state_->mutex);
+        state_->stop = true;

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(executor): support destruction from worker threads [paimon-cpp]

Posted by "lxy-9602 (via GitHub)" <gi...@apache.org>.
lxy-9602 merged PR #233:
URL: https://github.com/apache/paimon-cpp/pull/233


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org