[jira] [Created] (ZOOKEEPER-5073) C client hangs indefinitely when pthread_create fails during initialization

["Wenhai Wang (Jira)" <ji...@apache.org>]

Wenhai Wang created ZOOKEEPER-5073:
--------------------------------------

             Summary: C client hangs indefinitely when pthread_create fails during initialization
                 Key: ZOOKEEPER-5073
                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-5073
             Project: ZooKeeper
          Issue Type: Bug
    Affects Versions: 3.7.0, 3.5.9, 3.10.0
         Environment: OS: Debian10
Architecture: x86_64
Compiler: GCC 8.3.0
ZooKeeper version: 3.10.0
Client mode: multi-threaded
            Reporter: Wenhai Wang
         Attachments: README.md, build.sh, error.log, pthread_create_fail.c, run_fault_test.sh, zk_init_probe.c

The ZooKeeper multi-threaded C client may hang indefinitely when either of the worker threads fails to start during client initialization.
h3. Affected code

The issue is located in:

{{zookeeper-client/zookeeper-client-c/src/mt_adaptor.c}}

The {{start_threads()}} function initializes the startup barrier with:

{{adaptor->threadsToWait = 2;}}

It then creates the IO thread and completion thread:

 

{{rc = pthread_create(&adaptor->io, 0, do_io, zh);
assert("pthread_create() failed for the IO thread" && !rc);

rc = pthread_create(&adaptor->completion, 0, do_completion, zh);
assert("pthread_create() failed for the completion thread" && !rc);

wait_for_others(zh);}}
h3. Problem

When assertions are disabled, for example in a release build compiled with {{{}NDEBUG{}}}, the return values of {{pthread_create()}} are effectively ignored.

If either {{pthread_create()}} call fails, the corresponding worker thread is not created. However, {{threadsToWait}} remains initialized to {{{}2{}}}.

Each successfully created worker thread calls {{notify_thread_ready()}} and decrements {{threadsToWait}} once. The initialization thread calls {{wait_for_others()}} and waits until {{threadsToWait}} becomes zero.

For example, if the IO thread is created successfully but the completion thread fails to start:
 # {{threadsToWait}} is initialized to {{{}2{}}}.

 # The IO thread decrements it to {{{}1{}}}.

 # The completion thread does not exist and therefore cannot perform the second decrement.

 # The IO thread waits in {{{}notify_thread_ready(){}}}.

 # The initialization thread waits in {{{}wait_for_others(){}}}.

 # {{threadsToWait}} can never become zero.

As a result, {{zookeeper_init()}} does not return and the client initialization hangs indefinitely.

When assertions are enabled, the process aborts instead of returning an initialization error, which also means the runtime resource failure is not handled gracefully.
h3. Expected behavior

The C client should detect a non-zero return value from {{{}pthread_create(){}}}, stop and join any worker thread that was already created, clean up the initialized resources, and return an initialization error to the caller.
h3. Actual behavior

When assertions are disabled, the client waits indefinitely on the startup condition variable.

When assertions are enabled, the process terminates through {{{}assert(){}}}.
h3. Reproduction

The issue can be reproduced deterministically by:
 # Building the ZooKeeper C client with {{NDEBUG}} enabled.

 # Injecting an {{EAGAIN}} failure into either the first or second {{pthread_create()}} call.

 # Calling {{{}zookeeper_init(){}}}.

 # Observing that {{zookeeper_init()}} does not return.

A typical blocked thread state is:

 

{{Initialization thread:
pthread_cond_wait
wait_for_others
start_threads
adaptor_init
zookeeper_init

Successfully created worker thread:
pthread_cond_wait
notify_thread_ready
do_io or do_completion}}
h3. Root cause

{{start_threads()}} assumes that both {{pthread_create()}} calls always succeed and uses assertions instead of normal runtime error handling.

The startup barrier count is not adjusted when thread creation fails, and the failure cannot be propagated because {{start_threads()}} currently returns {{{}void{}}}.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)