[jira] [Commented] (THRIFT-6392) D client_pool_test occasionally hangs until the lib-d job times out

["Sylwester Lachiewicz (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/THRIFT-6392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18119882#comment-18119882 ] 

Sylwester Lachiewicz commented on THRIFT-6392:
----------------------------------------------

Root cause: {{FastestPoolJob}} in [lib/d/src/thrift/codegen/async_client_pool.d|https://github.com/apache/thrift/blob/master/lib/d/src/thrift/codegen/async_client_pool.d] stops registering completion callbacks at the first child future that has already completed, but a child that failed with an RPC fault does not set the pool's result, so the remaining children are never counted and the pool's future never completes. {{client_pool_test}} hits it in the all-clients-fail case when the 1 ms server answers before the pool is constructed. Fix and a unit test in [PR #3972|https://github.com/apache/thrift/pull/3972].

> D client_pool_test occasionally hangs until the lib-d job times out
> -------------------------------------------------------------------
>
>                 Key: THRIFT-6392
>                 URL: https://issues.apache.org/jira/browse/THRIFT-6392
>             Project: Thrift
>          Issue Type: Bug
>          Components: D - Library
>            Reporter: Sylwester Lachiewicz
>            Priority: Minor
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> {{lib/d/test/client_pool_test}} occasionally never finishes, so the lib-d job runs into its 60-minute limit. It happened twice in a row on [PR #3971|https://github.com/apache/thrift/pull/3971], which does not touch lib/d ([job 1|https://github.com/apache/thrift/actions/runs/36332377379], [job 2|https://github.com/apache/thrift/actions/runs/36337045757/job/108670208530]), while {{make -C lib/d check}} took about 2.5 minutes on the other PRs of the same day and on master.
> In both hung runs all 104 unit tests pass, then each of the six server threads logs, three seconds later:
> {noformat}
> src/thrift/server/simple.d:144: Client died unexpectedly: thrift.transport.base.TTransportException@src/thrift/transport/socket.d(340): Timed out
>   thrift.transport.socket.TSocket.read(ubyte[])
>   thrift.transport.buffered.TBufferedTransport.peek()
>   thrift.server.simple.TSimpleServer.serve(thrift.util.cancellation.TCancellation)
>   client_pool_test.ServerThread.run()
> {noformat}
> and nothing else is printed until the job is cancelled; the runner then kills an orphaned {{client_pool_test}}. A passing run prints none of these messages. So the test's own client side stopped while holding a connection to every server, and the servers' 3-second {{recvTimeout}} is the only timeout involved.
> The client side of the test has no bound anywhere, so any lost reply waits forever:
> * the synchronous clients are {{TSocket}}s without {{recvTimeout}};
> * the asynchronous tests use {{waitGet()}}, and the implicit {{waitGet}} of {{TFuture}}, instead of {{waitGet(Duration)}};
> * {{main}} ignores the result of {{sem.wait(dur!"seconds"(1))}}, so it goes on even if a server is not listening yet.
> What sets it off is not known yet. Bounding those waits would turn the hang into a test failure that names the call, and a step-level {{timeout-minutes}} on "Run make check for d" in build.yml would stop a hang well before the job limit. Related: THRIFT-4155.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)