[jira] [Commented] (CASSJAVA-11) threads lingering after driver shutdown: session close starts thread and doesn't await its stop

["Lukasz Antoniak (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSJAVA-11?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17909590#comment-17909590 ] 

Lukasz Antoniak commented on CASSJAVA-11:
-----------------------------------------

I am not sure of there is a win-win solution. {{GlobalEventExecutor.INSTANCE}} can be used by multiple frameworks, because it is a static instance. This implies that inactivity may not happen at all. Moreover, you cannot put await method inside {{DefaultNettOptions.onClose()}} method, because this will make function always blocking. You cannot also chain the await method in next future, because on which executor will it run? It cannot run on {{GlobalEventExecutor.INSTANCE}} and wait until the same executor has no tasks.

AFAIK we can consider switching from {{GlobalEventExecutor.INSTANCE}} to {{ImmediateEventExecutor.INSTANCE}} and make {{DefaultNettOptions.onClose()}} effectively synchronous, or run it as part of JVM's fork-join pool. Fundamental question is on which executor will you run all cleanup methods? Alternatively, we can create new executor that will always create new private thread to execute cleanup actions, and after this single task is completed, pool will be also closed.

> threads lingering after driver shutdown: session close starts thread and doesn't await its stop
> -----------------------------------------------------------------------------------------------
>
>                 Key: CASSJAVA-11
>                 URL: https://issues.apache.org/jira/browse/CASSJAVA-11
>             Project: Apache Cassandra Java driver
>          Issue Type: Bug
>            Reporter: Thomas Klambauer
>            Priority: Normal
>
> We are checking remaining/lingering threads during shutdown.
> we noticed some with naming pattern/thread factory: ""globalEventExecutor-1-2" Id=146 TIMED_WAITING"
> this one seems to be created during shutdown / session close and not awaited/shut down:
> {noformat}
> addTask:156, GlobalEventExecutor (io.netty.util.concurrent)
> execute0:225, GlobalEventExecutor (io.netty.util.concurrent)
> execute:221, GlobalEventExecutor (io.netty.util.concurrent)
> onClose:188, DefaultNettyOptions (com.datastax.oss.driver.internal.core.context)
> onChildrenClosed:589, DefaultSession$SingleThreaded (com.datastax.oss.driver.internal.core.session)
> lambda$close$9:552, DefaultSession$SingleThreaded (com.datastax.oss.driver.internal.core.session)
> run:-1, 860270832 (com.datastax.oss.driver.internal.core.session.DefaultSession$SingleThreaded$$Lambda$9508)
> tryFire$$$capture:783, CompletableFuture$UniRun (java.util.concurrent)
> tryFire:-1, CompletableFuture$UniRun (java.util.concurrent)
>  - Async stack trace
> addTask:-1, SingleThreadEventExecutor (io.netty.util.concurrent)
> execute:836, SingleThreadEventExecutor (io.netty.util.concurrent)
> execute0:827, SingleThreadEventExecutor (io.netty.util.concurrent)
> execute:817, SingleThreadEventExecutor (io.netty.util.concurrent)
> claim:568, CompletableFuture$UniCompletion (java.util.concurrent)
> tryFire$$$capture:780, CompletableFuture$UniRun (java.util.concurrent)
> tryFire:-1, CompletableFuture$UniRun (java.util.concurrent)
>  - Async stack trace
> <init>:767, CompletableFuture$UniRun (java.util.concurrent)
> uniRunStage:801, CompletableFuture (java.util.concurrent)
> thenRunAsync:2136, CompletableFuture (java.util.concurrent)
> thenRunAsync:143, CompletableFuture (java.util.concurrent)
> whenAllDone:75, CompletableFutures (com.datastax.oss.driver.internal.core.util.concurrent)
> close:551, DefaultSession$SingleThreaded (com.datastax.oss.driver.internal.core.session)
> access$1000:300, DefaultSession$SingleThreaded (com.datastax.oss.driver.internal.core.session)
> lambda$closeAsync$1:272, DefaultSession (com.datastax.oss.driver.internal.core.session)
> runTask:98, PromiseTask (io.netty.util.concurrent)
> run:106, PromiseTask (io.netty.util.concurrent)
> runTask$$$capture:174, AbstractEventExecutor (io.netty.util.concurrent)
> runTask:-1, AbstractEventExecutor (io.netty.util.concurrent)
>  - Async stack trace
> addTask:-1, SingleThreadEventExecutor (io.netty.util.concurrent)
> execute:836, SingleThreadEventExecutor (io.netty.util.concurrent)
> execute0:827, SingleThreadEventExecutor (io.netty.util.concurrent)
> execute:817, SingleThreadEventExecutor (io.netty.util.concurrent)
> submit:118, AbstractExecutorService (java.util.concurrent)
> submit:118, AbstractEventExecutor (io.netty.util.concurrent)
> on:57, RunOrSchedule (com.datastax.oss.driver.internal.core.util.concurrent)
> closeSafely:286, DefaultSession (com.datastax.oss.driver.internal.core.session)
> closeAsync:272, DefaultSession (com.datastax.oss.driver.internal.core.session)
> close:76, AsyncAutoCloseable (com.datastax.oss.driver.api.core)
> -- custom shutdown code
> run:829, Thread (java.lang)
> {noformat}
> the initial close here is called on com.datastax.oss.driver.api.core.CqlSession.
> netty framework suggests to call
> io.netty.util.concurrent.GlobalEventExecutor#awaitInactivity
> during shutdown to await event thread stopping
> (slightly related issue in netty: [https://github.com/netty/netty/issues/2084] )
> suggestion to add maybe GlobalEventExecutor.INSTANCE.awaitInactivity with some timeout during close around here:
> [https://github.com/apache/cassandra-java-driver/blob/4.x/core/src/main/java/com/datastax/oss/driver/internal/core/context/DefaultNettyOptions.java#L199]
> noting that this might slow down closing for up to 2 seconds if the netty issue comment is correct.
> this is on latest datastax java driver version: 4.17,



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org