[jira] [Comment Edited] (LIVY-1003) Interactive session - Setting large value of rsc.server.connect.timeout blocks other tasks

["wangdengshan (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/LIVY-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17856646#comment-17856646 ] 

wangdengshan edited comment on LIVY-1003 at 6/21/24 8:06 AM:
-------------------------------------------------------------

As you know, it will mainly get stuck at the startup of RemoteDriver, which is the pulling up of yarn. On the one hand, the waiting time can be reduced through configuration(reduce the wait time), and at the same time, the use of the thread pool is also adjusted in our case.


was (Author: JIRAUSER305117):
As you know, it will mainly get stuck at the startup of RemoteDriver, which is the pulling up of yarn. On the one hand, the waiting time can be reduced through configuration, and at the same time, the use of the thread pool is also adjusted in our case.

> Interactive session - Setting large value of rsc.server.connect.timeout blocks other tasks 
> -------------------------------------------------------------------------------------------
>
>                 Key: LIVY-1003
>                 URL: https://issues.apache.org/jira/browse/LIVY-1003
>             Project: Livy
>          Issue Type: Bug
>          Components: RSC
>    Affects Versions: 0.8.0
>            Reporter: Tom Chamnongvongse
>            Priority: Minor
>
> Problem:
> Livy is configured to deploy interactive sessions on YARN with `livy.rsc.server.connect.timeout` configured to a high value. Timeout is increased to allow more time for Livy session to be in YARN `ACCEPTED` state to prevent Livy server from killing the YARN app within the default timeout of 90 seconds.
> Until the app is in YARN `RUNNING` state, it takes up a thread in Scala's global execution context - I think due to https://github.com/apache/incubator-livy/blob/v0.8.0-incubating/server/src/main/scala/org/apache/livy/server/interactive/InteractiveSession.scala#L474. Creating too many of these sessions that are stuck in `ACCEPTED` state causes other tasks that use that global execution context to be queued up.
> How to reproduce:
> 1. Set `livy.rsc.server.connect.timeout` to something high like 24h.
> 2. Create enough interactive livy sessions in YARN so that they are queued in ACCEPTED state. The number of sessions that are stuck in ACCEPTED state should be equal to global execution context [thread pool size|https://docs.scala-lang.org/overviews/core/futures.html#the-global-execution-context] (Runtime.availableProcessors)
> 3. Try to delete a session using DELETE /sessions/{sessionId} and it should hang until one of the sessions is no longer stuck in ACCEPTED state.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)