You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@livy.apache.org by "Amit Kumar (Jira)" <ji...@apache.org> on 2020/09/25 06:28:00 UTC

[jira] [Commented] (LIVY-763) FD leak in InteractiveSession and PythonInterpreter

    [ https://issues.apache.org/jira/browse/LIVY-763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17201942#comment-17201942 ] 

Amit Kumar commented on LIVY-763:
---------------------------------

I am using EMR 5.30.0 which uses Livy 0.7.0 and still see this issue. I used the workaround and looks like it is working.

> FD leak in InteractiveSession and PythonInterpreter
> ---------------------------------------------------
>
>                 Key: LIVY-763
>                 URL: https://issues.apache.org/jira/browse/LIVY-763
>             Project: Livy
>          Issue Type: Bug
>          Components: Interpreter, Server
>    Affects Versions: 0.6.0, 0.7.0
>            Reporter: Dustin Koupal
>            Priority: Minor
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> We recently upgraded from Livy 0.4 to Livy 0.6 and started to run out of FDs (we fire up thousands of InteractiveSessions a day). Looking at the open FDs, they were all DIR handles for `/usr/lib/spark/python/lib`.
> I believe this change was introduced in https://github.com/apache/incubator-livy/commit/1bbefe601641aa4db74dbed1f7faf458b9a70a63#diff-7649a51ad4bddc91b6f1038e06479d41R259-R264
> The behavior without this change and the default config is Livy will open 2 FDs for each InteractiveSession, and they will never be closed (I think because `DirectoryStream` is 
> `AutoCloseable` and try-with-resource doesn't work with Scala's `Try`.
> As a workaround, in our production clusters, we have set `livy.rsc.pyspark.archives` to an empty string in `livy.conf`



--
This message was sent by Atlassian Jira
(v8.3.4#803005)