You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@livy.apache.org by "lakunma (Jira)" <ji...@apache.org> on 2020/06/19 10:04:00 UTC
[jira] [Comment Edited] (LIVY-763) FD leak in InteractiveSession
and PythonInterpreter
[ https://issues.apache.org/jira/browse/LIVY-763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17139469#comment-17139469 ]
lakunma edited comment on LIVY-763 at 6/19/20, 10:03 AM:
---------------------------------------------------------
-It is actually 'livy.pyspark.archives' in livy.conf (so not in livy-client.conf, and not livy.rsc.pyspark.archives option)-
Now, i'm suddenly not sure about that. It seems that indeed what is suggested in summary works. But interesting question is why that option is not in the template of livy.conf (but inside livy-client.conf).
was (Author: lakunma):
It is actually 'livy.pyspark.archives' in livy.conf (so not in livy-client.conf, and not livy.rsc.pyspark.archives option)
> FD leak in InteractiveSession and PythonInterpreter
> ---------------------------------------------------
>
> Key: LIVY-763
> URL: https://issues.apache.org/jira/browse/LIVY-763
> Project: Livy
> Issue Type: Bug
> Components: Interpreter, Server
> Affects Versions: 0.6.0, 0.7.0
> Reporter: Dustin Koupal
> Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We recently upgraded from Livy 0.4 to Livy 0.6 and started to run out of FDs (we fire up thousands of InteractiveSessions a day). Looking at the open FDs, they were all DIR handles for `/usr/lib/spark/python/lib`.
> I believe this change was introduced in https://github.com/apache/incubator-livy/commit/1bbefe601641aa4db74dbed1f7faf458b9a70a63#diff-7649a51ad4bddc91b6f1038e06479d41R259-R264
> The behavior without this change and the default config is Livy will open 2 FDs for each InteractiveSession, and they will never be closed (I think because `DirectoryStream` is
> `AutoCloseable` and try-with-resource doesn't work with Scala's `Try`.
> As a workaround, in our production clusters, we have set `livy.rsc.pyspark.archives` to an empty string in `livy.conf`
--
This message was sent by Atlassian Jira
(v8.3.4#803005)