You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@livy.apache.org by Lauren Spiegel <la...@scality.com> on 2017/12/01 00:55:02 UTC
Session Memory Management
Hi Livyers,
When I have an active session for a couple hours, the memory keeps growing
and I have to kill it. What do people do to manage ever-growing memory? Are
there any plans to limit the memory growth so that sessions can be long
living?
Thank you
--
<http://storage.scality.com/report-gartner-magic-quadrant-storage.html?utm_source=ScalityEmail&utm_medium=Email&utm_campaign=MagicQuadrant>
Re: Session Memory Management
Posted by Lauren Spiegel <la...@scality.com>.
Done -- https://issues.apache.org/jira/browse/LIVY-424
Thank you
On Thu, Nov 30, 2017 at 7:51 PM, Saisai Shao <sa...@gmail.com> wrote:
> Hi Lauren, we don't get such feedbacks regarding to this issue, can you
> please open a Livy JIRA and put up all your observations and snapshots to
> that JIRA, we can track and discuss this on the JIRA.
>
> Thanks
> Jerry
>
> 2017-12-01 11:43 GMT+08:00 Lauren Spiegel <la...@scality.com>:
>
>> Hi,
>>
>> Thanks for the response. I'm using Spark 2.1.1 with scala 2.11. We
>> noticed that memory would become an issue after an hour or so. See below
>> analysis from Eclipse. Are other people not running into this with long
>> living sessions?
>>
>>
>> Eclipse Memory Analyzer Tool shows:
>>
>>
>> [image: Inline image 1]
>>
>> Leak suspects (2):
>>
>> [image: Inline image 3][image: Inline image 4]
>>
>>
>>
>> Common root paths:
>>
>> [image: Inline image 2]
>>
>>
>>
>>
>>
>>
>>
>>
>> On Thu, Nov 30, 2017 at 7:33 PM, Saisai Shao <sa...@gmail.com>
>> wrote:
>>
>>> Hi Lauren,
>>>
>>> Thanks for the feedbacks. How do you identify this REPL memory issue? If
>>> it is a problem of Scala REPL, then I think we don't have pretty much
>>> solutions for it. Do you have any thought on it?
>>>
>>> BTW, which version of Spark are you using?
>>>
>>> @Jeff, if it is a problem of Scala REPL, then I believe Zeppelin also
>>> suffers from such issue, did you see any report from Zeppelin community?
>>>
>>> Thanks
>>> Jerry
>>>
>>> 2017-12-01 11:17 GMT+08:00 Lauren Spiegel <la...@scality.com>:
>>>
>>>> The spark driver. I think it is related to this since a session is a
>>>> spark shell/scala REPL: https://issues.scala-lang.org/browse/SI-4331
>>>>
>>>>
>>>> On Thu, Nov 30, 2017 at 6:11 PM, Jeff Zhang <zj...@gmail.com> wrote:
>>>>
>>>>>
>>>>> Which component's memory keeps growing ? livy server or the spark
>>>>> driver ?
>>>>>
>>>>>
>>>>> Lauren Spiegel <la...@scality.com>于2017年12月1日周五 上午8:55写道：
>>>>>
>>>>>> Hi Livyers,
>>>>>>
>>>>>> When I have an active session for a couple hours, the memory keeps
>>>>>> growing and I have to kill it. What do people do to manage ever-growing
>>>>>> memory? Are there any plans to limit the memory growth so that sessions can
>>>>>> be long living?
>>>>>>
>>>>>> Thank you
>>>>>>
>>>>>>
>>>>>> <http://storage.scality.com/report-gartner-magic-quadrant-storage.html?utm_source=ScalityEmail&utm_medium=Email&utm_campaign=MagicQuadrant>
>>>>>
>>>>>
>>>>
>>>>
>>>> <http://storage.scality.com/report-gartner-magic-quadrant-storage.html?utm_source=ScalityEmail&utm_medium=Email&utm_campaign=MagicQuadrant>
>>>
>>>
>>>
>>
>>
>> <http://storage.scality.com/report-gartner-magic-quadrant-storage.html?utm_source=ScalityEmail&utm_medium=Email&utm_campaign=MagicQuadrant>
>
>
>
--
<http://storage.scality.com/report-gartner-magic-quadrant-storage.html?utm_source=ScalityEmail&utm_medium=Email&utm_campaign=MagicQuadrant>
Re: Session Memory Management
Posted by Saisai Shao <sa...@gmail.com>.
Hi Lauren, we don't get such feedbacks regarding to this issue, can you
please open a Livy JIRA and put up all your observations and snapshots to
that JIRA, we can track and discuss this on the JIRA.
Thanks
Jerry
2017-12-01 11:43 GMT+08:00 Lauren Spiegel <la...@scality.com>:
> Hi,
>
> Thanks for the response. I'm using Spark 2.1.1 with scala 2.11. We noticed
> that memory would become an issue after an hour or so. See below analysis
> from Eclipse. Are other people not running into this with long living
> sessions?
>
>
> Eclipse Memory Analyzer Tool shows:
>
>
> [image: Inline image 1]
>
> Leak suspects (2):
>
> [image: Inline image 3][image: Inline image 4]
>
>
>
> Common root paths:
>
> [image: Inline image 2]
>
>
>
>
>
>
>
>
> On Thu, Nov 30, 2017 at 7:33 PM, Saisai Shao <sa...@gmail.com>
> wrote:
>
>> Hi Lauren,
>>
>> Thanks for the feedbacks. How do you identify this REPL memory issue? If
>> it is a problem of Scala REPL, then I think we don't have pretty much
>> solutions for it. Do you have any thought on it?
>>
>> BTW, which version of Spark are you using?
>>
>> @Jeff, if it is a problem of Scala REPL, then I believe Zeppelin also
>> suffers from such issue, did you see any report from Zeppelin community?
>>
>> Thanks
>> Jerry
>>
>> 2017-12-01 11:17 GMT+08:00 Lauren Spiegel <la...@scality.com>:
>>
>>> The spark driver. I think it is related to this since a session is a
>>> spark shell/scala REPL: https://issues.scala-lang.org/browse/SI-4331
>>>
>>>
>>> On Thu, Nov 30, 2017 at 6:11 PM, Jeff Zhang <zj...@gmail.com> wrote:
>>>
>>>>
>>>> Which component's memory keeps growing ? livy server or the spark
>>>> driver ?
>>>>
>>>>
>>>> Lauren Spiegel <la...@scality.com>于2017年12月1日周五 上午8:55写道：
>>>>
>>>>> Hi Livyers,
>>>>>
>>>>> When I have an active session for a couple hours, the memory keeps
>>>>> growing and I have to kill it. What do people do to manage ever-growing
>>>>> memory? Are there any plans to limit the memory growth so that sessions can
>>>>> be long living?
>>>>>
>>>>> Thank you
>>>>>
>>>>>
>>>>> <http://storage.scality.com/report-gartner-magic-quadrant-storage.html?utm_source=ScalityEmail&utm_medium=Email&utm_campaign=MagicQuadrant>
>>>>
>>>>
>>>
>>>
>>> <http://storage.scality.com/report-gartner-magic-quadrant-storage.html?utm_source=ScalityEmail&utm_medium=Email&utm_campaign=MagicQuadrant>
>>
>>
>>
>
>
> <http://storage.scality.com/report-gartner-magic-quadrant-storage.html?utm_source=ScalityEmail&utm_medium=Email&utm_campaign=MagicQuadrant>
Re: Session Memory Management
Posted by Lauren Spiegel <la...@scality.com>.
Hi,
Thanks for the response. I'm using Spark 2.1.1 with scala 2.11. We noticed
that memory would become an issue after an hour or so. See below analysis
from Eclipse. Are other people not running into this with long living
sessions?
Eclipse Memory Analyzer Tool shows:
[image: Inline image 1]
Leak suspects (2):
[image: Inline image 3][image: Inline image 4]
Common root paths:
[image: Inline image 2]
On Thu, Nov 30, 2017 at 7:33 PM, Saisai Shao <sa...@gmail.com> wrote:
> Hi Lauren,
>
> Thanks for the feedbacks. How do you identify this REPL memory issue? If
> it is a problem of Scala REPL, then I think we don't have pretty much
> solutions for it. Do you have any thought on it?
>
> BTW, which version of Spark are you using?
>
> @Jeff, if it is a problem of Scala REPL, then I believe Zeppelin also
> suffers from such issue, did you see any report from Zeppelin community?
>
> Thanks
> Jerry
>
> 2017-12-01 11:17 GMT+08:00 Lauren Spiegel <la...@scality.com>:
>
>> The spark driver. I think it is related to this since a session is a
>> spark shell/scala REPL: https://issues.scala-lang.org/browse/SI-4331
>>
>>
>> On Thu, Nov 30, 2017 at 6:11 PM, Jeff Zhang <zj...@gmail.com> wrote:
>>
>>>
>>> Which component's memory keeps growing ? livy server or the spark driver
>>> ?
>>>
>>>
>>> Lauren Spiegel <la...@scality.com>于2017年12月1日周五 上午8:55写道：
>>>
>>>> Hi Livyers,
>>>>
>>>> When I have an active session for a couple hours, the memory keeps
>>>> growing and I have to kill it. What do people do to manage ever-growing
>>>> memory? Are there any plans to limit the memory growth so that sessions can
>>>> be long living?
>>>>
>>>> Thank you
>>>>
>>>>
>>>> <http://storage.scality.com/report-gartner-magic-quadrant-storage.html?utm_source=ScalityEmail&utm_medium=Email&utm_campaign=MagicQuadrant>
>>>
>>>
>>
>>
>> <http://storage.scality.com/report-gartner-magic-quadrant-storage.html?utm_source=ScalityEmail&utm_medium=Email&utm_campaign=MagicQuadrant>
>
>
>
--
<http://storage.scality.com/report-gartner-magic-quadrant-storage.html?utm_source=ScalityEmail&utm_medium=Email&utm_campaign=MagicQuadrant>
Re: Session Memory Management
Posted by Saisai Shao <sa...@gmail.com>.
Hi Lauren,
Thanks for the feedbacks. How do you identify this REPL memory issue? If it
is a problem of Scala REPL, then I think we don't have pretty much
solutions for it. Do you have any thought on it?
BTW, which version of Spark are you using?
@Jeff, if it is a problem of Scala REPL, then I believe Zeppelin also
suffers from such issue, did you see any report from Zeppelin community?
Thanks
Jerry
2017-12-01 11:17 GMT+08:00 Lauren Spiegel <la...@scality.com>:
> The spark driver. I think it is related to this since a session is a spark
> shell/scala REPL: https://issues.scala-lang.org/browse/SI-4331
>
>
> On Thu, Nov 30, 2017 at 6:11 PM, Jeff Zhang <zj...@gmail.com> wrote:
>
>>
>> Which component's memory keeps growing ? livy server or the spark driver ?
>>
>>
>> Lauren Spiegel <la...@scality.com>于2017年12月1日周五 上午8:55写道：
>>
>>> Hi Livyers,
>>>
>>> When I have an active session for a couple hours, the memory keeps
>>> growing and I have to kill it. What do people do to manage ever-growing
>>> memory? Are there any plans to limit the memory growth so that sessions can
>>> be long living?
>>>
>>> Thank you
>>>
>>>
>>> <http://storage.scality.com/report-gartner-magic-quadrant-storage.html?utm_source=ScalityEmail&utm_medium=Email&utm_campaign=MagicQuadrant>
>>
>>
>
>
> <http://storage.scality.com/report-gartner-magic-quadrant-storage.html?utm_source=ScalityEmail&utm_medium=Email&utm_campaign=MagicQuadrant>
Re: Session Memory Management
Posted by Lauren Spiegel <la...@scality.com>.
The spark driver. I think it is related to this since a session is a spark
shell/scala REPL: https://issues.scala-lang.org/browse/SI-4331
On Thu, Nov 30, 2017 at 6:11 PM, Jeff Zhang <zj...@gmail.com> wrote:
>
> Which component's memory keeps growing ? livy server or the spark driver ?
>
>
> Lauren Spiegel <la...@scality.com>于2017年12月1日周五 上午8:55写道：
>
>> Hi Livyers,
>>
>> When I have an active session for a couple hours, the memory keeps
>> growing and I have to kill it. What do people do to manage ever-growing
>> memory? Are there any plans to limit the memory growth so that sessions can
>> be long living?
>>
>> Thank you
>>
>>
>> <http://storage.scality.com/report-gartner-magic-quadrant-storage.html?utm_source=ScalityEmail&utm_medium=Email&utm_campaign=MagicQuadrant>
>
>
--
<http://storage.scality.com/report-gartner-magic-quadrant-storage.html?utm_source=ScalityEmail&utm_medium=Email&utm_campaign=MagicQuadrant>
Re: Session Memory Management
Posted by Jeff Zhang <zj...@gmail.com>.
Which component's memory keeps growing ? livy server or the spark driver ?
Lauren Spiegel <la...@scality.com>于2017年12月1日周五 上午8:55写道：
> Hi Livyers,
>
> When I have an active session for a couple hours, the memory keeps growing
> and I have to kill it. What do people do to manage ever-growing memory? Are
> there any plans to limit the memory growth so that sessions can be long
> living?
>
> Thank you
>
>
> <http://storage.scality.com/report-gartner-magic-quadrant-storage.html?utm_source=ScalityEmail&utm_medium=Email&utm_campaign=MagicQuadrant>