You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Kohki Nishio <ta...@gmail.com> on 2021/09/17 15:27:17 UTC
Possibly a memory leak issue in Spark
Hello,
I'm seeing possible memory leak behavior in my spark application. According
to MAT, it looks like it's related to ElementTrackingStore ..
[image: Eclipse_Memory_Analyzer.png]
The increase is subtle, so it takes multiple days to actually cause some
impact, but I'm wondering if anybody has any idea about what this is about
... Below is the GC graph, yellow is the level after any GC kicks in.
[image: T2_G1GC_-_Grafana.png]
Thanks
--
Kohki Nishio
Re: Possibly a memory leak issue in Spark
Posted by Kohki Nishio <ta...@gmail.com>.
I'm seeing millions of Stage / Job / Task data
jcmd GC.class_histogram shows
--
6: 7835346 2444627952 org.apache.spark.status.TaskDataWrapper
25: 3765152 180727296 org.apache.spark.status.StageDataWrapper
88: 232255 9290200 org.apache.spark.status.JobDataWrapper
2nd column is the number of instances
-Kokhi
On Wed, Sep 22, 2021 at 11:06 AM Russell Spitzer <ru...@gmail.com>
wrote:
> As Sean said I believe you want to be setting
>
> spark.ui.retainedJobs 1000 How many jobs the Spark UI and status APIs
> remember before garbage collecting. This is a target maximum, and fewer
> elements may be retained in some circumstances. 1.2.0
> spark.ui.retainedStages 1000 How many stages the Spark UI and status APIs
> remember before garbage collecting. This is a target maximum, and fewer
> elements may be retained in some circumstances. 0.9.0
> spark.ui.retainedTasks 100000 How many tasks in one stage the Spark UI
> and status APIs remember before garbage collecting. This is a target
> maximum, and fewer elements may be retained in some circumstances. 2.0.1
>
> To lower numbers. If i remember correctly this is what controls how much
> metadata remains in the driver post task/stage/job competition.
>
> On Sep 22, 2021, at 12:42 PM, Kohki Nishio <ta...@gmail.com> wrote:
>
> I believe I have enough information, raised this
>
> https://issues.apache.org/jira/browse/SPARK-36827
>
> thanks
> -Kohki
>
>
> On Tue, Sep 21, 2021 at 9:30 PM Sean Owen <sr...@gmail.com> wrote:
>
>> No, that's just info Spark retains about finished jobs and tasks, likely.
>> You can limit how much is retained if desired with config.
>>
>> On Tue, Sep 21, 2021, 11:29 PM Kohki Nishio <ta...@gmail.com> wrote:
>>
>>> Just following up, it looks like task / stage / job data are not cleaned
>>> up
>>> --
>>> 6: 7835346 2444627952
>>> org.apache.spark.status.TaskDataWrapper
>>> 25: 3765152 180727296
>>> org.apache.spark.status.StageDataWrapper
>>> 88: 232255 9290200 org.apache.spark.status.JobDataWrapper
>>>
>>> UI is disabled, not sure why we need to have those data ..
>>>
>>> -Kohki
>>>
>>>
>>> On Fri, Sep 17, 2021 at 8:27 AM Kohki Nishio <ta...@gmail.com> wrote:
>>>
>>>> Hello,
>>>> I'm seeing possible memory leak behavior in my spark application.
>>>> According to MAT, it looks like it's related to ElementTrackingStore ..
>>>>
>>>> <Eclipse_Memory_Analyzer.png>
>>>>
>>>> The increase is subtle, so it takes multiple days to actually cause
>>>> some impact, but I'm wondering if anybody has any idea about what this is
>>>> about ... Below is the GC graph, yellow is the level after any GC kicks in.
>>>>
>>>> <T2_G1GC_-_Grafana.png>
>>>>
>>>> Thanks
>>>> --
>>>> Kohki Nishio
>>>>
>>>
>>>
>>> --
>>> Kohki Nishio
>>>
>>
>
> --
> Kohki Nishio
>
>
>
--
Kohki Nishio
Re: Possibly a memory leak issue in Spark
Posted by Russell Spitzer <ru...@gmail.com>.
As Sean said I believe you want to be setting
spark.ui.retainedJobs 1000 How many jobs the Spark UI and status APIs remember before garbage collecting. This is a target maximum, and fewer elements may be retained in some circumstances. 1.2.0
spark.ui.retainedStages 1000 How many stages the Spark UI and status APIs remember before garbage collecting. This is a target maximum, and fewer elements may be retained in some circumstances. 0.9.0
spark.ui.retainedTasks 100000 How many tasks in one stage the Spark UI and status APIs remember before garbage collecting. This is a target maximum, and fewer elements may be retained in some circumstances. 2.0.1
To lower numbers. If i remember correctly this is what controls how much metadata remains in the driver post task/stage/job competition.
> On Sep 22, 2021, at 12:42 PM, Kohki Nishio <ta...@gmail.com> wrote:
>
> I believe I have enough information, raised this
>
> https://issues.apache.org/jira/browse/SPARK-36827 <https://issues.apache.org/jira/browse/SPARK-36827>
>
> thanks
> -Kohki
>
>
> On Tue, Sep 21, 2021 at 9:30 PM Sean Owen <srowen@gmail.com <ma...@gmail.com>> wrote:
> No, that's just info Spark retains about finished jobs and tasks, likely. You can limit how much is retained if desired with config.
>
> On Tue, Sep 21, 2021, 11:29 PM Kohki Nishio <taroplus@gmail.com <ma...@gmail.com>> wrote:
> Just following up, it looks like task / stage / job data are not cleaned up
> --
> 6: 7835346 2444627952 org.apache.spark.status.TaskDataWrapper
> 25: 3765152 180727296 org.apache.spark.status.StageDataWrapper
> 88: 232255 9290200 org.apache.spark.status.JobDataWrapper
>
> UI is disabled, not sure why we need to have those data ..
>
> -Kohki
>
>
> On Fri, Sep 17, 2021 at 8:27 AM Kohki Nishio <taroplus@gmail.com <ma...@gmail.com>> wrote:
> Hello,
> I'm seeing possible memory leak behavior in my spark application. According to MAT, it looks like it's related to ElementTrackingStore ..
>
> <Eclipse_Memory_Analyzer.png>
>
> The increase is subtle, so it takes multiple days to actually cause some impact, but I'm wondering if anybody has any idea about what this is about ... Below is the GC graph, yellow is the level after any GC kicks in.
>
> <T2_G1GC_-_Grafana.png>
>
> Thanks
> --
> Kohki Nishio
>
>
> --
> Kohki Nishio
>
>
> --
> Kohki Nishio
Re: Possibly a memory leak issue in Spark
Posted by Kohki Nishio <ta...@gmail.com>.
I believe I have enough information, raised this
https://issues.apache.org/jira/browse/SPARK-36827
thanks
-Kohki
On Tue, Sep 21, 2021 at 9:30 PM Sean Owen <sr...@gmail.com> wrote:
> No, that's just info Spark retains about finished jobs and tasks, likely.
> You can limit how much is retained if desired with config.
>
> On Tue, Sep 21, 2021, 11:29 PM Kohki Nishio <ta...@gmail.com> wrote:
>
>> Just following up, it looks like task / stage / job data are not cleaned
>> up
>> --
>> 6: 7835346 2444627952
>> org.apache.spark.status.TaskDataWrapper
>> 25: 3765152 180727296
>> org.apache.spark.status.StageDataWrapper
>> 88: 232255 9290200 org.apache.spark.status.JobDataWrapper
>>
>> UI is disabled, not sure why we need to have those data ..
>>
>> -Kohki
>>
>>
>> On Fri, Sep 17, 2021 at 8:27 AM Kohki Nishio <ta...@gmail.com> wrote:
>>
>>> Hello,
>>> I'm seeing possible memory leak behavior in my spark application.
>>> According to MAT, it looks like it's related to ElementTrackingStore ..
>>>
>>> [image: Eclipse_Memory_Analyzer.png]
>>>
>>> The increase is subtle, so it takes multiple days to actually cause some
>>> impact, but I'm wondering if anybody has any idea about what this is about
>>> ... Below is the GC graph, yellow is the level after any GC kicks in.
>>>
>>> [image: T2_G1GC_-_Grafana.png]
>>>
>>> Thanks
>>> --
>>> Kohki Nishio
>>>
>>
>>
>> --
>> Kohki Nishio
>>
>
--
Kohki Nishio
Re: Possibly a memory leak issue in Spark
Posted by Sean Owen <sr...@gmail.com>.
No, that's just info Spark retains about finished jobs and tasks, likely.
You can limit how much is retained if desired with config.
On Tue, Sep 21, 2021, 11:29 PM Kohki Nishio <ta...@gmail.com> wrote:
> Just following up, it looks like task / stage / job data are not cleaned up
> --
> 6: 7835346 2444627952 org.apache.spark.status.TaskDataWrapper
> 25: 3765152 180727296 org.apache.spark.status.StageDataWrapper
> 88: 232255 9290200 org.apache.spark.status.JobDataWrapper
>
> UI is disabled, not sure why we need to have those data ..
>
> -Kohki
>
>
> On Fri, Sep 17, 2021 at 8:27 AM Kohki Nishio <ta...@gmail.com> wrote:
>
>> Hello,
>> I'm seeing possible memory leak behavior in my spark application.
>> According to MAT, it looks like it's related to ElementTrackingStore ..
>>
>> [image: Eclipse_Memory_Analyzer.png]
>>
>> The increase is subtle, so it takes multiple days to actually cause some
>> impact, but I'm wondering if anybody has any idea about what this is about
>> ... Below is the GC graph, yellow is the level after any GC kicks in.
>>
>> [image: T2_G1GC_-_Grafana.png]
>>
>> Thanks
>> --
>> Kohki Nishio
>>
>
>
> --
> Kohki Nishio
>
Re: Possibly a memory leak issue in Spark
Posted by Kohki Nishio <ta...@gmail.com>.
Just following up, it looks like task / stage / job data are not cleaned up
--
6: 7835346 2444627952 org.apache.spark.status.TaskDataWrapper
25: 3765152 180727296 org.apache.spark.status.StageDataWrapper
88: 232255 9290200 org.apache.spark.status.JobDataWrapper
UI is disabled, not sure why we need to have those data ..
-Kohki
On Fri, Sep 17, 2021 at 8:27 AM Kohki Nishio <ta...@gmail.com> wrote:
> Hello,
> I'm seeing possible memory leak behavior in my spark application.
> According to MAT, it looks like it's related to ElementTrackingStore ..
>
> [image: Eclipse_Memory_Analyzer.png]
>
> The increase is subtle, so it takes multiple days to actually cause some
> impact, but I'm wondering if anybody has any idea about what this is about
> ... Below is the GC graph, yellow is the level after any GC kicks in.
>
> [image: T2_G1GC_-_Grafana.png]
>
> Thanks
> --
> Kohki Nishio
>
--
Kohki Nishio