You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by "wyphao.2007" <wy...@163.com> on 2015/04/29 12:42:04 UTC
Driver memory leak?
Hi, Dear developer, I am using Spark Streaming to read data from kafka, the program already run about 120 hours, but today the program failed because of driver's OOM as follow:
Container [pid=49133,containerID=container_1429773909253_0050_02_000001] is running beyond physical memory limits. Current usage: 2.5 GB of 2.5 GB physical memory used; 3.2 GB of 50 GB virtual memory used. Killing container.
I set --driver-memory to 2g, In my mind, driver is responsibility for job scheduler and job monitor(Please correct me If I'm wrong), Why it using so much memory?
So I using jmap to monitor other program(already run about 48 hours):
sudo /home/q/java7/jdk1.7.0_45/bin/jmap -histo:live 31256, the result as follow:
the java.util.HashMap$Entry and java.lang.Long object using about 600Mb memory!
and I also using jmap to monitor other program(already run about 1 hours), the result as follow:
the java.util.HashMap$Entry and java.lang.Long object doesn't using so many memory, But I found, as time goes by, the java.util.HashMap$Entry and java.lang.Long object will occupied more and more memory,
It is driver's memory leak question? or other reason?
Thanks
Best Regards
Re: Driver memory leak?
Posted by Tathagata Das <td...@databricks.com>.
It could be related to this.
https://issues.apache.org/jira/browse/SPARK-6737
This was fixed in Spark 1.3.1.
On Wed, Apr 29, 2015 at 8:38 AM, Sean Owen <so...@cloudera.com> wrote:
> Not sure what you mean. It's already in CDH since 5.4 = 1.3.0
> (This isn't the place to ask about CDH)
> I also don't think that's the problem. The process did not run out of
> memory.
>
> On Wed, Apr 29, 2015 at 2:08 PM, Serega Sheypak <se...@gmail.com>
> wrote:
>
>> >The memory leak could be related to this
>> <https://issues.apache.org/jira/browse/SPARK-5967> defect that was
>> resolved in Spark 1.2.2 and 1.3.0.
>> @Sean
>> Will it be backported to CDH? I did't find that bug in CDH 5.4 release
>> notes.
>>
>> 2015-04-29 14:51 GMT+02:00 Conor Fennell <co...@altocloud.com>:
>>
>>> The memory leak could be related to this
>>> <https://issues.apache.org/jira/browse/SPARK-5967> defect that was
>>> resolved in Spark 1.2.2 and 1.3.0.
>>>
>>> It also was a HashMap causing the issue.
>>>
>>> -Conor
>>>
>>>
>>>
>>> On Wed, Apr 29, 2015 at 12:01 PM, Sean Owen <so...@cloudera.com> wrote:
>>>
>>>> Please use user@, not dev@
>>>>
>>>> This message does not appear to be from your driver. It also doesn't
>>>> say you ran out of memory. It says you didn't tell YARN to let it use the
>>>> memory you want. Look at the memory overhead param and please search first
>>>> for related discussions.
>>>> On Apr 29, 2015 11:43 AM, "wyphao.2007" <wy...@163.com> wrote:
>>>>
>>>>> Hi, Dear developer, I am using Spark Streaming to read data from
>>>>> kafka, the program already run about 120 hours, but today the program
>>>>> failed because of driver's OOM as follow:
>>>>>
>>>>> Container
>>>>> [pid=49133,containerID=container_1429773909253_0050_02_000001] is running
>>>>> beyond physical memory limits. Current usage: 2.5 GB of 2.5 GB physical
>>>>> memory used; 3.2 GB of 50 GB virtual memory used. Killing container.
>>>>>
>>>>> I set --driver-memory to 2g, In my mind, driver is responsibility for
>>>>> job scheduler and job monitor(Please correct me If I'm wrong), Why it using
>>>>> so much memory?
>>>>>
>>>>> So I using jmap to monitor other program(already run about 48 hours):
>>>>> sudo /home/q/java7/jdk1.7.0_45/bin/jmap -histo:live 31256, the result
>>>>> as follow:
>>>>> the java.util.HashMap$Entry and java.lang.Long object using about
>>>>> 600Mb memory!
>>>>>
>>>>> and I also using jmap to monitor other program(already run about 1
>>>>> hours), the result as follow:
>>>>> the java.util.HashMap$Entry and java.lang.Long object doesn't using so
>>>>> many memory, But I found, as time goes by, the
>>>>> java.util.HashMap$Entry and java.lang.Long object will occupied more
>>>>> and more memory,
>>>>> It is driver's memory leak question? or other reason?
>>>>>
>>>>> Thanks
>>>>> Best Regards
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>
>>
>
Re: Driver memory leak?
Posted by Sean Owen <so...@cloudera.com>.
Not sure what you mean. It's already in CDH since 5.4 = 1.3.0
(This isn't the place to ask about CDH)
I also don't think that's the problem. The process did not run out of
memory.
On Wed, Apr 29, 2015 at 2:08 PM, Serega Sheypak <se...@gmail.com>
wrote:
> >The memory leak could be related to this
> <https://issues.apache.org/jira/browse/SPARK-5967> defect that was
> resolved in Spark 1.2.2 and 1.3.0.
> @Sean
> Will it be backported to CDH? I did't find that bug in CDH 5.4 release
> notes.
>
> 2015-04-29 14:51 GMT+02:00 Conor Fennell <co...@altocloud.com>:
>
>> The memory leak could be related to this
>> <https://issues.apache.org/jira/browse/SPARK-5967> defect that was
>> resolved in Spark 1.2.2 and 1.3.0.
>>
>> It also was a HashMap causing the issue.
>>
>> -Conor
>>
>>
>>
>> On Wed, Apr 29, 2015 at 12:01 PM, Sean Owen <so...@cloudera.com> wrote:
>>
>>> Please use user@, not dev@
>>>
>>> This message does not appear to be from your driver. It also doesn't say
>>> you ran out of memory. It says you didn't tell YARN to let it use the
>>> memory you want. Look at the memory overhead param and please search first
>>> for related discussions.
>>> On Apr 29, 2015 11:43 AM, "wyphao.2007" <wy...@163.com> wrote:
>>>
>>>> Hi, Dear developer, I am using Spark Streaming to read data from
>>>> kafka, the program already run about 120 hours, but today the program
>>>> failed because of driver's OOM as follow:
>>>>
>>>> Container
>>>> [pid=49133,containerID=container_1429773909253_0050_02_000001] is running
>>>> beyond physical memory limits. Current usage: 2.5 GB of 2.5 GB physical
>>>> memory used; 3.2 GB of 50 GB virtual memory used. Killing container.
>>>>
>>>> I set --driver-memory to 2g, In my mind, driver is responsibility for
>>>> job scheduler and job monitor(Please correct me If I'm wrong), Why it using
>>>> so much memory?
>>>>
>>>> So I using jmap to monitor other program(already run about 48 hours):
>>>> sudo /home/q/java7/jdk1.7.0_45/bin/jmap -histo:live 31256, the result
>>>> as follow:
>>>> the java.util.HashMap$Entry and java.lang.Long object using about
>>>> 600Mb memory!
>>>>
>>>> and I also using jmap to monitor other program(already run about 1
>>>> hours), the result as follow:
>>>> the java.util.HashMap$Entry and java.lang.Long object doesn't using so
>>>> many memory, But I found, as time goes by, the java.util.HashMap$Entry
>>>> and java.lang.Long object will occupied more and more memory,
>>>> It is driver's memory leak question? or other reason?
>>>>
>>>> Thanks
>>>> Best Regards
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>
>
Re: Driver memory leak?
Posted by Serega Sheypak <se...@gmail.com>.
>The memory leak could be related to this
<https://issues.apache.org/jira/browse/SPARK-5967> defect that was resolved
in Spark 1.2.2 and 1.3.0.
@Sean
Will it be backported to CDH? I did't find that bug in CDH 5.4 release
notes.
2015-04-29 14:51 GMT+02:00 Conor Fennell <co...@altocloud.com>:
> The memory leak could be related to this
> <https://issues.apache.org/jira/browse/SPARK-5967> defect that was
> resolved in Spark 1.2.2 and 1.3.0.
>
> It also was a HashMap causing the issue.
>
> -Conor
>
>
>
> On Wed, Apr 29, 2015 at 12:01 PM, Sean Owen <so...@cloudera.com> wrote:
>
>> Please use user@, not dev@
>>
>> This message does not appear to be from your driver. It also doesn't say
>> you ran out of memory. It says you didn't tell YARN to let it use the
>> memory you want. Look at the memory overhead param and please search first
>> for related discussions.
>> On Apr 29, 2015 11:43 AM, "wyphao.2007" <wy...@163.com> wrote:
>>
>>> Hi, Dear developer, I am using Spark Streaming to read data from kafka,
>>> the program already run about 120 hours, but today the program failed
>>> because of driver's OOM as follow:
>>>
>>> Container [pid=49133,containerID=container_1429773909253_0050_02_000001]
>>> is running beyond physical memory limits. Current usage: 2.5 GB of 2.5 GB
>>> physical memory used; 3.2 GB of 50 GB virtual memory used. Killing
>>> container.
>>>
>>> I set --driver-memory to 2g, In my mind, driver is responsibility for
>>> job scheduler and job monitor(Please correct me If I'm wrong), Why it using
>>> so much memory?
>>>
>>> So I using jmap to monitor other program(already run about 48 hours):
>>> sudo /home/q/java7/jdk1.7.0_45/bin/jmap -histo:live 31256, the result as
>>> follow:
>>> the java.util.HashMap$Entry and java.lang.Long object using about 600Mb
>>> memory!
>>>
>>> and I also using jmap to monitor other program(already run about 1 hours
>>> ), the result as follow:
>>> the java.util.HashMap$Entry and java.lang.Long object doesn't using so
>>> many memory, But I found, as time goes by, the java.util.HashMap$Entry
>>> and java.lang.Long object will occupied more and more memory,
>>> It is driver's memory leak question? or other reason?
>>>
>>> Thanks
>>> Best Regards
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>
Re: Driver memory leak?
Posted by Conor Fennell <co...@altocloud.com>.
The memory leak could be related to this
<https://issues.apache.org/jira/browse/SPARK-5967> defect that was resolved
in Spark 1.2.2 and 1.3.0.
It also was a HashMap causing the issue.
-Conor
On Wed, Apr 29, 2015 at 12:01 PM, Sean Owen <so...@cloudera.com> wrote:
> Please use user@, not dev@
>
> This message does not appear to be from your driver. It also doesn't say
> you ran out of memory. It says you didn't tell YARN to let it use the
> memory you want. Look at the memory overhead param and please search first
> for related discussions.
> On Apr 29, 2015 11:43 AM, "wyphao.2007" <wy...@163.com> wrote:
>
>> Hi, Dear developer, I am using Spark Streaming to read data from kafka,
>> the program already run about 120 hours, but today the program failed
>> because of driver's OOM as follow:
>>
>> Container [pid=49133,containerID=container_1429773909253_0050_02_000001]
>> is running beyond physical memory limits. Current usage: 2.5 GB of 2.5 GB
>> physical memory used; 3.2 GB of 50 GB virtual memory used. Killing
>> container.
>>
>> I set --driver-memory to 2g, In my mind, driver is responsibility for
>> job scheduler and job monitor(Please correct me If I'm wrong), Why it using
>> so much memory?
>>
>> So I using jmap to monitor other program(already run about 48 hours):
>> sudo /home/q/java7/jdk1.7.0_45/bin/jmap -histo:live 31256, the result as
>> follow:
>> the java.util.HashMap$Entry and java.lang.Long object using about 600Mb
>> memory!
>>
>> and I also using jmap to monitor other program(already run about 1 hours
>> ), the result as follow:
>> the java.util.HashMap$Entry and java.lang.Long object doesn't using so
>> many memory, But I found, as time goes by, the java.util.HashMap$Entry
>> and java.lang.Long object will occupied more and more memory,
>> It is driver's memory leak question? or other reason?
>>
>> Thanks
>> Best Regards
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
Re: Driver memory leak?
Posted by Sean Owen <so...@cloudera.com>.
Please use user@, not dev@
This message does not appear to be from your driver. It also doesn't say
you ran out of memory. It says you didn't tell YARN to let it use the
memory you want. Look at the memory overhead param and please search first
for related discussions.
On Apr 29, 2015 11:43 AM, "wyphao.2007" <wy...@163.com> wrote:
> Hi, Dear developer, I am using Spark Streaming to read data from kafka,
> the program already run about 120 hours, but today the program failed
> because of driver's OOM as follow:
>
> Container [pid=49133,containerID=container_1429773909253_0050_02_000001]
> is running beyond physical memory limits. Current usage: 2.5 GB of 2.5 GB
> physical memory used; 3.2 GB of 50 GB virtual memory used. Killing
> container.
>
> I set --driver-memory to 2g, In my mind, driver is responsibility for job
> scheduler and job monitor(Please correct me If I'm wrong), Why it using so
> much memory?
>
> So I using jmap to monitor other program(already run about 48 hours):
> sudo /home/q/java7/jdk1.7.0_45/bin/jmap -histo:live 31256, the result as
> follow:
> the java.util.HashMap$Entry and java.lang.Long object using about 600Mb
> memory!
>
> and I also using jmap to monitor other program(already run about 1 hours
> ), the result as follow:
> the java.util.HashMap$Entry and java.lang.Long object doesn't using so
> many memory, But I found, as time goes by, the java.util.HashMap$Entry
> and java.lang.Long object will occupied more and more memory,
> It is driver's memory leak question? or other reason?
>
> Thanks
> Best Regards
>
>
>
>
>
>
>
>
>
>
Re:Re:Driver memory leak?
Posted by "wyphao.2007" <wy...@163.com>.
No, I am not collect the result to driver,I sample send the result to kafka.
BTW, the image address are:
https://cloud.githubusercontent.com/assets/5170878/7389463/ac03bf34-eea0-11e4-9e6b-1d2fba170c1c.png
and
https://cloud.githubusercontent.com/assets/5170878/7389480/c629d236-eea0-11e4-983a-dc5aa97c2554.png
At 2015-04-29 18:48:33,zhangxiongfei <zh...@163.com> wrote:
The mount of memory that the driver consumes depends on your program logic,did you try to collect the result of Spark job?
At 2015-04-29 18:42:04, "wyphao.2007" <wy...@163.com> wrote:
Hi, Dear developer, I am using Spark Streaming to read data from kafka, the program already run about 120 hours, but today the program failed because of driver's OOM as follow:
Container [pid=49133,containerID=container_1429773909253_0050_02_000001] is running beyond physical memory limits. Current usage: 2.5 GB of 2.5 GB physical memory used; 3.2 GB of 50 GB virtual memory used. Killing container.
I set --driver-memory to 2g, In my mind, driver is responsibility for job scheduler and job monitor(Please correct me If I'm wrong), Why it using so much memory?
So I using jmap to monitor other program(already run about 48 hours):
sudo /home/q/java7/jdk1.7.0_45/bin/jmap -histo:live 31256, the result as follow:
the java.util.HashMap$Entry and java.lang.Long object using about 600Mb memory!
and I also using jmap to monitor other program(already run about 1 hours), the result as follow:
the java.util.HashMap$Entry and java.lang.Long object doesn't using so many memory, But I found, as time goes by, the java.util.HashMap$Entry and java.lang.Long object will occupied more and more memory,
It is driver's memory leak question? or other reason?
Thanks
Best Regards