You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Ruslan Dautkhanov (JIRA)" <ji...@apache.org> on 2018/10/23 20:07:00 UTC
[jira] [Updated] (SPARK-25814) spark driver runs out of memory on
org.apache.spark.util.kvstore.InMemoryStore
[ https://issues.apache.org/jira/browse/SPARK-25814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ruslan Dautkhanov updated SPARK-25814:
--------------------------------------
Description:
We're looking into issue when even huge spark driver memory gets eventually exhausted and GC makes driver stop responding.
Used jxray.com tool and found that most of driver heap is used by
{noformat}
org.apache.spark.status.AppStatusStore
-> org.apache.spark.status.ElementTrackingStore
-> org.apache.spark.util.kvstore.InMemoryStore
{noformat}
Is there is a way to tune this particular spark driver's memory region down?
!image-2018-10-23-14-03-12-258.png!
!image-2018-10-23-14-06-53-722.png!
was:
We're looking into issue when even huge spark driver memory gets eventually exhausted and GC makes driver stop responding.
Used jxray.com tool and found that most of driver heap is used by
{noformat}
org.apache.spark.status.AppStatusStore
-> org.apache.spark.status.ElementTrackingStore
-> org.apache.spark.util.kvstore.InMemoryStore
{noformat}
Is there is a way to tune this particular spark driver's memory region down?
!image-2018-10-23-14-03-12-258.png!
> spark driver runs out of memory on org.apache.spark.util.kvstore.InMemoryStore
> ------------------------------------------------------------------------------
>
> Key: SPARK-25814
> URL: https://issues.apache.org/jira/browse/SPARK-25814
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.2.2, 2.3.2
> Reporter: Ruslan Dautkhanov
> Priority: Critical
> Labels: driver, memory-analysis, memory-leak, statestore
> Attachments: image-2018-10-23-14-06-53-722.png
>
>
> We're looking into issue when even huge spark driver memory gets eventually exhausted and GC makes driver stop responding.
> Used jxray.com tool and found that most of driver heap is used by
>
> {noformat}
> org.apache.spark.status.AppStatusStore
> -> org.apache.spark.status.ElementTrackingStore
> -> org.apache.spark.util.kvstore.InMemoryStore
>
> {noformat}
>
>
> Is there is a way to tune this particular spark driver's memory region down?
>
> !image-2018-10-23-14-03-12-258.png!
>
> !image-2018-10-23-14-06-53-722.png!
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org