You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Alessio (JIRA)" <ji...@apache.org> on 2016/06/12 18:38:20 UTC
[jira] [Updated] (SPARK-15904) High Memory Pressure using MLlib
K-means
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alessio updated SPARK-15904:
----------------------------
Description:
Running MLlib K-Means on a ~400MB dataset (12 partitions), persisted on Memory and Disk.
Everything's fine, although at the end of K-Means, after the number of iterations, the cost function value and the running time there's a nice "Removing RDD <idx> from persistent list" stage. However, during this stage there's a high memory pressure. Weird, since RDDs are about to be removed.
I'm running this cluster analysis on a 16GB machine, with Spark Context as local[*]. My machine has an i5 hyperthreaded dual-core, thus [*] means 4.
I'm launching this application though spark-submit with --driver-memory 10G
was:
Running MLlib K-Means on a ~400MB dataset, persisted on Memory and Disk.
Everything's fine, although at the end of K-Means, after the number of iterations, the cost function value and the running time there's a nice "Removing RDD <idx> from persistent list" stage. However, during this stage there's a high memory pressure. Weird, since RDDs are about to be removed.
I'm running this cluster analysis on a 16GB machine, with Spark Context as local[*]. My machine has an i5 hyperthreaded dual-core, thus [*] means 4.
I'm launching this application though spark-submit with --driver-memory 10G
> High Memory Pressure using MLlib K-means
> ----------------------------------------
>
> Key: SPARK-15904
> URL: https://issues.apache.org/jira/browse/SPARK-15904
> Project: Spark
> Issue Type: Bug
> Components: MLlib
> Affects Versions: 1.6.1
> Environment: Mac OS X 10.11.6beta on Macbook Pro 13" mid-2012. 16GB of RAM.
> Reporter: Alessio
>
> Running MLlib K-Means on a ~400MB dataset (12 partitions), persisted on Memory and Disk.
> Everything's fine, although at the end of K-Means, after the number of iterations, the cost function value and the running time there's a nice "Removing RDD <idx> from persistent list" stage. However, during this stage there's a high memory pressure. Weird, since RDDs are about to be removed.
> I'm running this cluster analysis on a 16GB machine, with Spark Context as local[*]. My machine has an i5 hyperthreaded dual-core, thus [*] means 4.
> I'm launching this application though spark-submit with --driver-memory 10G
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org