You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2019/05/05 22:15:00 UTC
[jira] [Assigned] (SPARK-27636) Remove cached RDD blocks after PIC
execution
[ https://issues.apache.org/jira/browse/SPARK-27636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-27636:
------------------------------------
Assignee: (was: Apache Spark)
> Remove cached RDD blocks after PIC execution
> --------------------------------------------
>
> Key: SPARK-27636
> URL: https://issues.apache.org/jira/browse/SPARK-27636
> Project: Spark
> Issue Type: Improvement
> Components: MLlib
> Affects Versions: 2.3.3, 2.4.2, 3.0.0
> Reporter: shahid
> Priority: Major
>
> Test steps to reproduce:
> 1) bin/spark-shell
> val dataset = spark.createDataFrame(Seq(
> (0L, 1L, 1.0),
> (1L,2L,1.0),
> (3L, 4L,1.0),
> (4L,0L,0.1))).toDF("src", "dst", "weight")
> val model = new PowerIterationClustering().
> setMaxIter(10).
> setInitMode("degree").
> setWeightCol("weight")
> val prediction = model.assignClusters(dataset).select("id", "cluster")
> 2) Open storage tab of the UI. we can see many RDD block cached, even after running the PIC.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org