[jira] [Created] (SPARK-27636) Remove cached RDD blocks after PIC execution

["shahid (JIRA)" <ji...@apache.org>]

shahid created SPARK-27636:
------------------------------

             Summary: Remove cached RDD blocks after PIC execution
                 Key: SPARK-27636
                 URL: https://issues.apache.org/jira/browse/SPARK-27636
             Project: Spark
          Issue Type: Improvement
          Components: MLlib
    Affects Versions: 2.4.2, 2.3.3, 3.0.0
            Reporter: shahid


Test steps to reproduce:
1) bin/spark-shell
val dataset = spark.createDataFrame(Seq(
     (0L, 1L, 1.0),
     (1L,2L,1.0),
     (3L, 4L,1.0),
     (4L,0L,0.1))).toDF("src", "dst", "weight")
val model = new PowerIterationClustering().
     setMaxIter(10).
     setInitMode("degree").
      setWeightCol("weight")     
 val prediction = model.assignClusters(dataset).select("id", "cluster")

2) Open storage tab of the UI. we can see many RDD block cached, even after running the PIC.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org