You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Liang-Chi Hsieh (JIRA)" <ji...@apache.org> on 2017/07/20 07:13:00 UTC
[jira] [Updated] (SPARK-21484) Wrong query plans of Dataset after
persist/unpersist
[ https://issues.apache.org/jira/browse/SPARK-21484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Liang-Chi Hsieh updated SPARK-21484:
------------------------------------
Summary: Wrong query plans of Dataset after persist/unpersist (was: Wrong query plans of Dataset after persist/unperesist)
> Wrong query plans of Dataset after persist/unpersist
> ----------------------------------------------------
>
> Key: SPARK-21484
> URL: https://issues.apache.org/jira/browse/SPARK-21484
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.2.0
> Reporter: Liang-Chi Hsieh
>
> After the cal of persist/unpersis, the query plans of a Dataset should be changed accordingly. But currently the query plans are the same. So you will see the inconsistent query plans like:
> {code}
> scala> val x1 = Seq(1).toDF()
> x1: org.apache.spark.sql.DataFrame = [value: int]
> scala> println(x1.queryExecution.executedPlan) // query plans are materialized before persist()
> LocalTableScan [value#1]
> println(x1.queryExecution.executedPlan)
> scala> x1.persist()
> scala> x1.count()
> scala> println(x1.queryExecution.executedPlan)
> LocalTableScan [value#1]
> {code}
> {code}
> scala> val x1 = Seq(1).toDF()
> x1: org.apache.spark.sql.DataFrame = [value: int]
> scala> x1.persist()
> scala> x1.count()
> scala> println(x1.queryExecution.executedPlan) // query plans are materialized after persist()
> InMemoryTableScan [value#24]
> +- InMemoryRelation [value#24], true, 10000, StorageLevel(disk, memory, deserialized, 1 replicas)
> +- LocalTableScan [value#1]
> scala> x1.unpersist()
> scala> println(x1.queryExecution.executedPlan)
> InMemoryTableScan [value#24]
> +- InMemoryRelation [value#24], true, 10000, StorageLevel(disk, memory, deserialized, 1 replicas)
> +- LocalTableScan [value#1]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org