You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Takeshi Yamamuro (JIRA)" <ji...@apache.org> on 2017/03/11 07:21:04 UTC
[jira] [Commented] (SPARK-19914) Spark Scala - Calling persist
after reading a parquet file makes certain spark.sql queries return empty
results
[ https://issues.apache.org/jira/browse/SPARK-19914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15906121#comment-15906121 ]
Takeshi Yamamuro commented on SPARK-19914:
------------------------------------------
I couldn't reproduce this and do I miss anything?
{code}
scala> Seq(("1", 2)).toDF("id", "value").write.parquet("/Users/maropu/Desktop/data")
scala> val df = spark.read.parquet("/Users/maropu/Desktop/data")
scala> df.createOrReplaceTempView("t")
scala> sql("SELECT * FROM t a WHERE a.id = '1'").show(10)
+---+-----+
| id|value|
+---+-----+
| 1| 2|
+---+-----+
scala> import org.apache.spark.storage.StorageLevel
scala> val df = spark.read.parquet("/Users/maropu/Desktop/data").persist(StorageLevel.DISK_ONLY)
scala> df.createOrReplaceTempView("t")
scala> sql("SELECT * FROM t a WHERE a.id = '1'").show(10)
+---+-----+
| id|value|
+---+-----+
| 1| 2|
+---+-----+
{code}
> Spark Scala - Calling persist after reading a parquet file makes certain spark.sql queries return empty results
> ---------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-19914
> URL: https://issues.apache.org/jira/browse/SPARK-19914
> Project: Spark
> Issue Type: Bug
> Components: Input/Output, SQL
> Affects Versions: 2.0.0, 2.1.0
> Reporter: Yifeng Li
>
> Hi There,
> It seems like calling .persist() after spark.read.parquet will make spark.sql statements return empty results if the query is written in a certain way.
> I have the following code here:
> val df = spark.read.parquet("C:\\...")
> df.createOrReplaceTempView("t1")
> spark.sql("select * from t1 a where a.id = '123456789'").show(10)
> Everything works fine here.
> Now, if I do:
> val df = spark.read.parquet("C:\\...").persist(StorageLevel.DISK_ONLY)
> df.createOrReplaceTempView("t1")
> spark.sql("select * from t1 a where a.id = '123456789'").show(10)
> I will get empty results.
> selecting individual columns works with persist, e.g.:
> val df = spark.read.parquet("C:\\...").persist(StorageLevel.DISK_ONLY)
> df.createOrReplaceTempView("t1")
> spark.sql("select a.id from t1 a where a.id = '123456789'").show(10)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org