You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (JIRA)" <ji...@apache.org> on 2018/08/24 23:45:00 UTC
[jira] [Commented] (SPARK-25206) Wrong data may be returned when
enable pushdown for Parquet
[ https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16592320#comment-16592320 ]
Dongjoon Hyun commented on SPARK-25206:
---------------------------------------
+1 for fixing this. However, actually, it isn't an issue with predicate pushdowns. Please see `where id is null`.
{code}
scala> sql("select * from p1 where id > 0").show
+---+
| ID|
+---+
+---+
scala> sql("select * from p1 where id is null").show
+----+
| ID|
+----+
|null|
|null|
|null|
|null|
|null|
|null|
|null|
|null|
|null|
|null|
+----+
{code}
And, are we going to backport this to Apache Spark 2.2.2?
> Wrong data may be returned when enable pushdown for Parquet
> -----------------------------------------------------------
>
> Key: SPARK-25206
> URL: https://issues.apache.org/jira/browse/SPARK-25206
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.2.2, 2.3.1
> Reporter: yucai
> Priority: Blocker
> Labels: correctness
> Attachments: image-2018-08-24-18-05-23-485.png, image-2018-08-24-22-33-03-231.png, image-2018-08-24-22-34-11-539.png, image-2018-08-24-22-46-05-346.png, pr22183.png
>
>
> In current Spark 2.3.1, below query returns wrong data silently.
> {code:java}
> spark.range(10).write.parquet("/tmp/data")
> sql("DROP TABLE t")
> sql("CREATE TABLE t (ID LONG) USING parquet LOCATION '/tmp/data'")
> scala> sql("select * from t where id > 0").show
> +---+
> | ID|
> +---+
> +---+{code}
>
> *Root Cause*
> Spark pushdowns FilterApi.gt(intColumn("{color:#ff0000}ID{color}"), 0: Integer) into parquet, but {color:#ff0000}ID{color} does not exist in /tmp/data (parquet is case sensitive, it has {color:#ff0000}id{color} actually).
> So no records are returned.
> In Spark 2.1, the user will get Exception:
> {code:java}
> Caused by: java.lang.IllegalArgumentException: Column [ID] was not found in schema!{code}
> But in Spark 2.3, they will get the wrong results sliently.
>
> Since SPARK-24716, Spark uses Parquet schema instead of Hive metastore schema to do the pushdown, perfect for this issue.
> [~yumwang], [~cloud_fan], [~smilegator], any thoughts? Should we backport it?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org