You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2021/05/21 01:35:00 UTC
[jira] [Commented] (SPARK-35479) Format PartitionFilters IN strings
in scan nodes
[ https://issues.apache.org/jira/browse/SPARK-35479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17348923#comment-17348923 ]
Apache Spark commented on SPARK-35479:
--------------------------------------
User 'maropu' has created a pull request for this issue:
https://github.com/apache/spark/pull/32615
> Format PartitionFilters IN strings in scan nodes
> ------------------------------------------------
>
> Key: SPARK-35479
> URL: https://issues.apache.org/jira/browse/SPARK-35479
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.2.0
> Reporter: Takeshi Yamamuro
> Priority: Major
>
> This ticket proposes to format strings correctly for `PushedFilters`. For example, `explain()` for a query below prints `v in (array('a'))` as `PushedFilters: [In(v, [WrappedArray(a)])]`;
> {code}
> scala> sql("create table t (v array<string>) using parquet")
> scala> sql("select * from t where v in (array('a'), null)").explain()
> == Physical Plan ==
> *(1) Filter v#4 IN ([a],null)
> +- FileScan parquet default.t[v#4] Batched: false, DataFilters: [v#4 IN ([a],null)], Format: Parquet, Location: InMemoryFileIndex[file:/Users/maropu/Repositories/spark/spark-3.1.1-bin-hadoop2.7/spark-warehouse/t], PartitionFilters: [], PushedFilters: [In(v, [WrappedArray(a),null])], ReadSchema: struct<v:array<string>>
> {code}
> This PR makes `explain()` print it as `PushedFilters: [In(v, [[a]])]`;
> {code}
> scala> sql("select * from t where v in (array('a'), null)").explain()
> == Physical Plan ==
> *(1) Filter v#4 IN ([a],null)
> +- FileScan parquet default.t[v#4] Batched: false, DataFilters: [v#4 IN ([a],null)], Format: Parquet, Location: InMemoryFileIndex[file:/Users/maropu/Repositories/spark/spark-3.1.1-bin-hadoop2.7/spark-warehouse/t], PartitionFilters: [], PushedFilters: [In(v, [[a],null])], ReadSchema: struct<v:array<string>>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org