You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Max Gekk (Jira)" <ji...@apache.org> on 2022/05/06 10:10:00 UTC
[jira] [Updated] (SPARK-37933) Limit push down for parquet datasource v2
[ https://issues.apache.org/jira/browse/SPARK-37933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Max Gekk updated SPARK-37933:
-----------------------------
Fix Version/s: (was: 3.3.0)
> Limit push down for parquet datasource v2
> -----------------------------------------
>
> Key: SPARK-37933
> URL: https://issues.apache.org/jira/browse/SPARK-37933
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.4.0
> Reporter: Jackey Lee
> Assignee: Jackey Lee
> Priority: Major
>
> Based on SPARK-37020, we can support limit push down to parquet datasource v2 reader. It can stop scanning parquet early, and reduce network and disk IO.
> Current limit parse status for parquet
> {code:java}
> == Parsed Logical Plan ==
> GlobalLimit 10
> +- LocalLimit 10
> +- RelationV2[a#0, b#1] parquet file:/datasources.db/test_push_down
> == Analyzed Logical Plan ==
> a: int, b: int
> GlobalLimit 10
> +- LocalLimit 10
> +- RelationV2[a#0, b#1] parquet file:/datasources.db/test_push_down
> == Optimized Logical Plan ==
> GlobalLimit 10
> +- LocalLimit 10
> +- RelationV2[a#0, b#1] parquet file:/datasources.db/test_push_down
> == Physical Plan ==
> CollectLimit 10
> +- *(1) ColumnarToRow
> +- BatchScan[a#0, b#1] ParquetScan DataFilters: [], Format: parquet, Location: InMemoryFileIndex(1 paths)[file:/datasources.db/test_push_down/par..., PartitionFilters: [], PushedAggregation: [], PushedFilters: [], PushedGroupBy: [], ReadSchema: struct<a:int,b:int>, PushedFilters: [], PushedAggregation: [], PushedGroupBy: [] RuntimeFilters: [] {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org