You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2022/08/31 19:13:00 UTC
[jira] [Commented] (SPARK-40280) Failure to create parquet predicate push down for ints and longs on some valid files
[ https://issues.apache.org/jira/browse/SPARK-40280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17598577#comment-17598577 ]
Apache Spark commented on SPARK-40280:
--------------------------------------
User 'revans2' has created a pull request for this issue:
https://github.com/apache/spark/pull/37747
> Failure to create parquet predicate push down for ints and longs on some valid files
> ------------------------------------------------------------------------------------
>
> Key: SPARK-40280
> URL: https://issues.apache.org/jira/browse/SPARK-40280
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.1.0, 3.2.0, 3.3.0, 3.4.0
> Reporter: Robert Joseph Evans
> Priority: Major
>
> The [parquet format|https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#signed-integers] specification states that...
> bq. {{{}INT(8, true){}}}, {{{}INT(16, true){}}}, and {{INT(32, true)}} must annotate an {{int32}} primitive type and {{INT(64, true)}} must annotate an {{int64}} primitive type. {{INT(32, true)}} and {{INT(64, true)}} are implied by the {{int32}} and {{int64}} primitive types if no other annotation is present and should be considered optional.
> But the code inside of [ParquetFilters.scala|https://github.com/apache/spark/blob/296fe49ec855ac8c15c080e7bab6d519fe504bd3/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala#L125-L126] requires that for {{int32}} and {{int64}} that there be no annotation. If there is an annotation for those columns and they are a part of a predicate push down, the hard coded types will not match and the corresponding filter ends up being {{None}}.
> This can be a huge performance penalty for a valid parquet file.
> I am happy to provide files that show the issue if needed for testing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org