You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Wenchen Fan (Jira)" <ji...@apache.org> on 2024/05/29 20:53:00 UTC

[jira] [Assigned] (SPARK-48431) Do not forward predicates on collated columns to file readers

     [ https://issues.apache.org/jira/browse/SPARK-48431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Wenchen Fan reassigned SPARK-48431:
-----------------------------------

    Assignee: Jan-Ole Sasse

> Do not forward predicates on collated columns to file readers
> -------------------------------------------------------------
>
>                 Key: SPARK-48431
>                 URL: https://issues.apache.org/jira/browse/SPARK-48431
>             Project: Spark
>          Issue Type: Task
>          Components: Spark Core
>    Affects Versions: 4.0.0
>            Reporter: Jan-Ole Sasse
>            Assignee: Jan-Ole Sasse
>            Priority: Major
>              Labels: pull-request-available
>
> SPARK-47657 allows to push filters on collated columns to file sources that support it. If such filters are pushed to file sources, those file sources must not push those filters to the actual file readers (i.e. parquet or csv readers), because there is no guarantee that those support collations.
> With this task, we are widening filters on collations to be AlwaysTrue when we translate filters for file sources.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org