[jira] [Resolved] (SPARK-33612) Add dataSourceRewriteRules batch to Optimizer

["Dongjoon Hyun (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-33612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dongjoon Hyun resolved SPARK-33612.
-----------------------------------
    Fix Version/s: 3.1.0
       Resolution: Fixed

Issue resolved by pull request 30558
[https://github.com/apache/spark/pull/30558]

> Add dataSourceRewriteRules batch to Optimizer
> ---------------------------------------------
>
>                 Key: SPARK-33612
>                 URL: https://issues.apache.org/jira/browse/SPARK-33612
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 3.1.0
>            Reporter: Anton Okolnychyi
>            Assignee: Anton Okolnychyi
>            Priority: Major
>             Fix For: 3.1.0
>
>
> Right now, we have a special place in the optimizer where we run rules for constructing v2 scans. As time shows, we need more rewrite rules for v2 tables and not all of them related to reads. One option is to rename the current batch into something more generic but it would require changing quite some places. Moreover, the batch contains some non-V2 logic as well. That's why it seems better to introduce a new batch and use it for all data source v2 rewrites and beyond.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org