You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Anton Okolnychyi (Jira)" <ji...@apache.org> on 2020/12/01 11:16:00 UTC
[jira] [Updated] (SPARK-33612) Add dataSourceRewriteRules batch to
Optimizer
[ https://issues.apache.org/jira/browse/SPARK-33612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Anton Okolnychyi updated SPARK-33612:
-------------------------------------
Summary: Add dataSourceRewriteRules batch to Optimizer (was: Add v2SourceRewriteRules batch to Optimizer)
> Add dataSourceRewriteRules batch to Optimizer
> ---------------------------------------------
>
> Key: SPARK-33612
> URL: https://issues.apache.org/jira/browse/SPARK-33612
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.1.0
> Reporter: Anton Okolnychyi
> Priority: Major
>
> Right now, we have a special place in the optimizer where we run rules for constructing v2 scans. As time shows, we need more rewrite rules for v2 tables and not all of them related to reads. One option is to rename the current batch into something more generic but it would require changing quite some places. Moreover, the batch contains some non-V2 logic as well. That's why it seems better to introduce a new batch and use it for all data source v2 rewrites and beyond.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org