You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "adriangb (via GitHub)" <gi...@apache.org> on 2025/12/01 18:54:33 UTC
[PR] Add upgrade guide for PhysicalOptimizerRule::optimize_plan [datafusion]
adriangb opened a new pull request, #19030:
URL: https://github.com/apache/datafusion/pull/19030
Closes #19029
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add upgrade guide for PhysicalOptimizerRule::optimize_plan [datafusion]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on code in PR #19030:
URL: https://github.com/apache/datafusion/pull/19030#discussion_r2580355110
##########
docs/source/library-user-guide/upgrading.md:
##########
@@ -332,6 +332,102 @@ let config = FileScanConfigBuilder::new(url, source)
**Handling projections in `FileSource`:**
+### `PhysicalOptimizerRule::optimize` deprecated in favor of `optimize_plan`
+
+The `PhysicalOptimizerRule` trait has been updated to provide optimizer rules with access to broader session context. A new method `optimize_plan` has been added that accepts an `OptimizerContext` parameter, and the existing `optimize` method has been deprecated.
+
+**Who is affected:**
+
+- Users who have implemented custom `PhysicalOptimizerRule` implementations
+
+**Breaking changes:**
+
+1. **New `optimize_plan` method**: This is the preferred method for implementing optimization rules. It provides access to the full `SessionConfig` through `OptimizerContext`, rather than just `ConfigOptions`.
+
+2. **`optimize` method deprecated**: The old `optimize` method that takes `&ConfigOptions` is now deprecated and will be removed in DataFusion 54.0.0.
+
+**Migration guide:**
+
+If you have a custom `PhysicalOptimizerRule` implementation, update it to implement `optimize_plan` instead of `optimize`:
+
+**Before:**
+
+```rust
+use datafusion::physical_optimizer::PhysicalOptimizerRule;
+use datafusion_common::config::ConfigOptions;
+use datafusion_common::Result;
+use datafusion_physical_plan::ExecutionPlan;
+use std::sync::Arc;
+
+#[derive(Debug)]
+struct MyOptimizerRule;
+
+#[allow(deprecated)]
+impl PhysicalOptimizerRule for MyOptimizerRule {
+ fn optimize(
+ &self,
+ plan: Arc<dyn ExecutionPlan>,
+ _config: &ConfigOptions,
+ ) -> Result<Arc<dyn ExecutionPlan>> {
+ // Use config.optimizer, config.execution, etc.
+ // ... optimization logic ...
+ Ok(plan)
+ }
+
+ fn name(&self) -> &str {
+ "my_optimizer_rule"
+ }
+
+ fn schema_check(&self) -> bool {
+ true
+ }
+}
+```
+
+**After:**
+
+```rust
+use datafusion::physical_optimizer::{OptimizerContext, PhysicalOptimizerRule};
+use datafusion_common::Result;
+use datafusion_physical_plan::ExecutionPlan;
+use std::sync::Arc;
+
+#[derive(Debug)]
+struct MyOptimizerRule;
+
+impl PhysicalOptimizerRule for MyOptimizerRule {
+ fn optimize_plan(
+ &self,
+ plan: Arc<dyn ExecutionPlan>,
+ context: &OptimizerContext,
+ ) -> Result<Arc<dyn ExecutionPlan>> {
+ // Access ConfigOptions through session_config
+ let _config = context.session_config().options();
+ // Or access extensions through session_config
+ let _extensions = context.session_config().extensions();
+ // ... optimization logic ...
+ Ok(plan)
+ }
+
+ fn name(&self) -> &str {
+ "my_optimizer_rule"
+ }
+
+ fn schema_check(&self) -> bool {
+ true
+ }
+}
+```
+
+**What is `OptimizerContext`?**
+
+`OptimizerContext` is a new struct that provides context during physical plan optimization, similar to how `TaskContext` provides context during execution. It wraps `SessionConfig`, giving optimizer rules access to:
+
+- Configuration options via `context.session_config().options()`
+- Session extensions via `context.session_config().extensions()`
+
+This enables optimizer rules to access custom extensions registered with the session, which was not possible with the old `&ConfigOptions` parameter.
+
Review Comment:
You’re right, the previous methods remain unchanged so from the user’s perspective nothing changes. My apologies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add upgrade guide for PhysicalOptimizerRule::optimize_plan [datafusion]
Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 commented on code in PR #19030:
URL: https://github.com/apache/datafusion/pull/19030#discussion_r2579747506
##########
docs/source/library-user-guide/upgrading.md:
##########
@@ -332,6 +332,102 @@ let config = FileScanConfigBuilder::new(url, source)
**Handling projections in `FileSource`:**
+### `PhysicalOptimizerRule::optimize` deprecated in favor of `optimize_plan`
+
+The `PhysicalOptimizerRule` trait has been updated to provide optimizer rules with access to broader session context. A new method `optimize_plan` has been added that accepts an `OptimizerContext` parameter, and the existing `optimize` method has been deprecated.
+
+**Who is affected:**
+
+- Users who have implemented custom `PhysicalOptimizerRule` implementations
Review Comment:
cc @zhuqi-lucas / @jcsherin , we need to make some changes for the next upgrade
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add upgrade guide for PhysicalOptimizerRule::optimize_plan [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #19030:
URL: https://github.com/apache/datafusion/pull/19030#discussion_r2579097445
##########
docs/source/library-user-guide/upgrading.md:
##########
@@ -332,6 +332,102 @@ let config = FileScanConfigBuilder::new(url, source)
**Handling projections in `FileSource`:**
+### `PhysicalOptimizerRule::optimize` deprecated in favor of `optimize_plan`
+
+The `PhysicalOptimizerRule` trait has been updated to provide optimizer rules with access to broader session context. A new method `optimize_plan` has been added that accepts an `OptimizerContext` parameter, and the existing `optimize` method has been deprecated.
+
+**Who is affected:**
+
+- Users who have implemented custom `PhysicalOptimizerRule` implementations
+
+**Breaking changes:**
+
+1. **New `optimize_plan` method**: This is the preferred method for implementing optimization rules. It provides access to the full `SessionConfig` through `OptimizerContext`, rather than just `ConfigOptions`.
+
+2. **`optimize` method deprecated**: The old `optimize` method that takes `&ConfigOptions` is now deprecated and will be removed in DataFusion 54.0.0.
+
+**Migration guide:**
+
+If you have a custom `PhysicalOptimizerRule` implementation, update it to implement `optimize_plan` instead of `optimize`:
+
+**Before:**
+
+```rust
+use datafusion::physical_optimizer::PhysicalOptimizerRule;
+use datafusion_common::config::ConfigOptions;
+use datafusion_common::Result;
+use datafusion_physical_plan::ExecutionPlan;
+use std::sync::Arc;
+
+#[derive(Debug)]
+struct MyOptimizerRule;
+
+#[allow(deprecated)]
+impl PhysicalOptimizerRule for MyOptimizerRule {
+ fn optimize(
+ &self,
+ plan: Arc<dyn ExecutionPlan>,
+ _config: &ConfigOptions,
+ ) -> Result<Arc<dyn ExecutionPlan>> {
+ // Use config.optimizer, config.execution, etc.
+ // ... optimization logic ...
+ Ok(plan)
+ }
+
+ fn name(&self) -> &str {
+ "my_optimizer_rule"
+ }
+
+ fn schema_check(&self) -> bool {
+ true
+ }
+}
+```
+
+**After:**
+
+```rust
+use datafusion::physical_optimizer::{OptimizerContext, PhysicalOptimizerRule};
+use datafusion_common::Result;
+use datafusion_physical_plan::ExecutionPlan;
+use std::sync::Arc;
+
+#[derive(Debug)]
+struct MyOptimizerRule;
+
+impl PhysicalOptimizerRule for MyOptimizerRule {
+ fn optimize_plan(
+ &self,
+ plan: Arc<dyn ExecutionPlan>,
+ context: &OptimizerContext,
+ ) -> Result<Arc<dyn ExecutionPlan>> {
+ // Access ConfigOptions through session_config
+ let _config = context.session_config().options();
+ // Or access extensions through session_config
+ let _extensions = context.session_config().extensions();
+ // ... optimization logic ...
+ Ok(plan)
+ }
+
+ fn name(&self) -> &str {
+ "my_optimizer_rule"
+ }
+
+ fn schema_check(&self) -> bool {
+ true
+ }
+}
+```
+
+**What is `OptimizerContext`?**
+
+`OptimizerContext` is a new struct that provides context during physical plan optimization, similar to how `TaskContext` provides context during execution. It wraps `SessionConfig`, giving optimizer rules access to:
+
+- Configuration options via `context.session_config().options()`
+- Session extensions via `context.session_config().extensions()`
+
+This enables optimizer rules to access custom extensions registered with the session, which was not possible with the old `&ConfigOptions` parameter.
+
Review Comment:
I think we just made something that was private public - is that really worth mentioning?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add upgrade guide for PhysicalOptimizerRule::optimize_plan [datafusion]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on code in PR #19030:
URL: https://github.com/apache/datafusion/pull/19030#discussion_r2578482427
##########
docs/source/library-user-guide/upgrading.md:
##########
@@ -332,6 +332,102 @@ let config = FileScanConfigBuilder::new(url, source)
**Handling projections in `FileSource`:**
+### `PhysicalOptimizerRule::optimize` deprecated in favor of `optimize_plan`
+
+The `PhysicalOptimizerRule` trait has been updated to provide optimizer rules with access to broader session context. A new method `optimize_plan` has been added that accepts an `OptimizerContext` parameter, and the existing `optimize` method has been deprecated.
+
+**Who is affected:**
+
+- Users who have implemented custom `PhysicalOptimizerRule` implementations
+
+**Breaking changes:**
+
+1. **New `optimize_plan` method**: This is the preferred method for implementing optimization rules. It provides access to the full `SessionConfig` through `OptimizerContext`, rather than just `ConfigOptions`.
+
+2. **`optimize` method deprecated**: The old `optimize` method that takes `&ConfigOptions` is now deprecated and will be removed in DataFusion 54.0.0.
+
+**Migration guide:**
+
+If you have a custom `PhysicalOptimizerRule` implementation, update it to implement `optimize_plan` instead of `optimize`:
+
+**Before:**
+
+```rust
+use datafusion::physical_optimizer::PhysicalOptimizerRule;
+use datafusion_common::config::ConfigOptions;
+use datafusion_common::Result;
+use datafusion_physical_plan::ExecutionPlan;
+use std::sync::Arc;
+
+#[derive(Debug)]
+struct MyOptimizerRule;
+
+#[allow(deprecated)]
+impl PhysicalOptimizerRule for MyOptimizerRule {
+ fn optimize(
+ &self,
+ plan: Arc<dyn ExecutionPlan>,
+ _config: &ConfigOptions,
+ ) -> Result<Arc<dyn ExecutionPlan>> {
+ // Use config.optimizer, config.execution, etc.
+ // ... optimization logic ...
+ Ok(plan)
+ }
+
+ fn name(&self) -> &str {
+ "my_optimizer_rule"
+ }
+
+ fn schema_check(&self) -> bool {
+ true
+ }
+}
+```
+
+**After:**
+
+```rust
+use datafusion::physical_optimizer::{OptimizerContext, PhysicalOptimizerRule};
+use datafusion_common::Result;
+use datafusion_physical_plan::ExecutionPlan;
+use std::sync::Arc;
+
+#[derive(Debug)]
+struct MyOptimizerRule;
+
+impl PhysicalOptimizerRule for MyOptimizerRule {
+ fn optimize_plan(
+ &self,
+ plan: Arc<dyn ExecutionPlan>,
+ context: &OptimizerContext,
+ ) -> Result<Arc<dyn ExecutionPlan>> {
+ // Access ConfigOptions through session_config
+ let _config = context.session_config().options();
+ // Or access extensions through session_config
+ let _extensions = context.session_config().extensions();
+ // ... optimization logic ...
+ Ok(plan)
+ }
+
+ fn name(&self) -> &str {
+ "my_optimizer_rule"
+ }
+
+ fn schema_check(&self) -> bool {
+ true
+ }
+}
+```
+
+**What is `OptimizerContext`?**
+
+`OptimizerContext` is a new struct that provides context during physical plan optimization, similar to how `TaskContext` provides context during execution. It wraps `SessionConfig`, giving optimizer rules access to:
+
+- Configuration options via `context.session_config().options()`
+- Session extensions via `context.session_config().extensions()`
+
+This enables optimizer rules to access custom extensions registered with the session, which was not possible with the old `&ConfigOptions` parameter.
+
Review Comment:
Maybe change to extension should be mentioned as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add upgrade guide for PhysicalOptimizerRule::optimize_plan [datafusion]
Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 commented on code in PR #19030:
URL: https://github.com/apache/datafusion/pull/19030#discussion_r2579437145
##########
docs/source/library-user-guide/upgrading.md:
##########
@@ -332,6 +332,102 @@ let config = FileScanConfigBuilder::new(url, source)
**Handling projections in `FileSource`:**
+### `PhysicalOptimizerRule::optimize` deprecated in favor of `optimize_plan`
+
+The `PhysicalOptimizerRule` trait has been updated to provide optimizer rules with access to broader session context. A new method `optimize_plan` has been added that accepts an `OptimizerContext` parameter, and the existing `optimize` method has been deprecated.
+
+**Who is affected:**
+
+- Users who have implemented custom `PhysicalOptimizerRule` implementations
+
+**Breaking changes:**
+
+1. **New `optimize_plan` method**: This is the preferred method for implementing optimization rules. It provides access to the full `SessionConfig` through `OptimizerContext`, rather than just `ConfigOptions`.
+
+2. **`optimize` method deprecated**: The old `optimize` method that takes `&ConfigOptions` is now deprecated and will be removed in DataFusion 54.0.0.
Review Comment:
```suggestion
2. **`optimize` method deprecated**: The old `optimize` method that takes `&ConfigOptions` is now deprecated and will be removed in DataFusion 58.0.0.
```
I think this will be released in `52.0.0`, and we have the policy to remove it after 6 major releases: https://datafusion.apache.org/contributor-guide/api-health.html#deprecation-guidelines
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add upgrade guide for PhysicalOptimizerRule::optimize_plan [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #19030:
URL: https://github.com/apache/datafusion/pull/19030#discussion_r2581643969
##########
docs/source/library-user-guide/upgrading.md:
##########
@@ -332,6 +332,102 @@ let config = FileScanConfigBuilder::new(url, source)
**Handling projections in `FileSource`:**
+### `PhysicalOptimizerRule::optimize` deprecated in favor of `optimize_plan`
+
+The `PhysicalOptimizerRule` trait has been updated to provide optimizer rules with access to broader session context. A new method `optimize_plan` has been added that accepts an `OptimizerContext` parameter, and the existing `optimize` method has been deprecated.
+
+**Who is affected:**
+
+- Users who have implemented custom `PhysicalOptimizerRule` implementations
Review Comment:
Sorry about this! At least it should be pretty mechanical and you'll have several releases...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add upgrade guide for PhysicalOptimizerRule::optimize_plan [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb merged PR #19030:
URL: https://github.com/apache/datafusion/pull/19030
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org