You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "geoffreyclaude (via GitHub)" <gi...@apache.org> on 2026/05/27 09:52:48 UTC
[I] ExecutionPlan downcasting no longer supports transparent wrapper nodes [datafusion]
geoffreyclaude opened a new issue, #22557:
URL: https://github.com/apache/datafusion/issues/22557
### Describe the bug
DataFusion 54 changed `ExecutionPlan` downcasting in PR #21263 by removing `ExecutionPlan::as_any`, making `ExecutionPlan` extend `Any`, and adding inherent `dyn ExecutionPlan::is::<T>()` / `downcast_ref::<T>()` helpers.
That removes boilerplate for normal execution plan nodes, but it also removes a behavior that transparent wrapper nodes could previously implement: exposing the wrapped inner node for downcasting while keeping the wrapper as an implementation detail.
A public example is [`datafusion-tracing`](https://github.com/datafusion-contrib/datafusion-tracing). It wraps `ExecutionPlan` nodes to record spans, metrics, and optional previews. Before DataFusion 54, its wrapper could keep itself mostly invisible by implementing `as_any()` to return the inner plan’s `as_any()` for normal callers, while using crate-internal logic to identify already-wrapped nodes during its own optimizer pass.
With the new implementation:
```rust
impl dyn ExecutionPlan {
pub fn is<T: ExecutionPlan>(&self) -> bool {
(self as &dyn Any).is::<T>()
}
pub fn downcast_ref<T: ExecutionPlan>(&self) -> Option<&T> {
(self as &dyn Any).downcast_ref()
}
}
```
a wrapper can no longer influence downcasting. `is::<T>()` and `downcast_ref::<T>()` always see the concrete wrapper type, not the wrapped plan type.
### To Reproduce
Conceptually:
```rust
#[derive(Debug)]
struct TransparentWrapper {
inner: Arc<dyn ExecutionPlan>,
}
impl ExecutionPlan for TransparentWrapper {
// delegates most methods to `inner`
// This was possible before DataFusion 54:
// fn as_any(&self) -> &dyn Any {
// self.inner.as_any()
// }
}
let inner: Arc<dyn ExecutionPlan> = Arc::new(SomeExec::new(...));
let wrapped: Arc<dyn ExecutionPlan> = Arc::new(TransparentWrapper { inner });
assert!(wrapped.downcast_ref::<SomeExec>().is_some()); // worked with delegated `as_any`
assert!(wrapped.downcast_ref::<SomeExec>().is_none()); // current DataFusion 54 behavior
```
The exact wrapper may be for tracing, metrics, streaming instrumentation, caching, distribution, or other cross-cutting behavior. The common pattern is that the wrapper is intended to preserve the wrapped plan’s public identity.
### Expected behavior
There should be a supported way for wrapper `ExecutionPlan` implementations to opt into transparent type introspection, or the new behavior should be documented as an intentional breaking semantic change for wrappers.
One possible API shape would be an explicit hook such as:
```rust
trait ExecutionPlan: Any + Debug + DisplayAs + Send + Sync {
fn transparent_child(&self) -> Option<&dyn ExecutionPlan> {
None
}
}
```
Then `dyn ExecutionPlan::is::<T>()` / `downcast_ref::<T>()` could check the concrete object first and, if needed, follow the transparent child. The exact API is not important; the useful property is an upstream contract for wrappers that intentionally preserve the wrapped node’s public identity.
### Additional context
This came up while preparing `datafusion-tracing` for DataFusion 54. The crate can keep its wrapper type private in Rust API terms, but it cannot preserve the old transparent downcasting behavior without upstream support because the new `dyn ExecutionPlan` methods are inherent methods, not trait methods that a wrapper can override.
This affects code that traverses physical plans and detects specific node types via `is::<T>()` or `downcast_ref::<T>()`, including optimizers, serializers/codecs, estimators, and observability tooling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] ExecutionPlan downcasting no longer supports transparent wrapper nodes [datafusion]
Posted by "geoffreyclaude (via GitHub)" <gi...@apache.org>.
geoffreyclaude commented on issue #22557:
URL: https://github.com/apache/datafusion/issues/22557#issuecomment-4553460932
Additional context: this is not only a `datafusion-tracing` concern. At DataDog, we also have internal `ExecutionPlan` wrappers that are intended to be transparent to downstream plan inspection code. I will avoid going into implementation details here, but the pattern is the same: the wrapper delegates execution-plan behavior to an inner node while preserving the inner node’s observable type identity for callers that use `as_any()` / downcasting.
In practice, the overridable `ExecutionPlan::as_any` had become more than boilerplate: it was a customization point for wrapper nodes. Removing it and replacing it with inherent `dyn ExecutionPlan::is::<T>()` / `downcast_ref::<T>()` changes that behavior because wrappers no longer have a way to opt into transparency.
So I think the compatibility concern is broader than one public crate: transparent wrappers appear to be a real extension pattern around `ExecutionPlan`, and it would be useful for DataFusion to either support that pattern explicitly or document that downcasting now intentionally reflects only the concrete outer wrapper type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] ExecutionPlan downcasting no longer supports transparent wrapper nodes [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #22557: ExecutionPlan downcasting no longer supports transparent wrapper nodes
URL: https://github.com/apache/datafusion/issues/22557
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] ExecutionPlan downcasting no longer supports transparent wrapper nodes [datafusion]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on issue #22557:
URL: https://github.com/apache/datafusion/issues/22557#issuecomment-4554198956
I'm not opposed to this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] ExecutionPlan downcasting no longer supports transparent wrapper nodes [datafusion]
Posted by "geoffreyclaude (via GitHub)" <gi...@apache.org>.
geoffreyclaude commented on issue #22557:
URL: https://github.com/apache/datafusion/issues/22557#issuecomment-4553481700
@timsaucer As owner of PR https://github.com/apache/datafusion/pull/21263 , what do you think of the proposed `transparent_child` API hook?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org