You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "90degs2infty (via GitHub)" <gi...@apache.org> on 2025/07/17 07:38:10 UTC
[I] `DataFusionError` leaks inner types to the user [datafusion]
90degs2infty opened a new issue, #16805:
URL: https://github.com/apache/datafusion/issues/16805
### Describe the bug
Some of `DataFusionError`'s variants leak the inner type to the user. E.g. [`DataFusionError::ObjectStore` simply wraps `object_store::Error`](https://docs.rs/datafusion/latest/datafusion/error/enum.DataFusionError.html#variant.ObjectStore) and propagates it to the user.
Please consider re-designing `DataFusionError` for at least two reasons:
1. To handle `DataFusionError::ObjectStore` the user is forced to add the otherwise transitive dependency `object_store` to their own dependencies.
2. `object_store` becomes part of `datafusion`'s public API. Whenever there is a breaking change in `object_store::Error` and `datafusion` updates `object_store` internally, this implicitly becomes a breaking change in `datafusion`'s API as well.
This also applies to other variants of `DataFusionError`.
### To Reproduce
_No response_
### Expected behavior
_No response_
### Additional context
See e.g. [mmapped.blog](https://mmapped.blog/posts/12-rust-error-handling) or [d34dl0ck.me](https://d34dl0ck.me/rust-bites-designing-error-types-in-rust-libraries/index.html) for some discussion of error types including inner type leakage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `DataFusionError` leaks inner types to the user [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on issue #16805:
URL: https://github.com/apache/datafusion/issues/16805#issuecomment-3163777346
Closing this.
Replacing with #17066
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `DataFusionError` leaks inner types to the user [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew closed issue #16805: `DataFusionError` leaks inner types to the user
URL: https://github.com/apache/datafusion/issues/16805
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `DataFusionError` leaks inner types to the user [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #16805:
URL: https://github.com/apache/datafusion/issues/16805#issuecomment-3141407372
> Thanks for raising this. DataFusionError::ObjectStore does indeed wrap object_store::Error directly, and the crate exposes several types such as ObjectStoreUrl. That means the object_store crate is effectively part of DataFusion’s public API
Yes I agree with this
Something we could do to make it easier on downstream crates would be to publically re-export the whole object store crate like we do for arrow:
https://github.com/apache/datafusion/blob/2a90ff606d697ca77c67da2ce72e40f2e67119e3/datafusion/common/src/lib.rs#L63
Something like
```rust
pub use object_store
```
That way users could just import anything from object_store like `use datafusion::object_store::ObjectStoreError`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `DataFusionError` leaks inner types to the user [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on issue #16805:
URL: https://github.com/apache/datafusion/issues/16805#issuecomment-3134720903
hi @90degs2infty ,
Thanks for raising this. `DataFusionError::ObjectStore` does indeed wrap `object_store::Error` directly, and the crate exposes several types such as `ObjectStoreUrl`. That means the `object_store` crate is effectively part of DataFusion’s public API, and users need it if they want to pattern‑match on `DataFusionError::ObjectStore` or interact with object stores directly. Since `object_store::Error` implements `std::error::Error`, one can still treat the inner error trait‑object without importing the type, but distinguishing variants or using concrete methods requires the dependency.
I think it is not necessary to 'hide' ObjectStore - (thanks to @crepererum's[ thoughts on this](https://github.com/apache/datafusion/pull/16947#issuecomment-3131816841))
What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `DataFusionError` leaks inner types to the user [datafusion]
Posted by "90degs2infty (via GitHub)" <gi...@apache.org>.
90degs2infty commented on issue #16805:
URL: https://github.com/apache/datafusion/issues/16805#issuecomment-3150877557
> What do you think?
I guess if `object_store` is treated as part of DataFusion's public API and this is a deliberate/concious decision, then exposing its types to the user is perfectly fine. As you can probably tell from the issue itself, I'm rather new to DataFusion and not super familiar with DataFusion's public API surface in general. Having just read on the design of error types, I might have been a little biased when opening the issue - sorry about that.
> Something we could do to make it easier on downstream crates would be to publically re-export the whole object store crate like we do for arrow:
This would also help users in places where DataFusion expects things from the `object_store` crate as input (e.g. [`SessionContext::register_object_store`](https://docs.rs/datafusion/latest/datafusion/execution/context/struct.SessionContext.html#method.register_object_store)).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `DataFusionError` leaks inner types to the user [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #16805:
URL: https://github.com/apache/datafusion/issues/16805#issuecomment-3113450971
Thanks for the idea @90degs2infty -- it sounds like a good idea to explore
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org