You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "linhr (via GitHub)" <gi...@apache.org> on 2025/05/07 09:38:59 UTC
[I] Injecting `ObjectStore` for storage [hudi-rs]
linhr opened a new issue, #330:
URL: https://github.com/apache/hudi-rs/issues/330
### Feature Description
Currently, a Hudi `Table` and the underlying `Storage` is constructed from the URL and storage options, and the storage options are kept in the struct. I found that `parse_url_opts` is used to determine the `ObjectStore` implementation for the URL, but this function only looks up built-in `ObjectStore` implementations from the `object_store` crate.
It would be great to support injecting `dyn ObjectStore` directly when constructing Hudi `Storage` and `Table`, since the user may have a custom `ObjectStore` implementation for custom URL schemes. Also I'm not sure if it's necessary to track storage options if the `ObjectStore` dependencies is injected from the outside.
For more context, I'm a maintainer of [Sail](https://github.com/lakehq/sail), a Rust library that offers a drop-in replacement for Apache Spark SQL and DataFrame APIs. We'd like to provide native Hudi support to our users. It would be great if we have the flexibility to integrate Hudi with the few custom `ObjectStore` implementations we have.
Thank you!
### Additional context
N/A
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Injecting `ObjectStore` for storage [hudi-rs]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on issue #330:
URL: https://github.com/apache/hudi-rs/issues/330#issuecomment-2864341367
@linhr thanks for raising this. we actually plan to adopt opendal (https://github.com/apache/hudi-rs/issues/131) for standardized storage layer access. cc @Xuanwo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Injecting `ObjectStore` for storage [hudi-rs]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on issue #330:
URL: https://github.com/apache/hudi-rs/issues/330#issuecomment-2918394083
@linhr Sure I'm open to it. I'll wait for more finalized solution from @Xuanwo since there is no rush right now to make such change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Injecting `ObjectStore` for storage [hudi-rs]
Posted by "linhr (via GitHub)" <gi...@apache.org>.
linhr commented on issue #330:
URL: https://github.com/apache/hudi-rs/issues/330#issuecomment-2865297861
Thanks @xushiyan.
Would there be a possibility to keep the object_store integration and make it more pluggable? There is a similar [discussion](https://github.com/apache/iceberg-rust/issues/172) in Iceberg that shows the desire to make object_store and OpenDAL co-exist. (And also see my [comment](https://github.com/apache/iceberg-rust/issues/172#issuecomment-2857585201) there about why supporting object_store may be needed in some cases.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org