You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "EmilyMatt (via GitHub)" <gi...@apache.org> on 2025/07/31 08:12:09 UTC
[I] request: Connect file groups in datasource to their object store [datafusion]
EmilyMatt opened a new issue, #16991:
URL: https://github.com/apache/datafusion/issues/16991
### Is your feature request related to a problem or challenge?
Currently, a filegroup will contain partitioned files, which when created from a path, will have the scheme and authority stripped from them in the parsing.
The datasourceexec itself only supports a single objectstoreurl/objectstore
therefore, if I create a filegroup from the following partitioned files
"s3://my-bucket/file" and "file:///my-local-file", I would get a file not found error from the object store, since the datasource itself can only be created with a single object store, either for the s3 or file.
For my use case, my partitioner found that the best partitioning scheme was to load files from those two locations in one partition, and the current exec doesn't let me do that.
### Describe the solution you'd like
For the datasource exec to maintain a map of ObjectStoreUrl -> ObjectStore, or just a set of registered ObjectStoreUrls which can then be used to fetch the object store from the task context.
And for the PartitionedFile itself to maintain an ObjectStoreUrl, as well as the file Path, so the right object store be selected for the file opener.
Additionally, when the datasource exec is created, it should verify that all the partitionedfiles's objectstore is available for the operator.
### Describe alternatives you've considered
Currently I need to either manually fix the partitioning, which is not great for my flow, or manually create a datasourceexec for each file/store and execute it, then have an operator to manually inject each record batch into the rest of the streams, but this ruins the ease-of-use for the datasourceexec's partitioning mechanism which would automatically generate the stream for each partition.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] request: Connect file groups in datasource to their object store [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #16991:
URL: https://github.com/apache/datafusion/issues/16991#issuecomment-3141439401
Thank you for bringing this up @EmilyMatt
I think there is currently an assumption in the `ListingTable` (and all the way down to the DataSourceExec) that all the files are read from the same underlying `ObjectStore` instance which I think is the root cause of the challenge
It would be pretty disruptive (aka a big and complicated PR) I think to try and wire in support for multiple object stores
Another idea I had work for you could be to make a "virtual" ObjectStore wrapper
Something like this (a sketch, not compiling):
```rust
struct VirtualObjectStore {
// Maps the first element of each path to a different ObjectStore
stores: HashMap<String, Arc<dyn ObjectStore>>.
}
impl ObjectStore for VirtualObjectStore {
// delegates to the correct store
// for example,
// get '/store1/my_data/1.parquet'
// would be mapped to a get call to `store1` at path `/my_data/1.parquet`
fn get(&self, path: Path) -> ... {
let store = path[0]; // first part of the path
let real_path = path[1..]; // remainder of the path
// delegate to the inner store
self.stores.get(store).get(real_path)
}
...
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] request: Connect file groups in datasource to their object store [datafusion]
Posted by "EmilyMatt (via GitHub)" <gi...@apache.org>.
EmilyMatt commented on issue #16991:
URL: https://github.com/apache/datafusion/issues/16991#issuecomment-3149437149
> Thank you for bringing this up [@EmilyMatt](https://github.com/EmilyMatt)
>
> I think there is currently an assumption in the `ListingTable` (and all the way down to the DataSourceExec) that all the files are read from the same underlying `ObjectStore` instance which I think is the root cause of the challenge
>
> It would be pretty disruptive (aka a big and complicated PR) I think to try and wire in support for multiple object stores
>
> Another idea I had work for you could be to make a "virtual" ObjectStore wrapper
>
> Something like this (a sketch, not compiling):
>
> struct VirtualObjectStore {
> // Maps the first element of each path to a different ObjectStore
> stores: HashMap<String, Arc<dyn ObjectStore>>.
> }
>
> impl ObjectStore for VirtualObjectStore {
> // delegates to the correct store
> // for example,
> // get '/store1/my_data/1.parquet'
> // would be mapped to a get call to `store1` at path `/my_data/1.parquet`
> fn get(&self, path: Path) -> ... {
> let store = path[0]; // first part of the path
> let real_path = path[1..]; // remainder of the path
> // delegate to the inner store
> self.stores.get(store).get(real_path)
> }
> ...
> }
Thank you for the suggestion, this is similar to what I'm working on :)
It just requires some micromanagement, thought i'd raise a flag on this^
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org