You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Dandandan (via GitHub)" <gi...@apache.org> on 2026/01/25 07:22:17 UTC
[I] Save file descriptors for LocalFileSystem object storage [datafusion]
Dandandan opened a new issue, #19983:
URL: https://github.com/apache/datafusion/issues/19983
### Is your feature request related to a problem or challenge?
The ObjectStore implementation is stateless and doesn't allow for passing `File` descriptors.
This means for every request it's doing a `File::open` (and close) call, which creates overhead (mainly when doing a lot of small requests, e.g. with predicate pushdown enabled).
### Describe the solution you'd like
Implement an object store implementation for DataFusion that opens / reuses the `File`s for each path.
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Reuse file descriptors for LocalFileSystem object storage [datafusion]
Posted by "AdamGS (via GitHub)" <gi...@apache.org>.
AdamGS commented on issue #19983:
URL: https://github.com/apache/datafusion/issues/19983#issuecomment-3804419944
glad to take this if you didn't get started
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Reuse file descriptors for LocalFileSystem object storage [datafusion]
Posted by "AdamGS (via GitHub)" <gi...@apache.org>.
AdamGS commented on issue #19983:
URL: https://github.com/apache/datafusion/issues/19983#issuecomment-3810142066
I do remember it being an issue in the past with Vortex, so I'm hoping I can see a difference
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Reuse file descriptors for LocalFileSystem object storage [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on issue #19983:
URL: https://github.com/apache/datafusion/issues/19983#issuecomment-3806951259
Sure! I did vibe-code something simple to test the performance - on the Clickbench benchmark couldn't see a dramatic performance, but I didn't yet check other areas like filter pushdown where I guess it could matter more (i.e. many small requests).
I do sometimes see `open` or `lstat` or (metadata) or seek in CPU profiles, most of the time it's only taking ~0.1% / 1 sample of total time, the most time is always in read and/or allocating the `Vec` to write to.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Reuse file descriptors for LocalFileSystem object storage [datafusion]
Posted by "alchemist51 (via GitHub)" <gi...@apache.org>.
alchemist51 commented on issue #19983:
URL: https://github.com/apache/datafusion/issues/19983#issuecomment-3816819095
We were facing file discriptor opening taking time when we were trying to open the file multiple times for smaller partitions. Using a Cached file reader did help us get over this bottleneck. Tagging @bharath-techie who was doing the experimentation on this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Reuse file descriptors for LocalFileSystem object storage [datafusion]
Posted by "AdamGS (via GitHub)" <gi...@apache.org>.
AdamGS commented on issue #19983:
URL: https://github.com/apache/datafusion/issues/19983#issuecomment-3817748140
@Dandandan Is there a reason to have it as part of DataFusion and not part of `object-store`? As I'm working through https://github.com/apache/arrow-rs-object-store/pull/628 it seems like there is already a lot of useful code there, and making the change there makes its impact spread wider.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Reuse file descriptors for LocalFileSystem object storage [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on issue #19983:
URL: https://github.com/apache/datafusion/issues/19983#issuecomment-3818292463
Ok! No if object store is willing to take this it might be better (and help other use cases as well potentially)!
I created the issue here as it's easier to experiment with DataFusion without changing the upstream code / interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org