You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2025/08/16 13:21:51 UTC
[I] [EPIC] ListingTable improvements [datafusion]
alamb opened a new issue, #17214:
URL: https://github.com/apache/datafusion/issues/17214
### Is your feature request related to a problem or challenge?
_No response_
### Describe the solution you'd like
_No response_
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] ListingTable improvements [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #17214:
URL: https://github.com/apache/datafusion/issues/17214#issuecomment-3312865807
I added myself and @BlakeOrth as people working on this epic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] ListingTable object store usage improvements [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #17214:
URL: https://github.com/apache/datafusion/issues/17214#issuecomment-3724259908
From my perspective, we are now done enough with this epic to claim success, so I am closing it out!
There is some additional work listed here but we can handle them as a follow on
- [ ] https://github.com/apache/datafusion/issues/19217
- [ ] https://github.com/apache/datafusion/issues/19052
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] ListingTable object store caching impovements [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #17214:
URL: https://github.com/apache/datafusion/issues/17214#issuecomment-3360257342
A brief update here is that I think the next steps for this project are:
1. Polish up https://github.com/apache/datafusion/issues/17207 so we can test / observe the improvements
2. Complete https://github.com/apache/datafusion/issues/17211
@BlakeOrth has prototype PRs for both, and they need some help polishing / testing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] ListingTable object store usage improvements [datafusion]
Posted by "BlakeOrth (via GitHub)" <gi...@apache.org>.
BlakeOrth commented on issue #17214:
URL: https://github.com/apache/datafusion/issues/17214#issuecomment-3756567968
@alamb (and everyone else who helped contribute to this effort!) I just wanted to quickly report back and issue a sincere "thank you" to all the work that went into designing, writing, reviewing and testing the changes this epic implemented. Here's some very "real world" results that show the difference this has made for our application that leverages DataFusion. I upgraded to DataFusion 52.0.0 at the end of the day on Jan 12, and the difference in our request access pattern to our backing storage has been very obvious!
<img width="1336" height="1533" alt="Image" src="https://github.com/user-attachments/assets/b20f36fd-b0e0-4d20-813c-64dc822eafe5" />
Even better, the Y-axis for this data is "request rate" (Sorry to omit the actual rates, I don't want to leak too many details of our proprietary software) which suggests that we have functionally replaced "waiting on list results" with "reading actual data"!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] ListingTable object store caching impovements [datafusion]
Posted by "BlakeOrth (via GitHub)" <gi...@apache.org>.
BlakeOrth commented on issue #17214:
URL: https://github.com/apache/datafusion/issues/17214#issuecomment-3362031834
@alamb I haven't pushed and opened a PR for #17211 yet, but I would be happy to do so if we want to start getting some feedback on the implementation. I actually think that code is more or less ready to go at this point, but I don't want to overload anyone on reviews when we still need #17207 to test/validate those changes.
I will comment in #17207 and cc you there so we can choose how to continue moving forward with that effort.
This all comes with a subtle caveat that I am currently focusing on some other efforts that unfortunately skyrocketed to the top of my priorities, so while I am more than happy to continue contributing to this effort I will be unlikely to do so for about the next week or so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] ListingTable object store usage improvements [datafusion]
Posted by "BlakeOrth (via GitHub)" <gi...@apache.org>.
BlakeOrth commented on issue #17214:
URL: https://github.com/apache/datafusion/issues/17214#issuecomment-3639444205
@alamb Can you add this ticket to the enhancements checklist for us?
- https://github.com/apache/datafusion/issues/19273
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] ListingTable object store usage improvements [datafusion]
Posted by "m09526 (via GitHub)" <gi...@apache.org>.
m09526 commented on issue #17214:
URL: https://github.com/apache/datafusion/issues/17214#issuecomment-3491527035
If ListingTable is undergoing some work, please could this bug be looked at: https://github.com/apache/datafusion/issues/15964
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] ListingTable object store usage improvements [datafusion]
Posted by "BlakeOrth (via GitHub)" <gi...@apache.org>.
BlakeOrth commented on issue #17214:
URL: https://github.com/apache/datafusion/issues/17214#issuecomment-3554313032
@alamb Can you add this ticket to the enhancements checklist for us?
- https://github.com/apache/datafusion/issues/18827
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] ListingTable object store usage improvements [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #17214: [EPIC] ListingTable object store usage improvements
URL: https://github.com/apache/datafusion/issues/17214
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] ListingTable object store caching impovements [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #17214:
URL: https://github.com/apache/datafusion/issues/17214#issuecomment-3366544969
Thanks @BlakeOrth -- what I am secretly (well not so secretly anymore) is that we have this ready for DataFusion 51 which won't be until Nov, so we have some time
- #17558
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] ListingTable object store usage improvements [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #17214:
URL: https://github.com/apache/datafusion/issues/17214#issuecomment-3759707380
> I upgraded to DataFusion 52.0.0 at the end of the day on Jan 12, and the difference in our request access pattern to our backing storage has been very obvious!
Thank you @BlakeOrth -- this is great to see. BTW I also think much of this work results in the improvements we are seeing in ClickBench 52, see https://github.com/apache/datafusion/issues/19826#issuecomment-3753896697
(thanks to @nuno-faria @jizezhang and @Yuvraj-cyborg as well 🙏 )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] ListingTable object store usage improvements [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #17214:
URL: https://github.com/apache/datafusion/issues/17214#issuecomment-3493700329
> If ListingTable is undergoing some work, please could this bug be looked at: [#15964](https://github.com/apache/datafusion/issues/15964)? ListingTable assumes objects come from a singular object store implementation.
Thanks @m09526 -- I would be happy to help review a PR for such an improvement
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org