Re: [I] ListingTable silently drops paths when given many files [datafusion]

["16pierre (via GitHub)" <gi...@apache.org>]

16pierre commented on issue #18242:
URL: https://github.com/apache/datafusion/issues/18242#issuecomment-3915285585

   Even when matching error pattern [object_store::Error::NotFound](https://docs.rs/object_store/latest/object_store/enum.Error.html#variant.NotFound%5D), silent data regressions are still possible - imagine an object-store implementation with relaxed data consistency guarantees, that can return transient `NotFound` errors for files recently written. From a Datafusion client perspective, it feels reasonable to expect a failfast by default rather than a data-regression in such cases - that was our understanding of the `listing_table` implementation before https://github.com/apache/datafusion/pull/17364.
   
   I wonder if there are ways to tweak the implementation of the quality-of-life PR https://github.com/apache/datafusion/pull/17364 so that it doesn't risk major silent data regressions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org