Re: [PR] Speed up Parquet filter pushdown v4 (Predicate evaluation cache for async_reader) [arrow-rs]

["alamb (via GitHub)" <gi...@apache.org>]

alamb commented on PR #7850:
URL: https://github.com/apache/arrow-rs/pull/7850#issuecomment-3118607666

   I thought a lot about next steps last night and here is a summary:
   1. I think caching the filter column results (this PR) is required in any solution 
   2. I don't think it is worth trying to also add the cache to the sync reader. Instead I think we should pursue a more general solution, see https://github.com/apache/arrow-rs/issues/8000)
   
   Thus I propose the following steps for this PR:
   1. Update some comments to make it clear the predicate cache only applies to the async decoder
   2. Update the code to avoid breaking API changes
   3. Break out the end to end tests + statistics into separate PR to make review tractable
   
   Once we have the end to end tests and statistics broken out, then we can get this PR ready to merge
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org