[GitHub] [arrow-rs] tustvold commented on issue #1626: Expose ArrowWriter row group flush in public API

[GitBox <gi...@apache.org>]

tustvold commented on issue #1626:
URL: https://github.com/apache/arrow-rs/issues/1626#issuecomment-1113803611

   I don't see any issue with exposing this, more power to the user, however, some thoughts:
   
   - I wonder if you could just set the max row group size smaller if you want greater row group granularity
   - For compressible data, more row groups will likely lead to larger files, which might actually be slower to read
   - Similar to the above, the reader is designed to amortise per-row group costs over many rows. This works less well with smaller row groups
   - It is possible to prune at a more granular level, it just hasn't been implemented yet -  https://github.com/apache/arrow-rs/issues/1191


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org