Re: [I] Allow per-column parquet dictionary page size limit [arrow-rs]

["alamb (via GitHub)" <gi...@apache.org>]

alamb commented on issue #7723:
URL: https://github.com/apache/arrow-rs/issues/7723#issuecomment-3025226729

   That does sound like a good way to optimize dictionary encoding. 
   
   One potential downside is that doing so  it likely consume significantly more resources on write (niavely you would have to buffer the entire unencoded dataset, do the analysis on it, update the parquet writer settings, and then write it)
   
   However, I actually think it would be very interesting to try these high-resource requirement algorithms as a secondary "rewrite" / "optimization" pass -- most of the disaggregated database systems that work object store have some sort of compaction process anyways . And we can leverage the disaggregated architecture to do this heavyweight optimization on different machines than are taking reads or writes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org