[GitHub] [iceberg] kbendick commented on pull request #3351: Bump Parquet from 1.12.0 to 1.12.2

[GitBox <gi...@apache.org>]

kbendick commented on pull request #3351:
URL: https://github.com/apache/iceberg/pull/3351#issuecomment-950214790


   > apache/parquet-mr@apache-parquet-1.12.0...apache-parquet-1.12.2
   
   Thanks for this @nastra! I’m hesitant to include this for 0.12.1 (though it’s not up to me, I’m just release manager). But I definitely do think it’s a good idea to upgrade at least for the next major version. Particularly the closing of zstd decompressor will fix a memory leak I’ve dealt with before in reading zstd compressed files.
   
   I wonder if some of the changes to stats will affect any of our operations though? Especially things like add_files and other procedures (where changes in parquet have caused issues before).
   
   Such as:
   - Change to column stats for min/max on binary types: https://github.com/apache/parquet-mr/commit/413d1dd4552dba5a7d66647747d17ef573d201c4
   - Changes of row group size from int to long: https://github.com/apache/parquet-mr/commit/aa132b3787b20170f3554cfeac214884cfecb820
   - Changing dictionary size from int to long (probably less of a concern): https://github.com/apache/parquet-mr/commit/7597c74c55c02980021eb9e8c5345ea0d86fa62a
   
   We should see if we have tests or any invalid assumptions around binary type min / max size (and if the row group type change will be an issue). I think the row group type change to long is actually more in line with our own code (and there’s likely an unnecessary cast to `long` that can be removed, unless it was never added).
   
   Happy to help take a look at these things as time permits 🙂


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org