[GitHub] [beam] kennknowles commented on issue #23179: [Bug]: Parquet size exploded for no apparent reason

[GitBox <gi...@apache.org>]

kennknowles commented on issue #23179:
URL: https://github.com/apache/beam/issues/23179#issuecomment-1246045232

   There are a *lot* of changes in that version range.
   
   `git log --oneline v2.35.0 ^v.2.20.0 -- sdks/java/io/parquet`
   
   In case this helps. I've deleted commits that have nothing to do with Parquet, like global formatting cleanups and stuff.
   
   ```
   2a94534a3d6 [BEAM-13157] add regression test for hadoop configuration on ParquetIO.Parse
   0bb6c1494fa [BEAM-12070] ParquetIO: use splittable reading by default
   d5cd15ab4de [BEAM-13157] support hadoop configuration on ParquetIO.Parse
   136eadc121e [BEAM-13104] ParquetIO: SplitReadFn must read the whole block
   15833448ad1 [BEAM-12165] Add support for ParquetIO Sink to specify an Avro model (#14644)
   f725953e295 [BEAM-12151] Bump Apache Parquet to 1.12.0
   3b77ff84738 [BEAM-12057] Add missing populateDisplayData methods to ParquetIO
   93d0be065e1 [BEAM-11969] Adds an option for setting row-group size in ParquetIO (#14227)
   8d0b19fa2f7 [BEAM-11972] Close all opened channels/readers on ParquetIO
   1de8ef9e0a6 [BEAM-11913] Add support for Hadoop configuration on ParquetIO
   d79cd82943c [BEAM-7929] Support column projection for Parquet Tables
   07e1e021250 [BEAM-11861] Add methods to explicitly provide coder for ParquetIO's Parse and ParseFiles (#14078)
   1a4d9a273a6 [BEAM-10961] enable strict dependency checking for sdks/java/io/parquet (#14062)
   4a0f056139c BEAM-4986: Bump to Apache Parquet 1.11.1
   39ee3ef170c [BEAM-8202] Support ParquetTable Writer
   34a8fcde615 [BEAM-11526] Cleanup Code for Schema support (#13646)
   1a617bed40b Add Beam schema support for ParquetIO reads.
   ff5a094be93 Consistency improvements and other fixes
   a7301b75179 [BEAM-11527] Add builder parameter to allow user defined Hadoop ReadSupport flags in Hadoop Configuration.
   0d344ba3b7a Make passthroughfunction singleton by making the class  with private constructor.
   6faeda36070 Make GenericRecordPassthroughFn singleton
   de6d2686e81 improve backward compatibility by creating separate `Parse<T>` and `ParseFiles<T>` implementation for supporting files with unknown schema.
   75e167ec34d [BEAM-11460] Implement reading unknown schema files for ParquetIO
   b0ac11db4b2 [BEAM-8615 BEAM-8569 BEAM-7937] Add hadoop 3 compatibility tests
   fb23d3f3707 [BEAM-8876] Run hadoop tests with different versions and enable HadoopFormatIT test in Java Postcommit
   f9492fb9ef3 [BEAM-8719 BEAM-8768 BEAM-8769 BEAM-8770 BEAM-8771] Update minor hadoop dependency
   a334bac48f8 [BEAM-7925]add schema encoder
   d5944974fbe [Beam-4379] Make ParquetIO read splittable (#12223)
   bd915c7af47 Implemented SchemaIO and SchemaIOProvider for Parquet
   4fb27bcc19d [BEAM-10284] Remove hadoop from ParquetIO.Sink public API
   9cabeae142f [BEAM-10284] Add option to pass configuration into ParquetIO.Sink
   ```
   
   It would help a lot of you could find a smaller version range.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org