You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "ccl125 (via GitHub)" <gi...@apache.org> on 2025/01/13 15:54:50 UTC

Re: [I] High Memory Usage and Long GC Times When Writing Parquet Files [parquet-java]

ccl125 commented on issue #3102:
URL: https://github.com/apache/parquet-java/issues/3102#issuecomment-2587505948

   > In general, dictionary encoding consumes a lot of memory due to buffering all entries. So yes, `withDictionaryEncoding(false)` is the right approach to reduce the memory footprint in your case. For the resulting file size and read performance, it depends on the data distribution or repetition.
   > 
   > > Each file has a fixed 500 rows
   > 
   > I'd say Parquet is not designed for small data in which case the metadata overhead is non-trivial. It is more suitable for 100,000+ rows of data to enjoy the columnar encoding and compression.
   
   Yes, in our business scenario, we split the total sample into multiple Parquet files, each with a fixed 500 rows but with varying column counts. When the number of columns is high (over 30,000), we encounter GC issues lasting over 1 minute. I modified the configuration to use dictionary encoding only for BINARY and BOOLEAN type columns, while setting withDictionaryEncoding(false) for other column types. After this modification, the GC time improved significantly, changing from minutes to normal millisecond levels.
   
   However, I encountered another issue: after setting withDictionaryEncoding(false), the size of all generated Parquet files increased substantially. For a task with 800,000 rows and 30,000+ columns, the total file size grew from around 20GB to 90GB. Our business requirements limit the maximum size to 50GB. To address this issue, I discovered that ParquetFileWriter doesn't configure file compression by default. After implementing builder.withCompressionCodec(CompressionCodecName.SNAPPY) for file compression, the file size reduced to around 30GB, which meets our business requirements while also solving the GC issue.
   
   However, we still occasionally encounter cases where file sizes exceed 50GB, which didn't happen before implementing withDictionaryEncoding(false). It seems that builder.withCompressionCodec(CompressionCodecName.SNAPPY) sometimes doesn't provide compression as effective as the original approach (without configuring withDictionaryEncoding(false) and without builder.withCompressionCodec). I suspect the compression ratio might be dependent on the file content.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org