[jira] [Created] (FLINK-40245) Wire parquet.page.size.row.check.min/max conf keys in FlinkParquetBuilder

["Nikita Singhvi (Jira)" <ji...@apache.org>]

Nikita Singhvi created FLINK-40245:
--------------------------------------

             Summary: Wire parquet.page.size.row.check.min/max conf keys in FlinkParquetBuilder
                 Key: FLINK-40245
                 URL: https://issues.apache.org/jira/browse/FLINK-40245
             Project: Flink
          Issue Type: Bug
          Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
            Reporter: Nikita Singhvi


*Problem:* 

FlinkParquetBuilder.createWriter() correctly reads parquet.page.size and 
parquet.block.size from conf, but never calls 
withMinRowCountForPageSizeCheck() / withMaxRowCountForPageSizeCheck().


This means parquet.page.size.row.check.min and parquet.page.size.row.check.max 
are silently ignored regardless of what values are set.

Without frequent page size checks, a single row with a large binary field can 
accumulate beyond Integer.MAX_VALUE bytes in the Parquet page buffer, causing: 
{code:java}
java.lang.OutOfMemoryError: Size of data exceeded Integer.MAX_VALUE     at org.apache.parquet.bytes.CapacityByteArrayOutputStream.addSlab     at org.apache.flink.formats.parquet.row.ParquetRowDataWriter$BinaryWriter.write  {code}
 

 

*Fix:*

Add two missing method calls in FlinkParquetBuilder.createWriter(): 
{code:java}
.withMinRowCountForPageSizeCheck(getMinRowCountForPageSizeCheck(conf)) .withMaxRowCountForPageSizeCheck(getMaxRowCountForPageSizeCheck(conf)) {code}
 

*Steps to Reproduce:*

1. Configure a Flink job writing large binary fields to Parquet
2. Set parquet.page.size.row.check.min=1 and parquet.page.size.row.check.max=1 in conf
3. Observe that the settings have no effect — Parquet still uses defaults (100, 10000)
4. With large enough binary fields, OOM occurs

 
{code:java}
 {code}
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)