You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "mdibaiee (via GitHub)" <gi...@apache.org> on 2026/07/27 18:00:39 UTC
[I] null_count is omitted for large columns in parquet files [parquet-java]
mdibaiee opened a new issue, #3574:
URL: https://github.com/apache/parquet-java/issues/3574
### Describe the bug, including details regarding any error messages, version, and platform.
Currently in [ParquetMetadataConverter.java](https://github.com/apache/parquet-java/blob/7be05b4702df78ae0c0c6b44adc6b7b7af2d931f/parquet-hadoop/src/main/java/org/apache/parquet/format/converter/ParquetMetadataConverter.java), there is a guard that prevents the writing of statistics such as min/max AND null_count when the stats are larger than the max allowed size under truncation. The rationale for this makes sense for omitting min/max, however null_count can be written on the file despite the size of its content. See the code below:
https://github.com/apache/parquet-java/blob/7be05b4702df78ae0c0c6b44adc6b7b7af2d931f/parquet-hadoop/src/main/java/org/apache/parquet/format/converter/ParquetMetadataConverter.java#L800-L807
The missing `null_count` metadata sometimes causes downstream consumers of the parquet files to error. For example in Snowflake we are seeing the following kind of error:
```
non-nullable column without default has null values according to file statistics
```
### Component(s)
Core
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [I] null_count is omitted for large columns in parquet files [parquet-java]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac closed issue #3574: null_count is omitted for large columns in parquet files
URL: https://github.com/apache/parquet-java/issues/3574
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [I] null_count is omitted for large columns in parquet files [parquet-java]
Posted by "gtrettenero (via GitHub)" <gi...@apache.org>.
gtrettenero commented on issue #3574:
URL: https://github.com/apache/parquet-java/issues/3574#issuecomment-5360471819
@Fokko Hi! I was wondering if we have anyone tackling the contract change on iceberg side (if there's an issue or PR already)? I agree that emitting the null counts is a good change and should not be skipped when the the min/max exceed the cap.
Also wondering if anyone has context on why the field is in the Statistics object. I know ColumnChunkMetaData holds values like total size and total values, so I was curious why null count (which is relatively small) is kept separate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org