You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "yangshangqing95 (via GitHub)" <gi...@apache.org> on 2026/08/13 14:33:18 UTC
[I] Parquet: notNaN filter throws NPE for columns missing from files [iceberg]
yangshangqing95 opened a new issue, #17632:
URL: https://github.com/apache/iceberg/issues/17632
### Apache Iceberg version
1.11.0 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
When a floating-point column with an initial default is added to an Iceberg table, querying older Parquet files with a `notNaN` predicate can throw a `NullPointerException`.
This can be reproduced in Spark Shell by writing data before adding the column:
```scala
val table = Spark3Util.loadIcebergTable(spark, tableName)
table
.updateSchema()
.addColumn(
"new_float",
Types.FloatType.get(),
Literal.of(java.lang.Float.valueOf(1.0f)))
.commit()
spark.catalog.refreshTable(tableName)
spark.sql(
s"""
|SELECT id, new_float
|FROM $tableName
|WHERE new_float <> CAST('NaN' AS FLOAT)
|""".stripMargin
).show(false)
```
The query fails with:
```text
java.lang.NullPointerException:
Cannot invoke "java.lang.Boolean.booleanValue()"
because the return value of "java.util.Map.get(Object)" is null
at ParquetDictionaryRowGroupFilter$EvalVisitor.notNaN
```
`mayContainNulls` is built from the physical columns in the Parquet row group. Because the newly added column is absent from older files, its field ID has no entry in the map.
The dictionary filter should conservatively retain the row group when null metadata for the column is unavailable.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Parquet: notNaN filter throws NPE for columns missing from files [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #17632:
URL: https://github.com/apache/iceberg/issues/17632#issuecomment-5284691630
I'm a little confused on how initial-default is related here. I think this is an issue whenever you add a new column?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Parquet: notNaN filter throws NPE for columns missing from files [iceberg]
Posted by "yangshangqing95 (via GitHub)" <gi...@apache.org>.
yangshangqing95 commented on issue #17632:
URL: https://github.com/apache/iceberg/issues/17632#issuecomment-5285154925
> I'm a little confused on how initial-default is related here. I think this is an issue whenever you add a new column?
Hi @RussellSpitzer
Yes, the underlying dictionary filter issue is not caused by the initial default itself. It can occur whenever a `FLOAT` or `DOUBLE` column exists in the current Iceberg schema but is absent from the Parquet file, and a `notNaN` predicate reaches `ParquetDictionaryRowGroupFilter`.
The initial default is relevant to the end-to-end Spark reproduction:
- Without an initial default, a newly added optional column has an implicit value of `null` for old files. Spark also pushes an `isNotNull` condition for `new_float <> NaN`, so the old file can be eliminated by the metrics filter before dictionary filtering.
- With a non-NaN initial default, old rows logically have that default value. The metrics filter cannot eliminate the file, so the `notNaN` predicate reaches the dictionary filter and exposes the missing map entry.
Therefore, adding a column alone does not necessarily produce a user visible failure. The lower level bug applies to any missing floating point column if `notNaN` reaches the dictionary filter; the initial default makes that path reachable in the Spark SQL reproduction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Parquet: notNaN filter throws NPE for columns missing from files [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer closed issue #17632: Parquet: notNaN filter throws NPE for columns missing from files
URL: https://github.com/apache/iceberg/issues/17632
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org