You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "bobmerevel (via GitHub)" <gi...@apache.org> on 2026/06/16 19:17:38 UTC
[I] ExpireSnapshotsSparkAction fails with NullPointerException when TableMetadata.metadataFileLocation() is null [iceberg]
bobmerevel opened a new issue, #16838:
URL: https://github.com/apache/iceberg/issues/16838
### Apache Iceberg version
1.9.2
### Query engine
Spark
### Please describe the bug 🐞
Iceberg Version
* Apache Iceberg: 1.9.2
* Spark Runtime: iceberg-spark-runtime-3.5
* Spark Version: 3.5.7
Description
While implementing a custom Iceberg REST Catalog for our product, I encountered a NullPointerException during snapshot expiration.
The root cause turned out to be that my REST Catalog accidentally returned:
{
"metadata_location": "..."
}
instead of the expected REST field:
{
"metadata-location": "..."
}
Because of this typo, TableMetadata.metadataFileLocation() becomes null.
Interestingly, normal table reads continue to work correctly because Iceberg is still able to discover and load the current metadata through the table location.
However, after a successful remove-snapshots commit, ExpireSnapshotsSparkAction reloads the table metadata from the catalog and later constructs a static table using:
protected Table newStaticTable(TableMetadata metadata, FileIO io) {
StaticTableOperations ops = new StaticTableOperations(metadata, io);
return new BaseTable(ops, metadata.metadataFileLocation());
}
Since metadata.metadataFileLocation() is null, the resulting BaseTable is created with a null metadata location.
Later, during file expiration (fileDS()), this causes an unexpected NullPointerException inside the Spark job execution.
Relevant code path
private Dataset<FileInfo> fileDS(TableMetadata metadata, Set<Long> snapshotIds) {
Table staticTable = this.newStaticTable(metadata, this.table.io());
return this.contentFileDS(staticTable, snapshotIds)
.union(this.manifestDS(staticTable, snapshotIds))
.union(this.manifestListDS(staticTable, snapshotIds))
.union(this.statisticsFileDS(staticTable, snapshotIds));
}
which eventually calls:
protected Table newStaticTable(TableMetadata metadata, FileIO io) {
StaticTableOperations ops = new StaticTableOperations(metadata, io);
return new BaseTable(ops, metadata.metadataFileLocation());
}
Expected behavior
If metadata.metadataFileLocation() is required for ExpireSnapshotsSparkAction, Iceberg should fail fast with a descriptive error such as:
Table metadata file location is null.
This may indicate an invalid REST Catalog response or missing metadata-location field.
instead of continuing and eventually failing with an unrelated NullPointerException.
Actual behavior
The action proceeds until later stages and eventually fails with a NullPointerException, making the root cause difficult to identify.
Root cause
In my case, the issue was an incorrect REST Catalog response field name (metadata_location instead of metadata-location).
After fixing the REST response to return the correct metadata-location property, the issue disappeared completely.
Suggestion
It would be helpful to add an explicit null check around metadata.metadataFileLocation() in BaseSparkAction.newStaticTable() (or earlier) and throw a meaningful exception explaining that the REST Catalog returned an invalid or incomplete table metadata response.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] ExpireSnapshotsSparkAction fails with NullPointerException when TableMetadata.metadataFileLocation() is null [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #16838:
URL: https://github.com/apache/iceberg/issues/16838#issuecomment-4722727953
Sure, I don't think the NPE is really a big deal since it could only come from a custom catalog at the moment which is in violation of the specification. I know we generally write parsers and such so that they won't fail on bad input and instead we let the data go as far as it can until it hits an error like this.
If you want to add a precondition though I don't think that's a huge deal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] ExpireSnapshotsSparkAction fails with NullPointerException when TableMetadata.metadataFileLocation() is null [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #16838:
URL: https://github.com/apache/iceberg/issues/16838#issuecomment-4722640763
At that level of the code we wouldn't know if it was a bad REST response, we would only know that the field was null. I think we could put a null check in here but it is a bit counter-intuitive at the moment because it is a required field. That said we are thinking of making it optional so at that point it would make sense to have a guard in the existing code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] ExpireSnapshotsSparkAction fails with NullPointerException when TableMetadata.metadataFileLocation() is null [iceberg]
Posted by "bobmerevel (via GitHub)" <gi...@apache.org>.
bobmerevel commented on issue #16838:
URL: https://github.com/apache/iceberg/issues/16838#issuecomment-4722699372
That makes sense. I agree that at this layer the implementation cannot distinguish whether the null value originated from a REST Catalog response, another catalog implementation, or some other source.
My main concern was the current failure mode: the action eventually fails with a NullPointerException, making the actual problem difficult to diagnose.
Even if the error message is generic (for example, "Table metadata file location must not be null"), I think a fail-fast validation would still significantly improve the debugging experience for catalog implementers and users.
If metadata-location becomes optional in the future, having such a guard would become even more valuable.
@RussellSpitzer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] ExpireSnapshotsSparkAction fails with NullPointerException when TableMetadata.metadataFileLocation() is null [iceberg]
Posted by "bobmerevel (via GitHub)" <gi...@apache.org>.
bobmerevel commented on issue #16838:
URL: https://github.com/apache/iceberg/issues/16838#issuecomment-4722667824
After investigating this further, I think there is an opportunity to improve the failure mode in ExpireSnapshotsSparkAction.
In my opinion, the action could validate metadata.metadataFileLocation() before performing the commit. This would prevent a successful metadata commit followed by a failure during expired file calculation due to an invalid TableMetadata.
For example, immediately after:
TableMetadata originalMetadata = ops.current();
the action could verify that originalMetadata.metadataFileLocation() is not null and fail with a descriptive error message if it is.
Similarly, after:
TableMetadata updatedMetadata = ops.refresh();
the refreshed metadata could be validated before it is used to construct static tables.
Additionally, it may be beneficial to add a defensive validation in BaseSparkAction.newStaticTable() itself, since it currently assumes that metadata.metadataFileLocation() is always present:
protected Table newStaticTable(TableMetadata metadata, FileIO io) {
StaticTableOperations ops = new StaticTableOperations(metadata, io);
return new BaseTable(ops, metadata.metadataFileLocation());
}
At the moment, if metadataFileLocation() is null, the code eventually fails with a NullPointerException, which makes the real root cause difficult to identify.
A fail-fast validation with an explicit error message such as:
“Table metadata file location is null. This may indicate an invalid catalog implementation or an invalid REST Catalog response missing the metadata-location field.”
would make debugging much easier for catalog implementers.
This would not change the behavior of valid implementations, but it would provide a much clearer diagnostic when the catalog contract is violated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] ExpireSnapshotsSparkAction fails with NullPointerException when TableMetadata.metadataFileLocation() is null [iceberg]
Posted by "bobmerevel (via GitHub)" <gi...@apache.org>.
bobmerevel commented on issue #16838:
URL: https://github.com/apache/iceberg/issues/16838#issuecomment-4727932154
I have prepared a fix for this issue and submitted a PR implementing a fail-fast validation in `BaseSparkAction.newStaticTable()` together with a corresponding unit test.
PR: #16846
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org