You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Planet-X (via GitHub)" <gi...@apache.org> on 2026/08/06 14:02:24 UTC

[I] Core: Discard changes when suppressing historical snapshots in CatalogHandlers [iceberg]

Planet-X opened a new issue, #17538:
URL: https://github.com/apache/iceberg/issues/17538

   ### Apache Iceberg version
   
   1.11.0 (latest release)
   
   ### Query engine
   
   None
   
   ### Please describe the bug 🐞
   
   ## Behavior
   In `snapshot-loading-mode=refs`, `CatalogHandlers.loadTable` fails with `IllegalArgumentException: Cannot set metadata location with changes to table metadata: 1 changes` when serving `GET .../tables/{table}?snapshots=refs` for a table that has a statistics file (or partition statistics file) attached to a historical (unreferenced) snapshot.
   
   ## Cause
   The `REFS` branch builds the response metadata like this ([CatalogHandlers.java#L526-L531](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/rest/CatalogHandlers.java#L526-L531)):
   
   ```java
   metadata =
       TableMetadata.buildFrom(loadedMetadata)
           .withMetadataLocation(loadedMetadata.metadataFileLocation())
           .suppressHistoricalSnapshots()
           .build();
   ```
   
   `suppressHistoricalSnapshots()` does not record `RemoveSnapshots` changes, but it removes the suppressed snapshots' statistics via `removeStatistics(...)` / `removePartitionStatistics(...)`, which do record `MetadataUpdate.RemoveStatistics` / `RemovePartitionStatistics` changes ([TableMetadata.java, rewriteSnapshotsInternal](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/TableMetadata.java#L1442-L1454)). `build()` then rejects the combination of pending changes and a set metadata location.
   
   The client-side equivalent in `RESTSessionCatalog.loadTable` already prevents this with [by calling `.discardChanges()`](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java#L553) when rebuilding metadata; `CatalogHandlers` performs the change-recording suppression without that measure.
   
   ## To Reproduce
   
   E.g. via `RESTCatalogAdapter` with `snapshot-loading-mode=refs` on the client:
   
   1. Create a table, commit snapshot A, attach a `StatisticsFile` to A (e.g. `UpdateStatistics`).
   2. Commit snapshot B so A becomes historical (only reachable via parent chain, not via refs).
   3. `loadTable` with `snapshots=refs` → 500 / `IllegalArgumentException: Cannot set metadata location with changes to table metadata: 1 changes`.
   
   Encountered in practice testing Trino's REST catalog against the in-memory test server with `snapshot-loading-mode=refs`: Trino writes statistics on INSERT by default, so the first `loadTable` after a stats-bearing snapshot becomes historical reliably fails.
   
   ## Suggested Fix
   Add `.discardChanges()` to the builder chain in `CatalogHandlers.loadTable` (matching the client-side wrapper), e.g.
   
   ```java
   TableMetadata.buildFrom(loadedMetadata)
       .withMetadataLocation(loadedMetadata.metadataFileLocation())
       .suppressHistoricalSnapshots()
       .discardChanges()
       .build();
   ```
   
   This would be the minimal fix, keeping the general behavior as is. Alternatively, the `suppress`-parameter that already prevents `MetadataUpdate.RemoveSnapshots` changes from being created [here](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/TableMetadata.java#L1449-L1450) could be incorporated into `removeStatistics()` and `removePartitionStatistics()` to prevent change creation there at the root.
   
   ## AI assistance
   Found this while working on a trino patch. **This root-cause analysis was AI-assisted.** I'm quite confident this is an actual bug and checked this in detail before opening the issue. I'll link to the related PR at trino shortly, whose test triggers this behavior.
   
   ### Willingness to contribute
   
   - [x] I can contribute a fix for this bug independently
   - [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
   - [ ] I cannot contribute a fix for this bug at this time


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Core: Discard changes when suppressing historical snapshots in CatalogHandlers [iceberg]

Posted by "Planet-X (via GitHub)" <gi...@apache.org>.
Planet-X commented on issue #17538:
URL: https://github.com/apache/iceberg/issues/17538#issuecomment-5206141638

   Related trino PR that triggered the issue can be found here: https://github.com/trinodb/trino/pull/30612
   This is the test that triggers the issue, where it is mitigated by not writing statistics: https://github.com/Planet-X/trino/blob/1602f911f9a7db96c95474aaf6228483c4a0ecbb/plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/catalog/rest/TestIcebergRestCatalogSnapshotLoadingMode.java#L72-L74  
   It fails without the `iceberg.extended-statistics.collect-on-write`-property.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org