[I] loadTable with snapshots=refs fails with 400 when a non-ref snapshot has statistics files [polaris]

["colivetree (via GitHub)" <gi...@apache.org>]

colivetree opened a new issue, #5810:
URL: https://github.com/apache/polaris/issues/5810

   ### Describe the bug
   
   `GET /v1/{prefix}/namespaces/{ns}/tables/{table}?snapshots=refs` returns HTTP 400 for any table where a snapshot outside the table's refs has a statistics file or a partition statistics file:
   
   ```
   Cannot set metadata location with changes to table metadata: N changes
   ```
   
   Trino writes a Puffin statistics file on every insert by default (`iceberg.extended-statistics.collect-on-write`). So on a Trino-written table, every historical snapshot has one, and `snapshots=refs` fails on effectively every such table. `snapshots=all` works.
   
   Cause: `IcebergCatalogHandler#filterResponseToSnapshots` (added in #5116 to keep `metadata-location` in the REFS response) does
   
   ```java
   TableMetadata.buildFrom(metadata)
       .withMetadataLocation(metadata.metadataFileLocation())
       .suppressHistoricalSnapshots()
       .build();
   ```
   
   In Iceberg 1.11 and 1.12, `suppressHistoricalSnapshots()` → `rewriteSnapshotsInternal(ids, true)` skips the `RemoveSnapshots` change for suppressed snapshots. It still calls `removeStatistics(id)` and `removePartitionStatistics(id)`, and each of those records a change whenever that snapshot had a file. `build()` then rejects the builder: `changes.isEmpty() || discardChanges || metadataLocation == null`.
   
   The same pattern in Iceberg's own `CatalogHandlers#loadTable` is tracked in apache/iceberg#17538.
   
   ### To Reproduce
   
   1. Create a table and append a few snapshots.
   2. For an older snapshot that no ref points at, commit a statistics file (`UpdateStatistics.setStatistics`), or insert through Trino with default settings.
   3. `GET .../tables/{table}?snapshots=refs`.
   
   Unit-level reproduction: add a `StatisticsFile` and a `PartitionStatisticsFile` for the historical snapshot in `IcebergCatalogHandlerTest#loadedTwoSnapshotTableMetadata` and call `filterResponseToSnapshots(response, "refs")`. It throws `IllegalArgumentException: Cannot set metadata location with changes to table metadata: 2 changes` at `IcebergCatalogHandler.filterResponseToSnapshots`.
   
   ### Actual Behavior
   
   HTTP 400. Measured against `apache/polaris:1.8.0` with pyiceberg:
   - a 5-snapshot table with per-snapshot statistics, one tag: 400 with `3 changes`;
   - a table with 1,600 snapshots and statistics on each: 400 with `1598 changes`.
   
   The same tables without statistics files return 200.
   
   ### Expected Behavior
   
   HTTP 200 with only the ref'd snapshots, their statistics, and the original `metadata-location`, as for a table without statistics.
   
   Adding `.discardChanges()` before `.build()` does this: the response is a view of an already-committed metadata file, not a new commit. With that one-line change applied to 1.8.0:
   - the same tables return 200;
   - the snapshots, refs, schema, `metadata-location` and the ref'd statistics match `snapshots=all` filtered to refs;
   - on the 1,600-snapshot table, the client receives 2 snapshots instead of 1,600 (client-side parse 29 ms → 4 ms).
   
   ### Additional context
   
   - Introduced by #5116 (fix for #4656); before it, the REFS response carried no `metadata-location` and did not hit this check.
   - Related upstream: apache/iceberg#17538.
   - We hit it while moving a pyiceberg reader to `snapshot-loading-mode=refs`, to avoid parsing 1,000–2,000 snapshots of history per table load.
   
   ### System information
   
   - Polaris: 1.8.0 (`apache/polaris:1.8.0` image); the code is unchanged on `main` at 49d2f9785 (Iceberg 1.12.0).
   - Runtime: the official container image on Linux amd64 (GKE), and Docker Desktop on macOS arm64 for the reproduction.
   - Persistence: relational-jdbc (PostgreSQL); storage GCS / local.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org