[I] Register overwrite drops live idempotency keys and lets retries undo newer updates [polaris]

["mattfaltyn (via GitHub)" <gi...@apache.org>]

mattfaltyn opened a new issue, #5789:
URL: https://github.com/apache/polaris/issues/5789

   ## Describe the bug
   
   With `polaris.idempotency.enabled=true`, overwriting a table registration replaces the table entity's internal properties and removes its live `polaris-idempotency-keys` window. This happens even when registration uses the identical metadata file and Iceberg table UUID.
   
   A retry of an already committed update can therefore execute again and undo a newer table-property update. A retry of an already committed create instead throws `AlreadyExistsException` (mapped to HTTP 409). These are two symptoms of the same lost retry state.
   
   ## To Reproduce
   
   Enable idempotency and use an internal catalog. Complete the following operations before the key TTL expires (default `PT5M`):
   
   1. Create a table.
   2. Commit an `updateTable` request setting `owner-note=older`, using idempotency key K and no table requirements.
   3. Commit a new request setting `owner-note=newer`, without a key.
   4. Register the table with `overwrite=true` and its current metadata URI. The metadata URI, table UUID and property value remain unchanged by this registration.
   5. Retry the exact request from step 2 with the same payload and key K.
   
   For the create symptom: create a table with key K, re-register its identical current metadata URI with `overwrite=true`, then retry the original create request with the same payload/key before TTL expiry.
   
   ## Actual Behavior
   
   The update retry commits `owner-note=older` again and advances the metadata version, losing the newer committed property value. The create retry throws `AlreadyExistsException`.
   
   The two cases were reproduced twice against `5cf029e0e8be89ee342061ba90611fac45fe6fa3`, using the existing in-process `TestServices` harness with local FILE storage. In each run, both new regression expectations failed and both existing same-key create/update replay controls passed. The affected overwrite code is unchanged in current main `33e0666fda55fd657b4b9bcb43f0e9c878781a49`.
   
   ## Expected Behavior
   
   Registration that retains the Iceberg table identity must retain existing retry keys with their original expiry. A same-key/same-payload retry within TTL should replay current table state instead of reapplying the mutation or returning a false create conflict.
   
   Intentional registration of a different Iceberg UUID must remain a table replacement and must not inherit keys belonging to the previous identity. Repairing a registration whose old metadata file is missing must continue to work without reading that old file.
   
   ## Additional context
   
   `LocalIcebergCatalog.overwriteRegisteredTable()` calls `setInternalProperties(storedProperties)` with metadata-derived properties only. Normal commits already carry the existing idempotency window forward in `idempotencyInternalProperties()`.
   
   Related prior work: #5037 and #5122 establish ordinary-commit preservation; #5551 supports overwrite repair when old metadata is missing. Open #5328 changes cache handling in the same method but leaves this key loss unchanged.
   
   This affects opt-in idempotency; it is disabled by default. The reproduction demonstrates lost table-property updates, not row corruption. No API, schema, encoding or new dependency is needed for the fix.
   
   ## System information
   
   - OS/architecture: macOS 27.0.1, arm64.
   - Polaris: reproduced on main `5cf029e0e8be89ee342061ba90611fac45fe6fa3`; affected code still present at current main `33e0666fda55fd657b4b9bcb43f0e9c878781a49`. Releases 1.7.0 and 1.8.0 also contain this overwrite path by source inspection.
   - Runtime/build: Java 21.0.12.1, Gradle wrapper 9.7.1.
   - Storage/setup: internal Polaris catalog, temporary local FILE storage and in-process test services; no external object store required.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org