You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "smaheshwar-pltr (via GitHub)" <gi...@apache.org> on 2026/09/07 03:57:51 UTC
[I] Concurrent commits through one `Table` can lose manifest-list encryption keys [iceberg]
smaheshwar-pltr opened a new issue, #17990:
URL: https://github.com/apache/iceberg/issues/17990
### Apache Iceberg version
1.11.0 (latest release)
### Query engine
None
### Please describe the bug 🐞
Concurrent commits through one shared, encrypted `Table` can commit a snapshot without the wrapped key required to read its manifest list. The commit succeeds, but the snapshot's key is lost, leaving the table unreadable to users and seemingly lost.
See https://github.com/apache/iceberg/pull/17982 for a reproduction, the user-facing error will be something like:
```
java.util.concurrent.ExecutionException: java.lang.NullPointerException:
Cannot invoke "org.apache.iceberg.encryption.EncryptedKey.encryptedKeyMetadata()"
because "manifestListKey" is null
Caused by: java.lang.NullPointerException
at org.apache.iceberg.encryption.EncryptionUtil.decryptManifestListKeyMetadata(...)
at org.apache.iceberg.BaseSnapshot.allManifests(...)
at org.apache.iceberg.FastAppend.apply(...)
at org.apache.iceberg.SnapshotProducer.commit(...)
```
This situation surfaces in https://github.com/apache/iceberg/issues/17989, but also more explicitly in https://github.com/apache/iceberg/blob/main/hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveTableConcurrency.java.
The root cause is that https://github.com/apache/iceberg/blob/main/hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java has race conditions which, for encryption, can lead to key loss. The issue here is that a writer can write a snapshot, mutating its encryption manager, when a concurrent refresh from another writer "installs" a different manager before the first writer's commit, losing the first writer's snapshot keys. Concretely, with two writers:
1. Writer A refreshes the table, constructs its manifest-list writer with encryption manager A, and blocks while that manager unwraps its key-encryption key before publishing the new manifest-list key.
2. Writer B refreshes the same table operations, "installs" encryption manager B, constructs its writer, and blocks at the same point.
3. Writer A resumes and publishes the new manifest-list key to manager A.
4. Writer A's Hive commit copies encryption keys from *manager B*.
5. Writer A's snapshot commits with a manifest-list key ID, but its key is absent from table metadata.
6. Writer B resumes. Its first commit is stale, and its retry fails in `FastAppend.apply` while reading writer A's manifest list because `manifestListKey` is null.
7. After reloading the table from the catalog, writer A's snapshot still lacks its key and planning it therefore fails with the same exception.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Concurrent commits through one `Table` can lose manifest-list encryption keys [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar closed issue #17990: Concurrent commits through one `Table` can lose manifest-list encryption keys
URL: https://github.com/apache/iceberg/issues/17990
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Concurrent commits through one `Table` can lose manifest-list encryption keys [iceberg]
Posted by "smaheshwar-pltr (via GitHub)" <gi...@apache.org>.
smaheshwar-pltr commented on issue #17990:
URL: https://github.com/apache/iceberg/issues/17990#issuecomment-5564815958
I think the fix we should proceed with initially involved preventing this key loss, not making everything e.g. the encryption manager thread safe (`TableOperations` themselves often have lack any synchronization - https://github.com/apache/iceberg/pull/13225/files#r3175544692, but those data races are "benign" and don't lead to corruption).
That fix is centred around point (5) in the steps above, and is similar to https://github.com/apache/iceberg/pull/16525. It involved committing keys along with snapshots in table metadata to avoid the two drifting - #17984. There may still be races on the `StandardEncryptionManager` that fail loudly (that's why I've avoided adding a full concurrency test in that PR that might flake), but unreadable snapshots will no longer be committed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org