You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "hectar-glitches (via GitHub)" <gi...@apache.org> on 2026/07/11 06:31:05 UTC
[PR] feat(encryption): add EncryptionManager, KeyManagementClient interfaces + in-memory KMS [iceberg-go]
hectar-glitches opened a new pull request, #1447:
URL: https://github.com/apache/iceberg-go/pull/1447
## Scope:
Landed the first slice of the encryption package per my earlier plan with the interfaces + no-op impl + in-memory KMS. Ready for review. EncryptingFileIO and manifest wiring intentionally deferred to follow-up PRs pending the key_metadata routing decision.
### Public API:
'EncryptionManager': coordination interface: 'NewEncryptedOutputFile(ctx, writer, keyID)' and 'NewDecryptedInputFile(ctx, file, keyMetadata)'
'KeyManagementClient': KMS interface: 'WrapKey' / 'UnwrapKey' / 'SupportsKeyGeneration' / 'GenerateKey'
'EncryptionKeyMetadata':opaque '[]byte' type for the per-file blob stored in 'DataFile.KeyMetadata' / 'ManifestFile.KeyMetadata' / 'StatisticsFile.KeyMetadata'
'EncryptedInputFile' / 'EncryptedOutputFile': file wrappers exposing 'KeyMetadata()'; names match the Java 'org.apache.iceberg.encryption' types
'NativeEncryptionInputFile' / 'NativeEncryptionOutputFile': extension interfaces returning 'any' for format-native crypto properties (so 'parquet.File{En,De}cryptionProperties' don't force a parquet import into a format-agnostic package)
'PlaintextEncryptionManager': no-op impl for unencrypted tables; matches Java's 'PlaintextEncryptionManager' (silently ignores 'keyID', passes bytes through)
'InMemoryKeyManagementClient': AES-GCM wrap/unwrap over an in-memory KEK map; documented as testing only
Sentinel errors: 'ErrUnknownKeyID', 'ErrInvalidKeyLength', 'ErrCiphertextTooShort', 'ErrAuthenticationFailed' (matches the 'Err*' convention used in 'io', 'catalog', root 'errors.go')
### Tests
'PlaintextEncryptionManager' round-trip (data & key metadata pass-through)
'InMemoryKeyManagementClient': wrap/unwrap round-trip, non-deterministic nonces, tamper detection, cross-key confusion, ciphertext-too-short, KEK overwrite, 'AddKey' input copy isolation, concurrent access via 'errgroup' (with race detector), key generation + round-trip, invalid length paths — all using 'errors.Is' and 't.Context()'
### Deferred to follow-ups (per our thread)
'EncryptingFileIO' wrapper — needs the 'key_metadata' routing decision
'encryption.Register' / 'LoadKMS' registry for 'encryption.kms-type' / 'encryption.kms-impl' catalog properties
Bridge between 'table.EncryptionKey.EncryptedKeyMetadata' (base64 'string') and 'EncryptionKeyMetadata' ('[]byte')
Parquet / Manifest / Puffin native encryption wiring
V3-only gating in table format
### Notes
'EncryptionKeyMetadata' is a defined type ('type EncryptionKeyMetadata []byte'), not a type alias. Every call site pulling 'KeyMetadata() []byte' from a manifest or data file will need an explicit conversion ('encryption.EncryptionKeyMetadata(f.KeyMetadata())'). If the group prefers a type alias instead, it's a one-line change here before anything downstream is wired.
'NativeEncryptionInputFile' / 'NativeEncryptionOutputFile' return 'any' for format-native properties. This avoids a parquet import in a format-agnostic package as callers type-assert to '*parquet.FileDecryptionProperties' etc. Happy to discuss alternatives.
'InMemoryKeyManagementClient' uses AES-GCM for key wrapping (matching the Java standard KMS). It is exported (not test-only file) so other packages can use it in their own tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(encryption): add EncryptionManager, KeyManagementClient interfaces + in-memory KMS [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade merged PR #1447:
URL: https://github.com/apache/iceberg-go/pull/1447
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(encryption): add EncryptionManager, KeyManagementClient interfaces + in-memory KMS [iceberg-go]
Posted by "hectar-glitches (via GitHub)" <gi...@apache.org>.
hectar-glitches commented on PR #1447:
URL: https://github.com/apache/iceberg-go/pull/1447#issuecomment-4963170917
@rambleraptor I am glad to follow your lead in dividing the work. I'll check the Docker issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(encryption): add EncryptionManager, KeyManagementClient interfaces + in-memory KMS [iceberg-go]
Posted by "hectar-glitches (via GitHub)" <gi...@apache.org>.
hectar-glitches commented on PR #1447:
URL: https://github.com/apache/iceberg-go/pull/1447#issuecomment-4997244672
@rambleraptor i made `PlaintextEncryptionManager` fail closed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(encryption): add EncryptionManager, KeyManagementClient interfaces + in-memory KMS [iceberg-go]
Posted by "hectar-glitches (via GitHub)" <gi...@apache.org>.
hectar-glitches commented on PR #1447:
URL: https://github.com/apache/iceberg-go/pull/1447#issuecomment-4972070455
@rambleraptor rerun the job and it looks good
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(encryption): add EncryptionManager, KeyManagementClient interfaces + in-memory KMS [iceberg-go]
Posted by "hectar-glitches (via GitHub)" <gi...@apache.org>.
hectar-glitches commented on PR #1447:
URL: https://github.com/apache/iceberg-go/pull/1447#issuecomment-4943102940
@rambleraptor first steps
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(encryption): add EncryptionManager, KeyManagementClient interfaces + in-memory KMS [iceberg-go]
Posted by "rambleraptor (via GitHub)" <gi...@apache.org>.
rambleraptor commented on PR #1447:
URL: https://github.com/apache/iceberg-go/pull/1447#issuecomment-4984459871
Is this ready for review? Could you take it off draft if it is?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(encryption): add EncryptionManager, KeyManagementClient interfaces + in-memory KMS [iceberg-go]
Posted by "rambleraptor (via GitHub)" <gi...@apache.org>.
rambleraptor commented on PR #1447:
URL: https://github.com/apache/iceberg-go/pull/1447#issuecomment-4961664376
@hectar-glitches this looks great! Let's try and divide up work in the issue. I don't want us to be stepping on each other's toes.
Looks like there was a Docker timeout and the integration test job needs to be re-run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(encryption): add EncryptionManager, KeyManagementClient interfaces + in-memory KMS [iceberg-go]
Posted by "hectar-glitches (via GitHub)" <gi...@apache.org>.
hectar-glitches commented on PR #1447:
URL: https://github.com/apache/iceberg-go/pull/1447#issuecomment-4987182084
i think it is ready, taking it off draft...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org