You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Alexander Bailey <za...@gmail.com> on 2026/09/02 20:54:02 UTC
[VOTE] [SPEC] Document standard encryption key metadata
Hi everyone,
Based on the conversation in
https://lists.apache.org/thread/4zm5jhwp42z10pfz2sths7q59r9dlf8l and on the
PR https://github.com/apache/iceberg/pull/16527, I’d like to raise a vote
to add a new encryption spec which documents the binary formats used by the
standard encryption scheme.
Today the spec describes the key_metadata field in manifest entries and the
encrypted-key-metadata field in table metadata as
“implementation-specific.” In practice these formats are fixed by the Java
implementation, and any other implementation has to reverse-engineer them
from the Java source to read or write encrypted tables. This came out of
implementing table encryption in iceberg-rust.
The PR adds a new encryption spec document covering:
-
The standard key metadata format: a version byte followed by an Avro
binary-encoded record holding the DEK, the AAD prefix, and the file length.
-
The two-tier key hierarchy in the encryption-keys list: KEK entries
wrapped by the KMS, and manifest list key entries encrypted by a KEK.
This documents existing behaviour rather than changing it, so no
implementation needs to change.
Vote
[ ] +1 Adopt the spec change
[ ] 0 No opinion
[ ] -1 Do not adopt, because…
Thanks,
Xander
Re: [VOTE] [SPEC] Document standard encryption key metadata
Posted by Alexander Bailey <za...@gmail.com>.
Thank both for the input, in agreement on the direction here. Will touch up
the PR to try and reflect both your comments.
Best,
Xander
On Thu, 3 Sep 2026 at 14:45, Gidon Gershinsky <gg...@gmail.com> wrote:
> Hi Gabor,
>
> Saw your comments in the PR. I agree the current work on encryption of
> stat files should be reflected there
> (both explicitly, and implicitly in the sense Manifest lists is an example
> of a category, not a single file type).
> Still, the doc should be precise enough to enable interop of
> implementations.
> In other words, I'm for updating the PR and putting it again for a vote.
>
> Cheers, Gidon
>
>
> On Thu, Sep 3, 2026 at 3:52 PM Gábor Kaszab <ga...@apache.org>
> wrote:
>
>> Hi Alexander,
>>
>> Thank you for preparing the spec change! This is a great addition to help
>> cross-implementation interworking in this area.
>> I went through the PR and left some comments that might be worth
>> discussing. In general I feel that there is too much coupling between the
>> table spec and the encryption spec, or in other words the encryption spec
>> makes too many assumptions on which table metadata struct field uses what
>> encryption method and how. We might want to be more generic with that in
>> the encryption spec.
>>
>> I'm wondering what others think.
>>
>> Thanks,
>> Gabor
>>
>> Alexander Bailey <za...@gmail.com> ezt írta (időpont: 2026. szept.
>> 2., Sze, 22:55):
>>
>>> Hi everyone,
>>>
>>> Based on the conversation in
>>> https://lists.apache.org/thread/4zm5jhwp42z10pfz2sths7q59r9dlf8l and on
>>> the PR https://github.com/apache/iceberg/pull/16527, I’d like to raise
>>> a vote to add a new encryption spec which documents the binary formats used
>>> by the standard encryption scheme.
>>>
>>> Today the spec describes the key_metadata field in manifest entries and
>>> the encrypted-key-metadata field in table metadata as
>>> “implementation-specific.” In practice these formats are fixed by the Java
>>> implementation, and any other implementation has to reverse-engineer them
>>> from the Java source to read or write encrypted tables. This came out of
>>> implementing table encryption in iceberg-rust.
>>>
>>> The PR adds a new encryption spec document covering:
>>>
>>> -
>>>
>>> The standard key metadata format: a version byte followed by an Avro
>>> binary-encoded record holding the DEK, the AAD prefix, and the file length.
>>> -
>>>
>>> The two-tier key hierarchy in the encryption-keys list: KEK entries
>>> wrapped by the KMS, and manifest list key entries encrypted by a KEK.
>>>
>>> This documents existing behaviour rather than changing it, so no
>>> implementation needs to change.
>>>
>>>
>>> Vote
>>>
>>> [ ] +1 Adopt the spec change
>>>
>>> [ ] 0 No opinion
>>>
>>> [ ] -1 Do not adopt, because…
>>>
>>> Thanks,
>>>
>>> Xander
>>>
>>
Re: [VOTE] [SPEC] Document standard encryption key metadata
Posted by Gidon Gershinsky <gg...@gmail.com>.
Hi Gabor,
Saw your comments in the PR. I agree the current work on encryption of stat
files should be reflected there
(both explicitly, and implicitly in the sense Manifest lists is an example
of a category, not a single file type).
Still, the doc should be precise enough to enable interop of
implementations.
In other words, I'm for updating the PR and putting it again for a vote.
Cheers, Gidon
On Thu, Sep 3, 2026 at 3:52 PM Gábor Kaszab <ga...@apache.org> wrote:
> Hi Alexander,
>
> Thank you for preparing the spec change! This is a great addition to help
> cross-implementation interworking in this area.
> I went through the PR and left some comments that might be worth
> discussing. In general I feel that there is too much coupling between the
> table spec and the encryption spec, or in other words the encryption spec
> makes too many assumptions on which table metadata struct field uses what
> encryption method and how. We might want to be more generic with that in
> the encryption spec.
>
> I'm wondering what others think.
>
> Thanks,
> Gabor
>
> Alexander Bailey <za...@gmail.com> ezt írta (időpont: 2026. szept.
> 2., Sze, 22:55):
>
>> Hi everyone,
>>
>> Based on the conversation in
>> https://lists.apache.org/thread/4zm5jhwp42z10pfz2sths7q59r9dlf8l and on
>> the PR https://github.com/apache/iceberg/pull/16527, I’d like to raise a
>> vote to add a new encryption spec which documents the binary formats used
>> by the standard encryption scheme.
>>
>> Today the spec describes the key_metadata field in manifest entries and
>> the encrypted-key-metadata field in table metadata as
>> “implementation-specific.” In practice these formats are fixed by the Java
>> implementation, and any other implementation has to reverse-engineer them
>> from the Java source to read or write encrypted tables. This came out of
>> implementing table encryption in iceberg-rust.
>>
>> The PR adds a new encryption spec document covering:
>>
>> -
>>
>> The standard key metadata format: a version byte followed by an Avro
>> binary-encoded record holding the DEK, the AAD prefix, and the file length.
>> -
>>
>> The two-tier key hierarchy in the encryption-keys list: KEK entries
>> wrapped by the KMS, and manifest list key entries encrypted by a KEK.
>>
>> This documents existing behaviour rather than changing it, so no
>> implementation needs to change.
>>
>>
>> Vote
>>
>> [ ] +1 Adopt the spec change
>>
>> [ ] 0 No opinion
>>
>> [ ] -1 Do not adopt, because…
>>
>> Thanks,
>>
>> Xander
>>
>
Re: [VOTE] [SPEC] Document standard encryption key metadata
Posted by Gábor Kaszab <ga...@apache.org>.
Hi Alexander,
Thank you for preparing the spec change! This is a great addition to help
cross-implementation interworking in this area.
I went through the PR and left some comments that might be worth
discussing. In general I feel that there is too much coupling between the
table spec and the encryption spec, or in other words the encryption spec
makes too many assumptions on which table metadata struct field uses what
encryption method and how. We might want to be more generic with that in
the encryption spec.
I'm wondering what others think.
Thanks,
Gabor
Alexander Bailey <za...@gmail.com> ezt írta (időpont: 2026. szept. 2.,
Sze, 22:55):
> Hi everyone,
>
> Based on the conversation in
> https://lists.apache.org/thread/4zm5jhwp42z10pfz2sths7q59r9dlf8l and on
> the PR https://github.com/apache/iceberg/pull/16527, I’d like to raise a
> vote to add a new encryption spec which documents the binary formats used
> by the standard encryption scheme.
>
> Today the spec describes the key_metadata field in manifest entries and
> the encrypted-key-metadata field in table metadata as
> “implementation-specific.” In practice these formats are fixed by the Java
> implementation, and any other implementation has to reverse-engineer them
> from the Java source to read or write encrypted tables. This came out of
> implementing table encryption in iceberg-rust.
>
> The PR adds a new encryption spec document covering:
>
> -
>
> The standard key metadata format: a version byte followed by an Avro
> binary-encoded record holding the DEK, the AAD prefix, and the file length.
> -
>
> The two-tier key hierarchy in the encryption-keys list: KEK entries
> wrapped by the KMS, and manifest list key entries encrypted by a KEK.
>
> This documents existing behaviour rather than changing it, so no
> implementation needs to change.
>
>
> Vote
>
> [ ] +1 Adopt the spec change
>
> [ ] 0 No opinion
>
> [ ] -1 Do not adopt, because…
>
> Thanks,
>
> Xander
>