You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "yigal-rozenberg (via GitHub)" <gi...@apache.org> on 2025/01/27 17:28:03 UTC
[I] Add support for encrypted/protected data type in iceberg table [iceberg-python]
yigal-rozenberg opened a new issue, #1582:
URL: https://github.com/apache/iceberg-python/issues/1582
### Feature Request / Improvement
I am working on extending Apache Iceberg supported data types with a new complex type: 'ProtctedType'.
This new data type internally is a StructType including a header and a payload.
The Header to include at minimum:
1. Encryption Provider ID
2. Encryption Key ID
3. Data Type
The payload to include the encrypted data as BinaryType.
The goal is to allow end user transparent interaction with the new type, allowing operations between encrypted data items, and clear text.
Further more, allow extension of puffin files to store aggregate data based on the clear text values, bloom filters, and optionally inverted index for gerex search without a full table scan.
Looking for guidance on how such data type can be introduced and what are the dependencies I would need to address with the various readers and writers.
[protected_type_merge.txt](https://github.com/user-attachments/files/18562222/protected_type_merge.txt)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Add support for encrypted/protected data type in iceberg table [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #1582:
URL: https://github.com/apache/iceberg-python/issues/1582#issuecomment-2640733293
Thanks for the additional context @yigal-rozenberg. I find this very interesting. Also, I know certain companies that use this pattern for GDPR, where they erase the decryption key in case a right-to-be-forgotten request is being issued.
If you want to have more feedback on this, I would highly encourage you to discuss this [on the dev-list](https://lists.apache.org/list.html?dev@iceberg.apache.org) this is the official place for communication for the Iceberg project (across languages).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Add support for encrypted/protected data type in iceberg table [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1582:
URL: https://github.com/apache/iceberg-python/issues/1582#issuecomment-2619864409
Hey there! Thanks for creating this issue. Typically for something like this we would want to create a [Improvement Proposals](https://iceberg.apache.org/contribute/?h=proposal#apache-iceberg-improvement-proposals) and get feedback from the community.
In this case, the proposal seem to be adding a new [data type to the table specification](https://iceberg.apache.org/spec/#schemas-and-data-types). (Note, the [table specification](https://iceberg.apache.org/spec/) should be language agnostic.)
Here are some somewhat related threads that i've found
https://lists.apache.org/thread/jm5xoy3fro4omlqlo476cf0118dcznkr
https://github.com/apache/iceberg/issues/10909
Hope this helps!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Add support for encrypted/protected data type in iceberg table [iceberg-python]
Posted by "yigal-rozenberg (via GitHub)" <gi...@apache.org>.
yigal-rozenberg commented on issue #1582:
URL: https://github.com/apache/iceberg-python/issues/1582#issuecomment-2631515059
Hi @Fokko,
File encryption is an essential functionality that adds an immortal security layer to data integrity and security on a file level.
Data Centric security addresses a different challenge in privacy and security of data.
The idea is based on a concept where you protect and classify data during creation and carry the security properties across all systems without the need to re-encrypt and reclassify it.
Traditional data security solutions rely on data discovery and classification based on the location of the data (server.db.schema.table.column), and the need to continuously track and re-identify data as it flows in the organization.
Once the base protected data type is identified by the engine as protected and not just a structure or binary buffer, the engine layer can set min/max and even aggregation values in puffin files to further improve performance.
Additionally these puffin fees can be extended to include bloom filters (based on the clear text values) and even inverted indexes to allow wild card and other operations over clear text data representation (this is where files encryption is important).
I am still trying to understand the code base, and identify how and where I can implement supported operators according to the input data type to decipher the data and decode it to the original data type (e.g., binary->str, binary->date, etc.).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Add support for encrypted/protected data type in iceberg table [iceberg-python]
Posted by "yigal-rozenberg (via GitHub)" <gi...@apache.org>.
yigal-rozenberg commented on issue #1582:
URL: https://github.com/apache/iceberg-python/issues/1582#issuecomment-2620105176
Before posting this as a proper improvement request, I would like to come up with a POC that demonstrate the desired functionality/ The thread you provided talks about the need for proper data centric security, and I have some years of experience in this topic.
IMHO the best way to secure data and centrally control access is to use data item encryption. In some cases this can also be referred to column level encryption, however, one can confuse this with file encryption in column based data.
When data items are encrypted, the cipher text can be sent and shared/accessed across multiple systems and engines.
The challenge is that cipher text by itself does not include metadata such as the key-id used to encrypt it, and the original data type of the clear text.
I am trying to understand, as a first phase, how in Iceberg Python interface I can crate a new Data Type, which has a different behavior when it stores and reads the data from the table storage, and a different behavior when data is inserted/updated/selected.
Do I need to implement different __str__ / __repr__ methods in the new data type, or I need to do it elsewhere?
Where to implement the operators to support operations between 2 encrypted types, and operations between encrypted and clear text?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Add support for encrypted/protected data type in iceberg table [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #1582:
URL: https://github.com/apache/iceberg-python/issues/1582#issuecomment-3157017845
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Add support for encrypted/protected data type in iceberg table [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #1582: Add support for encrypted/protected data type in iceberg table
URL: https://github.com/apache/iceberg-python/issues/1582
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Add support for encrypted/protected data type in iceberg table [iceberg-python]
Posted by "yigal-rozenberg (via GitHub)" <gi...@apache.org>.
yigal-rozenberg commented on issue #1582:
URL: https://github.com/apache/iceberg-python/issues/1582#issuecomment-2616475076
Sample program to read and write using ProtectedType data.
[app.txt](https://github.com/user-attachments/files/18562332/app.txt)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Add support for encrypted/protected data type in iceberg table [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #1582:
URL: https://github.com/apache/iceberg-python/issues/1582#issuecomment-2630481183
As @kevinjqliu already pointed out, this is a bigger thing than just implementing it on PyIceberg. There is also a proposal out for file-level encryption: https://github.com/apache/iceberg/pull/12162/. For the implementation, I think there are a few more things to take into account. For example, when doing query optimization we use the min/max of the column from the Manifest file, we need to make sure to decrypt these as well before evaluating the metrics.
> Do I need to implement different str / repr methods in the new data type, or I need to do it elsewhere?
I think this is being [done on the `PrimitiveType`](https://github.com/apache/iceberg-python/blob/5018efc203b6393898a000d98e6f305f287e42a7/pyiceberg/types.py#L184-L190).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Add support for encrypted/protected data type in iceberg table [iceberg-python]
Posted by "yigal-rozenberg (via GitHub)" <gi...@apache.org>.
yigal-rozenberg commented on issue #1582:
URL: https://github.com/apache/iceberg-python/issues/1582#issuecomment-2631602334
related technical publication: https://www.etsi.org/deliver/etsi_ts/103500_103599/103532/01.01.01_60/ts_103532v010101p.pdf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Add support for encrypted/protected data type in iceberg table [iceberg-python]
Posted by "yigal-rozenberg (via GitHub)" <gi...@apache.org>.
yigal-rozenberg commented on issue #1582:
URL: https://github.com/apache/iceberg-python/issues/1582#issuecomment-2627821210
added code change here: https://github.com/apache/iceberg-python/pull/1594
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Add support for encrypted/protected data type in iceberg table [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #1582:
URL: https://github.com/apache/iceberg-python/issues/1582#issuecomment-3599555290
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org