You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "daniel-adam-tfs (via GitHub)" <gi...@apache.org> on 2025/11/10 10:55:00 UTC
[I] [Go][Parquet] Encryption/Decryption failures [arrow-go]
daniel-adam-tfs opened a new issue, #566:
URL: https://github.com/apache/arrow-go/issues/566
### Describe the bug, including details regarding any error messages, version, and platform.
There seems to be a discrepancy between files encrypted by [https://arrow.apache.org/docs/python/index.html](PyArrow) and arrow-go. No scenario with encryption in one library and decryption in the other seems to work for me.
Related to https://github.com/apache/arrow-go/issues/540 , which is currently blocked until this is resolved.
### Component(s)
Parquet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Go][Parquet] Encryption/Decryption failures [arrow-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #566:
URL: https://github.com/apache/arrow-go/issues/566#issuecomment-4380879099
Sorry for the lack of response here, this fell off my radar. I think this could be useful
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Go][Parquet] Encryption/Decryption failures [arrow-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #566:
URL: https://github.com/apache/arrow-go/issues/566#issuecomment-3597458086
@daniel-adam-tfs Can you provide an example failure that we could try to debug?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Go][Parquet] Encryption/Decryption failures [arrow-go]
Posted by "daniel-adam-tfs (via GitHub)" <gi...@apache.org>.
daniel-adam-tfs commented on issue #566:
URL: https://github.com/apache/arrow-go/issues/566#issuecomment-3783484451
@zeroshade What do you think about the cross platform encryption/decryption tests with PyArrow?
Maybe something like this:
1. Run a python script that generates encrypted files to some folder
2. Run go test with a build tag or environmental variable that decrypts the files and checks the contents
3. The go test should also run the generates encrypted files (with the KMS-style metadata, so PyArrow can decrypt the data)
4. Run some python unit tests that tries to decrypt the Go generated encrypted files
All of this would be an integration test suite written in python (makes most sense to me), that can be run from CI or in a docker container.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Go][Parquet] Encryption/Decryption failures [arrow-go]
Posted by "EdwardShen125 (via GitHub) via github" <gi...@arrow.apache.org>.
EdwardShen125 commented on issue #566:
URL: https://github.com/apache/arrow-go/issues/566#issuecomment-6572676388
Thanks! The harness is up in #1439 — direct-key configs pass in both directions on current main; KMS/PKMT1 stays a documented skip pending the library-vs-example question.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Go][Parquet] Encryption/Decryption failures [arrow-go]
Posted by "daniel-adam-tfs (via GitHub)" <gi...@apache.org>.
daniel-adam-tfs commented on issue #566:
URL: https://github.com/apache/arrow-go/issues/566#issuecomment-3510865757
@zeroshade I might some help with this one. Once scenario that I can debug pretty easily is to generate an encrypted file by PyArrow and debug what is happening in arrow-go. But is the PyArrow implementation guaranteed to be correct? No point of doing it if that is the source of the problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Go][Parquet] Encryption/Decryption failures [arrow-go]
Posted by "daniel-adam-tfs (via GitHub) via github" <gi...@arrow.apache.org>.
daniel-adam-tfs commented on issue #566:
URL: https://github.com/apache/arrow-go/issues/566#issuecomment-6571958309
@EdwardShen125 Go for it. In our project we are still focusing on performance, and we don't even use encryption so this is low priority for us, so I wouldn't get to it any time soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Go][Parquet] Encryption/Decryption failures [arrow-go]
Posted by "daniel-adam-tfs (via GitHub)" <gi...@apache.org>.
daniel-adam-tfs commented on issue #566:
URL: https://github.com/apache/arrow-go/issues/566#issuecomment-3626561675
Yeah, sorry I was either sick or busy the past 3 weeks. Let's get this going before Christmas/New Years kicks in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Go][Parquet] Encryption/Decryption failures [arrow-go]
Posted by "EdwardShen125 (via GitHub) via github" <gi...@arrow.apache.org>.
EdwardShen125 commented on issue #566:
URL: https://github.com/apache/arrow-go/issues/566#issuecomment-6541019301
I'd like to take this. I tested current main against pyarrow 26: every direct-key config I tried works now, pyarrow files read fine in Go (uniform GCM/CTR, AAD prefix, plaintext footer), and pyarrow reads a Go-written file. My guess is #596/#1390/#1412 solved most of that.
What still fails is KMS files. pyarrow's CryptoFactory writes PKMT1 key material with a wrapped DEK, and arrow-go can't parse or unwrap it, so footer decryption fails unless you write a custom retriever (zeroshade posted one in #596).
I'll start with the harness from this thread: Go tests that call pyarrow to generate files for each config and check reads in both directions. Tests skip when pyarrow isn't installed. Not sure how you want this in CI, I'll ask in the PR. #1438-style fixtures test fixed files, this would also test the current pyarrow release.
After that, the KMS part: parsing and unwrapping PKMT1. Should that go into the library or stay an example retriever?
@daniel-adam-tfs still planning to take this, or can I take it over?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Go][Parquet] Encryption/Decryption failures [arrow-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #566:
URL: https://github.com/apache/arrow-go/issues/566#issuecomment-3514349617
In general, pyarrow and go should agree and be able to cross collaborate. So if you can generate a file from pyarrow and debug it in arrow go, that will still get us further to figure out what is happening.
Once we figure out the cause, we can look at the spec and determine which is correct, pyarrow or Go arrow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Go][Parquet] Encryption/Decryption failures [arrow-go]
Posted by "zeroshade (via GitHub) via github" <gi...@arrow.apache.org>.
zeroshade commented on issue #566:
URL: https://github.com/apache/arrow-go/issues/566#issuecomment-6585155263
@EdwardShen125 I'd be fine with adding an explicit implementation for KMS/PKMT1 inside the library to save users the trouble of having to build it themselves. Just make sure the documentation/examples show how to use it as an optional thing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Go][Parquet] Encryption/Decryption failures [arrow-go]
Posted by "EdwardShen125 (via GitHub) via github" <gi...@arrow.apache.org>.
EdwardShen125 commented on issue #566:
URL: https://github.com/apache/arrow-go/issues/566#issuecomment-6718675312
Update: #1478 adds read support for CryptoFactory-written files (single and double wrapping). Write support is a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org