You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "pitrou (via GitHub)" <gi...@apache.org> on 2026/08/24 15:02:37 UTC
[I] [C++][Parquet] "Keep redacted metadata version for old readers" [arrow]
pitrou opened a new issue, #50969:
URL: https://github.com/apache/arrow/issues/50969
### Describe the bug, including details regarding any error messages, version, and platform.
This code here looks slightly fishy:
https://github.com/apache/arrow/blob/8b469b179da3bda89d5e8d24c82523553f5aca5c/cpp/src/parquet/metadata.cc#L1787-L1796
Basically, when encrypting a Parquet file with plaintext footer, the Parquet writer _also_ keeps most of the column metadata in plaintext (except statistics).
### Component(s)
C++, Parquet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Parquet] "Keep redacted metadata version for old readers" [arrow]
Posted by "adamreeve (via GitHub)" <gi...@apache.org>.
adamreeve commented on issue #50969:
URL: https://github.com/apache/arrow/issues/50969#issuecomment-5401534599
The spec says that plaintext footer mode "allows legacy Parquet versions (released before the encryption support) to access unencrypted columns in encrypted files." But we (G-Research) like using plaintext footer mode because it makes it easy for engineers to inspect Parquet files and see non-sensitive columns without needing to configure the KMS and decryption settings. So I don't think plaintext footer mode should be removed, and agree with Gidon that we should probably update the spec wording.
If newer readers can handle the missing metadata OK, then maybe the plaintext column metadata for encrypted columns could be completely removed. But we should verify that first, I wouldn't be surprised if that caused problems for readers that do support encryption when they don't have access to those columns, even though it would be spec compliant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Parquet] "Keep redacted metadata version for old readers" [arrow]
Posted by "ggershinsky (via GitHub)" <gi...@apache.org>.
ggershinsky commented on issue #50969:
URL: https://github.com/apache/arrow/issues/50969#issuecomment-5397564807
Yep, maybe this needs to be re-evaluated (together with the spec, https://github.com/apache/parquet-format/blob/master/Encryption.md#55-plaintext-footer-mode), now that ~7 years have passed.
One of the questions, are there users that need to see footers of encrypted files.
Or users that need to access unencrypted columns with having any keys.
A low hanging fruit would be to update the comment "Keep redacted metadata version for old readers" to something like "Keep redacted metadata version for plaintext footer mode ".
A longer-term option is updating the spec ( to either deprecate the plaintext footer mode, or repurpose it to the usecases described above; the latter requires only text/comment updates).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Parquet] "Keep redacted metadata version for old readers" [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #50969:
URL: https://github.com/apache/arrow/issues/50969#issuecomment-5397118614
@ggershinsky @adamreeve @rok What do you think? Should this fallback "for old readers" be kept?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Parquet] "Keep redacted metadata version for old readers" [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #50969:
URL: https://github.com/apache/arrow/issues/50969#issuecomment-5399659065
Ah, I had forgotten about this tidbit:
> In the plaintext footer mode, the optional ColumnMetaData meta_data is set in the ColumnChunk structure for all columns, but is stripped of the statistics for the sensitive (encrypted) columns.
So this does seem spec-compliant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Parquet] "Keep redacted metadata version for old readers" [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on issue #50969:
URL: https://github.com/apache/arrow/issues/50969#issuecomment-5398311874
It does look shady at first glance, but I think this is just plaintext footer mode.
Regarding deprecating plaintext footer - we should probably have a ML discussion first to get an idea who actually uses it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org