[I] [Parquet] Column stats should be provided in encrypted part of the footer when writing in plaintext footer mode [arrow-rs]

["rok (via GitHub)" <gi...@apache.org>]

rok opened a new issue, #8304:
URL: https://github.com/apache/arrow-rs/issues/8304

   As per [spec](https://parquet.apache.org/docs/file-format/data-pages/encryption/#55-plaintext-footer-mode):
   
   > In the plaintext footer mode, the optional ColumnMetaData meta_data is set in the ColumnChunk structure for all columns, but is stripped of the statistics for the sensitive (encrypted) columns. These statistics are available for new readers with the column key - they decrypt the encrypted_column_metadata field, described in the section 5.3, and parse it to get statistics and all other column metadata values. The legacy readers are not aware of the encrypted metadata field; they parse the regular (plaintext) field as usual. While they can’t read the data of encrypted columns, they read their metadata to extract the offset and size of encrypted column data, required for column chunk vectorization.
   
   Current writer omits stats for plaintext footers all together. We would want to write stats as per the spec.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Parquet] Provide only encrypted column stats in plaintext footer [arrow-rs]

["adamreeve (via GitHub)" <gi...@apache.org>]

adamreeve commented on issue #8304:
URL: https://github.com/apache/arrow-rs/issues/8304#issuecomment-3331342778

   I've looked a bit into this and just want to document my findings.
   
   This behaviour is discussed in the format docs in [section 5.3](https://github.com/apache/parquet-format/blob/c3f7be7dc615de76910154d690c534768ba25d6f/Encryption.md#53-protection-of-sensitive-metadata).
   
   * [`encrypted_column_metadata`](https://github.com/apache/parquet-format/blob/c3f7be7dc615de76910154d690c534768ba25d6f/src/main/thrift/parquet.thrift#L998) in the `ColumnChunk` is the encrypted version of [`meta_data`](https://github.com/apache/parquet-format/blob/c3f7be7dc615de76910154d690c534768ba25d6f/src/main/thrift/parquet.thrift#L980) and should be set when a plaintext footer is used and the column is encrypted, or when a column is encrypted with a different key to the footer.
   * When reading with decryption properties provided, the `encrypted_column_metadata` should be decrypted and used to replace the `meta_data`. It looks like this should already [happen here](https://github.com/apache/arrow-rs/blob/aa626e12de8bc0d0f56b5349239cae1be8d1a195/parquet/src/file/metadata/mod.rs#L638), although I'm not 100% sure we hit this code path with a plaintext footer.
   * Currently, when a `ColumnChunk` is created for writing the encrypted metadata is [always None](https://github.com/apache/arrow-rs/blob/aa626e12de8bc0d0f56b5349239cae1be8d1a195/parquet/src/file/metadata/mod.rs#L1249), so we need to fix both the case of a plaintext footer and when a column is encrypted with a different key to the footer
   * It looks like [here](https://github.com/apache/arrow/blob/cbd36b817fc77812f8df1a15bf24314de3b27f29/cpp/src/parquet/metadata.cc#L1727-L1756) is where the logic for writing encrypted column metadata is handled in C++ Parquet
   	* When the footer is encrypted and a column is encrypted with another key, the whole `meta_data` field is removed. I guess this is because readers that support encryption are expected to handle when this is missing if they can't decrypt the column.
   	* When a plaintext footer is used, the unencrypted `meta_data` field is kept, but its `statistics` and `encoding_stats` are removed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Parquet] Provide only encrypted column stats in plaintext footer [arrow-rs]

["alamb (via GitHub)" <gi...@apache.org>]

alamb closed issue #8304: [Parquet] Provide only encrypted column stats in plaintext footer
URL: https://github.com/apache/arrow-rs/issues/8304


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org