Re: [PR] GH-41719: [C++][Parquet] Cannot read encrypted parquet datasets via _metadata file [arrow]

["rok (via GitHub)" <gi...@apache.org>]

rok commented on code in PR #41821:
URL: https://github.com/apache/arrow/pull/41821#discussion_r1792615650


##########
cpp/src/parquet/metadata.cc:
##########
@@ -1078,6 +1096,36 @@ void FileMetaData::WriteTo(::arrow::io::OutputStream* dst,
   return impl_->WriteTo(dst, encryptor);
 }
 
+::arrow::Result<std::shared_ptr<parquet::FileMetaData>> FileMetaData::CoalesceMetadata(
+    std::vector<std::shared_ptr<parquet::FileMetaData>>& metadata_list,
+    std::shared_ptr<parquet::WriterProperties>& writer_props) {
+  if (metadata_list.empty()) {
+    return ::arrow::Status::Invalid("No metadata to coalesce");
+  }
+
+  std::vector<std::string> values, keys;
+
+  // Read metadata from all dataset files and store AADs.
+  for (size_t i = 0; i < metadata_list.size(); i++) {
+    const auto& file_metadata = metadata_list[i];
+    keys.push_back("row_group_aad_" + std::to_string(i));
+    values.push_back(file_metadata->file_aad());
+    if (i > 0) {
+      metadata_list[0]->AppendRowGroups(*file_metadata);
+    }
+  }
+
+  // Create a new FileMetadata object with the created AADs as key_value_metadata.
+  auto fmd_builder =
+      parquet::FileMetaDataBuilder::Make(metadata_list[0]->schema(), writer_props);
+  const std::shared_ptr<const KeyValueMetadata> file_aad_metadata =
+      ::arrow::key_value_metadata(keys, values);
+  auto metadata = fmd_builder->Finish(file_aad_metadata);
+  metadata->AppendRowGroups(*metadata_list[0]);

Review Comment:
   @ggershinsky As proposed I now decrypt all footers and then coalesce them into a single footer. As decrypting data files at read times requires AADs I also store those into `key_value_metadata` with `row_group_aad_{i}` keys. Does this seem reasonable design?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org