You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "wjones127 (via GitHub)" <gi...@apache.org> on 2023/01/24 22:24:45 UTC

[GitHub] [arrow] wjones127 commented on a diff in pull request #10491: ARROW-9960: [C++] Enable external material and rotation for encryption keys

wjones127 commented on code in PR #10491:
URL: https://github.com/apache/arrow/pull/10491#discussion_r1085689450


##########
cpp/src/parquet/encryption/file_system_key_material_store.cc:
##########
@@ -0,0 +1,120 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/filesystem/filesystem.h"
+#include "arrow/testing/json_integration.h"
+#include "arrow/testing/json_internal.h"
+#include "arrow/util/io_util.h"
+#include "arrow/util/make_unique.h"
+
+#include "parquet/encryption/file_system_key_material_store.h"
+#include "parquet/encryption/key_material.h"
+
+namespace parquet {
+namespace encryption {
+
+constexpr const char FileSystemKeyMaterialStore::kKetMaterialFilePrefix[];
+constexpr const char FileSystemKeyMaterialStore::kTempFilePrefix[];
+constexpr const char FileSystemKeyMaterialStore::kKeyMaterialFileSuffix[];
+
+void FileSystemKeyMaterialStore::initialize(
+    const std::shared_ptr<FilePath>& parquet_file_path, bool temp_store) {
+  std::string full_prefix =
+      (temp_store ? std::string(FileSystemKeyMaterialStore::kTempFilePrefix) : "");
+  full_prefix =
+      full_prefix + std::string(FileSystemKeyMaterialStore::kKetMaterialFilePrefix);
+  std::string key_material_file_name =
+      full_prefix + parquet_file_path->base_name() +
+      std::string(FileSystemKeyMaterialStore::kKeyMaterialFileSuffix);
+  key_material_file_ = ::arrow::internal::make_unique<FilePath>(

Review Comment:
   We should probably switch this to `std::make_unique`



##########
cpp/src/parquet/encryption/file_system_key_material_store.h:
##########
@@ -0,0 +1,85 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <set>
+#include <string>
+#include <unordered_map>
+
+#include "arrow/filesystem/filesystem.h"
+
+#include "parquet/encryption/file_key_material_store.h"
+#include "parquet/encryption/file_path.h"
+
+namespace parquet {
+namespace encryption {
+
+/// Key material can be stored outside the Parquet file, for example in a separate small
+/// file in the same folder. This is important for “key rotation”, when MEKs have to be
+/// changed (if compromised; or periodically, just in case) - without modifying the
+/// Parquet files (often  immutable).
+class PARQUET_EXPORT FileSystemKeyMaterialStore : public FileKeyMaterialStore {
+ public:
+  static constexpr const char kKetMaterialFilePrefix[] = "_KEY_MATERIAL_FOR_";
+  static constexpr const char kTempFilePrefix[] = "_TMP";
+  static constexpr const char kKeyMaterialFileSuffix[] = ".json";
+
+  FileSystemKeyMaterialStore() {}
+
+  /// Initializes key material store for a parquet file.
+  void initialize(const std::shared_ptr<FilePath>& parquet_file_path, bool temp_store);

Review Comment:
   We should document what the purpose of `temp_store` is.



##########
cpp/src/parquet/encryption/file_key_wrapper.h:
##########
@@ -62,7 +62,8 @@ class PARQUET_EXPORT FileKeyWrapper {
   /// master key
   std::string GetEncryptionKeyMetadata(const std::string& data_key,
                                        const std::string& master_key_id,
-                                       bool is_footer_key);
+                                       bool is_footer_key,

Review Comment:
   In a doc comment above we say
   
   > Currently we don't support the case "key material" stores outside "key metadata" yet.
   
   But isn't the point of this PR to add support for that?



##########
cpp/src/parquet/encryption/file_path.h:
##########
@@ -0,0 +1,78 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/filesystem/filesystem.h"
+#include "arrow/result.h"
+#include "arrow/status.h"
+
+#include "parquet/exception.h"
+
+namespace parquet {
+
+/// \brief The path and filesystem where an actual file is located.
+class PARQUET_EXPORT FilePath {
+ public:
+  FilePath(std::string path, std::shared_ptr<::arrow::fs::FileSystem> filesystem)
+      : file_info_(std::move(path)), filesystem_(filesystem) {}
+
+  /// \brief The directory base name (component before the file base name

Review Comment:
   ```suggestion
     /// \brief The directory base name (component before the file base name)
   ```



##########
cpp/src/parquet/encryption/key_toolkit.cc:
##########
@@ -35,6 +46,89 @@ std::shared_ptr<KmsClient> KeyToolkit::GetKmsClient(
       });
 }
 
+// Filter out files that are not suitable for key rotation.
+inline bool filter_out_file(std::string child) {
+  // Filters out hidden files. A file is considered to
+  // be hidden, and should not be considered for processing, when the file name
+  // starts with a period ('.') or an underscore ('_').
+  if (child.at(0) == kPeriod || child.at(0) == kUnderscore) return true;
+  return false;
+}
+
+void KeyToolkit::RotateMasterKeys(const KmsConnectionConfig& kms_connection_config,
+                                  const std::shared_ptr<FilePath>& folder_path,
+                                  bool double_wrapping, double cache_lifetime_seconds) {
+  // If process wrote files with double-wrapped keys, clean KEK cache (since master keys
+  // are changing). Only once for each key rotation cycle; not for every folder
+  const auto now = internal::CurrentTimePoint();
+  auto lock = last_cache_clean_for_key_rotation_time_mutex_.Lock();
+  if (now > last_cache_clean_for_key_rotation_time_ +
+                std::chrono::duration<double>(kCacheCleanPeriodForKeyRotation)) {
+    kek_write_cache_per_token().Clear();
+    last_cache_clean_for_key_rotation_time_ = now;
+  }
+  lock.Unlock();
+  std::vector<::arrow::fs::FileInfo> parquet_files_in_folder;
+  ::arrow::fs::FileSelector s;
+  s.base_dir = folder_path->dir_name();
+  parquet_files_in_folder = folder_path->filesystem()->GetFileInfo(s).ValueOrDie();
+
+  if (parquet_files_in_folder.size() == 0) {
+    throw ParquetException("Couldn't rotate keys - no parquet files in folder " +
+                           folder_path->dir_name());
+  }
+
+  for (auto const& parquet_file : parquet_files_in_folder) {
+    if (parquet_file.type() != ::arrow::fs::FileType::File)
+      throw ParquetException("Expecting file type in " + folder_path->dir_name());

Review Comment:
   Why not ignore directories?



##########
cpp/src/parquet/encryption/key_toolkit.cc:
##########
@@ -35,6 +46,89 @@ std::shared_ptr<KmsClient> KeyToolkit::GetKmsClient(
       });
 }
 
+// Filter out files that are not suitable for key rotation.
+inline bool filter_out_file(std::string child) {
+  // Filters out hidden files. A file is considered to
+  // be hidden, and should not be considered for processing, when the file name
+  // starts with a period ('.') or an underscore ('_').
+  if (child.at(0) == kPeriod || child.at(0) == kUnderscore) return true;
+  return false;
+}
+
+void KeyToolkit::RotateMasterKeys(const KmsConnectionConfig& kms_connection_config,
+                                  const std::shared_ptr<FilePath>& folder_path,
+                                  bool double_wrapping, double cache_lifetime_seconds) {
+  // If process wrote files with double-wrapped keys, clean KEK cache (since master keys
+  // are changing). Only once for each key rotation cycle; not for every folder
+  const auto now = internal::CurrentTimePoint();
+  auto lock = last_cache_clean_for_key_rotation_time_mutex_.Lock();
+  if (now > last_cache_clean_for_key_rotation_time_ +
+                std::chrono::duration<double>(kCacheCleanPeriodForKeyRotation)) {
+    kek_write_cache_per_token().Clear();
+    last_cache_clean_for_key_rotation_time_ = now;
+  }
+  lock.Unlock();
+  std::vector<::arrow::fs::FileInfo> parquet_files_in_folder;
+  ::arrow::fs::FileSelector s;
+  s.base_dir = folder_path->dir_name();
+  parquet_files_in_folder = folder_path->filesystem()->GetFileInfo(s).ValueOrDie();

Review Comment:
   Should this be recursive? Or be allowing the user to supply the vector of fileinfos?



##########
cpp/src/parquet/encryption/file_key_unwrapper.cc:
##########
@@ -47,11 +55,26 @@ std::string FileKeyUnwrapper::GetKey(const std::string& key_metadata_bytes) {
   }
   KeyMetadata key_metadata = KeyMetadata::Parse(key_metadata_bytes);
 
-  if (!key_metadata.key_material_stored_internally()) {
-    throw ParquetException("External key material store is not supported yet.");
+  if (!checked_key_material_internal_storage_) {
+    if (!key_metadata.key_material_stored_internally()) {
+      key_material_store_ = std::make_shared<FileSystemKeyMaterialStore>();
+      key_material_store_->initialize(std::move(parquet_file_path_), false);
+    }
+    checked_key_material_internal_storage_ = true;
+  }
+  KeyMaterial key_material;
+  if (key_metadata.key_material_stored_internally()) {
+    key_material = key_metadata.key_material();
+  } else {
+    // External key material storage: key metadata contains a reference
+    // to a key in the material store
+    std::string key_id_in_file = key_metadata.key_reference();
+    std::string key_material_string = key_material_store_->GetKeyMaterial(key_id_in_file);
+    if (key_material_string.empty()) {
+      throw ParquetException("Null key material for keyIDinFile: " + key_id_in_file);

Review Comment:
   ```suggestion
         throw ParquetException("Null key material for key with ID: " + key_id_in_file);
   ```



##########
cpp/src/parquet/encryption/file_path.h:
##########
@@ -0,0 +1,78 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/filesystem/filesystem.h"
+#include "arrow/result.h"
+#include "arrow/status.h"
+
+#include "parquet/exception.h"
+
+namespace parquet {
+
+/// \brief The path and filesystem where an actual file is located.
+class PARQUET_EXPORT FilePath {

Review Comment:
   Do we really need this class?



##########
cpp/src/parquet/encryption/file_system_key_material_store.cc:
##########
@@ -0,0 +1,120 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/filesystem/filesystem.h"
+#include "arrow/testing/json_integration.h"
+#include "arrow/testing/json_internal.h"
+#include "arrow/util/io_util.h"
+#include "arrow/util/make_unique.h"
+
+#include "parquet/encryption/file_system_key_material_store.h"
+#include "parquet/encryption/key_material.h"
+
+namespace parquet {
+namespace encryption {
+
+constexpr const char FileSystemKeyMaterialStore::kKetMaterialFilePrefix[];
+constexpr const char FileSystemKeyMaterialStore::kTempFilePrefix[];
+constexpr const char FileSystemKeyMaterialStore::kKeyMaterialFileSuffix[];
+
+void FileSystemKeyMaterialStore::initialize(
+    const std::shared_ptr<FilePath>& parquet_file_path, bool temp_store) {
+  std::string full_prefix =
+      (temp_store ? std::string(FileSystemKeyMaterialStore::kTempFilePrefix) : "");
+  full_prefix =
+      full_prefix + std::string(FileSystemKeyMaterialStore::kKetMaterialFilePrefix);
+  std::string key_material_file_name =
+      full_prefix + parquet_file_path->base_name() +
+      std::string(FileSystemKeyMaterialStore::kKeyMaterialFileSuffix);
+  key_material_file_ = ::arrow::internal::make_unique<FilePath>(
+      parquet_file_path->dir_name() + "/" + key_material_file_name,
+      parquet_file_path->filesystem());
+}
+
+void FileSystemKeyMaterialStore::LoadKeyMaterialMap() {
+  ::arrow::Result<std::shared_ptr<::arrow::io::RandomAccessFile>> file =
+      key_material_file_->OpenReadable();
+  std::shared_ptr<::arrow::io::RandomAccessFile> input = file.ValueOrDie();
+  rj::Document document;
+  ::arrow::Result<std::shared_ptr<Buffer>> buff =
+      input->ReadAt(0, input->GetSize().ValueOrDie());
+  std::string buff_str = buff.ValueOrDie()->ToString();
+  ::arrow::util::string_view sv(buff_str);
+  document.Parse(sv.data(), sv.size());
+  for (rj::Value::ConstMemberIterator itr = document.MemberBegin();
+       itr != document.MemberEnd(); ++itr) {
+    key_material_map_.insert({itr->name.GetString(), itr->value.GetString()});
+  }
+}
+
+std::string FileSystemKeyMaterialStore::BuildKeyMaterialMapJson() {
+  rj::Document d;
+  auto& allocator = d.GetAllocator();
+  rj::Value json_map(rj::kObjectType);
+
+  rj::Value key(rj::kStringType);
+  rj::Value value(rj::kStringType);
+  for (auto it = key_material_map_.begin(); it != key_material_map_.end(); it++) {
+    key.SetString(it->first.c_str(), allocator);
+    value.SetString(it->second.c_str(), allocator);
+    json_map.AddMember(key, value, allocator);
+  }
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  json_map.Accept(writer);
+  return buffer.GetString();
+}
+
+void FileSystemKeyMaterialStore::SaveMaterial() {
+  ::arrow::Result<std::shared_ptr<::arrow::io::OutputStream>> sink =
+      key_material_file_->OpenWriteable();
+  auto stream = sink.ValueOrDie();
+  std::string key_material_json = BuildKeyMaterialMapJson();
+  if (!::arrow::internal::GenericToStatus(stream->Write(key_material_json)).ok())

Review Comment:
   Or simply use `PARQUET_THROW_NOT_OK`?



##########
cpp/src/parquet/encryption/file_system_key_material_store.cc:
##########
@@ -0,0 +1,120 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/filesystem/filesystem.h"
+#include "arrow/testing/json_integration.h"
+#include "arrow/testing/json_internal.h"
+#include "arrow/util/io_util.h"
+#include "arrow/util/make_unique.h"
+
+#include "parquet/encryption/file_system_key_material_store.h"
+#include "parquet/encryption/key_material.h"
+
+namespace parquet {
+namespace encryption {
+
+constexpr const char FileSystemKeyMaterialStore::kKetMaterialFilePrefix[];
+constexpr const char FileSystemKeyMaterialStore::kTempFilePrefix[];
+constexpr const char FileSystemKeyMaterialStore::kKeyMaterialFileSuffix[];
+
+void FileSystemKeyMaterialStore::initialize(
+    const std::shared_ptr<FilePath>& parquet_file_path, bool temp_store) {
+  std::string full_prefix =
+      (temp_store ? std::string(FileSystemKeyMaterialStore::kTempFilePrefix) : "");
+  full_prefix =
+      full_prefix + std::string(FileSystemKeyMaterialStore::kKetMaterialFilePrefix);
+  std::string key_material_file_name =
+      full_prefix + parquet_file_path->base_name() +
+      std::string(FileSystemKeyMaterialStore::kKeyMaterialFileSuffix);
+  key_material_file_ = ::arrow::internal::make_unique<FilePath>(
+      parquet_file_path->dir_name() + "/" + key_material_file_name,
+      parquet_file_path->filesystem());
+}
+
+void FileSystemKeyMaterialStore::LoadKeyMaterialMap() {
+  ::arrow::Result<std::shared_ptr<::arrow::io::RandomAccessFile>> file =
+      key_material_file_->OpenReadable();
+  std::shared_ptr<::arrow::io::RandomAccessFile> input = file.ValueOrDie();
+  rj::Document document;
+  ::arrow::Result<std::shared_ptr<Buffer>> buff =
+      input->ReadAt(0, input->GetSize().ValueOrDie());
+  std::string buff_str = buff.ValueOrDie()->ToString();
+  ::arrow::util::string_view sv(buff_str);

Review Comment:
   Also should upgrade `string_view` stuff.



##########
cpp/src/parquet/encryption/file_system_key_material_store.cc:
##########
@@ -0,0 +1,120 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/filesystem/filesystem.h"
+#include "arrow/testing/json_integration.h"
+#include "arrow/testing/json_internal.h"
+#include "arrow/util/io_util.h"
+#include "arrow/util/make_unique.h"
+
+#include "parquet/encryption/file_system_key_material_store.h"
+#include "parquet/encryption/key_material.h"
+
+namespace parquet {
+namespace encryption {
+
+constexpr const char FileSystemKeyMaterialStore::kKetMaterialFilePrefix[];
+constexpr const char FileSystemKeyMaterialStore::kTempFilePrefix[];
+constexpr const char FileSystemKeyMaterialStore::kKeyMaterialFileSuffix[];
+
+void FileSystemKeyMaterialStore::initialize(
+    const std::shared_ptr<FilePath>& parquet_file_path, bool temp_store) {
+  std::string full_prefix =
+      (temp_store ? std::string(FileSystemKeyMaterialStore::kTempFilePrefix) : "");
+  full_prefix =
+      full_prefix + std::string(FileSystemKeyMaterialStore::kKetMaterialFilePrefix);
+  std::string key_material_file_name =
+      full_prefix + parquet_file_path->base_name() +
+      std::string(FileSystemKeyMaterialStore::kKeyMaterialFileSuffix);
+  key_material_file_ = ::arrow::internal::make_unique<FilePath>(
+      parquet_file_path->dir_name() + "/" + key_material_file_name,
+      parquet_file_path->filesystem());
+}
+
+void FileSystemKeyMaterialStore::LoadKeyMaterialMap() {
+  ::arrow::Result<std::shared_ptr<::arrow::io::RandomAccessFile>> file =
+      key_material_file_->OpenReadable();
+  std::shared_ptr<::arrow::io::RandomAccessFile> input = file.ValueOrDie();
+  rj::Document document;
+  ::arrow::Result<std::shared_ptr<Buffer>> buff =
+      input->ReadAt(0, input->GetSize().ValueOrDie());
+  std::string buff_str = buff.ValueOrDie()->ToString();
+  ::arrow::util::string_view sv(buff_str);
+  document.Parse(sv.data(), sv.size());
+  for (rj::Value::ConstMemberIterator itr = document.MemberBegin();
+       itr != document.MemberEnd(); ++itr) {
+    key_material_map_.insert({itr->name.GetString(), itr->value.GetString()});
+  }
+}
+
+std::string FileSystemKeyMaterialStore::BuildKeyMaterialMapJson() {
+  rj::Document d;
+  auto& allocator = d.GetAllocator();
+  rj::Value json_map(rj::kObjectType);
+
+  rj::Value key(rj::kStringType);
+  rj::Value value(rj::kStringType);
+  for (auto it = key_material_map_.begin(); it != key_material_map_.end(); it++) {
+    key.SetString(it->first.c_str(), allocator);
+    value.SetString(it->second.c_str(), allocator);

Review Comment:
   I think if you set `#define RAPIDJSON_HAS_STDSTRING 1` you can just pass in the strings as is.



##########
cpp/src/parquet/encryption/crypto_factory.h:
##########
@@ -107,22 +107,29 @@ class PARQUET_EXPORT CryptoFactory {
 
   std::shared_ptr<FileEncryptionProperties> GetFileEncryptionProperties(
       const KmsConnectionConfig& kms_connection_config,
-      const EncryptionConfiguration& encryption_config);
+      const EncryptionConfiguration& encryption_config,
+      const std::shared_ptr<FilePath>& parquet_file_path = NULLPTR);
 
   /// The returned FileDecryptionProperties object will use the cache inside this
   /// CryptoFactory object, so please keep this
   /// CryptoFactory object alive along with the returned
   /// FileDecryptionProperties object.
   std::shared_ptr<FileDecryptionProperties> GetFileDecryptionProperties(
       const KmsConnectionConfig& kms_connection_config,
-      const DecryptionConfiguration& decryption_config);
+      const DecryptionConfiguration& decryption_config,
+      const std::shared_ptr<FilePath>& parquet_file_path = NULLPTR);
 
   void RemoveCacheEntriesForToken(const std::string& access_token) {
     key_toolkit_.RemoveCacheEntriesForToken(access_token);
   }
 
   void RemoveCacheEntriesForAllTokens() { key_toolkit_.RemoveCacheEntriesForAllTokens(); }
 
+  void RotateMasterKeys(const KmsConnectionConfig& kms_connection_config,

Review Comment:
   Since this seems to be a user-facing API, we should document this method.



##########
cpp/src/parquet/encryption/crypto_factory.cc:
##########
@@ -48,8 +51,20 @@ std::shared_ptr<FileEncryptionProperties> CryptoFactory::GetFileEncryptionProper
 
   std::shared_ptr<FileKeyMaterialStore> key_material_store = NULL;
   if (!encryption_config.internal_key_material) {
-    // TODO: using external key material store with Hadoop file system
-    throw ParquetException("External key material store is not supported yet.");
+    if (parquet_file_path == nullptr) {
+      std::stringstream ss;
+      ss << "Output file path cannot be null"
+         << "\n";
+      throw ParquetException(ss.str());
+    }
+    try {
+      key_material_store = std::make_shared<FileSystemKeyMaterialStore>();
+      key_material_store->initialize(parquet_file_path, false);
+    } catch (ParquetException& e) {
+      std::stringstream ss;
+      ss << "Failed to get key material store" << e.what() << "\n";

Review Comment:
   ```suggestion
         ss << "Failed to get key material store.\n" << e.what() << "\n";
   ```



##########
cpp/src/parquet/encryption/file_key_material_store.h:
##########
@@ -17,15 +17,48 @@
 
 #pragma once
 
+#include <set>
+#include <string>
+#include <unordered_map>
+
+#include "arrow/filesystem/filesystem.h"
+
+#include "parquet/encryption/file_path.h"
+
 namespace parquet {
 namespace encryption {
 
-// Key material can be stored outside the Parquet file, for example in a separate small
-// file in the same folder. This is important for “key rotation”, when MEKs have to be
-// changed (if compromised; or periodically, just in case) - without modifying the Parquet
-// files (often  immutable).
-// TODO: details will be implemented later
-class FileKeyMaterialStore {};
+/// Key material can be stored outside the Parquet file, for example in a separate small
+/// file in the same folder. This is important for “key rotation”, when MEKs have to be
+/// changed (if compromised; or periodically, just in case) - without modifying the
+/// Parquet files (often  immutable).
+class PARQUET_EXPORT FileKeyMaterialStore {
+ public:
+  /// Initializes key material store for a parquet file.
+  virtual void initialize(const std::shared_ptr<FilePath>& parquet_file_path,
+                          bool temp_store) = 0;

Review Comment:
   I think we typically create `Make()` methods instead of an `initialize()` method.



##########
cpp/src/parquet/encryption/key_management_test.cc:
##########
@@ -122,45 +130,96 @@ class TestEncryptionKeyManagement : public ::testing::Test {
         ARROW_LOG(FATAL) << "Invalid encryption_no";
     }
 
+    encryption.internal_key_material = internal_key_material;
     return encryption;
   }
 
   DecryptionConfiguration GetDecryptionConfiguration() {
     return DecryptionConfiguration();
   }
 
-  void WriteEncryptedParquetFile(bool double_wrapping, int encryption_no) {
-    std::string file_name = GetFileName(double_wrapping, wrap_locally_, encryption_no);
-    auto encryption_config = GetEncryptionConfiguration(double_wrapping, encryption_no);
-
-    auto file_encryption_properties = crypto_factory_.GetFileEncryptionProperties(
-        kms_connection_config_, encryption_config);
-    std::string file = temp_dir->path().ToString() + file_name;
-
-    encryptor_.EncryptFile(file, file_encryption_properties);
+  void WriteEncryptedParquetFile(bool double_wrapping, bool internal_key_material,
+                                 int encryption_no) {
+    std::string file_name =
+        GetFileName(double_wrapping, wrap_locally_, internal_key_material, encryption_no);
+
+    auto encryption_config =
+        GetEncryptionConfiguration(double_wrapping, internal_key_material, encryption_no);
+
+    std::string file = temp_dir_->path().ToString() + file_name;
+    if (internal_key_material) {
+      auto file_encryption_properties = crypto_factory_.GetFileEncryptionProperties(
+          kms_connection_config_, encryption_config);
+      encryptor_.EncryptFile(file, file_encryption_properties);
+    } else {
+      ::arrow::Result<std::shared_ptr<::arrow::fs::FileSystem>> filesystem =
+          MakeLocalFileSystem();
+      std::shared_ptr<FilePath> writable = std::make_shared<FilePath>(
+          temp_dir_->path().ToString() + file_name, filesystem.ValueOrDie());
+      auto file_encryption_properties = crypto_factory_.GetFileEncryptionProperties(
+          kms_connection_config_, encryption_config, writable);
+      encryptor_.EncryptFile(file, file_encryption_properties);
+    }
   }
 
-  void ReadEncryptedParquetFile(bool double_wrapping, int encryption_no) {
+  void ReadEncryptedParquetFile(bool double_wrapping, bool internal_key_material,
+                                int encryption_no) {
     auto decryption_config = GetDecryptionConfiguration();
-    std::string file_name = GetFileName(double_wrapping, wrap_locally_, encryption_no);
-
-    auto file_decryption_properties = crypto_factory_.GetFileDecryptionProperties(
-        kms_connection_config_, decryption_config);
-    std::string file = temp_dir->path().ToString() + file_name;
+    std::string file_name =
+        GetFileName(double_wrapping, wrap_locally_, internal_key_material, encryption_no);
+    std::string file = temp_dir_->path().ToString() + file_name;
+    if (internal_key_material) {
+      auto file_decryption_properties = crypto_factory_.GetFileDecryptionProperties(
+          kms_connection_config_, decryption_config);
+
+      decryptor_.DecryptFile(file, file_decryption_properties);
+    } else {
+      ::arrow::Result<std::shared_ptr<::arrow::fs::FileSystem>> filesystem =
+          MakeLocalFileSystem();
+      std::string file_name = GetFileName(double_wrapping, wrap_locally_,
+                                          internal_key_material, encryption_no);
+      std::shared_ptr<FilePath> readable = std::make_shared<FilePath>(
+          temp_dir_->path().ToString() + file_name, filesystem.ValueOrDie());
+      auto file_decryption_properties = crypto_factory_.GetFileDecryptionProperties(
+          kms_connection_config_, decryption_config, readable);
+
+      decryptor_.DecryptFile(file, file_decryption_properties);
+    }
+  }
 
-    decryptor_.DecryptFile(file, file_decryption_properties);
+  void RotateKeys(bool double_wrapping) {
+    ::arrow::Result<std::shared_ptr<::arrow::fs::FileSystem>> filesystem =
+        MakeLocalFileSystem();
+
+    std::shared_ptr<FilePath> readable =
+        std::make_shared<FilePath>(temp_dir_->path().ToString(), filesystem.ValueOrDie());

Review Comment:
   ```suggestion
       ::arrow::fs::FileSystem filesystem = MakeLocalFileSystem().ValueOrDie();
   
       std::shared_ptr<FilePath> readable =
           std::make_shared<FilePath>(temp_dir_->path().ToString(), filesystem);
   ```



##########
cpp/src/parquet/encryption/test_encryption_util.h:
##########
@@ -56,6 +59,8 @@ inline ::arrow::Result<std::unique_ptr<TemporaryDir>> temp_data_dir() {
   return TemporaryDir::Make("parquet-encryption-test-");
 }
 
+::arrow::Result<std::shared_ptr<::arrow::fs::FileSystem>> MakeLocalFileSystem();

Review Comment:
   Unused var?



##########
cpp/src/parquet/encryption/file_system_key_material_store.cc:
##########
@@ -0,0 +1,120 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/filesystem/filesystem.h"
+#include "arrow/testing/json_integration.h"
+#include "arrow/testing/json_internal.h"
+#include "arrow/util/io_util.h"
+#include "arrow/util/make_unique.h"
+
+#include "parquet/encryption/file_system_key_material_store.h"
+#include "parquet/encryption/key_material.h"
+
+namespace parquet {
+namespace encryption {
+
+constexpr const char FileSystemKeyMaterialStore::kKetMaterialFilePrefix[];
+constexpr const char FileSystemKeyMaterialStore::kTempFilePrefix[];
+constexpr const char FileSystemKeyMaterialStore::kKeyMaterialFileSuffix[];
+
+void FileSystemKeyMaterialStore::initialize(
+    const std::shared_ptr<FilePath>& parquet_file_path, bool temp_store) {
+  std::string full_prefix =
+      (temp_store ? std::string(FileSystemKeyMaterialStore::kTempFilePrefix) : "");
+  full_prefix =
+      full_prefix + std::string(FileSystemKeyMaterialStore::kKetMaterialFilePrefix);
+  std::string key_material_file_name =
+      full_prefix + parquet_file_path->base_name() +
+      std::string(FileSystemKeyMaterialStore::kKeyMaterialFileSuffix);
+  key_material_file_ = ::arrow::internal::make_unique<FilePath>(
+      parquet_file_path->dir_name() + "/" + key_material_file_name,
+      parquet_file_path->filesystem());
+}
+
+void FileSystemKeyMaterialStore::LoadKeyMaterialMap() {
+  ::arrow::Result<std::shared_ptr<::arrow::io::RandomAccessFile>> file =
+      key_material_file_->OpenReadable();
+  std::shared_ptr<::arrow::io::RandomAccessFile> input = file.ValueOrDie();
+  rj::Document document;
+  ::arrow::Result<std::shared_ptr<Buffer>> buff =
+      input->ReadAt(0, input->GetSize().ValueOrDie());
+  std::string buff_str = buff.ValueOrDie()->ToString();
+  ::arrow::util::string_view sv(buff_str);
+  document.Parse(sv.data(), sv.size());
+  for (rj::Value::ConstMemberIterator itr = document.MemberBegin();
+       itr != document.MemberEnd(); ++itr) {
+    key_material_map_.insert({itr->name.GetString(), itr->value.GetString()});
+  }
+}
+
+std::string FileSystemKeyMaterialStore::BuildKeyMaterialMapJson() {
+  rj::Document d;
+  auto& allocator = d.GetAllocator();
+  rj::Value json_map(rj::kObjectType);
+
+  rj::Value key(rj::kStringType);
+  rj::Value value(rj::kStringType);
+  for (auto it = key_material_map_.begin(); it != key_material_map_.end(); it++) {
+    key.SetString(it->first.c_str(), allocator);
+    value.SetString(it->second.c_str(), allocator);
+    json_map.AddMember(key, value, allocator);
+  }
+  rj::StringBuffer buffer;
+  rj::Writer<rj::StringBuffer> writer(buffer);
+  json_map.Accept(writer);
+  return buffer.GetString();
+}
+
+void FileSystemKeyMaterialStore::SaveMaterial() {
+  ::arrow::Result<std::shared_ptr<::arrow::io::OutputStream>> sink =
+      key_material_file_->OpenWriteable();
+  auto stream = sink.ValueOrDie();
+  std::string key_material_json = BuildKeyMaterialMapJson();
+  if (!::arrow::internal::GenericToStatus(stream->Write(key_material_json)).ok())

Review Comment:
   Why not check if the result itself is ok?'
   
   ```suggestion
     if (!stream->Write(key_material_json).ok())
   ```



##########
cpp/src/parquet/encryption/test_encryption_util.cc:
##########
@@ -48,6 +48,12 @@ std::string data_file(const char* file) {
   return ss.str();
 }
 
+::arrow::Result<std::shared_ptr<::arrow::fs::FileSystem>> MakeLocalFileSystem() {
+  std::shared_ptr<::arrow::fs::LocalFileSystem> local_fs =
+      std::make_shared<::arrow::fs::LocalFileSystem>();
+  return local_fs;
+}

Review Comment:
   Again, do we need to create fs utilities in Parquet? Especially if we are replacing a one-liner with one-liner?



##########
cpp/src/parquet/encryption/file_system_key_material_store.h:
##########
@@ -0,0 +1,85 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <set>
+#include <string>
+#include <unordered_map>
+
+#include "arrow/filesystem/filesystem.h"
+
+#include "parquet/encryption/file_key_material_store.h"
+#include "parquet/encryption/file_path.h"
+
+namespace parquet {
+namespace encryption {
+
+/// Key material can be stored outside the Parquet file, for example in a separate small
+/// file in the same folder. This is important for “key rotation”, when MEKs have to be
+/// changed (if compromised; or periodically, just in case) - without modifying the
+/// Parquet files (often  immutable).
+class PARQUET_EXPORT FileSystemKeyMaterialStore : public FileKeyMaterialStore {

Review Comment:
   It would be nice to put this into `file_key_material_store.h`. This doc comment is exactly the same as the base class, but we should make it clear how this differs from other approaches other concrete classes might take.



##########
cpp/src/parquet/encryption/file_key_unwrapper.cc:
##########
@@ -27,14 +27,22 @@ namespace encryption {
 
 using internal::KeyWithMasterId;
 
-FileKeyUnwrapper::FileKeyUnwrapper(KeyToolkit* key_toolkit,
-                                   const KmsConnectionConfig& kms_connection_config,
-                                   double cache_lifetime_seconds)
+FileKeyUnwrapper::FileKeyUnwrapper(
+    KeyToolkit* key_toolkit, const KmsConnectionConfig& kms_connection_config,
+    double cache_lifetime_seconds, const std::shared_ptr<FilePath>& parquet_file_path,
+    std::shared_ptr<FileKeyMaterialStore> key_material_store)
     : key_toolkit_(key_toolkit),
       kms_connection_config_(kms_connection_config),
-      cache_entry_lifetime_seconds_(cache_lifetime_seconds) {
+      cache_entry_lifetime_seconds_(cache_lifetime_seconds),
+      checked_key_material_internal_storage_(false),
+      parquet_file_path_(parquet_file_path) {
   kek_per_kek_id_ = key_toolkit_->kek_read_cache_per_token().GetOrCreateInternalCache(
       kms_connection_config.key_access_token(), cache_entry_lifetime_seconds_);
+
+  if (key_material_store != nullptr) {
+    key_material_store_ = key_material_store;
+    checked_key_material_internal_storage_ = true;

Review Comment:
   What if instead of having this bool we just checked `key_material_store != nullptr`?



##########
cpp/src/parquet/encryption/file_system_key_material_store.h:
##########
@@ -0,0 +1,85 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <set>
+#include <string>
+#include <unordered_map>
+
+#include "arrow/filesystem/filesystem.h"
+
+#include "parquet/encryption/file_key_material_store.h"
+#include "parquet/encryption/file_path.h"
+
+namespace parquet {
+namespace encryption {
+
+/// Key material can be stored outside the Parquet file, for example in a separate small
+/// file in the same folder. This is important for “key rotation”, when MEKs have to be
+/// changed (if compromised; or periodically, just in case) - without modifying the
+/// Parquet files (often  immutable).
+class PARQUET_EXPORT FileSystemKeyMaterialStore : public FileKeyMaterialStore {
+ public:
+  static constexpr const char kKetMaterialFilePrefix[] = "_KEY_MATERIAL_FOR_";
+  static constexpr const char kTempFilePrefix[] = "_TMP";
+  static constexpr const char kKeyMaterialFileSuffix[] = ".json";
+
+  FileSystemKeyMaterialStore() {}
+
+  /// Initializes key material store for a parquet file.
+  void initialize(const std::shared_ptr<FilePath>& parquet_file_path, bool temp_store);

Review Comment:
   If `temp_store` is true, then it uses the `kTempFilePrefix`. But when are temp files used?



##########
cpp/src/parquet/encryption/file_key_wrapper.cc:
##########
@@ -86,9 +88,22 @@ std::string FileKeyWrapper::GetEncryptionKeyMetadata(const std::string& data_key
   // Internal key material storage: key metadata and key material are the same
   if (store_key_material_internally) {
     return serialized_key_material;
-  } else {
-    throw ParquetException("External key material store is not supported yet.");
   }
+  // External key material storage: key metadata is a reference to a key in the material
+  // store
+  if (key_id_in_file.empty()) {
+    if (is_footer_key) {
+      key_id_in_file = KeyMaterial::kFooterKeyIdInFile;
+    } else {
+      key_id_in_file =
+          KeyMaterial::kColumnKeyIdInFilePrefix + std::to_string(key_counter_);
+      key_counter_++;

Review Comment:
   Why are we appending this integer? Who else needs to know about this ID? It seems like this detail deserves more comments.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org