You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "adamreeve (via GitHub)" <gi...@apache.org> on 2024/11/25 21:31:28 UTC
[I] [C++][Parquet] Decryptor errors when scanning a dataset that uses uniform encryption [arrow]
adamreeve opened a new issue, #44852:
URL: https://github.com/apache/arrow/issues/44852
### Describe the bug, including details regarding any error messages, version, and platform.
@pitrou pointed out that `InternalFileDecryptor` reusing the `footer_data_decryptor_` could be problematic for multi-threaded Parquet reads: https://github.com/apache/arrow/issues/43057#issuecomment-2497334650
I confirmed that this does lead to decryptor errors when scanning a Dataset with Parquet files that use uniform encryption by modifying the existing Parquet Dataset encryption tests:
```diff
diff --git a/cpp/src/arrow/dataset/file_parquet_encryption_test.cc b/cpp/src/arrow/dataset/file_parquet_encryption_test.cc
index 0287d593d1..6a13b1ee37 100644
--- a/cpp/src/arrow/dataset/file_parquet_encryption_test.cc
+++ b/cpp/src/arrow/dataset/file_parquet_encryption_test.cc
@@ -90,7 +90,7 @@ class DatasetEncryptionTestBase : public ::testing::Test {
auto encryption_config =
std::make_shared<parquet::encryption::EncryptionConfiguration>(
std::string(kFooterKeyName));
- encryption_config->column_keys = kColumnKeyMapping;
+ encryption_config->uniform_encryption = true;
auto parquet_encryption_config = std::make_shared<ParquetEncryptionConfig>();
// Directly assign shared_ptr objects to ParquetEncryptionConfig members
parquet_encryption_config->crypto_factory = crypto_factory_;
```
This causes `DatasetEncryptionTest::WriteReadDatasetWithEncryption` to fail with an error like:
```
/home/adam/dev/arrow/cpp/src/arrow/dataset/file_parquet_encryption_test.cc:159: Failure
Failed
'_error_or_value28.status()' failed with IOError: AesDecryptor was wiped outDeserializing page header failed.
/home/adam/dev/arrow/cpp/src/parquet/arrow/reader.cc:109 LoadBatch(batch_size)
/home/adam/dev/arrow/cpp/src/parquet/arrow/reader.cc:1263 ReadColumn(static_cast<int>(i), row_groups, reader.get(), &column)
/home/adam/dev/arrow/cpp/src/arrow/util/parallel.h:95 func(i, inputs[i])
/home/adam/dev/arrow/cpp/src/arrow/dataset/file_parquet_encryption_test.cc:208: Failure
Expected: TestScanDataset() doesn't generate new fatal failures in the current thread.
Actual: it does.
```
For `LargeRowEncryptionTest::ReadEncryptLargeRows`, I sometimes get the same `AesDecryptor was wiped out` error, but also see errors like:
```
/home/adam/dev/arrow/cpp/src/arrow/dataset/file_parquet_encryption_test.cc:159: Failure
Failed
'_error_or_value28.status()' failed with IOError: Failed decryption finalization
/home/adam/dev/arrow/cpp/src/parquet/arrow/reader.cc:109 LoadBatch(batch_size)
/home/adam/dev/arrow/cpp/src/parquet/arrow/reader.cc:1263 ReadColumn(static_cast<int>(i), row_groups, reader.get(), &column)
/home/adam/dev/arrow/cpp/src/arrow/util/parallel.h:95 func(i, inputs[i])
/home/adam/dev/arrow/cpp/src/arrow/dataset/file_parquet_encryption_test.cc:265: Failure
Expected: TestScanDataset() doesn't generate new fatal failures in the current thread.
Actual: it does.
```
I don't think it's possible to reproduce this from PyArrow only, as the `uniform_encryption` setting isn't exposed in PyArrow.
### Component(s)
C++, Parquet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Parquet] Decryptor errors when scanning a dataset that uses uniform encryption [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou closed issue #44852: [C++][Parquet] Decryptor errors when scanning a dataset that uses uniform encryption
URL: https://github.com/apache/arrow/issues/44852
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Parquet] Decryptor errors when scanning a dataset that uses uniform encryption [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou closed issue #44852: [C++][Parquet] Decryptor errors when scanning a dataset that uses uniform encryption
URL: https://github.com/apache/arrow/issues/44852
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org