You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "ChanTheDataExplorer (via GitHub)" <gi...@apache.org> on 2024/07/25 23:03:12 UTC
[I] Use Amazon KMS for encryption having error: "OSError: Incorrect key to columns mapping in column keys property:" [arrow]
ChanTheDataExplorer opened a new issue, #43426:
URL: https://github.com/apache/arrow/issues/43426
### Describe the usage question you have. Please include as many useful details as possible.
Hi, I'm trying to use the columnar encryption but having problem to make it work. Below is the sample code
```
KMS_KEY_ARN= 'arn:aws:kms:ap-southeast-1:643458469770:key/4c9195a3-bb54-40d3-b199-9f9bf6ea9dcf'
FOOTER_KEY_NAME = "footer_key"
COL_KEY_NAME = "column_key"
table = pa.Table.from_pydict({
'a': ['hello'],
'b': ['goodbye'],
'c': ['womp']
})
encryption_config = pe.EncryptionConfiguration(
footer_key=KMS_KEY_ARN,
column_keys={
KMS_KEY_ARN: ["a", "b", "c"],
},
encryption_algorithm="AES_GCM_V1",
cache_lifetime=timedelta(minutes=5.0),
data_key_length_bits=256
)
kms_connection_config = pe.KmsConnectionConfig(
custom_kms_conf={
FOOTER_KEY_NAME: FOOTER_KEY_ARN,
COL_KEY_NAME: FOOTER_KEY_ARN,
}
)
crypto_factory = pe.CryptoFactory(kms_factory)
file_encryption_properties = crypto_factory.file_encryption_properties(
kms_connection_config, encryption_config)
with pq.ParquetWriter(path, table.schema, encryption_properties=file_encryption_properties) as writer:
writer.write_table(table)
```
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Use Amazon KMS for encryption having error: "OSError: Incorrect key to columns mapping in column keys property:" [arrow]
Posted by "channingdata (via GitHub)" <gi...@apache.org>.
channingdata commented on issue #43426:
URL: https://github.com/apache/arrow/issues/43426#issuecomment-2259816283
i got encryption to work by modiying the kms client. now im having problem with the decryption. im having the following error:
`botocore.errorfactory.InvalidCiphertextException: An error occurred (InvalidCiphertextException) when calling the Decrypt operation: `
Possible reason:
it's just weird that im having that the wrapped key is having length of 3 when reading the encrypted file but should have 16 based on the encryption logs
aws_kms_client.py
`import logging
import base64
import boto3
import pyarrow.parquet.encryption as pe
import random
# Configure logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
class AwsKmsClient(pe.KmsClient):
def __init__(self, kms_client, key_arn_map):
super().__init__()
self.kms_client = kms_client
self.key_arn_map = key_arn_map
def wrap_key(self, key_bytes, master_key_identifier):
key_arn = self.key_arn_map.get(master_key_identifier, master_key_identifier)
logger.info(f"Wrapping key. Master key identifier: {master_key_identifier}")
logger.info(f"Key bytes length: {len(key_bytes)}")
logger.info(f"Key bytes to wrap: {key_bytes!r}")
key_arn = self.key_arn_map.get(master_key_identifier, master_key_identifier)
logger.info(f"Using KMS key ARN: {key_arn}")
try:
response = self.kms_client.encrypt(
KeyId=key_arn,
Plaintext=key_bytes
)
ciphertext_blob = response['CiphertextBlob']
logger.info(f"Ciphertext blob: {ciphertext_blob}")
logger.info(f"Ciphertext blob type: {type(ciphertext_blob)}")
return ciphertext_blob
except Exception as e:
logger.error(f"Error wrapping key: {e}")
raise
def unwrap_key(self, wrapped_key, master_key_identifier):
key_arn = self.key_arn_map.get(master_key_identifier, master_key_identifier)
logger.info(f"Unwrapping key. Master key identifier: {master_key_identifier}")
logger.info(f"Initial wrapped key: {wrapped_key!r}") # Use !r to see raw string representation
logger.info(f"Initial wrapped key type: {type(wrapped_key)}")
logger.info(f"Initial wrapped key length: {len(wrapped_key) if wrapped_key else 0}")
if not wrapped_key:
logger.error("Wrapped key is empty. Skipping decryption.")
return b''
if isinstance(wrapped_key, str):
wrapped_key = wrapped_key.encode('utf-8')
logger.info(f"Final wrapped key: {wrapped_key!r}")
logger.info(f"Final wrapped key type: {type(wrapped_key)}")
logger.info(f"Final wrapped key length: {len(wrapped_key)}")
try:
response = self.kms_client.decrypt(
KeyId=key_arn,
CiphertextBlob=wrapped_key
)
plaintext = response['Plaintext']
logger.info(f"Plaintext key: {plaintext}")
return plaintext
except Exception as e:
logger.error(f"Error unwrapping key: {e}")
raise
def kms_factory(kms_connection_configuration, key_arn_map):
session = boto3.Session(
profile_name='some-profile', # Replace with your AWS profile
region_name='ap-southeast-1' # Replace with your AWS region
)
kms_client = session.client('kms')
return AwsKmsClient(kms_client, key_arn_map)
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Use Amazon KMS for encryption having error: "OSError: Incorrect key to columns mapping in column keys property:" [arrow]
Posted by "channingdata (via GitHub)" <gi...@apache.org>.
channingdata closed issue #43426: Use Amazon KMS for encryption having error: "OSError: Incorrect key to columns mapping in column keys property:"
URL: https://github.com/apache/arrow/issues/43426
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Use Amazon KMS for encryption having error: "OSError: Incorrect key to columns mapping in column keys property:" [arrow]
Posted by "channingdata (via GitHub)" <gi...@apache.org>.
channingdata commented on issue #43426:
URL: https://github.com/apache/arrow/issues/43426#issuecomment-2259839447
Already figured the issue. Need to add base64 encoding and base64 decoding in wrap and unwrap_key functions for my KMS client.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org