You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "erg (via GitHub)" <gi...@apache.org> on 2025/11/21 22:03:12 UTC
[I] apache-airflow-providers-snowflake connector 6.6.0 not connecting -- "Could not deserialize key data" [airflow]
erg opened a new issue, #58573:
URL: https://github.com/apache/airflow/issues/58573
### Apache Airflow version
3.1.3
### If "Other Airflow 2/3 version" selected, which one?
_No response_
### What happened?
Looks like it's dying in openssl?
I have this same .pem file loading in Airflow 2.10.3 with `apache-airflow-providers-snowflake==5.7.0` and it works. It's possible that my openssl is missing some features that it needs to load this private key? It's aws linux on ec2.
```
[2025-11-21 15:19:26] INFO - Snowflake connection test failed: ('Could not deserialize key data. The data may be in an incorrect format, it may be encrypted with an unsupported algorithm, or it may be an unsupported key type (e.g. EC curves with explicit parameters).', [<OpenSSLError(code=503841036, lib=60, reason=524556, reason_text=unsupported)>]) source=unusual_prefix_265870fe9aeac5517b7f38b2a081d1e60a78a9d2_system_report_dag loc=system_report_dag.py:101
```
### What you think should happen instead?
_No response_
### How to reproduce
Copy/paste your creds into the Extra Fields JSON of the snowflake connector. I tried both the raw .pem format and the base64 version. Both have the above error. (Why doesn't it complain that my base64 doesn't decode?)
I have also tried setting up the connections with the command line and get the same error.
```
{"private_key_content":"-----BEGIN PRIVATE KEY-----\n...MEvgIB...a\n-----END PRIVATE KEY-----","warehouse":"GENERAL","account":"foobar.us-east-1"}
{"private_key_content":"LS0tLS1CRU...","warehouse":"GENERAL","account":"foobar.us-east-1"}
```
### Operating System
Amazon Linux
### Versions of Apache Airflow Providers
apache-airflow==3.1.3
apache-airflow-core==3.1.3
apache-airflow-providers-amazon==9.16.0
apache-airflow-providers-celery==3.13.0
apache-airflow-providers-cncf-kubernetes==10.9.0
apache-airflow-providers-common-compat==1.8.0
apache-airflow-providers-common-io==1.6.4
apache-airflow-providers-common-messaging==2.0.0
apache-airflow-providers-common-sql==1.28.2
apache-airflow-providers-docker==4.4.4
apache-airflow-providers-elasticsearch==6.3.4
apache-airflow-providers-fab==3.0.1
apache-airflow-providers-ftp==3.13.2
apache-airflow-providers-git==0.0.9
apache-airflow-providers-google==18.1.0
apache-airflow-providers-grpc==3.8.2
apache-airflow-providers-hashicorp==4.3.3
apache-airflow-providers-http==5.4.0
apache-airflow-providers-imap==3.9.3
apache-airflow-providers-microsoft-azure==12.8.0
apache-airflow-providers-microsoft-mssql==4.3.2
apache-airflow-providers-mysql==6.3.4
apache-airflow-providers-odbc==4.10.2
apache-airflow-providers-openai==1.6.3
apache-airflow-providers-openlineage==2.7.3
apache-airflow-providers-opensearch==1.7.4
apache-airflow-providers-postgres==6.4.0
apache-airflow-providers-qdrant==1.4.3
apache-airflow-providers-redis==4.3.2
apache-airflow-providers-sendgrid==4.1.4
apache-airflow-providers-sftp==5.4.1
apache-airflow-providers-slack==9.4.0
apache-airflow-providers-smtp==2.3.1
apache-airflow-providers-snowflake==6.6.0
apache-airflow-providers-sqlite==4.1.2
apache-airflow-providers-ssh==4.1.5
apache-airflow-providers-standard==1.9.1
### Deployment
Docker-Compose
### Deployment details
`docker compose up`
### Anything else?
_No response_
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] apache-airflow-providers-snowflake connector 6.6.0 not connecting -- "Could not deserialize key data" [airflow]
Posted by "hwang-cadent (via GitHub)" <gi...@apache.org>.
hwang-cadent commented on issue #58573:
URL: https://github.com/apache/airflow/issues/58573#issuecomment-3566817547
The code at line 361 in snowflake.py always base64-decodes private_key_content without checking if it's already raw PEM format. This causes failures when users provide raw PEM (starts with -----BEGIN PRIVATE KEY-----), as base64 decoding produces invalid bytes that fail to deserialize. I think the suggested Fix might add Auto-detect the format and handle both base64-encoded and raw PEM. This maintains backward compatibility with base64-encoded keys (as required since 6.3.0) while also supporting raw PEM for convenience.
The OpenSSL error may also indicate an unsupported key format or EC curve on AWS Linux. Ensure the key is PKCS#8 format (-----BEGIN PRIVATE KEY-----), not PKCS#1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] apache-airflow-providers-snowflake connector 6.6.0 not connecting -- "Could not deserialize key data" [airflow]
Posted by "SamanwayVolvo (via GitHub)" <gi...@apache.org>.
SamanwayVolvo commented on issue #58573:
URL: https://github.com/apache/airflow/issues/58573#issuecomment-3637220922
Ok. Got the problem now. Here is the breaking change: https://airflow.apache.org/docs/apache-airflow-providers-snowflake/6.7.0/changelog.html#breaking-changes
Basically the `private_key_content` should be base64 encoded. This is to avoid adding newline escape char problem in the JSON.
I think this issue should be closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] apache-airflow-providers-snowflake connector 6.6.0 not connecting -- "Could not deserialize key data" [airflow]
Posted by "erg (via GitHub)" <gi...@apache.org>.
erg commented on issue #58573:
URL: https://github.com/apache/airflow/issues/58573#issuecomment-3992927487
Still broken in `apache-airflow-providers-snowflake==6.9.0`
```
[2026-03-03 12:19:45] INFO - Snowflake connection test failed: ('Could not deserialize key data. The data may be in an incorrect format, the provided password may be incorrect, it may be encrypted with an unsupported algorithm, or it may be an unsupported key type (e.g. EC curves with explicit parameters).', [<OpenSSLError(code=503841036, lib=60, reason=524556, reason_text=unsupported)>])
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] apache-airflow-providers-snowflake connector 6.6.0 not connecting -- "Could not deserialize key data" [airflow]
Posted by "erg (via GitHub)" <gi...@apache.org>.
erg commented on issue #58573:
URL: https://github.com/apache/airflow/issues/58573#issuecomment-3647150919
I tried with 6.7.0 and the base64 encoded key -- it's super broken still. I did the few lines from my previous post above to encode the .pem and it does not work!
```
[2025-12-12 09:55:10] INFO - Snowflake connection test failed: ('Could not deserialize key data. The data may be in an incorrect format, it may be encrypted with an unsupported algorithm, or it may be an unsupported key type (e.g. EC curves with explicit parameters).', [<OpenSSLError(code=503841036, lib=60, reason=524556, reason_text=unsupported)>]) source=unusual_prefix_265870fe9aeac5517b7f38b2a081d1e60a78a9d2_system_report_dag loc=system_report_dag.py:96
```
Please reopen and fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] apache-airflow-providers-snowflake connector 6.6.0 not connecting -- "Could not deserialize key data" [airflow]
Posted by "erg (via GitHub)" <gi...@apache.org>.
erg commented on issue #58573:
URL: https://github.com/apache/airflow/issues/58573#issuecomment-3638857661
I tried base64 encoding it -- it's broken!
If you haven't personally base64 encoded your snowflake private rsa key and gotten the snowflake connector to work, then please reopen this bug.
https://airflow.apache.org/docs/apache-airflow-providers-snowflake/stable/connections/snowflake.html
```python
import base64
with open("path/to/your/private_key.pem", "rb") as key_file:
encoded_key = base64.b64encode(key_file.read()).decode("utf-8")
print(encoded_key)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] apache-airflow-providers-snowflake connector 6.6.0 not connecting -- "Could not deserialize key data" [airflow]
Posted by "hwang-cadent (via GitHub)" <gi...@apache.org>.
hwang-cadent commented on issue #58573:
URL: https://github.com/apache/airflow/issues/58573#issuecomment-3566915565
> If you look above, I also tried with a base64 encoded key. It matches the suggested base64 conversion in the docs.
>
> My key is PKCS#8.
>
> ```
> ❯ head -n1 rsa_key.pem
> -----BEGIN PRIVATE KEY-----
>
> # Convert to pkcs#8 and diff
> ❯ openssl pkcs8 -in rsa_key.pem -topk8 -nocrypt -out test_pkcs8.pem
> ❯ diff rsa_key.pem test_pkcs8.pem
> ```
>
> The bug report is that, despite my key being PKCS#8 and in the proper format, it still doesnt decode in openssl somehow.
Oh, in that way, the issue is likely in how the base64 string is decoded or passed to OpenSSL. base64.b64decode() at line 361 doesn't strip whitespace from the base64 string before decoding. I doubt when private_key_content is stored in JSON (especially with newlines like \n), the base64 string may contain whitespace that causes decoding to fail or produce invalid bytes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] apache-airflow-providers-snowflake connector 6.6.0 not connecting -- "Could not deserialize key data" [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #58573: apache-airflow-providers-snowflake connector 6.6.0 not connecting -- "Could not deserialize key data"
URL: https://github.com/apache/airflow/issues/58573
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] apache-airflow-providers-snowflake connector 6.6.0 not connecting -- "Could not deserialize key data" [airflow]
Posted by "erg (via GitHub)" <gi...@apache.org>.
erg commented on issue #58573:
URL: https://github.com/apache/airflow/issues/58573#issuecomment-3613156860
Yes, 5.7.0 worked first time for me. You have to pip install it separately, after the fact, without the constraints.txt file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] apache-airflow-providers-snowflake connector 6.6.0 not connecting -- "Could not deserialize key data" [airflow]
Posted by "erg (via GitHub)" <gi...@apache.org>.
erg commented on issue #58573:
URL: https://github.com/apache/airflow/issues/58573#issuecomment-3566899848
If you look above, I also tried with a base64 encoded key. It matches the suggested base64 conversion in the docs.
My key is PKCS#8.
```
❯ head -n1 rsa_key.pem
-----BEGIN PRIVATE KEY-----
# Convert to pkcs#8 and diff
❯ openssl pkcs8 -in rsa_key.pem -topk8 -nocrypt -out test_pkcs8.pem
❯ diff rsa_key.pem test_pkcs8.pem
```
The bug report is that, despite my key being PKCS#8 and in the proper format, it still doesnt decode in openssl somehow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] apache-airflow-providers-snowflake connector 6.6.0 not connecting -- "Could not deserialize key data" [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #58573:
URL: https://github.com/apache/airflow/issues/58573#issuecomment-3649853693
> Please reopen and fix. 5.7.0 works.
If you could fix, that would be greatm otherwise, I propose you create a nuew issue describing it well with latest version of the provider and reproduer.
You have to realise that the software released here comes without any guarantee. Bugs are fixed when there are some people who volunteer to fix it. You paid exactly 0 for the software - so you cannot expect things to be fixed because you wrote about it.
But if you describe it well in a new issue, with a clear reproducer, there is a chance (but only a chance) that someone will actually volunteer to work after hours - away from their family, or day job to actually fix a problem that you seem to have in your job. Nicely describing your problem with exact reproducer, is the least you ca do to increase chances of fixing the problem.
Of course - the best is that you make a fix yourself - with a PR. That would be a dream.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org