You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/03/28 20:06:48 UTC
[GitHub] [airflow] brettplarson opened a new issue #22576: SFTP connection hook not working when using inline Ed25519 key from Airflow connection
brettplarson opened a new issue #22576:
URL: https://github.com/apache/airflow/issues/22576
### Apache Airflow version
2.2.4 (latest released)
### What happened
I am trying to create an SFTP connection which includes the extra params of `private_key` which includes a txt output of my private key. Ie: `{"look_for_keys": "false", "no_host_key_check": "true", "private_key": "-----BEGIN OPENSSH PRIVATE KEY-----
keygoeshere==\n----END OPENSSH PRIVATE KEY-----"}`
When I test the connection I get the error `expected str, bytes or os.PathLike object, not Ed25519Key`
When I try and use this connection I get the following error:
```
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/sftp/hooks/sftp.py", line 208, in list_directory
conn = self.get_conn()
File "/home/airflow/.local/lib/python3.7/site-packages/tenacity/__init__.py", line 324, in wrapped_f
return self(f, *args, **kw)
File "/home/airflow/.local/lib/python3.7/site-packages/tenacity/__init__.py", line 404, in __call__
do = self.iter(retry_state=retry_state)
File "/home/airflow/.local/lib/python3.7/site-packages/tenacity/__init__.py", line 349, in iter
return fut.result()
File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 428, in result
return self.__get_result()
File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/home/airflow/.local/lib/python3.7/site-packages/tenacity/__init__.py", line 407, in __call__
result = fn(*args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/sftp/hooks/sftp.py", line 172, in get_conn
self.conn = pysftp.Connection(**conn_params)
File "/home/airflow/.local/lib/python3.7/site-packages/pysftp/__init__.py", line 142, in __init__
self._set_authentication(password, private_key, private_key_pass)
File "/home/airflow/.local/lib/python3.7/site-packages/pysftp/__init__.py", line 164, in _set_authentication
private_key_file = os.path.expanduser(private_key)
File "/usr/local/lib/python3.7/posixpath.py", line 235, in expanduser
path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not Ed25519Key
```
This only seems to happen for Ed25519 keys. RSA worked fine!
### What you think should happen instead
It should work, I don't specify this as an `Ed25519Key` I think the connection manager code is saving it as a paraminko key but when testing / using it as a DAG it is expecting a string!
I don't see why you can't save it as a paraminko key and use it in the connection.
Also it seems to work fine when using RSA keys, but super short keys are cooler!
### How to reproduce
Create a new Ed25519 ssh key and a new SFTP connection and copy the following into the extra field:
{"look_for_keys": "false", "no_host_key_check": "true", "private_key": "-----BEGIN RSA PRIVATE KEY----- Ed25519_key_goes_here -----END RSA PRIVATE KEY-----"}
Test should yield the failure `TypeError: expected str, bytes or os.PathLike object, not Ed25519Key`
### Operating System
RHEL 7.9 on host OS and Docker image for the rest.
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==3.0.0
apache-airflow-providers-celery==2.1.0
apache-airflow-providers-cncf-kubernetes==3.0.2
apache-airflow-providers-docker==2.4.1
apache-airflow-providers-elasticsearch==2.2.0
apache-airflow-providers-ftp==2.0.1
apache-airflow-providers-google==6.4.0
apache-airflow-providers-grpc==2.0.1
apache-airflow-providers-hashicorp==2.1.1
apache-airflow-providers-http==2.0.3
apache-airflow-providers-imap==2.2.0
apache-airflow-providers-microsoft-azure==3.6.0
apache-airflow-providers-mysql==2.2.0
apache-airflow-providers-odbc==2.0.1
apache-airflow-providers-postgres==3.0.0
apache-airflow-providers-redis==2.0.1
apache-airflow-providers-sendgrid==2.0.1
apache-airflow-providers-sftp==2.4.1
apache-airflow-providers-slack==4.2.0
apache-airflow-providers-sqlite==2.1.0
apache-airflow-providers-ssh==2.4.0
### Deployment
Other Docker-based deployment
### Deployment details
Docker image of 2.2.4 release with VERY minimal changes. (wget, curl, etc added)
### Anything else
RSA seems to work fine... only after a few hours of troubleshooting and writing this ticket did I learn that. 😿
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #22576: SFTP connection hook not working when using inline Ed25519 key from Airflow connection
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #22576:
URL: https://github.com/apache/airflow/issues/22576#issuecomment-1081102062
Feel free to make a PR to fix it (otherwise maybe someone can add this feature)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org