You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "MJB127 (via GitHub)" <gi...@apache.org> on 2023/08/21 16:11:18 UTC
[GitHub] [airflow] MJB127 opened a new issue, #33588: Impossible to use sftp provider on AWS MWAA 2.6.3
MJB127 opened a new issue, #33588:
URL: https://github.com/apache/airflow/issues/33588
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
I tried to use the sftp provider on MWAA versions 2.6.3 that is available with Python 3.10.
However, this provider depends on paramiko, which in turn depends on bcrypt library.
The assorted libs are provided in a plugins.zip bundle as I'm using the private server option (this has outgoing access disabled for MWAA).
Now bcrypt doesn't provide a release for 3.10 and MWAA enforces the minor Python version.
As such, installation of the requirements fails. SFTP provider can be used on workers, but its not available for the webserver (invisible in the provider list, can't set up a connection through the UI).
### What you think should happen instead
_No response_
### How to reproduce
1. Prepare requirements.txt using the following constraints https://raw.githubusercontent.com/apache/airflow/constraints-2.6.3/constraints-3.10.txt and containing apache-airflow-providers-sftp==4.3.1.
2. Download the dependencies with pip download
3. Pack the dependencies into plugins.zip. Note it should contain the bcrypt wheel. For me pip installs version for Python 3.6 while running on Python 3.10. No version of bcrypt is available for Python 3.10
4. Prepare another requirements.txt, remove constraints and add (force to use the libs from the zip)
--find-links /usr/local/airflow/plugins
--no-index
6. Upload plugins and requirements file to S3
7. Create an MWAA instance using these two files
8. Once it's created look into Cloudwatch logs, there should be a message "No matching distribution found for bcrypt>=3.2"
### Operating System
Amazon Linux
### Versions of Apache Airflow Providers
apache-airflow-providers-sftp==4.3.1
### Deployment
Amazon (AWS) MWAA
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #33588: Impossible to use sftp provider on AWS MWAA 2.6.3
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #33588:
URL: https://github.com/apache/airflow/issues/33588#issuecomment-1686624292
this is an issue with MWAA , not airflow you should open the issue with their support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #33588: Impossible to use sftp provider on AWS MWAA 2.6.3
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #33588: Impossible to use sftp provider on AWS MWAA 2.6.3
URL: https://github.com/apache/airflow/issues/33588
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #33588: Impossible to use sftp provider on AWS MWAA 2.6.3
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #33588:
URL: https://github.com/apache/airflow/issues/33588#issuecomment-1686619503
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org