You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/12/27 10:07:12 UTC
[GitHub] [airflow] ReneTC opened a new issue, #28603: SFTPOperator does not finish downloading
ReneTC opened a new issue, #28603:
URL: https://github.com/apache/airflow/issues/28603
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
I have an issue with the `SFTPOperator` from `airflow.providers.sftp.operators.sftp`.. Mostly it works perfectly.
However, sometimes it refuses to download the complete file and is just stuck "downloading". A job has been running for 7 days to download a 38 mb file. I can see in the output file there is 37.5 mb out of the 38 mb. Last output of the log is `Starting to transfer from ftp_server_file.csv to local_gcs_file.csv`
But most of the time it works perfectly - It has downalded a 39 mb file before without issues.
I'm downloading from a sftp server to google cloud storage in Airflow 2.3.3.
### What you think should happen instead
Somehow this operator still think it is downloading. Which makes sense since the file is not 100% downloaded. But the download speed must be close to 0 byte per second. Job should terminate.
### How to reproduce
_No response_
### Operating System
Google Cloud Composer
### Versions of Apache Airflow Providers
_No response_
### Deployment
Composer
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #28603: SFTPOperator does not finish downloading
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #28603:
URL: https://github.com/apache/airflow/issues/28603#issuecomment-1366737663
Of course, we can re-open it if you test both keepalive and migration to newer version and the issue is still there (though I think some more debugging info would be needed for that).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #28603: SFTPOperator does not finish downloading
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #28603:
URL: https://github.com/apache/airflow/issues/28603#issuecomment-1366733676
One of the reasons for paramico change was that pysftp did not allow for the kind of configuration that direct paramiko would. You should likely look at `keepalive_interval` and possibly other parameters there.
The problem that you might experience is because some networking equipment between you and the server could have been restarted (usual thing in the cloud) and if you do not enable keep-alives, you might as a client not even be aware that the other end has stopped responding.
The nature of TCP connection is such that there is no "physical" connection set up - those are just networking packets sent over the network to certain ports. So if you have no "keepalive" enabled, the client might simply not realize the other end is disconnected because if the other end did not have time to signal that it has died or the CLOSE message might not have been delivered.
Only [TCP keepalives](https://tldp.org/HOWTO/TCP-Keepalive-HOWTO/overview.html) can give you the certainty that you detect this situation. And they are disabled by default - you need to enable them - in case of SSHook (and SFTPHook derives from it in the new implementation) via the `keepalive_interval`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on issue #28603: SFTPOperator does not finish downloading
Posted by GitBox <gi...@apache.org>.
Taragolis commented on issue #28603:
URL: https://github.com/apache/airflow/issues/28603#issuecomment-1366540261
@ReneTC Could you provide a bit more information about which version of providers are you use?
Did you tried latest version of SFTP provider? Because I see in [provider changelog](https://airflow.apache.org/docs/apache-airflow-providers-sftp/stable/index.html#changelog) that we in 4.0.0 switch from `pysftp` to `paramico` and without actual version it is quite difficult to determine is it problem in old version of package or this error still persist in new one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #28603: SFTPOperator does not finish downloading
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #28603:
URL: https://github.com/apache/airflow/issues/28603#issuecomment-1366737096
I am closing it provisionally - because I believe keepalive is the solution. I believe you might also not need to upgrade the provider - because keepalive seems to be also available in previous version of the hook (the SFTP provider before also used SSHook as a base and I believe you have 3.0.0 version of ssh provider: https://airflow.apache.org/docs/apache-airflow-providers-ssh/3.0.0/_api/airflow/providers/ssh/hooks/ssh/index.html So likely you can test keepalive without migration (though of course it is advised).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ReneTC commented on issue #28603: SFTPOperator does not finish downloading
Posted by GitBox <gi...@apache.org>.
ReneTC commented on issue #28603:
URL: https://github.com/apache/airflow/issues/28603#issuecomment-1366555038
I see. I am using
Composer version:
2.0.28
Airflow version:
2.3.3
My providers settings are set to:
` apache-airflow-providers-sftp = "~=2.5.0"`
I will convince admin to update and report back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #28603: SFTPOperator does not finish downloading
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #28603: SFTPOperator does not finish downloading
URL: https://github.com/apache/airflow/issues/28603
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #28603: SFTPOperator does not finish downloading
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #28603:
URL: https://github.com/apache/airflow/issues/28603#issuecomment-1365777596
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org