You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/12/27 10:07:12 UTC

[GitHub] [airflow] ReneTC opened a new issue, #28603: SFTPOperator does not finish downloading

ReneTC opened a new issue, #28603:
URL: https://github.com/apache/airflow/issues/28603

   ### Apache Airflow version
   
   Other Airflow 2 version (please specify below)
   
   ### What happened
   
   I have an issue with the `SFTPOperator` from `airflow.providers.sftp.operators.sftp`.. Mostly it works perfectly.
   However, sometimes it refuses to download the complete file and is just stuck "downloading". A job has been running for 7 days to download a 38 mb file. I can see in the output file there is 37.5 mb out of the 38 mb. Last output of the log is `Starting to transfer from ftp_server_file.csv to local_gcs_file.csv`
   
   But most of the time it works perfectly - It has downalded a 39 mb file before without issues.
   
   I'm downloading from a sftp server to google cloud storage in Airflow 2.3.3.
   
   
   ### What you think should happen instead
   
   Somehow this operator still think it is downloading. Which makes sense since the file is not 100% downloaded. But the download speed must be close to 0 byte per second. Job should terminate. 
   
   ### How to reproduce
   
   _No response_
   
   ### Operating System
   
   Google Cloud Composer
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Composer
   
   ### Deployment details
   
   _No response_
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on issue #28603: SFTPOperator does not finish downloading

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #28603:
URL: https://github.com/apache/airflow/issues/28603#issuecomment-1366737663

   Of course, we can re-open it if you test both keepalive and migration to newer version and the issue is still there (though I think some more debugging info would be needed for that).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on issue #28603: SFTPOperator does not finish downloading

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #28603:
URL: https://github.com/apache/airflow/issues/28603#issuecomment-1366733676

    One of the reasons for paramico change was that pysftp did not allow for the kind of configuration that direct paramiko would. You should likely look at `keepalive_interval` and possibly other parameters there. 
   
   The problem that you might experience is because some networking equipment between you and the server could have been restarted (usual thing in the cloud) and if you do not enable keep-alives, you might as a client not even be aware that the other end has stopped responding.
   
   The nature of TCP connection is such that there is no "physical" connection set up - those are just networking packets sent over the network to certain ports. So if you have no "keepalive" enabled, the client might simply not realize the other end is disconnected because if the other end did not have time to signal that it has died or the CLOSE message might not have been delivered. 
   
   Only [TCP keepalives](https://tldp.org/HOWTO/TCP-Keepalive-HOWTO/overview.html) can give you the certainty  that you detect this situation. And they are disabled by default - you need to enable them - in case of SSHook (and SFTPHook derives from it in the new implementation) via the `keepalive_interval` 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] Taragolis commented on issue #28603: SFTPOperator does not finish downloading

Posted by GitBox <gi...@apache.org>.
Taragolis commented on issue #28603:
URL: https://github.com/apache/airflow/issues/28603#issuecomment-1366540261

   @ReneTC Could you provide a bit more information about which version of providers are you use?
   
   Did you tried latest version of SFTP provider? Because I see in [provider changelog](https://airflow.apache.org/docs/apache-airflow-providers-sftp/stable/index.html#changelog) that we in 4.0.0 switch from `pysftp` to `paramico` and without actual version it is quite difficult to determine is it problem in old version of package or  this error still persist in new one.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on issue #28603: SFTPOperator does not finish downloading

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #28603:
URL: https://github.com/apache/airflow/issues/28603#issuecomment-1366737096

   I am closing it provisionally - because I believe keepalive is the solution. I believe you might also not need to upgrade the provider - because keepalive seems to be also available in previous version of the hook (the SFTP provider before also used SSHook as a base and I believe you have 3.0.0 version of ssh provider: https://airflow.apache.org/docs/apache-airflow-providers-ssh/3.0.0/_api/airflow/providers/ssh/hooks/ssh/index.html  So likely you can test keepalive without migration (though of course it is advised).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ReneTC commented on issue #28603: SFTPOperator does not finish downloading

Posted by GitBox <gi...@apache.org>.
ReneTC commented on issue #28603:
URL: https://github.com/apache/airflow/issues/28603#issuecomment-1366555038

   I see. I am using
   
   Composer version:
   2.0.28
   
   Airflow version:
   2.3.3
   
   My providers settings are set to:
   ` apache-airflow-providers-sftp  = "~=2.5.0"`
   
   I will convince admin to update and report back. 
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk closed issue #28603: SFTPOperator does not finish downloading

Posted by GitBox <gi...@apache.org>.
potiuk closed issue #28603: SFTPOperator does not finish downloading
URL: https://github.com/apache/airflow/issues/28603


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] boring-cyborg[bot] commented on issue #28603: SFTPOperator does not finish downloading

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #28603:
URL: https://github.com/apache/airflow/issues/28603#issuecomment-1365777596

   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org