You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "kandharvishnu (via GitHub)" <gi...@apache.org> on 2024/08/14 11:25:28 UTC
[I] Airflow SFTPToGCSOperator sftp file exist check [airflow]
kandharvishnu opened a new issue, #41472:
URL: https://github.com/apache/airflow/issues/41472
### Description
Currently, the operator fails if a file is not found on the SFTP server. To make this behavior more flexible, we could introduce a new parameter, `fail_on_sftp_file_not_exist`, allowing users to customize the operator's response in such scenarios. By setting this parameter, users can choose whether the operator should raise an error when a file is missing or simply log a message and continue processing.
### Use case/motivation
This enhancement would give users greater control over the operator's behaviour, particularly in environments where missing files are expected or where continued processing is preferred.
### Related issues
A similar [issue](https://github.com/apache/airflow/issues/40576) has been reported for the `SFTPToS3Operator`.
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow SFTPToGCSOperator sftp file exist check [airflow]
Posted by "geraj1010 (via GitHub)" <gi...@apache.org>.
geraj1010 commented on issue #41472:
URL: https://github.com/apache/airflow/issues/41472#issuecomment-2290195349
Is there an issue with using an `SFTPSensor` for checking file existence, prior to running `SFTPToGCSOperator`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow SFTPToGCSOperator sftp file exist check [airflow]
Posted by "geraj1010 (via GitHub)" <gi...@apache.org>.
geraj1010 commented on issue #41472:
URL: https://github.com/apache/airflow/issues/41472#issuecomment-2300595119
> > Is there an issue with using an `SFTPSensor` for checking file existence, prior to running `SFTPToGCSOperator`?
>
> The SFTP sensor will wait until the file is detected. This issue is to determine whether the task should fail if the file is not found.
I see. In that case, I don't think it makes sense to have the task succeed if a file is not found. A 'success' would not have much meaning anymore besides that the task simply ran. The interpretation of 'success' would become ambiguous i.e. how can you tell if a file was indeed moved or not without having to further dig into the logs? In addition, it doesn't feel very 'airflow-like' since we would be overloading the operator with functionality that already exists through a sensor.
I'm not sure what your DAG looks like, but perhaps you can use a `SFTPSensor` and trigger rules for your downstream tasks? There is `soft_fail` for sensors too, if you want the sensor to 'skip' instead of 'fail'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow SFTPToGCSOperator sftp file exist check [airflow]
Posted by "kandharvishnu (via GitHub)" <gi...@apache.org>.
kandharvishnu commented on issue #41472:
URL: https://github.com/apache/airflow/issues/41472#issuecomment-2301174472
> > > Is there an issue with using an `SFTPSensor` for checking file existence, prior to running `SFTPToGCSOperator`?
> >
> >
> > The SFTP sensor will wait until the file is detected. This issue is to determine whether the task should fail if the file is not found.
>
> I see. In that case, I don't think it makes sense to have the task succeed if a file is not found. A 'success' would not have much meaning anymore besides that the task simply ran. The interpretation of 'success' would become ambiguous i.e. how can you tell if a file was indeed moved or not without having to further dig into the logs? In addition, it doesn't feel very 'airflow-like' since we would be overloading the operator with functionality that already exists through a sensor.
>
There is `soft_fail` for sensors too, if you want the sensor to 'skip' instead of 'fail'.
I agree with your point, but this is an optional parameter. By default, it will fail the task if the file is not found. If the user sets the `fail_on_sftp_file_not_exist` to False, it would make some difference.
> I'm not sure what your DAG looks like, but perhaps you can use a `SFTPSensor` and trigger rules for your downstream tasks?
The final task of my DAG is looking for a file, and the dag_run should not fail, though the file is not present. And I don't want to use the sensor to poke for a file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow SFTPToGCSOperator sftp file exist check [airflow]
Posted by "gdivya6028 (via GitHub)" <gi...@apache.org>.
gdivya6028 commented on issue #41472:
URL: https://github.com/apache/airflow/issues/41472#issuecomment-2795053389
PR submitted for the issue got merged .. we can close this issue @kandharvishnu @potiuk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow SFTPToGCSOperator sftp file exist check [airflow]
Posted by "kandharvishnu (via GitHub)" <gi...@apache.org>.
kandharvishnu commented on issue #41472:
URL: https://github.com/apache/airflow/issues/41472#issuecomment-2795911683
The [#44320](https://github.com/apache/airflow/pull/44320) PR addresses the SFTPToS3Operator, whereas this one is for the SFTPToGCSOperator — please correct me if I'm mistaken. I'll take a closer look and work on it over the next weekend.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow SFTPToGCSOperator sftp file exist check [airflow]
Posted by "nguy4130 (via GitHub)" <gi...@apache.org>.
nguy4130 commented on issue #41472:
URL: https://github.com/apache/airflow/issues/41472#issuecomment-3387815089
I have made a PR to fix this https://github.com/apache/airflow/pull/56528. Please review and give feedbacks.. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow SFTPToGCSOperator sftp file exist check [airflow]
Posted by "geraj1010 (via GitHub)" <gi...@apache.org>.
geraj1010 commented on issue #41472:
URL: https://github.com/apache/airflow/issues/41472#issuecomment-2300591945
I see. In that case, I don't think it makes sense to have the task succeed if a file is not found. A 'success' would not have much meaning anymore besides that the task simply ran. The interpretation of 'success' would become ambiguous i.e. how can you tell if a file was indeed moved or not without having to further dig into the logs? In addition, it doesn't feel very 'airflow-like' since we would be overloading the operator with functionality that already exists through a sensor.
I'm not sure what your DAG looks like, but perhaps you can use a `SFTPSensor` and trigger rules for your downstream tasks? There is `soft_fail` for sensors too, if you want the sensor to 'skip' instead of 'fail'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow SFTPToGCSOperator sftp file exist check [airflow]
Posted by "kandharvishnu (via GitHub)" <gi...@apache.org>.
kandharvishnu commented on issue #41472:
URL: https://github.com/apache/airflow/issues/41472#issuecomment-2296575917
> Is there an issue with using an `SFTPSensor` for checking file existence, prior to running `SFTPToGCSOperator`?
The SFTP sensor will wait until the file is detected. This issue is to determine whether the task should fail if the file is not found.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow SFTPToGCSOperator sftp file exist check [airflow]
Posted by "geraj1010 (via GitHub)" <gi...@apache.org>.
geraj1010 commented on issue #41472:
URL: https://github.com/apache/airflow/issues/41472#issuecomment-2303477666
> > > > Is there an issue with using an `SFTPSensor` for checking file existence, prior to running `SFTPToGCSOperator`?
> > >
> > >
> > > The SFTP sensor will wait until the file is detected. This issue is to determine whether the task should fail if the file is not found.
> >
> >
> > I see. In that case, I don't think it makes sense to have the task succeed if a file is not found. A 'success' would not have much meaning anymore besides that the task simply ran. The interpretation of 'success' would become ambiguous i.e. how can you tell if a file was indeed moved or not without having to further dig into the logs? In addition, it doesn't feel very 'airflow-like' since we would be overloading the operator with functionality that already exists through a sensor.
> > There is `soft_fail` for sensors too, if you want the sensor to 'skip' instead of 'fail'.
>
> I agree with your point, but this is an optional parameter. By default, it will fail the task if the file is not found. If the user sets the `fail_on_sftp_file_not_exist` to False, it would make some difference.
>
> > I'm not sure what your DAG looks like, but perhaps you can use a `SFTPSensor` and trigger rules for your downstream tasks?
>
> The final task of my DAG is looking for a file, and the dag_run should not fail, though the file is not present. And I don't want to use the sensor to poke for a file.
That's unfortunate you don't want to use a sensor for checking for file existence. If the sensor failed (i.e. file didn't exist after some time), you could have it `soft_fail` and it would skip (instead of fail) and you could choose which downstream tasks get skipped or ran based on trigger rules.
At any rate, I think as long as you use the trigger rule `all_done` for one (or more) of the downstream tasks, the dag_run will still be marked as successful. So you could have your `SFTPToGCSOperator` task fail (because the file doesn't exist on SFTP), and have the trigger rule for all other downstream tasks set to `all_done`, and they will still run and the dag_run will still be considered successful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow SFTPToGCSOperator sftp file exist check [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #41472: Airflow SFTPToGCSOperator sftp file exist check
URL: https://github.com/apache/airflow/issues/41472
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org