You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "dgrewal-tc (via GitHub)" <gi...@apache.org> on 2024/09/25 12:30:02 UTC
[I] FivetranOperator stuck in deferred state till it times out [airflow]
dgrewal-tc opened a new issue, #42465:
URL: https://github.com/apache/airflow/issues/42465
### Apache Airflow version
Other Airflow 2 version (please specify below)
### If "Other Airflow 2 version" selected, which one?
2.9.2
### What happened?
This is in relation to https://github.com/astronomer/airflow-provider-fivetran-async/issues/109
As mentioned in the [comments](https://github.com/astronomer/airflow-provider-fivetran-async/issues/109#issuecomment-2368036922), When we upgraded Airflow from 2.8.1 to 2.9.2, airflow-provider-fivetran-async operator started giving issues abruptly.
We are using the FivetranOperator to trigger a Fivetran connector. The connector completes in 5-10 minutes, but the Airflow task does not finish. It remains in a deferred state until it times out and fails.
Airflow Version : 2.9.2
airflow-provider-fivetran-async==2.1.0
Failed logs
```
2024-09-15, 10:26:53 UTC] INFO - Pausing task as DEFERRED.
[2024-09-15, 10:26:54 UTC] INFO - Task exited with return code 100 (task deferral)
[2024-09-15, 11:26:58 UTC] INFO - Resuming after deferral
[2024-09-15, 11:26:59 UTC] INFO - Executing <Task(FivetranOperator)> on 2024-09-14 08:00:00+00:00
[2024-09-15, 11:26:59 UTC] INFO - Started process 9527 to run task
[2024-09-15, 11:26:59 UTC] INFO - Job 918648: Subtask
[2024-09-15, 11:26:59 UTC] ERROR - Task failed with exception
Traceback (most recent call last):
airflow.exceptions.AirflowTaskTimeout
```
Success logs
```
[2024-09-15, 12:12:51 UTC] INFO - Pausing task as DEFERRED.
[2024-09-15, 12:12:52 UTC] INFO - Task exited with return code 100 (task deferral)
[2024-09-15, 12:12:52 UTC] INFO - Using connection ID 'id' for task execution.
[2024-09-15, 12:12:52 UTC] INFO - sync is still running...
[2024-09-15, 12:12:52 UTC] INFO - sleeping for 120 seconds.
[2024-09-15, 12:14:53 UTC] INFO - sync is still running...
[2024-09-15, 12:14:53 UTC] INFO - sleeping for 120 seconds.
[2024-09-15, 12:16:53 UTC] INFO - sync is still running...
[2024-09-15, 12:16:53 UTC] INFO - sleeping for 120 seconds.
[2024-09-15, 12:19:00 UTC] INFO - Resuming after deferral
[2024-09-15, 12:19:00 UTC] INFO - Executing <Task(FivetranOperator):> on 2024-09-14 08:00:00+00:00
[2024-09-15, 12:19:00 UTC] INFO - Started process 11601 to run task
[2024-09-15, 12:19:00 UTC] INFO - Job 918720: Subtask
[2024-09-15, 12:19:00 UTC] INFO - Fivetran connector finished syncing at 2024-09-15 12:17:07.840000+00:00
[2024-09-15, 12:19:00 UTC] INFO - Marking task as SUCCESS.
[2024-09-15, 12:19:00 UTC] INFO - Task exited with return code 0
```
Can someone please have a look ?
Thanks
### What you think should happen instead?
_No response_
### How to reproduce
Use FivetranOperator.
This issue does not happen every time we run it. It is very random.
### Operating System
Amazon Linux. VERSION=2023
### Versions of Apache Airflow Providers
_No response_
### Deployment
Amazon (AWS) MWAA
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] FivetranOperator stuck in deferred state till it times out [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #42465:
URL: https://github.com/apache/airflow/issues/42465#issuecomment-2440050123
Please elaborate why this is a bug in Airflow?
There were no similar reports on other providers thus my conclusion is that this is a bug in the specific operator that you are using. `FivetranOperator` is official operator of Apache-Airflow.
We are not investigating 3rd party code. If you believe the issue is with Airflow itself please provide reproduce example on a service that we can check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] FivetranOperator stuck in deferred state till it times out [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #42465:
URL: https://github.com/apache/airflow/issues/42465#issuecomment-2373951361
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] FivetranOperator stuck in deferred state till it times out [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #42465: FivetranOperator stuck in deferred state till it times out
URL: https://github.com/apache/airflow/issues/42465
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org