You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "dgrewal-tc (via GitHub)" <gi...@apache.org> on 2024/10/27 22:13:22 UTC

[D] FivetranOperator stuck in deferred state till it times out [airflow]

GitHub user dgrewal-tc created a discussion: FivetranOperator stuck in deferred state till it times out

### Apache Airflow version

Other Airflow 2 version (please specify below)

### If "Other Airflow 2 version" selected, which one?

2.9.2

### What happened?

This is in relation to https://github.com/astronomer/airflow-provider-fivetran-async/issues/109

As mentioned in the [comments](https://github.com/astronomer/airflow-provider-fivetran-async/issues/109#issuecomment-2368036922), When we upgraded Airflow from 2.8.1 to 2.9.2, FivetranOperator in airflow-provider-fivetran-async package started giving issues abruptly.

We are using the FivetranOperator to trigger a Fivetran connector. The connector completes in 5-10 minutes, but the Airflow task does not finish. It remains in a deferred state until it times out and fails.

Airflow Version : 2.9.2
airflow-provider-fivetran-async==2.1.0

Failed logs
```
2024-09-15, 10:26:53 UTC] INFO - Pausing task as DEFERRED.
[2024-09-15, 10:26:54 UTC] INFO - Task exited with return code 100 (task deferral)

[2024-09-15, 11:26:58 UTC] INFO - Resuming after deferral
[2024-09-15, 11:26:59 UTC] INFO - Executing <Task(FivetranOperator)> on 2024-09-14 08:00:00+00:00
[2024-09-15, 11:26:59 UTC] INFO - Started process 9527 to run task
[2024-09-15, 11:26:59 UTC] INFO - Job 918648: Subtask
[2024-09-15, 11:26:59 UTC] ERROR - Task failed with exception
Traceback (most recent call last):
airflow.exceptions.AirflowTaskTimeout
```

Success logs
```
[2024-09-15, 12:12:51 UTC] INFO - Pausing task as DEFERRED. 
[2024-09-15, 12:12:52 UTC] INFO - Task exited with return code 100 (task deferral)

[2024-09-15, 12:12:52 UTC] INFO - Using connection ID 'id' for task execution.
[2024-09-15, 12:12:52 UTC] INFO - sync is still running...
[2024-09-15, 12:12:52 UTC] INFO - sleeping for 120 seconds.
[2024-09-15, 12:14:53 UTC] INFO - sync is still running...
[2024-09-15, 12:14:53 UTC] INFO - sleeping for 120 seconds.
[2024-09-15, 12:16:53 UTC] INFO - sync is still running...
[2024-09-15, 12:16:53 UTC] INFO - sleeping for 120 seconds.

[2024-09-15, 12:19:00 UTC] INFO - Resuming after deferral
[2024-09-15, 12:19:00 UTC] INFO - Executing <Task(FivetranOperator):> on 2024-09-14 08:00:00+00:00
[2024-09-15, 12:19:00 UTC] INFO - Started process 11601 to run task
[2024-09-15, 12:19:00 UTC] INFO - Job 918720: Subtask
[2024-09-15, 12:19:00 UTC] INFO - Fivetran connector finished syncing at 2024-09-15 12:17:07.840000+00:00
[2024-09-15, 12:19:00 UTC] INFO - Marking task as SUCCESS.
[2024-09-15, 12:19:00 UTC] INFO - Task exited with return code 0
```

Can someone please have a look ?

Thanks

### What you think should happen instead?

_No response_

### How to reproduce

Use FivetranOperator.
This issue does not happen every time we run it. It is very random.


### Operating System

Amazon Linux. VERSION=2023

### Versions of Apache Airflow Providers

_No response_

### Deployment

Amazon (AWS) MWAA

### Deployment details

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)


GitHub link: https://github.com/apache/airflow/discussions/43423

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org


Re: [D] FivetranOperator stuck in deferred state till it times out [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
GitHub user boring-cyborg[bot] added a comment to the discussion: FivetranOperator stuck in deferred state till it times out

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.


GitHub link: https://github.com/apache/airflow/discussions/43423#discussioncomment-11068951

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org


Re: [D] FivetranOperator stuck in deferred state till it times out [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: FivetranOperator stuck in deferred state till it times out

Converting it to a discussion - this does not seem likely it's an Airflow issue, it's more of a 3rd-party - while we can still discuss it, this is not clear if it is an "airflow issue" - more evidence needs to be provided - if you can't do it @dgrewal-tc - reach out to whoever supports the provider you use so that they can investingate and provide more evidences.

GitHub link: https://github.com/apache/airflow/discussions/43423#discussioncomment-11068957

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org


Re: [D] FivetranOperator stuck in deferred state till it times out [airflow]

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
GitHub user eladkal added a comment to the discussion: FivetranOperator stuck in deferred state till it times out

Please elaborate why this is a bug in Airflow?
There were no similar reports on other providers thus my conclusion is that this is a bug in the specific operator that you are using. `FivetranOperator` is official operator of Apache-Airflow.

We are not investigating 3rd party code. If you believe the issue is with Airflow itself please provide reproduce example on a service that we can check.

GitHub link: https://github.com/apache/airflow/discussions/43423#discussioncomment-11068952

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org


Re: [D] FivetranOperator stuck in deferred state till it times out [airflow]

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
GitHub user eladkal edited a comment on the discussion: FivetranOperator stuck in deferred state till it times out

Please elaborate why this is a bug in Airflow?
There were no similar reports on other providers thus my conclusion is that this is a bug in the specific operator that you are using. `FivetranOperator` is not official operator of Apache-Airflow.

We are not investigating 3rd party code. If you believe the issue is with Airflow itself please provide reproduce example on a service that we can check.

GitHub link: https://github.com/apache/airflow/discussions/43423#discussioncomment-11068952

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org