You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "SameerMesiah97 (via GitHub)" <gi...@apache.org> on 2026/02/04 21:48:50 UTC
[PR] Implement execution_timeout semantics for DbtCloudRunJobOperator in deferrable mode [airflow]
SameerMesiah97 opened a new pull request, #61472:
URL: https://github.com/apache/airflow/pull/61472
**Description**
This change implements `execution_timeout` **semantics** for `DbtCloudRunJobOperator` when running in deferrable mode.
Previously, when the operator deferred execution, `execution_timeout` was not enforced because the task process is no longer running and the scheduler cannot terminate it via SIGTERM. As a result, dbt Cloud jobs could continue running after the Airflow task exceeded its execution timeout.
This update restores parity with non-deferrable execution by explicitly enforcing execution timeouts through the trigger/operator interaction. The operator now computes an absolute execution deadline derived from `execution_timeout` before deferring, the trigger emits a timeout event when that deadline is exceeded, and the operator cancels the dbt Cloud job and fails the task when the event is received.
**Rationale**
In non-deferrable mode, `execution_timeout` is enforced by the scheduler, which terminates the task process and invokes `on_kill()` to cancel the external dbt Cloud job.
In deferrable mode, execution is handed off to a trigger running in the triggerer process, which does not have an associated worker process that can be terminated. However, **the absence of a worker process does not remove the requirement to honor task-level execution semantics**. From a user perspective, `execution_timeout` represents a hard task-level limit that should behave consistently regardless of whether execution is deferrable or not.
Without explicit handling in the trigger/operator interaction, `execution_timeout` silently stops working in deferrable mode, leading to leaked dbt Cloud jobs and inconsistent behavior between deferrable and non-deferrable execution. Deferrable execution should adapt how timeouts are enforced, but not whether they are enforced.
This change ensures `execution_timeout` semantics are preserved in deferrable mode and remain consistent with non-deferrable execution.
**Notes**
* The existing `timeout` parameter continues to limit only how long the operator waits for job completion and does not imply cancellation.
* When both `execution_timeout` and `timeout` are set, the earlier deadline takes precedence.
**Tests**
* Added trigger-level tests asserting timeout events when the execution deadline is exceeded.
* Added operator-level tests verifying job cancellation and task failure on execution timeout.
* Extended trigger serialization tests to ensure the optional `execution_deadline` field is always serialized.
**Documentation**
* The docstring for `DbtCloudRunJobTrigger` has been updated to document the new `execution_deadline` parameter and clarify its behavior.
* The docstring for `DbtCloudRunJobOperator` has been updated to clarify the behavior of the `timeout` parameter and distinguish it from task-level `execution_timeout`.
**Backwards Compatibility**
This change does not alter public APIs or method signatures. The runtime behavior changes in that dbt Cloud jobs are now explicitly cancelled when `execution_timeout` is reached during deferrable execution, whereas previously the job could continue running after the task timed out.
Closes: #61467
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement execution_timeout semantics for DbtCloudRunJobOperator in deferrable mode [airflow]
Posted by "josh-fell (via GitHub)" <gi...@apache.org>.
josh-fell commented on PR #61472:
URL: https://github.com/apache/airflow/pull/61472#issuecomment-4040752426
@vincbeck It's similar to what's described in https://github.com/apache/airflow/issues/62516 -- which I just ran into myself assuming setting `deferrable=True` would be a quick and non-breaking update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement execution_timeout semantics for DbtCloudRunJobOperator in deferrable mode [airflow]
Posted by "SameerMesiah97 (via GitHub)" <gi...@apache.org>.
SameerMesiah97 commented on PR #61472:
URL: https://github.com/apache/airflow/pull/61472#issuecomment-3939553975
> in case there is something a bit more complex withich requires access to services
I was just wondering whether screenshots are strictly required for this PR to be validated?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement execution_timeout semantics for DbtCloudRunJobOperator in deferrable mode [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #61472:
URL: https://github.com/apache/airflow/pull/61472#issuecomment-3905914904
> For future reference, would you advise including screenshots by default for any bugs? Or are explicit steps sufficient and this is more of a sanity check on a case-by-case basis?
in case there is something a bit more complex withich requires access to services
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement execution_timeout semantics for DbtCloudRunJobOperator in deferrable mode [airflow]
Posted by "SameerMesiah97 (via GitHub)" <gi...@apache.org>.
SameerMesiah97 commented on PR #61472:
URL: https://github.com/apache/airflow/pull/61472#issuecomment-3877003669
Requesting review for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement execution_timeout semantics for DbtCloudRunJobOperator in deferrable mode [airflow]
Posted by "SameerMesiah97 (via GitHub)" <gi...@apache.org>.
SameerMesiah97 commented on PR #61472:
URL: https://github.com/apache/airflow/pull/61472#issuecomment-4039588713
@potiuk
I have provided the screenshots you requested for this. Please review (and merge if appropriate).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement execution_timeout semantics for DbtCloudRunJobOperator in deferrable mode [airflow]
Posted by "SameerMesiah97 (via GitHub)" <gi...@apache.org>.
SameerMesiah97 commented on PR #61472:
URL: https://github.com/apache/airflow/pull/61472#issuecomment-4024725280
> This looks like more a bug/something we missed in Airflow in general rather than one bug in a provider? Am I wrong?
That’s definitely possible but at the moment I cannot see any opportunities for a generalisable abstraction because cancellation semantics may differ between operators. And whilst I suspect other operators might have this class of bug, I don’t have concrete evidence. So based on what I know now, this is the most appropriate solution I can think of for this operator. However, I will definitely check other operators to see if I can reproduce this type of bug for them as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement execution_timeout semantics for DbtCloudRunJobOperator in deferrable mode [airflow]
Posted by "SameerMesiah97 (via GitHub)" <gi...@apache.org>.
SameerMesiah97 commented on code in PR #61472:
URL: https://github.com/apache/airflow/pull/61472#discussion_r2920315446
##########
providers/dbt/cloud/src/airflow/providers/dbt/cloud/operators/dbt.py:
##########
@@ -212,16 +214,26 @@ def execute(self, context: Context):
raise DbtCloudJobRunException(f"Job run {self.run_id} has failed or has been cancelled.")
return self.run_id
+
+ # Derive absolute deadlines for deferrable execution.
+ # execution_timeout is a hard task-level limit (cancels the job),
+ # while timeout only limits how long we wait for the job to finish.
+ # If both are set, the earliest deadline wins.
end_time = time.time() + self.timeout
+ execution_deadline = None
+ if self.execution_timeout:
Review Comment:
I have added an entry for `execution_timeout` in the docstring.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement execution_timeout semantics for DbtCloudRunJobOperator in deferrable mode [airflow]
Posted by "SameerMesiah97 (via GitHub)" <gi...@apache.org>.
SameerMesiah97 commented on PR #61472:
URL: https://github.com/apache/airflow/pull/61472#issuecomment-4019207871
> That looks good. Though I would feel more comfortable if I saw some output of running job/ screenshot showing the behaviour. Possible @SameerMesiah97 ?
I used the following DAG to test the behavior of `execution_timeout` before and after my implementation:
```
from airflow import DAG
from airflow.providers.dbt.cloud.operators.dbt import DbtCloudRunJobOperator
from datetime import datetime, timedelta
with DAG(
dag_id="dbt_deferrable_timeout_none_repro",
start_date=datetime(2024, 1, 1),
schedule=None,
catchup=False,
) as dag:
run_dbt = DbtCloudRunJobOperator(
dbt_cloud_conn_id = "test_dbt_conn",
task_id="run_dbt",
job_id= **********0798,
wait_for_termination=True,
deferrable=True,
execution_timeout=timedelta(seconds=30),
)
```
**Behavior before the fix**
DAG run:
<img width="1353" height="681" alt="image" src="https://github.com/user-attachments/assets/8996a7ff-97b4-4bf1-b69d-f4d1b5514bfd" />
Job run status after `execution_timeout` period (30s) has elapsed:
<img width="1121" height="761" alt="image" src="https://github.com/user-attachments/assets/c2259506-5997-4814-84e6-1b69edfa6a4b" />
You can see that the job keeps running even after the timeout period.
**Behavior after the fix**
DAG run:
<img width="1663" height="841" alt="image" src="https://github.com/user-attachments/assets/d76eedbc-11ef-4fb6-bbe6-fef527cfda73" />
Job run status after `execution_timeout` period (30s) has elapsed:
<img width="1353" height="817" alt="image" src="https://github.com/user-attachments/assets/15ce77ab-862f-491a-8661-f8ed238b5e7f" />
The Job is cancelled via a request by the operator due to my implementation.
_Note: Only the last 4 digits of the JOB ID and RUN ID have been provided for security reasons but it should be sufficient for you to match the DAG runs to the Job runs. Additional private information has been blacked out as well for the same reason._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement execution_timeout semantics for DbtCloudRunJobOperator in deferrable mode [airflow]
Posted by "josh-fell (via GitHub)" <gi...@apache.org>.
josh-fell merged PR #61472:
URL: https://github.com/apache/airflow/pull/61472
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement execution_timeout semantics for DbtCloudRunJobOperator in deferrable mode [airflow]
Posted by "josh-fell (via GitHub)" <gi...@apache.org>.
josh-fell commented on code in PR #61472:
URL: https://github.com/apache/airflow/pull/61472#discussion_r2919792657
##########
providers/dbt/cloud/src/airflow/providers/dbt/cloud/operators/dbt.py:
##########
@@ -212,16 +214,26 @@ def execute(self, context: Context):
raise DbtCloudJobRunException(f"Job run {self.run_id} has failed or has been cancelled.")
return self.run_id
+
+ # Derive absolute deadlines for deferrable execution.
+ # execution_timeout is a hard task-level limit (cancels the job),
+ # while timeout only limits how long we wait for the job to finish.
+ # If both are set, the earliest deadline wins.
end_time = time.time() + self.timeout
+ execution_deadline = None
+ if self.execution_timeout:
Review Comment:
Let's explicitly call this out in the docstring since it seems important enough to bubble-up to users. The curse of inheritance here hurts us with API docs not including subclassed parameters.
##########
providers/dbt/cloud/src/airflow/providers/dbt/cloud/operators/dbt.py:
##########
@@ -70,7 +70,9 @@ class DbtCloudRunJobOperator(BaseOperator):
enabled but could be disabled to perform an asynchronous wait for a long-running job run execution
using the ``DbtCloudJobRunSensor``.
:param timeout: Time in seconds to wait for a job run to reach a terminal status for non-asynchronous
- waits. Used only if ``wait_for_termination`` is True. Defaults to 7 days.
+ waits. Used only if ``wait_for_termination`` is True.This limits how long the operator waits for the
Review Comment:
```suggestion
waits. Used only if ``wait_for_termination`` is True. This limits how long the operator waits for the
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement execution_timeout semantics for DbtCloudRunJobOperator in deferrable mode [airflow]
Posted by "SameerMesiah97 (via GitHub)" <gi...@apache.org>.
SameerMesiah97 commented on PR #61472:
URL: https://github.com/apache/airflow/pull/61472#issuecomment-3905034806
> That looks good. Though I would feel more comfortable if I saw some output of running job/ screenshot showing the behaviour. Possible @SameerMesiah97 ?
My free trial has expired but I will see if I can still use the DBT Cloud API. I will update the PR description with the screenshots once I am able to reproduce it again.
There is a link to the issue which has comprehensive reproduction steps so I believe it can be easily verified by someone with DBT Cloud API access.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org