You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "SameerMesiah97 (via GitHub)" <gi...@apache.org> on 2026/05/22 21:14:49 UTC
[PR] Fix deferrable execution_timeout regression in DbtCloudRunJobOperator [airflow]
SameerMesiah97 opened a new pull request, #67360:
URL: https://github.com/apache/airflow/pull/67360
**Description**
This change fixes a regression introduced in PR #66449 affecting `DbtCloudRunJobOperator` in deferrable mode.
Previously, `execution_timeout` was passed directly to `defer()`. The operator now explicitly passes `timeout=None` to `defer()` while still preserving `execution_deadline` handling within the trigger/operator flow.
**Rationale**
In deferrable mode, dbt job cancellation is performed within `execute_complete()` when the trigger emits a timeout event.
However, framework-level deferred timeout handling bypasses `execute_complete()` entirely and does not invoke `on_kill()` in the triggerer process. As a result, dbt jobs could continue running after the Airflow task timed out, leading to leaked workloads and excessive resource consumption.
While this weakens the framework-level deferred timeout guarantee, in this case preventing leaked external workloads takes precedence because there is currently no equivalent cleanup path in the triggerer process.
**Tests**
Added an operator-level regression test verifying that `defer()` is called with `timeout=None` while `execution_deadline` handling remains preserved for dbt job timeout cancellation processing.
**Backwards Compatibility**
This change does not modify public APIs or method signatures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix deferrable execution_timeout regression in DbtCloudRunJobOperator [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl merged PR #67360:
URL: https://github.com/apache/airflow/pull/67360
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix deferrable execution_timeout regression in DbtCloudRunJobOperator [airflow]
Posted by "SameerMesiah97 (via GitHub)" <gi...@apache.org>.
SameerMesiah97 commented on code in PR #67360:
URL: https://github.com/apache/airflow/pull/67360#discussion_r3293218180
##########
providers/dbt/cloud/tests/unit/dbt/cloud/operators/test_dbt.py:
##########
@@ -180,6 +180,58 @@ def test_execute_failed_before_getting_deferred(
dbt_op.execute(MagicMock())
assert not mock_defer.called
+ @patch(
+ "airflow.providers.dbt.cloud.hooks.dbt.DbtCloudHook.get_job_run_status",
+ return_value=DbtCloudJobRunStatus.QUEUED.value,
+ )
+ @patch("airflow.providers.dbt.cloud.operators.dbt.DbtCloudRunJobOperator.defer")
+ @patch("airflow.providers.dbt.cloud.operators.dbt.DbtCloudRunJobTrigger")
+ @patch("airflow.providers.dbt.cloud.hooks.dbt.DbtCloudHook.get_connection")
+ @patch(
+ "airflow.providers.dbt.cloud.hooks.dbt.DbtCloudHook.trigger_job_run",
+ return_value=mock_response_json(DEFAULT_ACCOUNT_JOB_RUN_RESPONSE),
+ )
+ def test_execute_deferrable_does_not_pass_execution_timeout_to_defer(
+ self,
+ mock_trigger_job_run,
+ mock_dbt_hook,
+ mock_dbt_trigger,
+ mock_defer,
+ mock_job_run_status,
+ ):
+ dbt_op = DbtCloudRunJobOperator(
+ dbt_cloud_conn_id=ACCOUNT_ID_CONN,
+ task_id=TASK_ID,
+ job_id=JOB_ID,
+ check_interval=1,
+ timeout=3,
+ dag=self.dag,
+ deferrable=True,
+ execution_timeout=timedelta(seconds=3),
Review Comment:
I have no reservations about doing that. But I am unsure of what value that would provide considering that the key behavior being covered here is the argument for the `timeout` parameter for `defer` always being set to `None`, regardless of the value of `execution_timeout`. If you could clarify your reasoning, that would be great. I understand that it is a trivial addition but I think we should avoid adding cases if they are not strictly needed.
Thank you for the quick review anyway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix deferrable execution_timeout regression in DbtCloudRunJobOperator [airflow]
Posted by "Shrividya (via GitHub)" <gi...@apache.org>.
Shrividya commented on code in PR #67360:
URL: https://github.com/apache/airflow/pull/67360#discussion_r3291387101
##########
providers/dbt/cloud/tests/unit/dbt/cloud/operators/test_dbt.py:
##########
@@ -180,6 +180,58 @@ def test_execute_failed_before_getting_deferred(
dbt_op.execute(MagicMock())
assert not mock_defer.called
+ @patch(
+ "airflow.providers.dbt.cloud.hooks.dbt.DbtCloudHook.get_job_run_status",
+ return_value=DbtCloudJobRunStatus.QUEUED.value,
+ )
+ @patch("airflow.providers.dbt.cloud.operators.dbt.DbtCloudRunJobOperator.defer")
+ @patch("airflow.providers.dbt.cloud.operators.dbt.DbtCloudRunJobTrigger")
+ @patch("airflow.providers.dbt.cloud.hooks.dbt.DbtCloudHook.get_connection")
+ @patch(
+ "airflow.providers.dbt.cloud.hooks.dbt.DbtCloudHook.trigger_job_run",
+ return_value=mock_response_json(DEFAULT_ACCOUNT_JOB_RUN_RESPONSE),
+ )
+ def test_execute_deferrable_does_not_pass_execution_timeout_to_defer(
+ self,
+ mock_trigger_job_run,
+ mock_dbt_hook,
+ mock_dbt_trigger,
+ mock_defer,
+ mock_job_run_status,
+ ):
+ dbt_op = DbtCloudRunJobOperator(
+ dbt_cloud_conn_id=ACCOUNT_ID_CONN,
+ task_id=TASK_ID,
+ job_id=JOB_ID,
+ check_interval=1,
+ timeout=3,
+ dag=self.dag,
+ deferrable=True,
+ execution_timeout=timedelta(seconds=3),
Review Comment:
Might be better to add another case with execution_timeout = None as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix deferrable execution_timeout regression in DbtCloudRunJobOperator [airflow]
Posted by "Shrividya (via GitHub)" <gi...@apache.org>.
Shrividya commented on code in PR #67360:
URL: https://github.com/apache/airflow/pull/67360#discussion_r3293286457
##########
providers/dbt/cloud/tests/unit/dbt/cloud/operators/test_dbt.py:
##########
@@ -180,6 +180,58 @@ def test_execute_failed_before_getting_deferred(
dbt_op.execute(MagicMock())
assert not mock_defer.called
+ @patch(
+ "airflow.providers.dbt.cloud.hooks.dbt.DbtCloudHook.get_job_run_status",
+ return_value=DbtCloudJobRunStatus.QUEUED.value,
+ )
+ @patch("airflow.providers.dbt.cloud.operators.dbt.DbtCloudRunJobOperator.defer")
+ @patch("airflow.providers.dbt.cloud.operators.dbt.DbtCloudRunJobTrigger")
+ @patch("airflow.providers.dbt.cloud.hooks.dbt.DbtCloudHook.get_connection")
+ @patch(
+ "airflow.providers.dbt.cloud.hooks.dbt.DbtCloudHook.trigger_job_run",
+ return_value=mock_response_json(DEFAULT_ACCOUNT_JOB_RUN_RESPONSE),
+ )
+ def test_execute_deferrable_does_not_pass_execution_timeout_to_defer(
+ self,
+ mock_trigger_job_run,
+ mock_dbt_hook,
+ mock_dbt_trigger,
+ mock_defer,
+ mock_job_run_status,
+ ):
+ dbt_op = DbtCloudRunJobOperator(
+ dbt_cloud_conn_id=ACCOUNT_ID_CONN,
+ task_id=TASK_ID,
+ job_id=JOB_ID,
+ check_interval=1,
+ timeout=3,
+ dag=self.dag,
+ deferrable=True,
+ execution_timeout=timedelta(seconds=3),
Review Comment:
I understand. The addition of test case was suggested more of about documenting the default behavior explicitly in the test suite, but I'll leave it to your discretion!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org