You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "joern-m-h (via GitHub)" <gi...@apache.org> on 2025/12/24 06:03:01 UTC

[I] Add parameter `deferrable` to `AzureBatchOperator` [airflow]

joern-m-h opened a new issue, #59779:
URL: https://github.com/apache/airflow/issues/59779

   ### Description
   
   The AzureBatchOperator currently has no option to set it to `deferrable`, which would allow the task to be run by an async low-resource worker ("triggerer"). Therefore Azure Batch tasks block a regular worker, even though they only ping the Azure platform from time to time. 
   
   I propose to add this parameter and changing the operator to use asynchronous calls.
   
   As a reference, the AWS Batch operator has a parameter `deferrable`. 
   
   ### Use case/motivation
   
   Adding this parameter and changing the operator to use asynchronous calls would reduce resource consumption when using Azure Batch.
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Add parameter `deferrable` to `AzureBatchOperator` [airflow]

Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl closed issue #59779: Add parameter `deferrable` to `AzureBatchOperator`
URL: https://github.com/apache/airflow/issues/59779


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Add parameter `deferrable` to `AzureBatchOperator` [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #59779:
URL: https://github.com/apache/airflow/issues/59779#issuecomment-3986420084

   @Ankurdeewan We are unassigning you from this issue as part of our updated [assignment policy](https://github.com/apache/airflow/blob/main/contributing-docs/04_how_to_contribute.rst#contribute-code-changes).
   
   This is not meant to discourage your contribution — quite the opposite! You are still very welcome to work on this issue and submit a PR for it. Simply comment that you are working on it and open a PR when ready.
   
   We found that formal assignments were not working well, as they often prevented others from contributing when the assignee was not actively working on the issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Add parameter `deferrable` to `AzureBatchOperator` [airflow]

Posted by "SameerMesiah97 (via GitHub)" <gi...@apache.org>.
SameerMesiah97 commented on issue #59779:
URL: https://github.com/apache/airflow/issues/59779#issuecomment-4433394597

   @potiuk @Ankurdeewan 
   
   I have opened a draft PR to address this issue building upon the work in PR #59798 (which as now been closed for over a month). Credit has been given to @Ankurdeewan in the PR description.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Add parameter `deferrable` to `AzureBatchOperator` [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #59779:
URL: https://github.com/apache/airflow/issues/59779#issuecomment-3688778296

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Add parameter `deferrable` to `AzureBatchOperator` [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #59779:
URL: https://github.com/apache/airflow/issues/59779#issuecomment-3691854232

   > I’m happy to take this on and implement it following the AWSBatchOperator/BatchJobTrigger pattern, unless there’s a different direction that would be preferred.
   
   Feel free. I think when PR is raised, the implementation can be assessed, but "consistency with other, similar things" is important - you should look for other Azure operators if there are other similar patterns, but if not - borrowing the patterns from other providers is as good as it can get.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Add parameter `deferrable` to `AzureBatchOperator` [airflow]

Posted by "Ankurdeewan (via GitHub)" <gi...@apache.org>.
Ankurdeewan commented on issue #59779:
URL: https://github.com/apache/airflow/issues/59779#issuecomment-4011126166

   Hi @potiuk — just to flag that I already have an open PR addressing this issue. I had it assigned and the fix is ready, but it hasn't had a reviewer yet. Could you help get a reviewer on the PR? Happy to address any feedback! #59798


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Add parameter `deferrable` to `AzureBatchOperator` [airflow]

Posted by "Ankurdeewan (via GitHub)" <gi...@apache.org>.
Ankurdeewan commented on issue #59779:
URL: https://github.com/apache/airflow/issues/59779#issuecomment-3689180235

   Looking at the current implementation, `AzureBatchOperator` submits the job and then spends the remaining of its execution synchronously polling Azure Batch. Both pool readiness and task completion are handled via repeated SDK GET/list calls wrapped in sleep loops (`wait_for_all_node_state()` and `wait_for_job_tasks_to_complete()`), with no callbacks, streams, or long-lived connections involved.
   
   That control flow is inherently deferrable and effectively matches the AWS Batch operator’s model prior to deferral. Refactoring this to submit once, defer to a trigger for async polling, and resume on a terminal state would remove unnecessary worker blocking while keeping behavior unchanged, with `deferrable=False` remaining the default for compatibility.
   
   I’m happy to take this on and implement it following the AWSBatchOperator/BatchJobTrigger pattern, unless there’s a different direction that would be preferred.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org