You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Shashwatpandey4 (via GitHub)" <gi...@apache.org> on 2025/05/18 18:47:30 UTC
[PR] Fix: Support .expand(overrides=...) in AWS BatchOperator for dynamic task mapping [airflow]
Shashwatpandey4 opened a new pull request, #50751:
URL: https://github.com/apache/airflow/pull/50751
This PR adds support for dynamically mapping the `overrides` argument in `BatchOperator` via `.expand()`.
Currently, attempting to call `.expand(overrides=[...])` on `BatchOperator` raises:
TypeError: BatchOperator.expand() got an unexpected keyword argument 'overrides'
This happens because `overrides` is not listed in the operator's `template_fields`, and not handled for dynamic task mapping.
- Adds `overrides` as a supported field in the constructor
- Updates `template_fields` and `template_fields_renderers`
- Uses the new overrides field (or falls back to container_overrides) when calling submit_job() with containerOverrides.
Closes #50535
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix: Support .expand(overrides=...) in AWS BatchOperator for dynamic task mapping [airflow]
Posted by "Shashwatpandey4 (via GitHub)" <gi...@apache.org>.
Shashwatpandey4 closed pull request #50751: Fix: Support .expand(overrides=...) in AWS BatchOperator for dynamic task mapping
URL: https://github.com/apache/airflow/pull/50751
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix: Support .expand(overrides=...) in AWS BatchOperator for dynamic task mapping [airflow]
Posted by "davidcroda (via GitHub)" <gi...@apache.org>.
davidcroda commented on PR #50751:
URL: https://github.com/apache/airflow/pull/50751#issuecomment-2894765106
This seems to change the property from `container_overrides` to `overrides`. The correct parameter is `container_overrides` and it works with expand already (though there is another bug preventing this from working in airflow 3 documented here: https://github.com/apache/airflow/issues/50535`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix: Support .expand(overrides=...) in AWS BatchOperator for dynamic task mapping [airflow]
Posted by "davidcroda (via GitHub)" <gi...@apache.org>.
davidcroda commented on code in PR #50751:
URL: https://github.com/apache/airflow/pull/50751#discussion_r2098223266
##########
providers/amazon/src/airflow/providers/amazon/aws/operators/batch.py:
##########
@@ -310,7 +316,7 @@ def submit_job(self, context: Context):
"arrayProperties": self.array_properties,
"parameters": self.parameters,
"tags": self.tags,
- "containerOverrides": self.container_overrides,
+ "containerOverrides": self.overrides,
Review Comment:
Why is this being renamed? `container_overrides` is the correct parameter name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix: Support .expand(overrides=...) in AWS BatchOperator for dynamic task mapping [airflow]
Posted by "Shashwatpandey4 (via GitHub)" <gi...@apache.org>.
Shashwatpandey4 commented on PR #50751:
URL: https://github.com/apache/airflow/pull/50751#issuecomment-2895652534
I've reverted the `batch.py` change so the `container_overrides` property remains untouched. I’ve instead added a unit test to validate that `expand(container_overrides=[...])` works as expected, which gives us regression coverage without modifying the operator itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix: Support .expand(overrides=...) in AWS BatchOperator for dynamic task mapping [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #50751:
URL: https://github.com/apache/airflow/pull/50751#issuecomment-2889760816
Can you add unit test to avoid regression?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix: Support .expand(overrides=...) in AWS BatchOperator for dynamic task mapping [airflow]
Posted by "Shashwatpandey4 (via GitHub)" <gi...@apache.org>.
Shashwatpandey4 commented on PR #50751:
URL: https://github.com/apache/airflow/pull/50751#issuecomment-2891904253
sure :+1:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org