You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "dylanschw (via GitHub)" <gi...@apache.org> on 2026/02/17 19:17:04 UTC
[PR] Use Task SDK BaseHook in Amazon executors [airflow]
dylanschw opened a new pull request, #62090:
URL: https://github.com/apache/airflow/pull/62090
Closes #58205
## What
Update Amazon Batch and ECS executors to use the Task SDK BaseHook import path. (Update Amazon Batch and ECS executors to use the Task SDK `BaseHook` import path instead of the deprecated `airflow.hooks.base.BaseHook`.)
## Why
Avoid deprecated BaseHook import usage and align these executors with the Airflow Task SDK hook location. The previous import path is deprecated. This change aligns the Amazon executors with the current Task SDK structure and removes usage of deprecated core hook imports.
## Tests
Verified by running Amazon provider executor unit tests:
AIRFLOW__SECRETS__BACKEND="airflow.secrets.environment_variables.EnvironmentVariablesBackend" \
AIRFLOW_CONN_SSH_DEFAULT='ssh://airflowtest:airflowtest@127.0.0.1:22?__extra__={"look_for_keys":false,"allow_agent":false,"allow_host_key_change":true}' \
PYTHONPATH="$PWD/airflow-core/src:$PWD/task-sdk/src:$PWD/devel-common/src:$PWD/dev:$PWD:$PROVIDERS_SRC" \
.venv-breeze/bin/pytest -q providers/amazon/tests -k "batch_executor or ecs_executor"
All relevant Batch and ECS executor tests pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Use Task SDK BaseHook in Amazon executors [airflow]
Posted by "SameerMesiah97 (via GitHub)" <gi...@apache.org>.
SameerMesiah97 commented on code in PR #62090:
URL: https://github.com/apache/airflow/pull/62090#discussion_r2819869700
##########
providers/amazon/src/airflow/providers/amazon/aws/executors/ecs/ecs_executor.py:
##########
@@ -69,6 +69,19 @@
class AwsEcsExecutor(BaseExecutor):
+ """Executor that runs Airflow tasks on AWS ECS."""
+
+ def _ti_to_execute_workload_cmd(self, ti):
Review Comment:
Maybe this functionality can be included in the executor/utils folder to avoid duplication.. I do not think this is a massive issue as it is only duplicated once but I think it should be considered. I would wait for the codeowner to weigh in here.
##########
providers/amazon/src/airflow/providers/amazon/aws/executors/ecs/ecs_executor.py:
##########
@@ -69,6 +69,19 @@
class AwsEcsExecutor(BaseExecutor):
+ """Executor that runs Airflow tasks on AWS ECS."""
+
+ def _ti_to_execute_workload_cmd(self, ti):
+ from airflow.executors.workloads import ExecuteTask
+
+ workload_json = ExecuteTask.make(ti).model_dump_json()
+ return [
+ "python3",
+ "-m",
+ "airflow.sdk.execution_time.execute_workload",
+ "--json-string",
+ workload_json,
+ ]
Review Comment:
Is this compatible with airflow 2.x? It seems like this will not work for versions before Airflow 3.0 as `airflow.sdk` was introduced then. I think you should have a look at PR #58207. There are issues with that but I think the implementation in that PR was more backwards compatible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Use Task SDK BaseHook in Amazon executors [airflow]
Posted by "dylanschw (via GitHub)" <gi...@apache.org>.
dylanschw commented on code in PR #62090:
URL: https://github.com/apache/airflow/pull/62090#discussion_r2822752613
##########
providers/amazon/src/airflow/providers/amazon/aws/executors/ecs/ecs_executor.py:
##########
@@ -69,6 +69,19 @@
class AwsEcsExecutor(BaseExecutor):
+ """Executor that runs Airflow tasks on AWS ECS."""
+
+ def _ti_to_execute_workload_cmd(self, ti):
+ from airflow.executors.workloads import ExecuteTask
+
+ workload_json = ExecuteTask.make(ti).model_dump_json()
+ return [
+ "python3",
+ "-m",
+ "airflow.sdk.execution_time.execute_workload",
+ "--json-string",
+ workload_json,
+ ]
Review Comment:
Thanks for calling this out. As written, using airflow.sdk.execution_time.execute_workload is not compatible with Airflow 2.x since airflow.sdk is only available in Airflow 3.x. I will update the executor command construction to branch on Airflow version, using the SDK entrypoint for Airflow 3.x and the legacy task runner entrypoint for Airflow 2.x. I will add unit tests that assert the exact command built for both Airflow >= 3.0 and Airflow < 3.0, including the entrypoint and arguments like --json-string. Since the regression occurred there, I will take another look at test_try_adopt_task_instances for ECS and Batch. If unskipping is not feasible due to environment assumptions, I will add a targeted test to cover the same behavior with mocking.
On factoring this into executor/utils, that sounds reasonable. I can do it in this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Use Task SDK BaseHook in Amazon executors [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on PR #62090:
URL: https://github.com/apache/airflow/pull/62090#issuecomment-4000676547
Closing this one since @vincbeck put in a fix already for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Use Task SDK BaseHook in Amazon executors [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas closed pull request #62090: Use Task SDK BaseHook in Amazon executors
URL: https://github.com/apache/airflow/pull/62090
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Use Task SDK BaseHook in Amazon executors [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #62090:
URL: https://github.com/apache/airflow/pull/62090#issuecomment-3916636626
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:
- Pay attention to the quality of your code (ruff, mypy and type annotations). Our [prek-hooks]( https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst#prerequisites-for-prek-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/airflow-core/docs/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/dev/breeze/doc/README.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#coding-style-and-best-practices).
- Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org