You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "vincbeck (via GitHub)" <gi...@apache.org> on 2026/02/20 21:04:34 UTC
[I] [Multi-team] [Testing] Executor - AWS ECS [airflow]
vincbeck opened a new issue, #62246:
URL: https://github.com/apache/airflow/issues/62246
# Test 1. Same executor
- Have the same executor across two teams but different config
- Ensure each team gets the correct config/behaviour
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Multi-team] [Testing] Executor - AWS ECS [airflow]
Posted by "shivaam (via GitHub)" <gi...@apache.org>.
shivaam commented on issue #62246:
URL: https://github.com/apache/airflow/issues/62246#issuecomment-3986397354
I will try to reproduce it on my end. Since I was setting up a completely new environment, I wonder if it was a configuration issue on my side. Let me see if I can reproduce it on my end, before you deep dive into it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Multi-team] [Testing] Executor - AWS ECS [airflow]
Posted by "shivaam (via GitHub)" <gi...@apache.org>.
shivaam commented on issue #62246:
URL: https://github.com/apache/airflow/issues/62246#issuecomment-3978952332
@vincbeck — for the ECS executor integration test, what's your setup? Do you have existing infra/scripts for the AWS side (clusters, task defs, RDS), or do you spin it up manually? Want to make sure I'm not rebuilding something that already exists.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Multi-team] [Testing] Executor - AWS ECS [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on issue #62246:
URL: https://github.com/apache/airflow/issues/62246#issuecomment-4006622784
Cool, so I think we're happy with this behaviour then @shivaam. Thanks for calling it out though! It was good to double check it 🙂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Multi-team] [Testing] Executor - AWS ECS [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on issue #62246:
URL: https://github.com/apache/airflow/issues/62246#issuecomment-4001138789
> Do the executors also need `AIRFLOW__CORE__MULTI_TEAM` = True environment
> variables?
>
> Steps to reproduce
> - Run airflow with multi_team=True.
> - Start ecs executor without AIRFLOW__CORE__MULTI_TEAM` = True environment flag.
>
> Error
> ```
> {"timestamp":"2026-03-02T04:34:49.374043Z","level":"error","event":"Top level error","logger":"task","filename":"task_runner.py","lineno":1827,"error_detail":[{"exc_type":"AirflowConfigException","exc_value":"Section `dag_processor` key `dag_bundle_config_list` cannot have a team name when multi-team mode is disabled.
>
> ```
This seems like a reasonable expectation and the error message seems pretty clear on what the user needs to do to fix it. Do you agree @vincbeck?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Multi-team] [Testing] Executor - AWS ECS [airflow]
Posted by "shivaam (via GitHub)" <gi...@apache.org>.
shivaam commented on issue #62246:
URL: https://github.com/apache/airflow/issues/62246#issuecomment-3988158386
@o-nikolas if the executor does not have the AIRFLOW__CORE__MULTI_TEAM: 'True' environment variables they fail. Is this expected or by design? Curious, about the reason.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Multi-team] [Testing] Executor - AWS ECS [airflow]
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on issue #62246:
URL: https://github.com/apache/airflow/issues/62246#issuecomment-3986372844
> When using dag_bundle_config_list with team_name in multi-team mode, the dag-processor's
sync_bundles_to_db method logs "Added new DAG bundle" for team-scoped bundles but the records
are not persisted to the dag_bundle table. Non-team bundles (like shared_dags and example_dags)
persist correctly. The dag-processor then continuously logs "Bundle model not found" for the
team bundles, and their DAGs are never processed.
I am really surprised by this one. For each of my multi-team testing I have a setup with multiple dag bundles and I never saw this error. I can successfully see the Dags that are in the different Dag bundles I set-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Multi-team] [Testing] Executor - AWS ECS [airflow]
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on issue #62246:
URL: https://github.com/apache/airflow/issues/62246#issuecomment-3986409275
> I will try to reproduce it on my end. Since I was setting up a completely new environment, I wonder if it was a configuration issue on my side. Let me see if I can reproduce it on my end, before you deep dive into it.
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Multi-team] [Testing] Executor - AWS ECS [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on issue #62246:
URL: https://github.com/apache/airflow/issues/62246#issuecomment-3986137875
> Thanks for your testing [@shivaam](https://github.com/shivaam). Pinging [@o-nikolas](https://github.com/o-nikolas) here to answer your questions more accurately than me
The ECS testing looks good, but the issues are bundle related? I think you might be the best to examine that one Vincent 😬
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Multi-team] [Testing] Executor - AWS ECS [airflow]
Posted by "shivaam (via GitHub)" <gi...@apache.org>.
shivaam commented on issue #62246:
URL: https://github.com/apache/airflow/issues/62246#issuecomment-4001014798
Do the executors also need `AIRFLOW__CORE__MULTI_TEAM` = True environment
variables?
On Wed, 4 Mar 2026 at 14:20, Niko Oliveira ***@***.***> wrote:
> *o-nikolas* left a comment (apache/airflow#62246)
> <https://github.com/apache/airflow/issues/62246#issuecomment-4000666058>
>
> @o-nikolas <https://github.com/o-nikolas> if the executor does not have
> the AIRFLOW__CORE__MULTI_TEAM: 'True' environment variables they fail. Is
> this expected or by design? Curious, about the reason.
>
> Can you say more? Is this when running tasks that are from a scheduler
> that is using multi-team? Or they just fail always and in all cases when
> multi-team is not enabled?
>
> It happens only when running tasks with a scheduler with multi-team
> enabled. All I need to fix it is add the env variables in executors as well.
>
> Can you share the traceback you're seeing and rough steps to reproduce it?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/airflow/issues/62246#issuecomment-4000666058>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABRJ7GISW2QTYQRN7YHFNVL4PCT4JAVCNFSM6AAAAACV25DPBSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DAMBQGY3DMMBVHA>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Multi-team] [Testing] Executor - AWS ECS [airflow]
Posted by "shivaam (via GitHub)" <gi...@apache.org>.
shivaam commented on issue #62246:
URL: https://github.com/apache/airflow/issues/62246#issuecomment-3981615565
Tested multi-team ECS executor config isolation with two teams sharing the same AwsEcsExecutor class but with independent configurations.
Setup:
Airflow running from main (3.2.0dev) on EC2
Two teams: team_alpha and team_beta
Each team has its own [team_name=aws_ecs_executor] config section pointing to separate ECS clusters and task definitions
Worker images built from source using breeze prod-image build and pushed to ECR
Config:
```
[core]
executor = LocalExecutor;team_alpha=AwsEcsExecutor;team_beta=AwsEcsExecutor
[team_alpha=aws_ecs_executor]
cluster = alpha-cluster
task_definition = alpha-task-def:2
[team_beta=aws_ecs_executor]
cluster = beta-cluster
task_definition = beta-task-def:2
```
Results:
Triggered alpha_simple_dag → ECS task launched in alpha-cluster with alpha-task-def:2 ✅
Triggered beta_simple_dag → ECS task launched in beta-cluster with beta-task-def:2 ✅
No cross-contamination: alpha tasks only appear in alpha-cluster, beta tasks only in beta-cluster ✅
Bundle routing confirmed from ECS task command payloads:
- Alpha: bundle_info.name=team_alpha_dags, dag_rel_path=alpha_dag.py ✅
- Beta: bundle_info.name=team_beta_dags, dag_rel_path=beta_dag.py ✅
My tasks failed as the worker image was older but the task were triggered correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Multi-team] [Testing] Executor - AWS ECS [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #62246:
URL: https://github.com/apache/airflow/issues/62246#issuecomment-3986362136
@shivaam We are unassigning you from this issue as part of our updated [assignment policy](https://github.com/apache/airflow/blob/main/contributing-docs/04_how_to_contribute.rst#contribute-code-changes).
This is not meant to discourage your contribution — quite the opposite! You are still very welcome to work on this issue and submit a PR for it. Simply comment that you are working on it and open a PR when ready.
We found that formal assignments were not working well, as they often prevented others from contributing when the assignee was not actively working on the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Multi-team] [Testing] Executor - AWS ECS [airflow]
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on issue #62246:
URL: https://github.com/apache/airflow/issues/62246#issuecomment-3986350302
Oops, I read too fast :| My bad. I'll try to reproduce it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Multi-team] [Testing] Executor - AWS ECS [airflow]
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on issue #62246:
URL: https://github.com/apache/airflow/issues/62246#issuecomment-4005446629
> > Do the executors also need `AIRFLOW__CORE__MULTI_TEAM` = True environment
> > variables?
> > Steps to reproduce
> >
> > * Run airflow with multi_team=True.
> > * Start ecs executor without AIRFLOW__CORE__MULTI_TEAM` = True environment flag.
> >
> > Error
> > ```
> > {"timestamp":"2026-03-02T04:34:49.374043Z","level":"error","event":"Top level error","logger":"task","filename":"task_runner.py","lineno":1827,"error_detail":[{"exc_type":"AirflowConfigException","exc_value":"Section `dag_processor` key `dag_bundle_config_list` cannot have a team name when multi-team mode is disabled.
> > ```
>
> This seems like a reasonable expectation and the error message seems pretty clear on what the user needs to do to fix it. Do you agree [@vincbeck](https://github.com/vincbeck)?
Looks good to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Multi-team] [Testing] Executor - AWS ECS [airflow]
Posted by "shivaam (via GitHub)" <gi...@apache.org>.
shivaam commented on issue #62246:
URL: https://github.com/apache/airflow/issues/62246#issuecomment-3988247316
@vincbeck I was able to replicate the issue by resetting my DB and restarting airflow. Here is an analysis from Kiro and I can attache the whole log file if you need that.
```
Startup — all 4 bundles load and get "added":
02:23:06.165 [info] DAG bundles loaded: team_alpha_dags, team_beta_dags, shared_dags, example_dags
02:23:06.182 [info] Added new DAG bundle team_alpha_dags to the database
02:23:06.188 [info] Added new DAG bundle team_beta_dags to the database
02:23:06.192 [info] Added new DAG bundle shared_dags to the database
02:23:06.193 [info] Added new DAG bundle example_dags to the database
Less than 1 second later — team bundles can't be found in the DB:
02:23:07.121 [warning] Bundle model not found for team_alpha_dags
02:23:07.997 [warning] Bundle model not found for team_beta_dags
Meanwhile the non-team bundles work fine:
02:23:08.497 [info] Refreshing bundle shared_dags
02:23:08.522 [info] Searching for files in shared_dags at /tmp/airflow/dag_bundles/shared_dags
02:23:08.523 [info] Found 1 files for bundle shared_dags
02:23:08.539 [info] Refreshing bundle example_dags
02:23:08.558 [info] Found 61 files for bundle example_dags
Then it repeats every ~5 seconds, never resolving — all the way to the end of the log at 02:32:02:
02:23:11.300 [warning] Bundle model not found for team_alpha_dags
02:23:11.306 [warning] Bundle model not found for team_beta_dags
02:23:16.552 [warning] Bundle model not found for team_alpha_dags
02:23:16.554 [warning] Bundle model not found for team_beta_dags
...
02:32:02.184 [warning] Bundle model not found for team_alpha_dags
02:32:02.188 [warning] Bundle model not found for team_beta_dags
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Multi-team] [Testing] Executor - AWS ECS [airflow]
Posted by "shivaam (via GitHub)" <gi...@apache.org>.
shivaam commented on issue #62246:
URL: https://github.com/apache/airflow/issues/62246#issuecomment-3970315840
Are there instructions on which branch to test or how to setup the environment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Multi-team] [Testing] Executor - AWS ECS [airflow]
Posted by "shivaam (via GitHub)" <gi...@apache.org>.
shivaam commented on issue #62246:
URL: https://github.com/apache/airflow/issues/62246#issuecomment-3982068060
Findings:
[Seems like a bug]
1. When using `dag_bundle_config_list` with `team_name` in multi-team mode, the dag-processor's
`sync_bundles_to_db` method logs "Added new DAG bundle" for team-scoped bundles but the records
are not persisted to the `dag_bundle` table. Non-team bundles (like `shared_dags` and `example_dags`)
persist correctly. The dag-processor then continuously logs "Bundle model not found" for the
team bundles, and their DAGs are never processed.
```
dag_bundle_config_list = [
{"name": "team_alpha_dags", "classpath": "airflow.providers.amazon.aws.bundles.s3.S3DagBundle", "kwargs": {"bucket_name": "airflow-ecs-dags-ACCOUNT-REGION", "prefix": "team_alpha"}, "team_name": "team_alpha"},
{"name": "team_beta_dags", "classpath": "airflow.providers.amazon.aws.bundles.s3.S3DagBundle", "kwargs": {"bucket_name": "airflow-ecs-dags-ACCOUNT-REGION", "prefix": "team_beta"}, "team_name": "team_beta"},
{"name": "shared_dags", "classpath": "airflow.providers.amazon.aws.bundles.s3.S3DagBundle", "kwargs": {"bucket_name": "airflow-ecs-dags-ACCOUNT-REGION", "prefix": "shared"}}
]
```
2. If multi team config in the executor is not set it throws an error. My scheduler config has multi_team=True though.
```
what is the last log {"timestamp":"2026-03-02T04:34:49.374043Z","level":"error","event":"Top level error","logger":"task","filename":"task_runner.py","lineno":1827,"error_detail":[{"exc_type":"AirflowConfigException","exc_value":"Section `dag_processor` key `dag_bundle_config_list` cannot have a team name when multi-team mode is disabled.
To enable multi-team, you need to update section `core` key `multi_team` in your config.","exc_notes":[],"syntax_error":null,"is_cause":false,"frames":[{"filename":"/opt/airflow/task-sdk/src/airflow/sdk/execution_time/task_runner.py","lineno":1806,"name":"main"},{"filename":"/opt/airflow/task-s is that the start
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Multi-team] [Testing] Executor - AWS ECS [airflow]
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on issue #62246:
URL: https://github.com/apache/airflow/issues/62246#issuecomment-3984995682
Thanks for your testing @shivaam. Pinging @o-nikolas here to answer your questions more accurately than me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Multi-team] [Testing] Executor - AWS ECS [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on issue #62246:
URL: https://github.com/apache/airflow/issues/62246#issuecomment-3988307070
> @o-nikolas if the executor does not have the AIRFLOW__CORE__MULTI_TEAM: 'True' environment variables they fail. Is this expected or by design? Curious, about the reason.
Can you say more? Is this when running tasks that are from a scheduler that is using multi-team? Or they just fail always and in all cases when multi-team is not enabled?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Multi-team] [Testing] Executor - AWS ECS [airflow]
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck closed issue #62246: [Multi-team] [Testing] Executor - AWS ECS
URL: https://github.com/apache/airflow/issues/62246
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Multi-team] [Testing] Executor - AWS ECS [airflow]
Posted by "shivaam (via GitHub)" <gi...@apache.org>.
shivaam commented on issue #62246:
URL: https://github.com/apache/airflow/issues/62246#issuecomment-3988336717
> > [@o-nikolas](https://github.com/o-nikolas) if the executor does not have the AIRFLOW__CORE__MULTI_TEAM: 'True' environment variables they fail. Is this expected or by design? Curious, about the reason.
>
> Can you say more? Is this when running tasks that are from a scheduler that is using multi-team? Or they just fail always and in all cases when multi-team is not enabled?
It happens only when running tasks with a scheduler with multi-team enabled. All I need to fix it is add the env variables in executors as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Multi-team] [Testing] Executor - AWS ECS [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on issue #62246:
URL: https://github.com/apache/airflow/issues/62246#issuecomment-4000666058
> > > [@o-nikolas](https://github.com/o-nikolas) if the executor does not have the AIRFLOW__CORE__MULTI_TEAM: 'True' environment variables they fail. Is this expected or by design? Curious, about the reason.
> >
> >
> > Can you say more? Is this when running tasks that are from a scheduler that is using multi-team? Or they just fail always and in all cases when multi-team is not enabled?
>
> It happens only when running tasks with a scheduler with multi-team enabled. All I need to fix it is add the env variables in executors as well.
Can you share the traceback you're seeing and rough steps to reproduce it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Multi-team] [Testing] Executor - AWS ECS [airflow]
Posted by "shivaam (via GitHub)" <gi...@apache.org>.
shivaam commented on issue #62246:
URL: https://github.com/apache/airflow/issues/62246#issuecomment-3981544121
I was able to setup the whole stack and trigger ecs tasks using multi team. One issue I am facing in my setup is to build the 3.2 worker images for ecs executor. Any tips for that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org