You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "mweisshaupt1988 (via GitHub)" <gi...@apache.org> on 2025/09/29 11:00:28 UTC
[I] TriggerDagRunOperator.execute cannot be called outside of the Task Runner! [airflow]
mweisshaupt1988 opened a new issue, #56204:
URL: https://github.com/apache/airflow/issues/56204
### Apache Airflow version
Other Airflow 2 version (please specify below)
### If "Other Airflow 2 version" selected, which one?
3.0.6
### What happened?
The TriggerDagRunOperator can not be executed from a task anymore
### What you think should happen instead?
Either it should work like in Airflow 2.11 or there should be some documentation how to migrate existing DAGs
### How to reproduce
I have added an example with two minimal Dags that reproduce the problem
[x_task_run_demo.py](https://github.com/user-attachments/files/22595855/x_task_run_demo.py)
### Operating System
Ubuntu 24.04
### Versions of Apache Airflow Providers
_No response_
### Deployment
Docker-Compose
### Deployment details
It does not matter if I run it in docker compose or on OpenShift. The error is the same.
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] TriggerDagRunOperator.execute cannot be called outside of the Task Runner! [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #56204:
URL: https://github.com/apache/airflow/issues/56204#issuecomment-3346339807
This message no longer exist in the code base after https://github.com/apache/airflow/pull/47523
please try with Airflow 3.1.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] TriggerDagRunOperator.execute cannot be called outside of the Task Runner! [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on issue #56204:
URL: https://github.com/apache/airflow/issues/56204#issuecomment-3557016884
Easiest way is to update the code so that the TriggerDagRunOperator is run as a task:
```python
with DAG(
dag_id="Task-Run-Demo",
schedule="0 0 * * *", # Daily at midnight (00:00)
catchup=False,
params={"schedule_date": Param(f"{datetime.date.today()}", type="string", format="date")},
) as dag:
@task
def schedule_dags(**kwargs):
return {"status": "ready"}
trigger_tasks = []
for i in range(5):
execution_date = utcnow() + datetime.timedelta(minutes=5)
trigger_tasks.append(
TriggerDagRunOperator(
task_id=f"create_reports_dag_{i}",
trigger_dag_id="Create-Reports",
conf={"counter": i},
logical_date=execution_date.replace(tzinfo=pytz.UTC),
)
)
schedule_dags() >> trigger_tasks
with DAG(
dag_id="Create-Reports",
schedule=None,
catchup=False,
params={"counter": None},
) as target_dag:
@task.python
def report(params: dict):
counter = params["counter"]
logging.info(f"Current counter: {counter}")
return {"counter": counter, "status": "processed"}
report()
```
Let me know what you think about this approach and we can update the doc about it. @hussein-awala @mweisshaupt1988
Another option is to use `SUPERVISOR_COMMS` which I do not recommend (and not sure if it's made for public use)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] TriggerDagRunOperator.execute cannot be called outside of the Task Runner! [airflow]
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on issue #56204:
URL: https://github.com/apache/airflow/issues/56204#issuecomment-3349166639
I confirm the bug — I was able to reproduce it on the main branch. It looks like the issue is related to the `try/except` context (or any context). The other DAG is triggered with 3.1.0+ if we remove the `try/except` block and keep only the `TriggerDagRunOperator`, but the operator is called once, which means there’s a short-circuit in the task runner that stops the loop:
```python
@task
def schedule_dags(**kwargs):
for i in range(5):
execution_date = datetime.datetime.utcnow() + datetime.timedelta(minutes=5)
logging.info(f"Trigger report for counter #{i} at '{execution_date}'")
TriggerDagRunOperator(task_id=f"create_reports_dag_{i}"
, trigger_dag_id="Create-Reports"
, conf={"counter": i}
# , logical_date=execution_date.replace(tzinfo=pytz.UTC),
).execute(context=kwargs)
```
logs:
```
Log message source details sources=["/root/airflow/logs/dag_id=Task-Run-Demo/run_id=manual__2025-09-29T21:30:50.712933+00:00/task_id=schedule_dags/attempt=1.log"]
INFO - DAG bundles loaded: dags-folder
INFO - Filling up the DagBag from /files/dags/56204.py
INFO - Trigger report for counter #0 at '2025-09-29 21:35:51.792317'
WARNING - TriggerDagRunOperator.execute cannot be called outside of the Task Runner!
INFO - Triggering Dag Run. trigger_dag_id=Create-Reports
INFO - Dag Run triggered successfully. trigger_dag_id=Create-Reports
```
Why the warning message?
in this code:
```
if not cls.test_mode and sentinel is not self:
```
self is `<Task(TriggerDagRunOperator): create_reports_dag_0>` but `sentinel` is `<Task(_PythonDecoratedOperator): schedule_dags>`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] TriggerDagRunOperator.execute cannot be called outside of the Task Runner! [airflow]
Posted by "mweisshaupt1988 (via GitHub)" <gi...@apache.org>.
mweisshaupt1988 commented on issue #56204:
URL: https://github.com/apache/airflow/issues/56204#issuecomment-3589826489
I have tested the workaround and this time it works for our use case. But I'm not really sold on it if it is not officially supported. If the bug will still be fixed, I can live with it for now though.
@ashb The version 3.1.3 still shows the same problems if the workaround is not used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] TriggerDagRunOperator.execute cannot be called outside of the Task Runner! [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #56204:
URL: https://github.com/apache/airflow/issues/56204#issuecomment-3346319150
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] TriggerDagRunOperator.execute cannot be called outside of the Task Runner! [airflow]
Posted by "mweisshaupt1988 (via GitHub)" <gi...@apache.org>.
mweisshaupt1988 commented on issue #56204:
URL: https://github.com/apache/airflow/issues/56204#issuecomment-3347841512
I have tried the same code with Airflow 3.1.0 but the same error occurs. I have also tried to fix the deprecation warnings by using the "airflow.sdk" and "airflow.providers.standard.operators.trigger_dagrun" imports instead. But this was also not helping.
I'll upload a screenshot of the log and the updated DAGs
<img width="865" height="362" alt="Image" src="https://github.com/user-attachments/assets/35534a43-349e-47d7-a5fd-10062ebf93e6" />
[x_task_run_demo.py](https://github.com/user-attachments/files/22601734/x_task_run_demo.py)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] TriggerDagRunOperator.execute cannot be called outside of the Task Runner! [airflow]
Posted by "phanikumv (via GitHub)" <gi...@apache.org>.
phanikumv commented on issue #56204:
URL: https://github.com/apache/airflow/issues/56204#issuecomment-3521084201
@ephraimbuddy could you please check this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] TriggerDagRunOperator.execute cannot be called outside of the Task Runner! [airflow]
Posted by "hlukashonak (via GitHub)" <gi...@apache.org>.
hlukashonak commented on issue #56204:
URL: https://github.com/apache/airflow/issues/56204#issuecomment-3659812628
Hi, we experience a very similar problem and the suggested workaround with SUPERVISOR_COMMS works, but we also would like to find a solution with "official" features that will stably work in the long term. As pointed by @hussein-awala:
>...and keep only the TriggerDagRunOperator, but the operator is called once, which means there’s a short-circuit in the task runner that stops the loop
@ephraimbuddy is it planned to remove this short-circuit in future versions so that it will be possible to run TriggerDagRunOperator.excute() from a loop?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] TriggerDagRunOperator.execute cannot be called outside of the Task Runner! [airflow]
Posted by "mweisshaupt1988 (via GitHub)" <gi...@apache.org>.
mweisshaupt1988 commented on issue #56204:
URL: https://github.com/apache/airflow/issues/56204#issuecomment-3558760439
@ephraimbuddy I'm afraid that won't solve my problem.
This solution calculates the number of tasks to run during loading the DAG. I need it to do that while running the schedule_dags task because the configuration which reports to run is extracted from the database.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] TriggerDagRunOperator.execute cannot be called outside of the Task Runner! [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on issue #56204:
URL: https://github.com/apache/airflow/issues/56204#issuecomment-3571515141
FYI using `SUPERVISOR_COMMS` directly is "unsupported" -- we don't plan to break it, but also we make no guarantees that it will stay working from patch to patch version either.
Does the original dag still fail on Airflow 3.1.x?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] TriggerDagRunOperator.execute cannot be called outside of the Task Runner! [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on issue #56204:
URL: https://github.com/apache/airflow/issues/56204#issuecomment-3570910192
> [@ephraimbuddy](https://github.com/ephraimbuddy) I'm afraid that won't solve my problem. This solution calculates the number of tasks to run during loading the DAG. I need it to do that while running the schedule_dags task because the configuration which reports to run is extracted from the database.
Ok. Here's another approach using SUPERVISOR_COMMS:
```python
# This DAG should schedule the other one multiple times
with DAG(
dag_id="Task-Run-Demo",
schedule="0 0 * * *", # Daily at midnight (00:00)
catchup=False,
params={"schedule_date": Param(f"{datetime.date.today()}", type="string", format="date")},
) as dag:
@task
def schedule_dags(**kwargs):
from airflow.models.dagrun import DagRun, DagRunType
from airflow.sdk.execution_time.comms import TriggerDagRun
from airflow.sdk.execution_time.task_runner import SUPERVISOR_COMMS
for i in range(5):
execution_date = utcnow() + datetime.timedelta(minutes=5)
logging.info(f"Trigger report for counter #{i} at '{execution_date}'")
execution_date_utc = execution_date.replace(tzinfo=pytz.UTC)
run_id = DagRun.generate_run_id(
run_type=DagRunType.MANUAL,
logical_date=execution_date_utc,
run_after=execution_date_utc,
)
logger.info(
"Triggering report #%s at '%s' with run_id '%s'",
i,
execution_date,
run_id,
)
try:
# Send trigger request via supervisor communication
SUPERVISOR_COMMS.send(
TriggerDagRun(
dag_id="Create-Reports",
run_id=run_id,
conf={"counter": i},
logical_date=execution_date.replace(tzinfo=pytz.UTC),
reset_dag_run=False,
)
)
except Exception as ex:
error_message = str(ex)
logging.error(f"Failed to trigger DAG #{i}: {error_message}")
schedule_dags()
# This DAG should not be executed directly but triggered by the Task-Run-Demo DAG
with DAG(
dag_id="Create-Reports",
schedule=None,
catchup=False,
params={"counter": None},
) as target_dag:
@task.python
def report(params: dict):
counter = params["counter"]
logging.info(f"Current counter: {counter}")
report()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org