You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "LinasData (via GitHub)" <gi...@apache.org> on 2026/02/04 14:58:44 UTC
[I] I am upgrading from Airflow 2.11.0 to Airflow 3.1.6 and noticed a change in behavior regarding the `ExternalTaskMarker ` or maybe I am just using Airflow 3.1.6 wrong, so any help counts. [airflow]
LinasData opened a new issue, #61451:
URL: https://github.com/apache/airflow/issues/61451
I am upgrading from Airflow 2.11.0 to Airflow 3.1.6 and noticed a change in behavior regarding the `ExternalTaskMarker ` or maybe I am just using Airflow 3.1.6 wrong, so any help counts.
In Airflow 2.11.0, when I cleared a task upstream of an `ExternalTaskMarker `(and selected the "Recursive" or "Downstream" clearing options), the corresponding ExternalTaskSensor in the child DAG would also be cleared. In Airflow 3, this relationship seems broken; clearing the parent task does not trigger a clear on the child task.
We want to maintain task-level cross-DAG dependencies. While we are aware of the new Assets feature in Airflow 3, we specifically need the granular control provided by `ExternalTaskMarker` to ensure that re-running a specific parent task forces the child sensor to reset.
**Environment**:
Airflow Version: 3.0.x
Executor: CeleryExecutor
Python Version: 3.12
Ran on docker container setup locally
Minimal working example:
```
# --- PARENT DAG ---
from datetime import datetime
from airflow import DAG
from airflow.operators.bash import BashOperator
from airflow.sensors.external_task import ExternalTaskMarker
with DAG(
dag_id='parent_dag',
start_date=datetime(2026, 1, 18),
schedule='@daily',
catchup=False
) as parent_dag:
task_1 = BashOperator(
task_id='echo_hello',
bash_command='echo HELLO!!!!'
)
# Marker to point to child_dag/receive_call
parent_trigger = ExternalTaskMarker(
task_id='parent_trigger',
external_dag_id='child_dag',
external_task_id='receive_call'
)
task_1 >> parent_trigger
# --- CHILD DAG ---
from airflow.sensors.external_task import ExternalTaskSensor
with DAG(
dag_id='child_dag',
start_date=datetime(2026, 1, 18),
schedule='@daily',
catchup=False
) as child_dag:
receive_call = ExternalTaskSensor(
task_id='receive_call',
external_dag_id='parent_dag',
external_task_id='parent_trigger'
)
child_task = BashOperator(
task_id='child_echo',
bash_command='echo DONE'
)
receive_call >> child_task
```
_Originally posted by @LinasData in https://github.com/apache/airflow/discussions/60905_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] I am upgrading from Airflow 2.11.0 to Airflow 3.1.6 and noticed a change in behavior regarding the `ExternalTaskMarker` [airflow]
Posted by "shahar1 (via GitHub) via commits" <co...@airflow.apache.org>.
shahar1 commented on issue #61451:
URL: https://github.com/apache/airflow/issues/61451#issuecomment-6705974020
> [@shahar1](https://github.com/shahar1) is this going to be pushed on version 3.4.1?
Doing my best that it will
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] I am upgrading from Airflow 2.11.0 to Airflow 3.1.6 and noticed a change in behavior regarding the `ExternalTaskMarker` [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on issue #61451:
URL: https://github.com/apache/airflow/issues/61451#issuecomment-4244766415
I'm also working with a team that's seeing this same issue. Here's what they have reported:
> I’ve been using ExternalTaskMarker in Airflow 2 to connect our DAGs and propagate clears downstream without much concern. However, after upgrading to Airflow 3, this no longer seems to work as expected.
When attempting to clear tasks (with “Downstream” selected), the downstream dependencies don’t appear in the UI, and the clears are not being propagated. This suggests it’s not just a UI issue. I’ve attached a screenshot for reference.
Additionally, the Airflow docs mention: “child_task1 will only be cleared if ‘Recursive’ is selected when the user clears parent_task.”
https://airflow.apache.org/docs/apache-airflow-providers-standard/stable/sensors/external_task_sensor.html
However, I’m not seeing any “Recursive” option in the UI, so I’m unsure whether this is a UI issue or a problem with the operator behavior itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] I am upgrading from Airflow 2.11.0 to Airflow 3.1.6 and noticed a change in behavior regarding the `ExternalTaskMarker ` or maybe I am just using Airflow 3.1.6 wrong, so any help counts. [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #61451:
URL: https://github.com/apache/airflow/issues/61451#issuecomment-3847956580
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] I am upgrading from Airflow 2.11.0 to Airflow 3.1.6 and noticed a change in behavior regarding the `ExternalTaskMarker` [airflow]
Posted by "LinasData (via GitHub) via commits" <co...@airflow.apache.org>.
LinasData commented on issue #61451:
URL: https://github.com/apache/airflow/issues/61451#issuecomment-6705886832
@shahar1 is this going to be pushed on version 3.4.1?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] I am upgrading from Airflow 2.11.0 to Airflow 3.1.6 and noticed a change in behavior regarding the `ExternalTaskMarker` [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on issue #61451:
URL: https://github.com/apache/airflow/issues/61451#issuecomment-4245488954
@uranusjr - seems like these changes to remove `include_dependent_dag` may be the cause here. Do you know why this was removed and not added back elsewhere in the codebase?
https://github.com/apache/airflow/pull/54383/changes#diff-c3bdd41526f052b55dea0ebcbcfb2bda387039201c8d78feb71c42f0ce81450eL974
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] I am upgrading from Airflow 2.11.0 to Airflow 3.1.6 and noticed a change in behavior regarding the `ExternalTaskMarker` [airflow]
Posted by "kaxil (via GitHub) via commits" <co...@airflow.apache.org>.
kaxil closed issue #61451: I am upgrading from Airflow 2.11.0 to Airflow 3.1.6 and noticed a change in behavior regarding the `ExternalTaskMarker`
URL: https://github.com/apache/airflow/issues/61451
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org