You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "atul-astronomer (via GitHub)" <gi...@apache.org> on 2025/04/11 12:46:29 UTC
[I] ExternalTaskSensor Dag is getting timed out poking for tasks [airflow]
atul-astronomer opened a new issue, #49122:
URL: https://github.com/apache/airflow/issues/49122
### Apache Airflow version
3.0.0
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
```
INFO - Poking for tasks ['child_dummy1', 'child_dummy2'] in dag external_task_sensor_child_dag on 2025-04-11T12:23:00+00:00 ... : source="airflow.task.operators.providers.standard.src.airflow.providers.standard.sensors.external_task.ExternalTaskSensor"
[2025-04-11, 12:39:33] ERROR - Task failed with exception: source="task"
AirflowSensorTimeout: Sensor has timed out; run duration of 420.7096250[26](http://localhost:28080/dags/external_task_sensor_parent/runs/manual__2025-04-11T12:32:31.707987+00:00/tasks/check_ids?try_number=1#26)0074 seconds exceeds the specified timeout of 420.0.
File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/task_runner.py", line 775 in run
File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/task_runner.py", line 1038 in _execute_task
File "/opt/airflow/task-sdk/src/airflow/sdk/bases/operator.py", line 409 in wrapper
File "/opt/airflow/providers/standard/src/airflow/providers/standard/sensors/external_task.py", line 420 in execute
File "/opt/airflow/task-sdk/src/airflow/sdk/bases/operator.py", line 409 in wrapper
File "/opt/airflow/task-sdk/src/airflow/sdk/bases/sensor.py", line 244 in execute
```
### What you think should happen instead?
_No response_
### How to reproduce
Use below 2 Dags to reproduce the scenario:
```python
from datetime import datetime, timedelta
from airflow.providers.standard.operators.python import PythonOperator
from airflow.sdk import DAG
from airflow.utils.state import TaskInstanceState
from dags.plugins.airflow_dag_introspection import assert_the_task_states
from airflow.providers.standard.sensors.external_task import ExternalTaskSensor
docs = """
####Purpose
This dag tests that the ExternalTaskSensor is able to sense multiple tasks with the parameter 'external_task_ids=['list', 'of', 'task', 'ids']'.\n
The ExternalTaskSensor senses only task_ids with a specific execution date. by default it searches for task_ids with the same execution date as that of the dag with the sensor.\n
If there is a different execution date than that of the the dag that has the sensor then either the execution_delta or the execution_delta_fn needs to be set.\n
####Expected Behahvior
This dag has 1 tasks that are both expected to succeed.\n
The first task senses the completion of the tasks child_dummy1 and child_dummy2 in the dag with a python file of multiple_tasks_ids_retake_child.py and with a dag_id of external_task_sensor_child_dag .\n
"""
def ex_date(dt):
new_dt = dt - timedelta(days=1)
return new_dt
with DAG(
dag_id="external_task_sensor_parent",
start_date=datetime(2022, 3, 12, 3, 35, 0),
schedule=timedelta(days=1),
# is_paused_upon_creation=False,
doc_md=docs,
tags=["core", "sensor"]
) as dag:
ets1 = ExternalTaskSensor(
task_id="check_ids",
external_dag_id="external_task_sensor_child_dag",
external_task_ids=["child_dummy1", "child_dummy2"],
allowed_states=[TaskInstanceState.SUCCESS, TaskInstanceState.SKIPPED],
failed_states=[TaskInstanceState.FAILED, TaskInstanceState.REMOVED,],
execution_delta=timedelta(minutes=7),
#execution_date_fn=ex_date,
check_existence=True,
poke_interval=20.0,
timeout=420.0,
)
py1 = PythonOperator(
task_id="check_state_of_sensor",
python_callable=assert_the_task_states,
op_args=[{"check_ids": "success"}],
)
ets1 >> py1
```
```python
from datetime import datetime, timedelta
from airflow.providers.standard.operators.empty import EmptyOperator
from airflow.sdk import DAG
docs = """
####Purpose
This dag is a simple dag whose only purpose is to be ran at a specific execution date so it can be checked by the ExternalTaskSensor.\n
The validity of the success of this dag comes from the ExternalTaskSensor checking the task_id's.
####Expected Behavior
All tasks expected to succeed.
"""
with DAG(
dag_id="external_task_sensor_child_dag",
start_date=datetime(2022, 3, 12, 3, 28, 0),
schedule=timedelta(days=1),
is_paused_upon_creation=False,
doc_md=docs,
tags=["core", "sensor"],
) as dag:
d0 = EmptyOperator(task_id="child_dummy1")
d2 = EmptyOperator(task_id="child_dummy2")
d0 >> d2
```
### Operating System
Linux
### Versions of Apache Airflow Providers
_No response_
### Deployment
Other
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ExternalTaskSensor Dag is getting timed out poking for tasks [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on issue #49122:
URL: https://github.com/apache/airflow/issues/49122#issuecomment-2797216123
Yeah, can you try with `catchup=True` please. This is expected but will let you confirm after re-testing with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ExternalTaskSensor Dag is getting timed out poking for tasks [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on issue #49122:
URL: https://github.com/apache/airflow/issues/49122#issuecomment-2797410483
closing this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ExternalTaskSensor Dag is getting timed out poking for tasks [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 closed issue #49122: ExternalTaskSensor Dag is getting timed out poking for tasks
URL: https://github.com/apache/airflow/issues/49122
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ExternalTaskSensor Dag is getting timed out poking for tasks [airflow]
Posted by "gopidesupavan (via GitHub)" <gi...@apache.org>.
gopidesupavan commented on issue #49122:
URL: https://github.com/apache/airflow/issues/49122#issuecomment-2797092603
I think this is the slight behaviour change in AF 3 the backfill not happen automatically i believe.
Add the catchup=True for both dags it should work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ExternalTaskSensor Dag is getting timed out poking for tasks [airflow]
Posted by "atul-astronomer (via GitHub)" <gi...@apache.org>.
atul-astronomer commented on issue #49122:
URL: https://github.com/apache/airflow/issues/49122#issuecomment-2797343204
Yes, its passing with catchup=True
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org