You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "HaythemJ (via GitHub)" <gi...@apache.org> on 2025/09/04 01:03:25 UTC
[D] DAG Manual Trigger queue for a long time [airflow]
GitHub user HaythemJ created a discussion: DAG Manual Trigger queue for a long time
### Apache Airflow version
3.0.6
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
I deployed a DAG that is working OK with a schedule of 30 seconds. It works very well but When I am trying to trigger it manually it is queued for a too long time.
### What you think should happen instead?
My understaing is that the manual trigger should be launched immediately or at least queued for a while and start execution.
### How to reproduce
Just Trigger DAG Manually.
### Operating System
Ubuntu
### Versions of Apache Airflow Providers
_No response_
### Deployment
Docker-Compose
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
GitHub link: https://github.com/apache/airflow/discussions/55239
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] DAG Manual Trigger queue for a long time [airflow]
Posted by "HaythemJ (via GitHub)" <gi...@apache.org>.
GitHub user HaythemJ edited a comment on the discussion: DAG Manual Trigger queue for a long time
Could you please tell me what could I share with you as helpful information?
In api-server logs I can see that the http query is handled:
<img width="1746" height="206" alt="image" src="https://github.com/user-attachments/assets/d2e35841-0196-4e92-b238-c813d4cdb0f7" />
But nothing happening at scheduler and worker side.
After a while (more than 10 minutes) I can see the entry appears in the scheduler and the worker.
<img width="1896" height="582" alt="image" src="https://github.com/user-attachments/assets/76eb39a3-7397-4c06-8a17-ef7fbc5d88c6" />
<img width="1898" height="198" alt="image" src="https://github.com/user-attachments/assets/98ca800e-7e0c-4407-aa53-b0fce82f3c15" />
I am running the docker compose file shared in the airflow documentation. I am running airflow on docker on an Ubuntu machine 32G and 8 Cores.
My dag header:
`
default_args = {
'depends_on_past': False,
'start_date': datetime(2023, 1, 1),
# Ensure these settings don't block manual triggers
'wait_for_downstream': False
}
@dag(
default_args=default_args,
schedule=timedelta(seconds=60),
catchup=False,
max_active_runs=1,
max_active_tasks=1,
tags=["CM"],
)
`
My configuration:
parallelism = 32
max_active_tasks_per_dag = 16
max_active_runs_per_dag = 16
Could you please support?
GitHub link: https://github.com/apache/airflow/discussions/55239#discussioncomment-14309562
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] DAG Manual Trigger queue for a long time [airflow]
Posted by "HaythemJ (via GitHub)" <gi...@apache.org>.
GitHub user HaythemJ added a comment to the discussion: DAG Manual Trigger queue for a long time
I figured out the root cause !
It was related to my local time on my machine. It was wrong 15 minutes ahead. That's why it does not trigger immediatly. So the Logical Time was 15 minutes ahead.
I noticed that this is not the case with Airflow 2.11.0. When I Trigger manually it is triggered immédiatly nevetheless my clock is 15 minute ahead.
GitHub link: https://github.com/apache/airflow/discussions/55239#discussioncomment-14317345
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] DAG Manual Trigger queue for a long time [airflow]
Posted by "HaythemJ (via GitHub)" <gi...@apache.org>.
GitHub user HaythemJ added a comment to the discussion: DAG Manual Trigger queue for a long time
Additional Information: when using the CLI airflow dags trigger <dag_id> the dag is triggered immediatly and it is visible in the UI. It is not the case when I push the button "Trigger" in the UI: The DAG kept queued too long time.
GitHub link: https://github.com/apache/airflow/discussions/55239#discussioncomment-14301801
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] DAG Manual Trigger queue for a long time [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: DAG Manual Trigger queue for a long time
Yes. It is a requirement to have the time synchronized on all machines. Its mentioned in a numner of places in our docs - for example: https://airflow.apache.org/docs/apache-airflow/stable/faq.html#what-to-do-if-you-see-disappearing-dags-on-ui (and it's absolutely normal to expect you do NOT have drifts)
GitHub link: https://github.com/apache/airflow/discussions/55239#discussioncomment-14328199
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] DAG Manual Trigger queue for a long time [airflow]
Posted by "HaythemJ (via GitHub)" <gi...@apache.org>.
GitHub user HaythemJ added a comment to the discussion: DAG Manual Trigger queue for a long time
Could you please tell me what could I share with you as helpful information?
I api-server logs I can see that the http query is handled:
<img width="1746" height="206" alt="image" src="https://github.com/user-attachments/assets/d2e35841-0196-4e92-b238-c813d4cdb0f7" />
But nothing happening at scheduler and worker side.
After a while (more than 10 minutes) I can see the entry appears in the scheduler and the worker.
<img width="1896" height="582" alt="image" src="https://github.com/user-attachments/assets/76eb39a3-7397-4c06-8a17-ef7fbc5d88c6" />
<img width="1898" height="198" alt="image" src="https://github.com/user-attachments/assets/98ca800e-7e0c-4407-aa53-b0fce82f3c15" />
I am running the docker compose file shared in the airflow documentation. I am running airflow on docker on an Ubuntu machine 32G and 8 Cores.
My dag header:
`
default_args = {
'depends_on_past': False,
'start_date': datetime(2023, 1, 1),
# Ensure these settings don't block manual triggers
'wait_for_downstream': False
}
@dag(
default_args=default_args,
schedule=timedelta(seconds=60),
catchup=False,
max_active_runs=1,
max_active_tasks=1,
tags=["CM"],
)
`
My configuration:
parallelism = 32
max_active_tasks_per_dag = 16
max_active_runs_per_dag = 16
Could you please support?
GitHub link: https://github.com/apache/airflow/discussions/55239#discussioncomment-14309562
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] DAG Manual Trigger queue for a long time [airflow]
Posted by "HaythemJ (via GitHub)" <gi...@apache.org>.
GitHub user HaythemJ added a comment to the discussion: DAG Manual Trigger queue for a long time
Additional Information: I am getting the same behavior while triggering the DAG using REST API
curl --location 'http://<host>/api/v2/dags/<dag_id>/dagRuns' \
--header 'Content-Type: application/json' \
--header 'Authorization: ******' \
--data '{
"logical_date": "2025-09-03T21:55:22Z"
}'
The DAG is kept queued for a long time.
GitHub link: https://github.com/apache/airflow/discussions/55239#discussioncomment-14301802
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] DAG Manual Trigger queue for a long time [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: DAG Manual Trigger queue for a long time
Not reproducible with the information thaat you provided. You might want to try to find something in logs that is relevant to your case but without any way to reproduce it, there is no way your problem can be helped by anyone.
Converted it into a discussion if you want to provide more data here and discuss it - if we find that this might ba an airflow issue we can always convert it to issue back.
GitHub link: https://github.com/apache/airflow/discussions/55239#discussioncomment-14301810
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org