You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ElXaxe (via GitHub)" <gi...@apache.org> on 2025/02/19 15:29:21 UTC
[I] Queued Runs not running in expected order [airflow]
ElXaxe opened a new issue, #46898:
URL: https://github.com/apache/airflow/issues/46898
### Apache Airflow version
Other Airflow 2 version (please specify below)
### If "Other Airflow 2 version" selected, which one?
2.9.2
### What happened?
Hi,
I have a DAG in production in AWS MWAA that is called by a AWS Lambda after creating a CSV file with infor received by AWS SQS. Lamda passes the file name as a DAG param.
**_max_active_runs_** is set to 1, as I need to process every file in order. E.g.: create hierarchy 205, delete hierarchy 205, create hierarchy 206, create hierarchy 207, delete hierarchy 206, etc.
Yesterdey, we received 200 petitions in a small range of time, so there were a lot of runs queued. From this amount of queued runs of the DAG, some were executed without respecting the FIFO order.
Next tables show some of the latest runs, in descending queued order. I marked the start dates with bold and italic the ones that started when they shouldn't.
|Logical Date|Run Id|Queued At|Start Date|End Date|
|------------|------|---------|----------|--------|
|2025-02-18, 19:02:59|manual__2025-02-18T18:02:59+00:00|2025-02-18, 19:02:59|2025-02-18, 19:58:51|2025-02-18, 20:00:04|
|2025-02-18, 19:02:52|manual__2025-02-18T18:02:52+00:00|2025-02-18, 19:02:52|2025-02-18, 19:58:18|2025-02-18, 19:58:50|
|2025-02-18, 19:02:33|manual__2025-02-18T18:02:33+00:00|2025-02-18, 19:02:33|2025-02-18, 19:57:00|2025-02-18, 19:58:17|
|2025-02-18, 19:02:28|manual__2025-02-18T18:02:28+00:00|2025-02-18, 19:02:28|2025-02-18, 19:56:24|2025-02-18, 19:56:59|
|2025-02-18, 19:01:28|manual__2025-02-18T18:01:28+00:00|2025-02-18, 19:01:28|2025-02-18, 19:54:06|2025-02-18, 19:56:23|
|2025-02-18, 19:01:18|manual__2025-02-18T18:01:18+00:00|2025-02-18, 19:01:18|2025-02-18, 19:53:32|2025-02-18, 19:54:05|
|2025-02-18, 19:00:32|manual__2025-02-18T18:00:32+00:00|2025-02-18, 19:00:32|2025-02-18, 19:52:19|2025-02-18, 19:53:32|
|2025-02-18, 19:00:27|manual__2025-02-18T18:00:27+00:00|2025-02-18, 19:00:27|2025-02-18, 19:51:45|2025-02-18, 19:52:18|
|2025-02-18, 18:59:54|manual__2025-02-18T17:59:54+00:00|2025-02-18, 18:59:54|2025-02-18, 19:50:34|2025-02-18, 19:51:45|
|2025-02-18, 18:59:46|manual__2025-02-18T17:59:46+00:00|2025-02-18, 18:59:46|2025-02-18, 19:49:54|2025-02-18, 19:50:33|
|2025-02-18, 18:59:28|manual__2025-02-18T17:59:28+00:00|2025-02-18, 18:59:28|2025-02-18, 19:48:42|2025-02-18, 19:49:54|
|2025-02-18, 18:59:19|manual__2025-02-18T17:59:19+00:00|2025-02-18, 18:59:19|2025-02-18, 19:48:11|2025-02-18, 19:48:42|
|2025-02-18, 18:58:51|manual__2025-02-18T17:58:51+00:00|2025-02-18, 18:58:51|2025-02-18, 19:46:55|2025-02-18, 19:48:11|
|2025-02-18, 18:58:46|manual__2025-02-18T17:58:46+00:00|2025-02-18, 18:58:46|**_2025-02-18, 19:27:51_**|2025-02-18, 19:28:56|
|2025-02-18, 18:58:17|manual__2025-02-18T17:58:17+00:00|2025-02-18, 18:58:17|2025-02-18, 19:45:44|2025-02-18, 19:46:55|
|2025-02-18, 18:58:12|manual__2025-02-18T17:58:12+00:00|2025-02-18, 18:58:12|2025-02-18, 19:45:11|2025-02-18, 19:45:43|
|2025-02-18, 18:57:58|manual__2025-02-18T17:57:58+00:00|2025-02-18, 18:57:58|2025-02-18, 19:43:59|2025-02-18, 19:45:10|
|2025-02-18, 18:57:50|manual__2025-02-18T17:57:50+00:00|2025-02-18, 18:57:50|2025-02-18, 19:43:24|2025-02-18, 19:43:59|
|2025-02-18, 18:57:33|manual__2025-02-18T17:57:33+00:00|2025-02-18, 18:57:33|2025-02-18, 19:42:11|2025-02-18, 19:43:24|
|2025-02-18, 18:57:27|manual__2025-02-18T17:57:27+00:00|2025-02-18, 18:57:28|2025-02-18, 19:41:34|2025-02-18, 19:42:10|
|2025-02-18, 18:57:03|manual__2025-02-18T17:57:03+00:00|2025-02-18, 18:57:03|2025-02-18, 19:40:23|2025-02-18, 19:41:34|
|2025-02-18, 18:56:56|manual__2025-02-18T17:56:56+00:00|2025-02-18, 18:56:56|2025-02-18, 19:39:48|2025-02-18, 19:40:22|
|2025-02-18, 18:56:34|manual__2025-02-18T17:56:34+00:00|2025-02-18, 18:56:34|2025-02-18, 19:38:39|2025-02-18, 19:39:47|
|2025-02-18, 18:56:27|manual__2025-02-18T17:56:27+00:00|2025-02-18, 18:56:27|2025-02-18, 19:38:06|2025-02-18, 19:38:39|
|2025-02-18, 18:51:32|manual__2025-02-18T17:51:32+00:00|2025-02-18, 18:51:32|2025-02-18, 19:36:52|2025-02-18, 19:38:06|
|2025-02-18, 18:51:27|manual__2025-02-18T17:51:27+00:00|2025-02-18, 18:51:27|2025-02-18, 19:36:19|2025-02-18, 19:36:52|
|2025-02-18, 18:51:05|manual__2025-02-18T17:51:05+00:00|2025-02-18, 18:51:05|_**2025-02-18, 19:16:06**_|2025-02-18, 19:18:34|
|2025-02-18, 18:50:59|manual__2025-02-18T17:50:59+00:00|2025-02-18, 18:50:59|2025-02-18, 19:35:46|2025-02-18, 19:36:19|
|2025-02-18, 18:50:38|manual__2025-02-18T17:50:38+00:00|2025-02-18, 18:50:38|2025-02-18, 19:34:37|2025-02-18, 19:35:45|
|2025-02-18, 18:50:31|manual__2025-02-18T17:50:31+00:00|2025-02-18, 18:50:31|2025-02-18, 19:34:03|2025-02-18, 19:34:36|
|2025-02-18, 18:50:04|manual__2025-02-18T17:50:04+00:00|2025-02-18, 18:50:04|2025-02-18, 19:32:46|2025-02-18, 19:34:02|
|2025-02-18, 18:49:56|manual__2025-02-18T17:49:56+00:00|2025-02-18, 18:49:56|2025-02-18, 19:32:11|2025-02-18, 19:32:45|
|2025-02-18, 18:49:27|manual__2025-02-18T17:49:27+00:00|2025-02-18, 18:49:27|2025-02-18, 19:30:24|2025-02-18, 19:32:11|
|2025-02-18, 18:49:19|manual__2025-02-18T17:49:19+00:00|2025-02-18, 18:49:19|2025-02-18, 19:29:40|2025-02-18, 19:30:24|
|2025-02-18, 18:48:49|manual__2025-02-18T17:48:49+00:00|2025-02-18, 18:48:49|2025-02-18, 19:27:51|2025-02-18, 19:29:40|
|2025-02-18, 18:48:43|manual__2025-02-18T17:48:43+00:00|2025-02-18, 18:48:43|2025-02-18, 19:27:16|2025-02-18, 19:27:50|
|2025-02-18, 18:48:13|manual__2025-02-18T17:48:13+00:00|2025-02-18, 18:48:13|2025-02-18, 19:26:04|2025-02-18, 19:27:15|
|2025-02-18, 18:48:07|manual__2025-02-18T17:48:07+00:00|2025-02-18, 18:48:07|2025-02-18, 19:25:32|2025-02-18, 19:26:04|
|2025-02-18, 18:39:45|manual__2025-02-18T17:39:45+00:00|2025-02-18, 18:39:45|2025-02-18, 19:24:18|2025-02-18, 19:25:32|
|2025-02-18, 18:39:39|manual__2025-02-18T17:39:39+00:00|2025-02-18, 18:39:39|2025-02-18, 19:23:43|2025-02-18, 19:24:18|
|2025-02-18, 18:39:20|manual__2025-02-18T17:39:20+00:00|2025-02-18, 18:39:20|2025-02-18, 19:22:38|2025-02-18, 19:23:43|
|2025-02-18, 18:39:14|manual__2025-02-18T17:39:14+00:00|2025-02-18, 18:39:14|2025-02-18, 19:22:04|2025-02-18, 19:22:37|
|2025-02-18, 18:38:47|manual__2025-02-18T17:38:47+00:00|2025-02-18, 18:38:47|2025-02-18, 19:20:50|2025-02-18, 19:22:03|
|2025-02-18, 18:38:42|manual__2025-02-18T17:38:42+00:00|2025-02-18, 18:38:42|2025-02-18, 19:20:18|2025-02-18, 19:20:50|
|2025-02-18, 18:38:23|manual__2025-02-18T17:38:23+00:00|2025-02-18, 18:38:23|2025-02-18, 19:19:09|2025-02-18, 19:20:18|
|2025-02-18, 18:38:17|manual__2025-02-18T17:38:17+00:00|2025-02-18, 18:38:17|2025-02-18, 19:18:35|2025-02-18, 19:19:09|
|2025-02-18, 18:37:52|manual__2025-02-18T17:37:52+00:00|2025-02-18, 18:37:52|2025-02-18, 19:16:06|2025-02-18, 19:18:34|
|2025-02-18, 18:37:47|manual__2025-02-18T17:37:47+00:00|2025-02-18, 18:37:47|2025-02-18, 19:15:33|2025-02-18, 19:16:05|
|2025-02-18, 18:37:31|manual__2025-02-18T17:37:31+00:00|2025-02-18, 18:37:31|2025-02-18, 19:14:19|2025-02-18, 19:15:32|
|2025-02-18, 18:37:25|manual__2025-02-18T17:37:25+00:00|2025-02-18, 18:37:25|2025-02-18, 19:13:38|2025-02-18, 19:14:19|
|2025-02-18, 18:37:03|manual__2025-02-18T17:37:03+00:00|2025-02-18, 18:37:03|2025-02-18, 19:11:51|2025-02-18, 19:13:38|
|2025-02-18, 18:36:57|manual__2025-02-18T17:36:57+00:00|2025-02-18, 18:36:57|2025-02-18, 19:11:18|2025-02-18, 19:11:50|
|2025-02-18, 18:36:33|manual__2025-02-18T17:36:33+00:00|2025-02-18, 18:36:33|2025-02-18, 19:10:07|2025-02-18, 19:11:17|
|2025-02-18, 18:36:28|manual__2025-02-18T17:36:28+00:00|2025-02-18, 18:36:28|2025-02-18, 19:09:32|2025-02-18, 19:10:07|
|2025-02-18, 18:36:04|manual__2025-02-18T17:36:04+00:00|2025-02-18, 18:36:04|2025-02-18, 19:08:17|2025-02-18, 19:09:31|
|2025-02-18, 18:35:58|manual__2025-02-18T17:35:58+00:00|2025-02-18, 18:35:58|2025-02-18, 19:07:38|2025-02-18, 19:08:17|
|2025-02-18, 18:35:23|manual__2025-02-18T17:35:23+00:00|2025-02-18, 18:35:23|2025-02-18, 19:05:10|2025-02-18, 19:07:38|
|2025-02-18, 18:35:17|manual__2025-02-18T17:35:17+00:00|2025-02-18, 18:35:17|2025-02-18, 19:04:37|2025-02-18, 19:05:09|
|2025-02-18, 18:35:03|manual__2025-02-18T17:35:03+00:00|2025-02-18, 18:35:03|**_2025-02-18, 18:44:51_**|2025-02-18, 18:47:20|
|2025-02-18, 18:34:57|manual__2025-02-18T17:34:57+00:00|2025-02-18, 18:34:57|2025-02-18, 19:04:04|2025-02-18, 19:04:37|
|2025-02-18, 18:34:40|manual__2025-02-18T17:34:40+00:00|2025-02-18, 18:34:40|2025-02-18, 19:02:47|2025-02-18, 19:04:03|
|2025-02-18, 18:34:34|manual__2025-02-18T17:34:34+00:00|2025-02-18, 18:34:34|2025-02-18, 19:02:15|2025-02-18, 19:02:47|
|2025-02-18, 18:34:15|manual__2025-02-18T17:34:15+00:00|2025-02-18, 18:34:15|2025-02-18, 19:00:38|2025-02-18, 19:02:14|
|2025-02-18, 18:34:10|manual__2025-02-18T17:34:10+00:00|2025-02-18, 18:34:10|2025-02-18, 18:59:42|2025-02-18, 19:00:37|
|2025-02-18, 18:33:46|manual__2025-02-18T17:33:46+00:00|2025-02-18, 18:33:46|2025-02-18, 18:58:31|2025-02-18, 18:59:41|
|2025-02-18, 18:33:40|manual__2025-02-18T17:33:40+00:00|2025-02-18, 18:33:40|2025-02-18, 18:57:53|2025-02-18, 18:58:30|
|2025-02-18, 18:33:22|manual__2025-02-18T17:33:22+00:00|2025-02-18, 18:33:22|2025-02-18, 18:56:43|2025-02-18, 18:57:53|
|2025-02-18, 18:33:15|manual__2025-02-18T17:33:15+00:00|2025-02-18, 18:33:15|2025-02-18, 18:56:12|2025-02-18, 18:56:43|
|2025-02-18, 18:33:04|manual__2025-02-18T17:33:04+00:00|2025-02-18, 18:33:04|2025-02-18, 18:55:01|2025-02-18, 18:56:11|
|2025-02-18, 18:32:58|manual__2025-02-18T17:32:58+00:00|2025-02-18, 18:32:58|2025-02-18, 18:54:29|2025-02-18, 18:55:00|
|2025-02-18, 18:32:34|manual__2025-02-18T17:32:34+00:00|2025-02-18, 18:32:34|2025-02-18, 18:53:15|2025-02-18, 18:54:29|
|2025-02-18, 18:32:29|manual__2025-02-18T17:32:29+00:00|2025-02-18, 18:32:29|2025-02-18, 18:52:39|2025-02-18, 18:53:14|
|2025-02-18, 18:32:09|manual__2025-02-18T17:32:09+00:00|2025-02-18, 18:32:09|2025-02-18, 18:51:31|2025-02-18, 18:52:39|
|2025-02-18, 18:32:04|manual__2025-02-18T17:32:04+00:00|2025-02-18, 18:32:04|2025-02-18, 18:50:57|2025-02-18, 18:51:30|
|2025-02-18, 18:31:45|manual__2025-02-18T17:31:45+00:00|2025-02-18, 18:31:45|2025-02-18, 18:49:44|2025-02-18, 18:50:57|
|2025-02-18, 18:31:40|manual__2025-02-18T17:31:40+00:00|2025-02-18, 18:31:40|2025-02-18, 18:49:12|2025-02-18, 18:49:44|
|2025-02-18, 18:31:16|manual__2025-02-18T17:31:16+00:00|2025-02-18, 18:31:16|2025-02-18, 18:47:58|2025-02-18, 18:49:12|
|2025-02-18, 18:31:12|manual__2025-02-18T17:31:12+00:00|2025-02-18, 18:31:12|2025-02-18, 18:47:21|2025-02-18, 18:47:58|
|2025-02-18, 18:30:57|manual__2025-02-18T17:30:57+00:00|2025-02-18, 18:30:57|2025-02-18, 18:44:50|2025-02-18, 18:47:20|
|2025-02-18, 18:30:48|manual__2025-02-18T17:30:48+00:00|2025-02-18, 18:30:48|2025-02-18, 18:44:17|2025-02-18, 18:44:50|
|2025-02-18, 18:30:30|manual__2025-02-18T17:30:30+00:00|2025-02-18, 18:30:30|2025-02-18, 18:42:22|2025-02-18, 18:44:17|
|2025-02-18, 18:30:21|manual__2025-02-18T17:30:21+00:00|2025-02-18, 18:30:21|2025-02-18, 18:41:45|2025-02-18, 18:42:20|
|2025-02-18, 18:30:08|manual__2025-02-18T17:30:08+00:00|2025-02-18, 18:30:08|2025-02-18, 18:40:40|2025-02-18, 18:41:45|
|2025-02-18, 18:30:02|manual__2025-02-18T17:30:02+00:00|2025-02-18, 18:30:02|2025-02-18, 18:40:07|2025-02-18, 18:40:39|
|2025-02-18, 18:29:42|manual__2025-02-18T17:29:42+00:00|2025-02-18, 18:29:42|2025-02-18, 18:38:52|2025-02-18, 18:40:06|
|2025-02-18, 18:29:37|manual__2025-02-18T17:29:37+00:00|2025-02-18, 18:29:37|2025-02-18, 18:38:18|2025-02-18, 18:38:52|
|2025-02-18, 18:29:11|manual__2025-02-18T17:29:11+00:00|2025-02-18, 18:29:11|2025-02-18, 18:37:05|2025-02-18, 18:38:18|
|2025-02-18, 18:29:05|manual__2025-02-18T17:29:05+00:00|2025-02-18, 18:29:05|2025-02-18, 18:35:45|2025-02-18, 18:37:04|
|2025-02-18, 18:28:42|manual__2025-02-18T17:28:42+00:00|2025-02-18, 18:28:42|2025-02-18, 18:34:06|2025-02-18, 18:35:44|
|2025-02-18, 18:28:37|manual__2025-02-18T17:28:37+00:00|2025-02-18, 18:28:37|2025-02-18, 18:33:32|2025-02-18, 18:34:06|
### What you think should happen instead?
_No response_
### How to reproduce
Don't know how.
### Operating System
MWAA
### Versions of Apache Airflow Providers
_No response_
### Deployment
Amazon (AWS) MWAA
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Queued Runs not running in expected order [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal closed issue #46898: Queued Runs not running in expected order
URL: https://github.com/apache/airflow/issues/46898
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Queued Runs not running in expected order [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #46898:
URL: https://github.com/apache/airflow/issues/46898#issuecomment-2668991019
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Queued Runs not running in expected order [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #46898:
URL: https://github.com/apache/airflow/issues/46898#issuecomment-2675137219
Closing for now. Should there be new information we can reopen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Queued Runs not running in expected order [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #46898:
URL: https://github.com/apache/airflow/issues/46898#issuecomment-2669646219
> executed without respecting the FIFO order.
That is a question for MWAA support.
Your report does not present a specific bug with Airflow.
To my knowledge MWAA use CeleryExecutor so it seems more of a question how they configure consiming tasks from the broker queue.
Please constact MWAA support with your query.
If you/MWAA confirm there is a bug with open source code we need to see reproduce example from scratch on vanila Airflow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org