You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2023/01/12 02:25:57 UTC
[GitHub] [airflow] aprettyloner opened a new issue, #28872: Grid view doesn't show recent dag runs for some pipelines in 2.4.0
aprettyloner opened a new issue, #28872:
URL: https://github.com/apache/airflow/issues/28872
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
# Context
We performed an upgrade today from 2.2.3 to 2.4.0, with a chart upgrade from 8.4.1 to 8.6.1.
# Issue Encountered
The grid UI at `/dags/{dag_id}/grid` endpoint has very unusual display behavior.
## 1. Missing recent dag runs in grid
This dag has run hourly in production since 2021-08-19. For some reason, the grid endpoint is displaying dag runs from January 2022. Even when a dag run is in progress, the grid default view still shows old data.
### Data Mismatch
#### Expected
We would expect to see in the **DAG Runs Summary** table:
- First Run Start: 2023-01-11, 02:00:09 UTC
- Last Run Start: 2023-01-12, 02:00:03 UTC
#### Observed
In the **DAG Runs Summary** table, we see:
- First Run Start: 2022-01-24, 04:00:05 UTC
- Last Run Start: 2022-01-25, 04:00:04 UTC
This doesn't align with the database for the same query (default 25 most recent runs).
### Workaround
If and only if a dag is running, then it appears in the grid using the `/dags/{dag_id}/grid?run_state=running` endpoint.
## 2. Peculiar ordering for dag runs in grid
This dag has been running daily since 2022-01-07. Unlike the above example, the recent 2023 dag runs do appear in the grid. However, the ordering is very peculiar. We would expect this week's runs to be all the way to the right.
Runs from 2023-01-07 to 2023-01-12 appear left of first run (2022-01-06, in the failed state).
#### Expected
We would expect to see in the **DAG Runs Summary** table:
- First Run Start: 2022-12-19, 00:05:00 UTC
- Last Run Start: 2023-01-12, 00:05:00 UTC
#### Observed
As above, there is a mismatch in the **DAG Runs Summary** table.
- First Run Start: 2023-01-07, 00:05:01 UTC <---- Why would "first" be a later date? 🤔 🤔 🤔 🤔 🤔 🤔
- Last Run Start: 2022-01-25, 00:05:04 UTC
This doesn't align with the database for the same query (default 25 most recent runs).
### What you think should happen instead
# What possibly went wrong
There seems to be some incorrect date filtering/ordering against the database.
# Observations
Interestingly, our stage instance for the same DAG shows recent runs correctly and in a logical order. The deployments process, chart configuration, and airflow version are identical across our environments.
## DAG History
The only discernable difference between behavior is when the dags were first turned on.
### Not missing recent runs
Staging DAG has run since 2022-05-05
Prod DAG has run since 2022-01-07 - **Note that this one had the additional left -> right ordering issue**
### Missing recent runs
Prod DAG has run since 2021-08-19
### How to reproduce
1. Spin up with 2.4.0 and run multiple dags with varying `start_date` and with `catchup` enabled.
2. Inspect the `/dags/{dag_id}/grid` endpoint for running dags.
### Operating System
Debian GNU/Linux - 11 (bullseye)
### Versions of Apache Airflow Providers
```
apache-airflow-providers-amazon 5.1.0
apache-airflow-providers-celery 3.0.0
apache-airflow-providers-cncf-kubernetes 4.3.0
apache-airflow-providers-common-sql 1.3.1
apache-airflow-providers-docker 3.1.0
apache-airflow-providers-elasticsearch 4.2.0
apache-airflow-providers-ftp 3.2.0
apache-airflow-providers-google 8.3.0
apache-airflow-providers-grpc 3.0.0
apache-airflow-providers-hashicorp 3.1.0
apache-airflow-providers-http 4.1.0
apache-airflow-providers-imap 3.1.0
apache-airflow-providers-microsoft-azure 4.2.0
apache-airflow-providers-mysql 3.2.0
apache-airflow-providers-odbc 3.1.1
apache-airflow-providers-pagerduty 3.1.0
apache-airflow-providers-postgres 5.2.1
apache-airflow-providers-redis 3.0.0
apache-airflow-providers-salesforce 5.2.0
apache-airflow-providers-sendgrid 3.0.0
apache-airflow-providers-sftp 4.0.0
apache-airflow-providers-slack 5.1.0
apache-airflow-providers-sqlite 3.3.1
apache-airflow-providers-ssh 3.1.0
```
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
Deployed on GKE via helm:v3.2.0.
### Anything else
I am willing to submit a PR but I need to dig into the codebase a bit more.
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] aprettyloner closed issue #28872: Peculiar grid view behavior in 2.4.0, recent dag runs fail to show
Posted by GitBox <gi...@apache.org>.
aprettyloner closed issue #28872: Peculiar grid view behavior in 2.4.0, recent dag runs fail to show
URL: https://github.com/apache/airflow/issues/28872
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #28872: Grid view doesn't show recent dag runs for some pipelines in 2.4.0
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #28872:
URL: https://github.com/apache/airflow/issues/28872#issuecomment-1379731428
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] aprettyloner commented on issue #28872: Peculiar grid view behavior in 2.4.0, recent dag runs fail to show
Posted by GitBox <gi...@apache.org>.
aprettyloner commented on issue #28872:
URL: https://github.com/apache/airflow/issues/28872#issuecomment-1380011776
Update: Bumping to version 2.4.3 fixed the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org