You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/09/19 20:28:51 UTC
[GitHub] [airflow] zachliu opened a new issue, #26505: The grid view shows incorrect DAG runs
zachliu opened a new issue, #26505:
URL: https://github.com/apache/airflow/issues/26505
### Apache Airflow version
2.4.0
### What happened
Airflow 2.4.0's grid view has two issues:
1. Grid view stop showing the latest DAG runs in 2.4.0
3. Grid view's calendar is inconsistent with graph view
these 2 screenshots were taken within minute
### What you think should happen instead
_No response_
### How to reproduce
1. Go to a DAG with >365 dag runs
2. The place where it's supposed to show the latest dag run in the "grid view" now shows an old dag run ~1 year ago
### Operating System
Linux Mint 20.3 Una
### Versions of Apache Airflow Providers
_No response_
### Deployment
Other Docker-based deployment
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] zachliu commented on issue #26505: The grid view shows incorrect DAG runs
Posted by GitBox <gi...@apache.org>.
zachliu commented on issue #26505:
URL: https://github.com/apache/airflow/issues/26505#issuecomment-1254239166
> @kxepal Creating a minimal stand-alone reproduction case if you can ("here's a dag, run it/backfill it for X" sort of thing)
i tried that, running the same dag many times in a short period (backfilling) doesn't help, they will be ordered properly. i guess because they are all converted to timetables uniformly
it seems the issue occurs on dags that are older than the introduction of timetable, which suggests that during the conversion, older dag runs lack attribute(s) for a proper sorting :thinking:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #26505: The grid view shows incorrect DAG runs
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #26505:
URL: https://github.com/apache/airflow/issues/26505#issuecomment-1253542313
@uranusjr Can you take a look at this ASAP please so we can get a fix for this out in 2.4.1?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] zachliu commented on issue #26505: The grid view shows incorrect DAG runs
Posted by GitBox <gi...@apache.org>.
zachliu commented on issue #26505:
URL: https://github.com/apache/airflow/issues/26505#issuecomment-1253773464
@kxepal same here, i'm waiting for the distant future "calendar"-like scheduling system :wink:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kxepal commented on issue #26505: The grid view shows incorrect DAG runs
Posted by GitBox <gi...@apache.org>.
kxepal commented on issue #26505:
URL: https://github.com/apache/airflow/issues/26505#issuecomment-1254270205
> Creating a minimal stand-alone reproduction case if you can ("here's a dag, run it/backfill it for X" sort of thing)
Oblivious, but it be quite a hard to provide. @zachliu feels like walk on this path with no luck. But it looks like the problem could be solved by a simple new migration script. But what the cases it should handle and how it should fix them?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kxepal commented on issue #26505: The grid view shows incorrect DAG runs
Posted by GitBox <gi...@apache.org>.
kxepal commented on issue #26505:
URL: https://github.com/apache/airflow/issues/26505#issuecomment-1253536779
Hi there! Follow up from dev ML.
We faced exactly the same issue after upgrade from 2.3.4 to 2.4.0 for all our dags. All of them has quite a history up to 2016 year, so a lot of dag runs. Somehow grid show for us latest runs at 2021.11 while other views (calendar, graph) shows all of them and the most recent ones.
The exact fix which solved this problem is:
```
--- airflow/www/views.py
+++ airflow/www/views.py
@@ -3453,8 +3460,9 @@ class Airflow(AirflowBaseView):
if run_state:
query = query.filter(DagRun.state == run_state)
- ordering = (DagRun.__table__.columns[name].desc() for name in dag.timetable.run_ordering)
- dag_runs = query.order_by(*ordering, DagRun.id.desc()).limit(num_runs).all()
+ dag_runs = query.order_by(DagRun.execution_date.desc()).limit(num_runs).all(
dag_runs.reverse()
encoded_runs = [wwwutils.encode_dag_run(dr) for dr in dag_runs]
```
It feels like there is a problem between data and logic that operates it, but not sure where and what is it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #26505: The grid view shows incorrect DAG runs
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #26505:
URL: https://github.com/apache/airflow/issues/26505#issuecomment-1255933559
Filling all data interval with execution date would be excruciatingly slow, which is why it’s not done. Doing some fancy `coalesce` for ordering is probably possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] zachliu commented on issue #26505: The grid view shows incorrect DAG runs
Posted by GitBox <gi...@apache.org>.
zachliu commented on issue #26505:
URL: https://github.com/apache/airflow/issues/26505#issuecomment-1253756455
@kxepal doesn't the fix actually revert the core of https://github.com/apache/airflow/pull/25633? i also did that because i "accidentally" deployed airflow 2.4.0 to our production :shushing_face:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] zachliu commented on issue #26505: The grid view shows incorrect DAG runs
Posted by GitBox <gi...@apache.org>.
zachliu commented on issue #26505:
URL: https://github.com/apache/airflow/issues/26505#issuecomment-1254274262
can the migration script simply fills all the NULL `data_interval_end` with `execution_date`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #26505: The grid view shows incorrect DAG runs
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #26505:
URL: https://github.com/apache/airflow/issues/26505#issuecomment-1255989365
I created a PR for this. Would be awesome if someone could test it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kxepal commented on issue #26505: The grid view shows incorrect DAG runs
Posted by GitBox <gi...@apache.org>.
kxepal commented on issue #26505:
URL: https://github.com/apache/airflow/issues/26505#issuecomment-1253770098
@zachliu probably. We don't use timetables, so it was clear fix for us.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] zachliu commented on issue #26505: The grid view shows incorrect DAG runs
Posted by GitBox <gi...@apache.org>.
zachliu commented on issue #26505:
URL: https://github.com/apache/airflow/issues/26505#issuecomment-1253901238
aww... wrong guess :crying_cat_face: it was based on the fact that some old dag runs don't have the `Data interval start/end` parameters (see case 2 screenshots)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] zachliu commented on issue #26505: The grid view shows incorrect DAG runs
Posted by GitBox <gi...@apache.org>.
zachliu commented on issue #26505:
URL: https://github.com/apache/airflow/issues/26505#issuecomment-1253768506
@kxepal taking a wild guess here, maybe a `if` can solve all: if timetables are used, do that; else, do this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #26505: The grid view shows incorrect DAG runs
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #26505:
URL: https://github.com/apache/airflow/issues/26505#issuecomment-1253861070
_Everything_ is converted to a timetable now under the hood (and had been since timetables were introduced) so it's not as simple as an `if`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kxepal commented on issue #26505: The grid view shows incorrect DAG runs
Posted by GitBox <gi...@apache.org>.
kxepal commented on issue #26505:
URL: https://github.com/apache/airflow/issues/26505#issuecomment-1253645416
@ashb we have good old timedelta schedule interval value everywhere. No timetables at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #26505: The grid view shows incorrect DAG runs
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #26505:
URL: https://github.com/apache/airflow/issues/26505#issuecomment-1253639185
@kxepal What is the timetable/schedule parameter for the DAG(s) with this problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #26505: The grid view shows incorrect DAG runs
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #26505:
URL: https://github.com/apache/airflow/issues/26505#issuecomment-1258003976
Fixed by #26626
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kxepal commented on issue #26505: The grid view shows incorrect DAG runs
Posted by GitBox <gi...@apache.org>.
kxepal commented on issue #26505:
URL: https://github.com/apache/airflow/issues/26505#issuecomment-1254207835
@ashb What information could be useful to debug this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb closed issue #26505: The grid view shows incorrect DAG runs
Posted by GitBox <gi...@apache.org>.
ashb closed issue #26505: The grid view shows incorrect DAG runs
URL: https://github.com/apache/airflow/issues/26505
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kxepal commented on issue #26505: The grid view shows incorrect DAG runs
Posted by GitBox <gi...@apache.org>.
kxepal commented on issue #26505:
URL: https://github.com/apache/airflow/issues/26505#issuecomment-1254278656
But why it already doesn't? It doesn't feels like a problem with current logic - it works fine, but somewhere in changes between two versions. Actually, running new dagruns with 2.4 without a fix doesn't change anything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] zachliu commented on issue #26505: The grid view shows incorrect DAG runs
Posted by GitBox <gi...@apache.org>.
zachliu commented on issue #26505:
URL: https://github.com/apache/airflow/issues/26505#issuecomment-1252959481
most likely due to https://github.com/apache/airflow/pull/25633 and https://github.com/apache/airflow/pull/25880
@uranusjr @bbovenzi any idea how to solve this? i'm not familiar with the "timetable" part :sweat_smile:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #26505: The grid view shows incorrect DAG runs
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #26505:
URL: https://github.com/apache/airflow/issues/26505#issuecomment-1254232672
@kxepal Creating a minimal stand-alone reproduction case if you can ("here's a dag, run it/backfill it for X" sort of thing)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kxepal commented on issue #26505: The grid view shows incorrect DAG runs
Posted by GitBox <gi...@apache.org>.
kxepal commented on issue #26505:
URL: https://github.com/apache/airflow/issues/26505#issuecomment-1253760660
@zachliu That PR contains a lot of changes, but for us was enough to revert those changes that in the diff. The rest of them causes no troubles.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org