You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "yimingpeng (via GitHub)" <gi...@apache.org> on 2025/09/01 09:39:40 UTC
[PR] #55020 gantt view is not getting updatedin realtime [airflow]
yimingpeng opened a new pull request, #55130:
URL: https://github.com/apache/airflow/pull/55130
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
Hello there,
This PR is an attempt to fix #55020 where the gantt chart wasn't showing real-time progress for running tasks.
After applying the fix:
https://github.com/user-attachments/assets/5a121f1b-f8ab-4395-9c69-929f937d4309
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] #55020 gantt view is not getting updatedin realtime [airflow]
Posted by "bbovenzi (via GitHub)" <gi...@apache.org>.
bbovenzi commented on code in PR #55130:
URL: https://github.com/apache/airflow/pull/55130#discussion_r2345124121
##########
airflow-core/src/airflow/ui/src/layouts/Details/Gantt/Gantt.tsx:
##########
@@ -157,14 +160,17 @@ export const Gantt = ({ limit }: Props) => {
const taskInstance = taskInstances.find((ti) => ti.task_id === node.id);
if (taskInstance) {
+ const hasTaskRunning = isStatePending(taskInstance.state);
+ const endTime = hasTaskRunning ? currentTime : taskInstance.end_date;
+
return {
isGroup: node.isGroup,
isMapped: node.is_mapped,
state: taskInstance.state,
taskId: taskInstance.task_id,
x: [
formatDate(taskInstance.start_date, selectedTimezone, "YYYY-MM-DD HH:mm:ss.SSS"),
- formatDate(taskInstance.end_date, selectedTimezone, "YYYY-MM-DD HH:mm:ss.SSS"),
+ formatDate(endTime, selectedTimezone, "YYYY-MM-DD HH:mm:ss.SSS"),
Review Comment:
```suggestion
formatDate(endTime, selectedTimezone, defaultFormatWithTZ),
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] #55020 gantt view is not getting updatedin realtime [airflow]
Posted by "yimingpeng (via GitHub)" <gi...@apache.org>.
yimingpeng commented on PR #55130:
URL: https://github.com/apache/airflow/pull/55130#issuecomment-3289396385
hello @bbovenzi, thanks for the code review, I guess you meant `DEFAULT_DATETIME_FORMAT_WITH_TZ `? Can you have another review pls? if I misunderstood anything here, please correct me, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] #55020 gantt view is not getting updatedin realtime [airflow]
Posted by "guan404ming (via GitHub)" <gi...@apache.org>.
guan404ming commented on code in PR #55130:
URL: https://github.com/apache/airflow/pull/55130#discussion_r2313564243
##########
airflow-core/src/airflow/ui/src/layouts/Details/Gantt/Gantt.tsx:
##########
@@ -123,6 +124,20 @@ export const Gantt = ({ limit }: Props) => {
const isLoading = runsLoading || structureLoading || summariesLoading || tiLoading;
+ useEffect(() => {
Review Comment:
`taskInstancesData` would auto-refresh here so I think it could work without this `useEffect`.
https://github.com/apache/airflow/blob/2b91be8e490714b4fef68a6d89a695caa1003745/airflow-core/src/airflow/ui/src/layouts/Details/Gantt/Gantt.tsx#L109-L120
##########
airflow-core/src/airflow/ui/src/layouts/Details/Gantt/utils.ts:
##########
@@ -81,93 +81,98 @@ export const createChartOptions = ({
selectedRun,
selectedTimezone,
translate,
-}: ChartOptionsParams) => ({
- animation: {
- duration: 100,
- },
- indexAxis: "y" as const,
- maintainAspectRatio: false,
- onClick: handleBarClick,
- onHover: (event: ChartEvent, elements: Array<ActiveElement>) => {
- const target = event.native?.target as HTMLElement | undefined;
+}: ChartOptionsParams) => {
+ const isRunning = selectedRun?.state === "running";
+ const effectiveEndDate = isRunning ? new Date().toISOString() : selectedRun?.end_date;
Review Comment:
Also here
##########
airflow-core/src/airflow/ui/src/layouts/Details/Gantt/Gantt.tsx:
##########
@@ -123,6 +124,20 @@ export const Gantt = ({ limit }: Props) => {
const isLoading = runsLoading || structureLoading || summariesLoading || tiLoading;
+ useEffect(() => {
+ const hasRunningTasks = taskInstancesData?.task_instances.some((ti) => isStatePending(ti.state));
+
+ if (hasRunningTasks) {
+ const interval = setInterval(() => {
+ setCurrentTime(new Date().toISOString());
Review Comment:
We could get current time with `selectedTimezone` easily with `dayjs`
```js
const { selectedTimezone } = useTimezone();
// Make the value timezone-aware
setCurrentTime(dayjs().tz(selectedTimezone).format("YYYY-MM-DD HH:mm:ss"))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] #55020 gantt view is not getting updatedin realtime [airflow]
Posted by "guan404ming (via GitHub)" <gi...@apache.org>.
guan404ming commented on code in PR #55130:
URL: https://github.com/apache/airflow/pull/55130#discussion_r2319391503
##########
airflow-core/src/airflow/ui/src/layouts/Details/Gantt/Gantt.tsx:
##########
@@ -33,6 +33,9 @@ import {
import "chart.js/auto";
import "chartjs-adapter-dayjs-4/dist/chartjs-adapter-dayjs-4.esm";
import annotationPlugin from "chartjs-plugin-annotation";
+import dayjs from "dayjs";
+import timezone from "dayjs/plugin/timezone";
+import utc from "dayjs/plugin/utc";
Review Comment:
Is there any special reason for using this?
##########
airflow-core/src/airflow/ui/src/layouts/Details/Gantt/Gantt.tsx:
##########
@@ -33,6 +33,9 @@ import {
import "chart.js/auto";
import "chartjs-adapter-dayjs-4/dist/chartjs-adapter-dayjs-4.esm";
import annotationPlugin from "chartjs-plugin-annotation";
+import dayjs from "dayjs";
+import timezone from "dayjs/plugin/timezone";
+import utc from "dayjs/plugin/utc";
Review Comment:
Is there any special reason for using this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] #55020 gantt view is not getting updatedin realtime [airflow]
Posted by "yimingpeng (via GitHub)" <gi...@apache.org>.
yimingpeng commented on PR #55130:
URL: https://github.com/apache/airflow/pull/55130#issuecomment-3244886073
Hello @guan404ming,
Thanks for all the great suggestions, I believe I have addressed them, also made a few more changes, e..g, expanding to other pending tasks (`isStatePending`) which was checking `running` only. Please help review again, thanks.
Here is the expanded test dag from the original issue post:
```python
from datetime import datetime, timedelta
from airflow import DAG
from airflow.providers.standard.operators.bash import BashOperator
from airflow.providers.standard.operators.python import PythonOperator
from airflow.providers.standard.sensors.filesystem import FileSensor
from airflow.utils.task_group import TaskGroup
def task_that_fails():
raise Exception("This task always fails for testing")
def task_that_succeeds():
print("This task succeeds")
return "success"
dag = DAG(
'test_gantt_chart_dag',
start_date=datetime(2025, 7, 1),
schedule=None,
is_paused_upon_creation=False,
catchup=False,
max_active_runs=1,
default_args={
'retries': 2,
'retry_delay': timedelta(seconds=10),
}
)
long_running_task = BashOperator(
task_id="long_running_task",
bash_command="echo 'Starting long task'; sleep 180; echo 'Long task completed'",
dag=dag
)
queued_task = BashOperator(
task_id="queued_task",
bash_command="echo 'Queued task executed'; sleep 30",
dag=dag,
pool='default_pool',
pool_slots=10
)
failing_task = PythonOperator(
task_id="failing_task",
python_callable=task_that_fails,
dag=dag
)
waiting_sensor = FileSensor(
task_id="waiting_sensor",
filepath="/tmp/nonexistent_file_for_testing.txt",
timeout=300,
poke_interval=30,
dag=dag
)
scheduled_task = BashOperator(
task_id="scheduled_task",
bash_command="echo 'Scheduled task executed'; sleep 15",
dag=dag
)
quick_success_task = PythonOperator(
task_id="quick_success_task",
python_callable=task_that_succeeds,
dag=dag
)
upstream_failed_task = BashOperator(
task_id="upstream_failed_task",
bash_command="echo 'This should not run due to upstream failure'",
dag=dag
)
with TaskGroup("processing_group", dag=dag) as processing_group:
group_start = BashOperator(
task_id="group_start",
bash_command="echo 'Group starting'; sleep 5",
dag=dag,
)
parallel_task_1 = BashOperator(
task_id="parallel_task_1",
bash_command="echo 'Parallel task 1 running'; sleep 60",
dag=dag,
)
parallel_task_2 = PythonOperator(
task_id="parallel_task_2",
python_callable=task_that_fails,
dag=dag,
)
parallel_task_3 = BashOperator(
task_id="parallel_task_3",
bash_command="echo 'Parallel task 3 running'; sleep 45",
dag=dag,
)
group_end = BashOperator(
task_id="group_end",
bash_command="echo 'Group completed'; sleep 10",
trigger_rule="none_failed_min_one_success",
dag=dag,
)
group_start >> [parallel_task_1, parallel_task_2, parallel_task_3] >> group_end
final_task = BashOperator(
task_id="final_task",
bash_command="echo 'All processing completed'",
dag=dag
)
long_running_task >> scheduled_task
quick_success_task >> queued_task
failing_task >> upstream_failed_task
waiting_sensor >> scheduled_task
processing_group >> final_task
```
Here is the new recording after the fix is applied:
https://github.com/user-attachments/assets/f93615d0-8fba-4842-bebc-f5d580c16730
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] #55020 gantt view is not getting updatedin realtime [airflow]
Posted by "bbovenzi (via GitHub)" <gi...@apache.org>.
bbovenzi merged PR #55130:
URL: https://github.com/apache/airflow/pull/55130
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] #55020 gantt view is not getting updatedin realtime [airflow]
Posted by "yimingpeng (via GitHub)" <gi...@apache.org>.
yimingpeng commented on code in PR #55130:
URL: https://github.com/apache/airflow/pull/55130#discussion_r2321369331
##########
airflow-core/src/airflow/ui/src/layouts/Details/Gantt/Gantt.tsx:
##########
@@ -33,6 +33,9 @@ import {
import "chart.js/auto";
import "chartjs-adapter-dayjs-4/dist/chartjs-adapter-dayjs-4.esm";
import annotationPlugin from "chartjs-plugin-annotation";
+import dayjs from "dayjs";
+import timezone from "dayjs/plugin/timezone";
+import utc from "dayjs/plugin/utc";
Review Comment:
oh, good catch, those imports are redundant, as well as L58-L60 down below. Sorry, not very familiar with `dayjs` so missed that. Fixing it now, thanks for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] #55020 gantt view is not getting updatedin realtime [airflow]
Posted by "guan404ming (via GitHub)" <gi...@apache.org>.
guan404ming commented on PR #55130:
URL: https://github.com/apache/airflow/pull/55130#issuecomment-3283704042
@yimingpeng please help rebase and fix the tests, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] #55020 gantt view is not getting updatedin realtime [airflow]
Posted by "bbovenzi (via GitHub)" <gi...@apache.org>.
bbovenzi commented on code in PR #55130:
URL: https://github.com/apache/airflow/pull/55130#discussion_r2345123646
##########
airflow-core/src/airflow/ui/src/layouts/Details/Gantt/Gantt.tsx:
##########
@@ -127,6 +128,8 @@ export const Gantt = ({ limit }: Props) => {
const isLoading = runsLoading || structureLoading || summariesLoading || tiLoading;
+ const currentTime = dayjs().tz(selectedTimezone).format("YYYY-MM-DD HH:mm:ss");
Review Comment:
Let's use the `defaultFormatWithTZ` var
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] #55020 gantt view is not getting updatedin realtime [airflow]
Posted by "guan404ming (via GitHub)" <gi...@apache.org>.
guan404ming commented on code in PR #55130:
URL: https://github.com/apache/airflow/pull/55130#discussion_r2319392221
##########
airflow-core/src/airflow/ui/src/layouts/Details/Gantt/Gantt.tsx:
##########
@@ -33,6 +33,9 @@ import {
import "chart.js/auto";
import "chartjs-adapter-dayjs-4/dist/chartjs-adapter-dayjs-4.esm";
import annotationPlugin from "chartjs-plugin-annotation";
+import dayjs from "dayjs";
+import timezone from "dayjs/plugin/timezone";
+import utc from "dayjs/plugin/utc";
Review Comment:
Is there any special reason for using this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] #55020 gantt view is not getting updatedin realtime [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #55130:
URL: https://github.com/apache/airflow/pull/55130#issuecomment-3241636619
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:
- Pay attention to the quality of your code (ruff, mypy and type annotations). Our [prek-hooks]( https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst#prerequisites-for-prek-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/airflow-core/docs/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/dev/breeze/doc/README.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#coding-style-and-best-practices).
- Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org