You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "huymq1710 (via GitHub)" <gi...@apache.org> on 2024/05/31 01:52:23 UTC
[PR] fix: publish schedule_delay metrics unless run_type is 'backfill' [airflow]
huymq1710 opened a new pull request, #39958:
URL: https://github.com/apache/airflow/pull/39958
The `schedule_delay` metric is crucial for defining SLOs, but it often includes noise from backfill DagRuns.
This PR aims to exclude backfill DagRuns from the `schedule_delay` metrics to enhance data accuracy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: publish schedule_delay metrics unless run_type is 'backfill' [airflow]
Posted by "huymq1710 (via GitHub)" <gi...@apache.org>.
huymq1710 commented on PR #39958:
URL: https://github.com/apache/airflow/pull/39958#issuecomment-2141723227
@uranusjr
Coud you check this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Do not publish schedule_delay metrics if run_type is 'backfill' [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39958:
URL: https://github.com/apache/airflow/pull/39958#issuecomment-2375486688
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Do not publish schedule_delay metrics if run_type is 'backfill' [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39958:
URL: https://github.com/apache/airflow/pull/39958#issuecomment-2251611770
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Do not publish schedule_delay metrics if run_type is 'backfill' [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on PR #39958:
URL: https://github.com/apache/airflow/pull/39958#issuecomment-2144232206
p.s. Changing the title; the original implies weโre adding this metric. In reality the metric already exists, and this removes its emission in backfills.
> it often includes noise from backfill DagRuns
Would you mind elaborating on this? Some real cases would be helpful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Do not publish schedule_delay metrics if run_type is 'backfill' [airflow]
Posted by "huymq1710 (via GitHub)" <gi...@apache.org>.
huymq1710 commented on PR #39958:
URL: https://github.com/apache/airflow/pull/39958#issuecomment-2251964453
@uranusjr
Coud you check this ๐๐ผ
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Do not publish schedule_delay metrics if run_type is 'backfill' [airflow]
Posted by "huymq1710 (via GitHub)" <gi...@apache.org>.
huymq1710 commented on PR #39958:
URL: https://github.com/apache/airflow/pull/39958#issuecomment-2144268691
> The `schedule_delay` value is the time diff between start_date of a DagRun (the time it is started) and the data_interval_end (the time it is expected to run)
https://github.com/apache/airflow/blob/2.9.1/airflow/jobs/scheduler_job_runner.py#L1358-L1359
<img width="669" alt="image" src="https://github.com/apache/airflow/assets/57699454/1d0f9bfe-4be2-4440-b178-c97d3b626fc7">
- Normal DagRuns should have a delay less than 5 mins
- We are setting our delay SLO to 10 mins
- When a schedule delay value exceeds 30 mins threshold, it is considered noisy data, and it is very likely that the data is generated by a backfill DagRun. In this case, we need to check manually
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org