You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2026/07/12 23:29:14 UTC
[PR] Discount CI image-build time from duration-trend alerts [airflow]
potiuk opened a new pull request, #69789:
URL: https://github.com/apache/airflow/pull/69789
The CI duration monitor folded the "Prepare breeze & CI image" step into each
job's total. That step occasionally balloons on a one-off cache miss (a full
image rebuild instead of a cached pull), which repeatedly flagged unrelated
jobs as regressed — e.g. a Non-DB core job reported +190% driven almost
entirely by an 18m image build, while its actual test time was flat.
Image-build time is now excluded from the run and per-job durations used for
the trend, and the image build is watched on its own: it is only reported
when it has stayed slow for more than two days (`IMAGE_BUILD_PERSISTENCE_DAYS`,
default 2), so a transient rebuild spike no longer produces a false alert.
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Opus 4.8)
Generated-by: Claude Code (Opus 4.8) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Discount CI image-build time from duration-trend alerts [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69789:
URL: https://github.com/apache/airflow/pull/69789#issuecomment-5017272489
### Backport successfully created: v3-3-test
Note: As of [Merging PRs targeted for Airflow 3.X](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#merging-prs-targeted-for-airflow-3x)
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
In matter of doubt please ask in [#release-management](https://apache-airflow.slack.com/archives/C03G9H97MM2) Slack channel.
<table>
<tr>
<th>Status</th>
<th>Branch</th>
<th>Result</th>
</tr>
<tr>
<td>✅</td>
<td>v3-3-test</td>
<td><a href="https://github.com/apache/airflow/pull/70102"><img src="https://img.shields.io/badge/PR-70102-blue" alt="PR Link"></a></td>
</tr>
</table>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Discount CI image-build time from duration-trend alerts [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #69789:
URL: https://github.com/apache/airflow/pull/69789
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Discount CI image-build time from duration-trend alerts [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #69789:
URL: https://github.com/apache/airflow/pull/69789#discussion_r3609667358
##########
scripts/ci/analyze_ci_job_durations.py:
##########
@@ -263,6 +259,34 @@ def get_prepare_breeze_step_duration(job: dict) -> float | None:
return None
+def work_duration(job_data: JobDuration) -> float:
Review Comment:
```suggestion
def calculate_work_duration(job_data: JobDuration) -> float:
```
##########
scripts/ci/analyze_ci_job_durations.py:
##########
@@ -587,32 +699,43 @@ def main() -> None:
f"Not enough runs to establish a trend "
f"(found {len(runs)}, need {latest_runs_count + min_baseline_runs}). Skipping."
)
- _write_outputs(False, False, 0)
+ _write_outputs(False, False, 0, False)
sys.exit(0)
latest_runs = runs[:latest_runs_count]
baseline_runs = runs[latest_runs_count:]
print(f"Latest runs: {len(latest_runs)}; baseline runs: {len(baseline_runs)}.")
+ # Fetch each run's jobs once: they feed the image-build discount below, the
+ # per-job trend, and the image-build persistence check.
+ jobs_by_run_id = fetch_run_jobs_map(repo, runs) if do_analyze_jobs else {}
+
+ def adjusted_run_duration(run: dict) -> float:
Review Comment:
```suggestion
def calculate_adjusted_run_duration(run: dict) -> float:
```
##########
scripts/ci/analyze_ci_job_durations.py:
##########
@@ -263,6 +259,34 @@ def get_prepare_breeze_step_duration(job: dict) -> float | None:
return None
+def work_duration(job_data: JobDuration) -> float:
+ """Return a job's wall-clock with the image-build (prepare breeze) step removed.
+
+ The image build occasionally balloons — a cache miss forces a full rebuild
+ (minutes → tens of minutes) — which would otherwise inflate the job's total
+ and flag an unrelated job as "slower". The duration trend should track the
+ actual test/work time, so image build is discounted from it and watched
+ separately by :func:`detect_image_build_regression`.
+ """
+ prepare_breeze = job_data["prepare_breeze_duration"] or 0.0
+ return max(job_data["duration"] - prepare_breeze, 0.0)
+
+
+def run_image_build_seconds(jobs: dict[str, JobDuration]) -> float | None:
Review Comment:
```suggestion
def calcuate_image_build_seconds(jobs: dict[str, JobDuration]) -> float | None:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org