You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/09/18 14:23:09 UTC
(datafusion-comet) branch gh-readonly-queue/main/pr-6000-9bc07ef0110dbcc1dd003030abb1fbe97936e46d created (now 67168ca1f4)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-6000-9bc07ef0110dbcc1dd003030abb1fbe97936e46d
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
at 67168ca1f4 docs: how to check the scheduled CI runs are actually running (#6000)
This branch includes the following new commits:
new 67168ca1f4 docs: how to check the scheduled CI runs are actually running (#6000)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion-comet) 01/01: docs: how to check the scheduled CI runs are actually running (#6000)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-6000-9bc07ef0110dbcc1dd003030abb1fbe97936e46d
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit 67168ca1f494572e4f01959d04bb51331c1ef53d
Author: Andy Grove <ag...@apache.org>
AuthorDate: Fri Sep 18 14:20:33 2026 +0000
docs: how to check the scheduled CI runs are actually running (#6000)
* docs: how to check the scheduled CI runs are actually running
Most of Comet's coverage of the non-default Spark and Iceberg versions moved
to the nightly tier in #5963, and a scheduled run has no pull request to turn
red. `ci.yml` opens a `ci-nightly-failure` issue when it goes red, but
`miri.yml` and `publish_snapshot.yml` report nothing at all, which is how
Miri came to fail 79 nights in a row unnoticed (issue #5999).
Add a section to the CI guide covering the two distinct failures: a schedule
that stopped firing, and a streak of red nights. For `ci.yml` specifically,
green does not mean the suites ran -- a quiet day and a drifted diff base both
report the same green `Required Checks` -- so include a second command that
counts suite jobs by conclusion to tell them apart.
Hook it into the release process as a preparation step, so a release does not
go out on the assumption that the nightly tier has been covering the versions
the tier table says it covers.
Use `gh api` rather than `gh run list --event=schedule` in the documented
commands: the latter returned three-week-old runs as the latest three while
investigating this.
* docs: add a CI tier diagram, pin the job lookup to the upstream repo
Review feedback on #6000.
comphead asked for a diagram of how CI splits across the PR, merge queue and
nightly tiers. The docs site is Sphinx and had no mermaid support, so a bare
```mermaid fence would have rendered as a literal code block there while
rendering as a diagram on github.com. Add sphinxcontrib-mermaid and set
myst_fence_as_directive so the one fence renders as a diagram in both places.
The extension loads mermaid.js from cdn.jsdelivr.net at the version it pins.
sunchao found that the job-conclusion command mixes repositories: the run ID
comes from an explicit `gh api` call against apache/datafusion-comet, but the
`gh run view` that consumes it resolves against whatever repository the caller
has selected, so it 404s for anyone working from a fork. Pass --repo.
---
docs/requirements.txt | 1 +
docs/source/conf.py | 9 +++
docs/source/contributor-guide/ci.md | 72 ++++++++++++++++++++++++
docs/source/contributor-guide/release_process.md | 9 +++
4 files changed, 91 insertions(+)
diff --git a/docs/requirements.txt b/docs/requirements.txt
index 80505f2aa5..10eb4f693d 100644
--- a/docs/requirements.txt
+++ b/docs/requirements.txt
@@ -20,5 +20,6 @@ sphinx>=7.0,<8.0
sphinx-reredirects
pydata-sphinx-theme>=0.16.1,<0.17.0
myst-parser>=2.0,<4.0
+sphinxcontrib-mermaid>=1.0,<3.0
maturin
jinja2
diff --git a/docs/source/conf.py b/docs/source/conf.py
index 63f8f0114b..ba325734e0 100644
--- a/docs/source/conf.py
+++ b/docs/source/conf.py
@@ -53,6 +53,10 @@ extensions = [
'sphinx.ext.napoleon',
'myst_parser',
'sphinx_reredirects',
+ # Renders the ```mermaid fences in the contributor guide. The diagrams are drawn in
+ # the browser by mermaid.js, which sphinxcontrib-mermaid loads from cdn.jsdelivr.net
+ # at the version pinned by its `mermaid_version` default.
+ 'sphinxcontrib.mermaid',
]
source_suffix = {
@@ -60,6 +64,11 @@ source_suffix = {
'.md': 'markdown',
}
+# Route ```mermaid fences to the mermaid directive rather than treating them as a literal
+# code block, so the same source renders as a diagram here and on github.com, which
+# understands that fence natively.
+myst_fence_as_directive = ['mermaid']
+
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
diff --git a/docs/source/contributor-guide/ci.md b/docs/source/contributor-guide/ci.md
index 20f2eedf80..585d0c589e 100644
--- a/docs/source/contributor-guide/ci.md
+++ b/docs/source/contributor-guide/ci.md
@@ -32,6 +32,26 @@ preflight checks first (license headers, Markdown formatting, workflow linting,
checks), computes which heavy jobs the changed files are relevant to, and fans out to those jobs.
Which jobs run also depends on the event:
+```mermaid
+flowchart LR
+ PR([pull request]) --> PRTIER
+ PR -. with label .-> QUEUE
+ PR -. with label .-> NIGHTLY
+ PR -. with label .-> S34
+ MQ([merge queue]) --> PRTIER
+ MQ --> QUEUE
+ CRON([schedule<br>06:00 UTC]) --> NIGHTLY
+ PUSH([push to main]) --> CACHE
+
+ PRTIER["PR tier<br>Linux build, lint, Rust tests<br>TPC-H / TPC-DS<br>Comet suites, Spark 4.1"]
+ QUEUE["Queue tier, on top of the PR tier<br>Spark SQL, Spark 4.1<br>Iceberg 1.11<br>macOS build and Comet suites<br>Benchmark check, Delta gate<br>PyArrow UDF, Spark 4.0 / 4.1 / 4.2"]
+ NIGHTLY["Nightly tier<br>Comet suites, Spark 3.4 / 3.5 / 4.0 / 4.2<br>Spark SQL, Spark 3.5 / 4.0<br>Iceberg 1.8 / 1.9 / 1.10"]
+ S34["Neither tier<br>Spark SQL, Spark 3.4"]
+ CACHE["Cache-refresh-only mode<br>the four cache-writing jobs, plus Lint"]
+```
+
+Suite by suite:
+
| Suite | Pull request | Merge queue | Nightly |
| ------------------------------------------------- | ------------ | ----------- | ------- |
| Linux build, lint, Rust tests, TPC-H/TPC-DS | yes | yes | no |
@@ -244,6 +264,58 @@ stays in one place. When you pick up a nightly failure:
Dispatching `ci.yml` from the Actions page with **Run workflow** runs every tier, including the
nightly suites, if you need a result before the next scheduled run.
+## Checking that the scheduled runs are healthy
+
+A scheduled run has no pull request to turn red, so when one breaks, nothing puts it in front of
+anyone. Comet has three daily schedules plus a weekly one, and two of the daily ones report nothing
+when they fail:
+
+| Workflow | Cron (UTC) | Reports a failure? |
+| ---------------------- | ------------ | ------------------------------------- |
+| `publish_snapshot.yml` | `0 3 * * *` | no |
+| `miri.yml` | `0 4 * * *` | no |
+| `ci.yml` nightly tier | `0 6 * * *` | yes, a `ci-nightly-failure` issue |
+| `codeql.yml` | `16 4 * * 1` | yes, to the repository's Security tab |
+
+So the two without a reporting step have to be looked at deliberately. Check all of them at once:
+
+```sh
+for wf in ci.yml miri.yml publish_snapshot.yml; do
+ gh api "repos/apache/datafusion-comet/actions/workflows/$wf/runs?event=schedule&per_page=5" \
+ --jq ".workflow_runs[] | \"$wf\t\(.created_at[0:10])\t\(.conclusion)\t\(.html_url)\""
+done
+```
+
+Read the output for two different things:
+
+- **A missing row.** If the most recent run is not from the last day or two, the schedule itself
+ stopped firing. GitHub drops scheduled runs under load and disables them entirely in a repository
+ with no activity for 60 days, and it does not announce either. Confirm the workflow is still
+ enabled with `gh api repos/apache/datafusion-comet/actions/workflows --jq '.workflows[] | "\(.state)\t\(.path)"'`,
+ and re-enable it from the Actions page if it is `disabled_inactivity`.
+- **A run of failures.** One red night is a flake or a real regression, and for `ci.yml` there is an
+ issue open about it. Several consecutive red nights on `miri.yml` or `publish_snapshot.yml` means
+ nobody has looked; treat the streak, not the newest run, as the thing to explain.
+
+For the `ci.yml` nightly, green on its own does not mean the suites ran. Path filters and the diff
+base are both allowed to select nothing — a documentation-only day legitimately runs no suite at
+all — and a run that tested nothing reports exactly the same green `Required Checks` as a run that
+tested everything. Confirm the suites actually ran:
+
+```sh
+run=$(gh api "repos/apache/datafusion-comet/actions/workflows/ci.yml/runs?event=schedule&per_page=1" \
+ --jq '.workflow_runs[0].id')
+gh run view "$run" --repo apache/datafusion-comet --json jobs \
+ --jq '[.jobs[] | select(.name | test("^(Spark SQL|Iceberg Spark SQL) Tests"))]
+ | group_by(.conclusion)[] | "\(length)\t\(.[0].conclusion)"'
+```
+
+A healthy run over a day of normal merges reports about 40 successes, with the handful of skips
+being the suites that belong to the queue tier rather than the nightly one — Spark 3.4, Spark 4.1
+and Iceberg 1.11. If everything is skipped, open the run's `Detect changes` job: it logs the
+`Nightly base:` commit it diffed against and the list of changed files, which is enough to tell a
+genuinely quiet day from a base that has drifted.
+
## Reproducing a suite failure locally
The Spark SQL suites outside the PR tier run Spark's own test suite against Comet, with the
diff --git a/docs/source/contributor-guide/release_process.md b/docs/source/contributor-guide/release_process.md
index 2cc430a148..ae364b3692 100644
--- a/docs/source/contributor-guide/release_process.md
+++ b/docs/source/contributor-guide/release_process.md
@@ -29,6 +29,7 @@ The following is a quick-reference checklist for the full release process. See t
instructions on each step.
- [ ] Release preparation: review expression support status and user guide
+- [ ] Release preparation: check the scheduled CI runs are healthy
- [ ] Create release branch
- [ ] Protect the release branch in `.asf.yaml`
- [ ] Generate release documentation
@@ -70,6 +71,14 @@ It is also recommended to run benchmarks (such as TPC-H and TPC-DS) comparing pe
release to check for regressions. See the
[Comet Benchmarking Guide](benchmarking.md) for instructions.
+Check that the scheduled CI runs have been healthy over the release window. Most of Comet's coverage of the
+non-default Spark and Iceberg versions runs nightly rather than on pull requests, so a nightly run that has been
+failing — or one that silently stopped firing — means the release is going out with less testing behind it than the
+tier table suggests. A scheduled run has no pull request to turn red, and `miri.yml` and `publish_snapshot.yml` do
+not report their own failures at all, so this has to be looked at deliberately. See
+[Checking that the scheduled runs are healthy](ci.md#checking-that-the-scheduled-runs-are-healthy) for the commands
+and for how to tell a genuinely quiet night from a broken one.
+
These are tasks where agentic coding tools can be particularly helpful — for example, scanning the codebase for
newly registered expressions and cross-referencing them against the documented list, or generating test queries to
verify expression support status.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org