You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/09/22 17:49:02 UTC
(datafusion-comet) branch asf-site updated: Publish built docs triggered by b2d06787824baeb08ebb2893678f7710615488db
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 497070bc3f Publish built docs triggered by b2d06787824baeb08ebb2893678f7710615488db
497070bc3f is described below
commit 497070bc3f99ffe177f26c8250303aaadc71b85f
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Tue Sep 22 17:48:57 2026 +0000
Publish built docs triggered by b2d06787824baeb08ebb2893678f7710615488db
---
...id-013012468fa27c9e9fdb0a14196d357c619c68a8.svg | 2 +-
_sources/contributor-guide/ci.md.txt | 38 ++++++++++++++++++----
_sources/contributor-guide/release_process.md.txt | 4 +--
contributor-guide/ci.html | 36 ++++++++++++++++----
contributor-guide/release_process.html | 4 +--
searchindex.js | 2 +-
6 files changed, 66 insertions(+), 20 deletions(-)
diff --git a/_images/mermaid-013012468fa27c9e9fdb0a14196d357c619c68a8.svg b/_images/mermaid-013012468fa27c9e9fdb0a14196d357c619c68a8.svg
index d6570e9ffb..d7670fedb1 100644
--- a/_images/mermaid-013012468fa27c9e9fdb0a14196d357c619c68a8.svg
+++ b/_images/mermaid-013012468fa27c9e9fdb0a14196d357c619c68a8.svg
@@ -1 +1 @@
-<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 942" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
+<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 942" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
diff --git a/_sources/contributor-guide/ci.md.txt b/_sources/contributor-guide/ci.md.txt
index 103b89f3c1..cea032370e 100644
--- a/_sources/contributor-guide/ci.md.txt
+++ b/_sources/contributor-guide/ci.md.txt
@@ -264,16 +264,40 @@ stays in one place. When you pick up a nightly failure:
Dispatching `ci.yml` from the Actions page with **Run workflow** runs every tier, including the
nightly suites, if you need a result before the next scheduled run.
+## Miri safety checks
+
+`miri.yml` runs nightly and supports manual dispatch. It runs the shuffle crate's
+`spark_unsafe::` tests and the expression crate's `hash_funcs::` tests in separate jobs,
+so a failure in one does not cancel the other. These cover Comet's unsafe row decoding
+and hash kernels. The job also fails if its filter selects no passing tests. A failed
+scheduled run opens or updates the same `ci-nightly-failure` issue used by nightly CI.
+
+Run either suite locally after installing nightly Rust with the Miri component:
+
+```sh
+cd native
+cargo +nightly miri setup
+MIRIFLAGS="-Zmiri-disable-isolation" cargo +nightly miri test --locked \
+ -p datafusion-comet-shuffle --lib 'spark_unsafe::'
+MIRIFLAGS="-Zmiri-disable-isolation" cargo +nightly miri test --locked \
+ -p datafusion-comet-spark-expr --lib 'hash_funcs::'
+```
+
+Miri cannot execute the JVM, cloud clients, or compression libraries through FFI.
+Running the whole workspace reaches those dependencies or dependency errors in
+platform synchronization before completing the safety checks. Regular Rust CI still runs the
+full tests. When adding unsafe code, add suitable tests to the Miri matrix as well;
+these focused suites do not cover every unsafe operation in Comet.
+
## Checking that the scheduled runs are healthy
-A scheduled run has no pull request to turn red, so when one breaks, nothing puts it in front of
-anyone. Comet has three daily schedules plus a weekly one, and two of the daily ones report nothing
-when they fail:
+A scheduled run has no pull request to turn red, so check both whether it ran and whether it
+reported a failure. Comet has three daily schedules plus a weekly one:
| Workflow | Cron (UTC) | Reports a failure? |
| ---------------------- | ------------ | ------------------------------------- |
| `publish_snapshot.yml` | `0 3 * * *` | no |
-| `miri.yml` | `0 4 * * *` | no |
+| `miri.yml` | `0 4 * * *` | yes |
| `ci.yml` nightly tier | `0 6 * * *` | yes, a `ci-nightly-failure` issue |
| `codeql.yml` | `16 4 * * 1` | yes, to the repository's Security tab |
@@ -293,9 +317,9 @@ Read the output for two different things:
with no activity for 60 days, and it does not announce either. Confirm the workflow is still
enabled with `gh api repos/apache/datafusion-comet/actions/workflows --jq '.workflows[] | "\(.state)\t\(.path)"'`,
and re-enable it from the Actions page if it is `disabled_inactivity`.
-- **A run of failures.** One red night is a flake or a real regression, and for `ci.yml` there is an
- issue open about it. Several consecutive red nights on `miri.yml` or `publish_snapshot.yml` means
- nobody has looked; treat the streak, not the newest run, as the thing to explain.
+- **A run of failures.** One red night is a flake or a real regression. `ci.yml` and `miri.yml`
+ open or update a failure issue; `publish_snapshot.yml` does not. Investigate repeated failures
+ even when a report already exists; treat the streak, not the newest run, as the thing to explain.
For the `ci.yml` nightly, green on its own does not mean the suites ran. Path filters and the diff
base are both allowed to select nothing — a documentation-only day legitimately runs no suite at
diff --git a/_sources/contributor-guide/release_process.md.txt b/_sources/contributor-guide/release_process.md.txt
index ae364b3692..e490cbec25 100644
--- a/_sources/contributor-guide/release_process.md.txt
+++ b/_sources/contributor-guide/release_process.md.txt
@@ -74,8 +74,8 @@ release to check for regressions. See the
Check that the scheduled CI runs have been healthy over the release window. Most of Comet's coverage of the
non-default Spark and Iceberg versions runs nightly rather than on pull requests, so a nightly run that has been
failing — or one that silently stopped firing — means the release is going out with less testing behind it than the
-tier table suggests. A scheduled run has no pull request to turn red, and `miri.yml` and `publish_snapshot.yml` do
-not report their own failures at all, so this has to be looked at deliberately. See
+tier table suggests. A scheduled run has no pull request to turn red, and `publish_snapshot.yml` does
+not report its own failures, so this has to be looked at deliberately. See
[Checking that the scheduled runs are healthy](ci.md#checking-that-the-scheduled-runs-are-healthy) for the commands
and for how to tell a genuinely quiet night from a broken one.
diff --git a/contributor-guide/ci.html b/contributor-guide/ci.html
index 1c48d21a3e..47a1d4fadf 100644
--- a/contributor-guide/ci.html
+++ b/contributor-guide/ci.html
@@ -856,11 +856,32 @@ be infrastructure. The next red nightly opens a new issue.</p></li>
<p>Dispatching <code class="docutils literal notranslate"><span class="pre">ci.yml</span></code> from the Actions page with <strong>Run workflow</strong> runs every tier, including the
nightly suites, if you need a result before the next scheduled run.</p>
</section>
+<section id="miri-safety-checks">
+<h2>Miri safety checks<a class="headerlink" href="#miri-safety-checks" title="Link to this heading">#</a></h2>
+<p><code class="docutils literal notranslate"><span class="pre">miri.yml</span></code> runs nightly and supports manual dispatch. It runs the shuffle crate’s
+<code class="docutils literal notranslate"><span class="pre">spark_unsafe::</span></code> tests and the expression crate’s <code class="docutils literal notranslate"><span class="pre">hash_funcs::</span></code> tests in separate jobs,
+so a failure in one does not cancel the other. These cover Comet’s unsafe row decoding
+and hash kernels. The job also fails if its filter selects no passing tests. A failed
+scheduled run opens or updates the same <code class="docutils literal notranslate"><span class="pre">ci-nightly-failure</span></code> issue used by nightly CI.</p>
+<p>Run either suite locally after installing nightly Rust with the Miri component:</p>
+<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span><span class="nb">cd</span><span class="w"> </span>native
+cargo<span class="w"> </span>+nightly<span class="w"> </span>miri<span class="w"> </span>setup
+<span class="nv">MIRIFLAGS</span><span class="o">=</span><span class="s2">"-Zmiri-disable-isolation"</span><span class="w"> </span>cargo<span class="w"> </span>+nightly<span class="w"> </span>miri<span class="w"> </span><span class="nb">test</span><span class="w"> </span>--locked<span class="w"> </span><span class="se">\</span>
+<span class="w"> </span>-p<span class="w"> </span>datafusion-comet-shuffle<span class="w"> </span>--lib<span class="w"> </span><span class="s1">'spark_unsafe::'</span>
+<span class="nv">MIRIFLAGS</span><span class="o">=</span><span class="s2">"-Zmiri-disable-isolation"</span><span class="w"> </span>cargo<span class="w"> </span>+nightly<span class="w"> </span>miri<span class="w"> </span><span class="nb">test</span><span class="w"> </span>--locked<span class="w"> </span><span class="se">\</span>
+<span class="w"> </span>-p<span class="w"> </span>datafusion-comet-spark-expr<span class="w"> </span>--lib<span class="w"> </span><span class="s1">'hash_funcs::'</span>
+</pre></div>
+</div>
+<p>Miri cannot execute the JVM, cloud clients, or compression libraries through FFI.
+Running the whole workspace reaches those dependencies or dependency errors in
+platform synchronization before completing the safety checks. Regular Rust CI still runs the
+full tests. When adding unsafe code, add suitable tests to the Miri matrix as well;
+these focused suites do not cover every unsafe operation in Comet.</p>
+</section>
<section id="checking-that-the-scheduled-runs-are-healthy">
<h2>Checking that the scheduled runs are healthy<a class="headerlink" href="#checking-that-the-scheduled-runs-are-healthy" title="Link to this heading">#</a></h2>
-<p>A scheduled run has no pull request to turn red, so when one breaks, nothing puts it in front of
-anyone. Comet has three daily schedules plus a weekly one, and two of the daily ones report nothing
-when they fail:</p>
+<p>A scheduled run has no pull request to turn red, so check both whether it ran and whether it
+reported a failure. Comet has three daily schedules plus a weekly one:</p>
<div class="pst-scrollable-table-container"><table class="table">
<thead>
<tr class="row-odd"><th class="head"><p>Workflow</p></th>
@@ -875,7 +896,7 @@ when they fail:</p>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">miri.yml</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">0</span> <span class="pre">4</span> <span class="pre">*</span> <span class="pre">*</span> <span class="pre">*</span></code></p></td>
-<td><p>no</p></td>
+<td><p>yes</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">ci.yml</span></code> nightly tier</p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">0</span> <span class="pre">6</span> <span class="pre">*</span> <span class="pre">*</span> <span class="pre">*</span></code></p></td>
@@ -902,9 +923,9 @@ stopped firing. GitHub drops scheduled runs under load and disables them entirel
with no activity for 60 days, and it does not announce either. Confirm the workflow is still
enabled with <code class="docutils literal notranslate"><span class="pre">gh</span> <span class="pre">api</span> <span class="pre">repos/apache/datafusion-comet/actions/workflows</span> <span class="pre">--jq</span> <span class="pre">'.workflows[]</span> <span class="pre">|</span> <span class="pre">"\(.state)\t\(.path)"'</span></code>,
and re-enable it from the Actions page if it is <code class="docutils literal notranslate"><span class="pre">disabled_inactivity</span></code>.</p></li>
-<li><p><strong>A run of failures.</strong> One red night is a flake or a real regression, and for <code class="docutils literal notranslate"><span class="pre">ci.yml</span></code> there is an
-issue open about it. Several consecutive red nights on <code class="docutils literal notranslate"><span class="pre">miri.yml</span></code> or <code class="docutils literal notranslate"><span class="pre">publish_snapshot.yml</span></code> means
-nobody has looked; treat the streak, not the newest run, as the thing to explain.</p></li>
+<li><p><strong>A run of failures.</strong> One red night is a flake or a real regression. <code class="docutils literal notranslate"><span class="pre">ci.yml</span></code> and <code class="docutils literal notranslate"><span class="pre">miri.yml</span></code>
+open or update a failure issue; <code class="docutils literal notranslate"><span class="pre">publish_snapshot.yml</span></code> does not. Investigate repeated failures
+even when a report already exists; treat the streak, not the newest run, as the thing to explain.</p></li>
</ul>
<p>For the <code class="docutils literal notranslate"><span class="pre">ci.yml</span></code> nightly, green on its own does not mean the suites ran. Path filters and the diff
base are both allowed to select nothing — a documentation-only day legitimately runs no suite at
@@ -1046,6 +1067,7 @@ list first.</p>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#merging-through-the-queue">Merging through the queue</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#when-a-queue-run-fails">When a queue run fails</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#nightly-runs">Nightly runs</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#miri-safety-checks">Miri safety checks</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#checking-that-the-scheduled-runs-are-healthy">Checking that the scheduled runs are healthy</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#reproducing-a-suite-failure-locally">Reproducing a suite failure locally</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#changing-ci-itself">Changing CI itself</a></li>
diff --git a/contributor-guide/release_process.html b/contributor-guide/release_process.html
index c9d1112f8f..062fce91f4 100644
--- a/contributor-guide/release_process.html
+++ b/contributor-guide/release_process.html
@@ -608,8 +608,8 @@ release to check for regressions. See the
<p>Check that the scheduled CI runs have been healthy over the release window. Most of Comet’s coverage of the
non-default Spark and Iceberg versions runs nightly rather than on pull requests, so a nightly run that has been
failing — or one that silently stopped firing — means the release is going out with less testing behind it than the
-tier table suggests. A scheduled run has no pull request to turn red, and <code class="docutils literal notranslate"><span class="pre">miri.yml</span></code> and <code class="docutils literal notranslate"><span class="pre">publish_snapshot.yml</span></code> do
-not report their own failures at all, so this has to be looked at deliberately. See
+tier table suggests. A scheduled run has no pull request to turn red, and <code class="docutils literal notranslate"><span class="pre">publish_snapshot.yml</span></code> does
+not report its own failures, so this has to be looked at deliberately. See
<a class="reference internal" href="ci.html#checking-that-the-scheduled-runs-are-healthy"><span class="std std-ref">Checking that the scheduled runs are healthy</span></a> for the commands
and for how to tell a genuinely quiet night from a broken one.</p>
<p>These are tasks where agentic coding tools can be particularly helpful — for example, scanning the codebase for
diff --git a/searchindex.js b/searchindex.js
index 66f1711b22..67bb2433dc 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[56, "id1"]], "%": [[54, "id1"]], "&": [[44, "id1"]], "*": [[54, "id2"]], "+": [[54, "id3"]], "-": [[54, "id4"]], "/": [[54, "id5"]], "1. Format Your Code": [[41, "format-your-code"]], "1. Install Comet": [[62, "install-comet"], [73, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[41, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[62, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[56, "id1"]], "%": [[54, "id1"]], "&": [[44, "id1"]], "*": [[54, "id2"]], "+": [[54, "id3"]], "-": [[54, "id4"]], "/": [[54, "id5"]], "1. Format Your Code": [[41, "format-your-code"]], "1. Install Comet": [[62, "install-comet"], [73, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[41, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[62, "clone-iceberg-and-apply- [...]
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org