You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/09/13 19:20:55 UTC
(datafusion-comet) branch asf-site updated: Publish built docs triggered by 451c99963206fa6bf0387239aa12887a16255516
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 2cf440f875 Publish built docs triggered by 451c99963206fa6bf0387239aa12887a16255516
2cf440f875 is described below
commit 2cf440f875182d2d339b0b9398972caf214ae3f3
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Sun Sep 13 19:20:51 2026 +0000
Publish built docs triggered by 451c99963206fa6bf0387239aa12887a16255516
---
_sources/contributor-guide/development.md.txt | 11 +++++++++++
_sources/user-guide/latest/datasources.md.txt | 22 ++++++++++++++++++++++
_sources/user-guide/latest/iceberg.md.txt | 4 +++-
contributor-guide/development.html | 8 ++++++++
searchindex.js | 2 +-
user-guide/latest/datasources.html | 19 +++++++++++++++++++
user-guide/latest/iceberg.html | 3 ++-
7 files changed, 66 insertions(+), 3 deletions(-)
diff --git a/_sources/contributor-guide/development.md.txt b/_sources/contributor-guide/development.md.txt
index b04a3fdd42..3025e259e8 100644
--- a/_sources/contributor-guide/development.md.txt
+++ b/_sources/contributor-guide/development.md.txt
@@ -658,6 +658,17 @@ Choose the group that best matches the area your test covers:
`*Suite.scala` files in the repository and verifies that each one appears in both workflow files.
If any suite is missing, this check will fail and block the PR.
+A small number of suites are deliberately **not** run in CI, because they need infrastructure CI
+does not have or because running them there is not worth the cost. These are listed in the
+`ignore_list` in `dev/ci/check-suites.py`, and each one documents in its own scaladoc why it is
+excluded and how to run it. Run a manual suite with:
+
+```sh
+./mvnw test -Dtest=none -Dsuites="org.apache.comet.parquet.ParquetReadFromFakeHadoopFsSuite"
+```
+
+Only add a suite to that list with a good reason; the default is that a new suite runs in CI.
+
The macOS suites only run in the merge queue by default. See
[Continuous Integration](ci.md) for the two tiers and the labels that opt a pull request into a
queue-only suite.
diff --git a/_sources/user-guide/latest/datasources.md.txt b/_sources/user-guide/latest/datasources.md.txt
index 84994c00c2..2f368a6b06 100644
--- a/_sources/user-guide/latest/datasources.md.txt
+++ b/_sources/user-guide/latest/datasources.md.txt
@@ -61,6 +61,20 @@ Comet supports most standard storage systems, such as local file system and obje
The Apache DataFusion Comet Rust-based reader seamlessly scans files from remote HDFS for [supported formats](#supported-spark-data-sources)
+```{warning}
+HDFS support is experimental and is not covered by continuous integration. Comet reads HDFS through
+`libhdfs`, which registers a thread-local destructor that detaches the calling thread from the JVM
+regardless of which component attached it
+([HDFS-16021](https://issues.apache.org/jira/browse/HDFS-16021), still open upstream). Comet
+attaches its own worker threads, so a worker that has read from HDFS can crash the JVM with a
+`SIGSEGV` when it later exits
+([#5023](https://github.com/apache/datafusion-comet/issues/5023)). The crash surfaces well after
+the HDFS read itself, typically while an unrelated query is running.
+```
+
+Native Iceberg scans do not support HDFS-backed tables; those scans fall back to Spark. See the
+[Comet and Iceberg Guide](iceberg.md).
+
### Building Comet with HDFS support
To build Comet with remote HDFS support it is required to have a JDK installed.
@@ -165,6 +179,14 @@ JAVA_HOME="/opt/homebrew/opt/openjdk@17" make release PROFILES="-Pspark-4.1" RUS
Or use `spark-shell` with HDFS support as described [above](#building-comet-with-hdfs-support)
+Comet also has a test suite that exercises a native scan through `libhdfs` against a fake Hadoop
+filesystem, so it needs no cluster. Because of the crash described above it is excluded from CI and
+run by hand:
+
+```shell
+./mvnw test -Dtest=none -Dsuites="org.apache.comet.parquet.ParquetReadFromFakeHadoopFsSuite"
+```
+
## S3
Comet's Parquet scan completely offloads data loading to Rust. It uses the
diff --git a/_sources/user-guide/latest/iceberg.md.txt b/_sources/user-guide/latest/iceberg.md.txt
index 9d2694544b..96e19ca11b 100644
--- a/_sources/user-guide/latest/iceberg.md.txt
+++ b/_sources/user-guide/latest/iceberg.md.txt
@@ -110,8 +110,10 @@ The native Iceberg reader supports the following features:
**Storage:**
- Local filesystem
-- Hadoop Distributed File System (HDFS)
- S3-compatible storage (AWS S3, MinIO)
+- Google Cloud Storage (`gs`) and Alibaba Cloud OSS (`oss`)
+
+HDFS-backed tables are not supported by the native Iceberg reader and fall back to Spark.
### REST Catalog
diff --git a/contributor-guide/development.html b/contributor-guide/development.html
index 0a33c68fc7..4f6be74fe2 100644
--- a/contributor-guide/development.html
+++ b/contributor-guide/development.html
@@ -1183,6 +1183,14 @@ appropriate group. For example, if you add a new expression test suite, add it t
<code class="docutils literal notranslate"><span class="pre">.github/workflows/ci.yml</span></code> runs <code class="docutils literal notranslate"><span class="pre">dev/ci/check-suites.py</span></code> on every pull request. It scans for all
<code class="docutils literal notranslate"><span class="pre">*Suite.scala</span></code> files in the repository and verifies that each one appears in both workflow files.
If any suite is missing, this check will fail and block the PR.</p>
+<p>A small number of suites are deliberately <strong>not</strong> run in CI, because they need infrastructure CI
+does not have or because running them there is not worth the cost. These are listed in the
+<code class="docutils literal notranslate"><span class="pre">ignore_list</span></code> in <code class="docutils literal notranslate"><span class="pre">dev/ci/check-suites.py</span></code>, and each one documents in its own scaladoc why it is
+excluded and how to run it. Run a manual suite with:</p>
+<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>./mvnw<span class="w"> </span><span class="nb">test</span><span class="w"> </span>-Dtest<span class="o">=</span>none<span class="w"> </span>-Dsuites<span class="o">=</span><span class="s2">"org.apache.comet.parquet.ParquetReadFromFakeHadoopFsSuite"</span>
+</pre></div>
+</div>
+<p>Only add a suite to that list with a good reason; the default is that a new suite runs in CI.</p>
<p>The macOS suites only run in the merge queue by default. See
<a class="reference internal" href="ci.html"><span class="std std-doc">Continuous Integration</span></a> for the two tiers and the labels that opt a pull request into a
queue-only suite.</p>
diff --git a/searchindex.js b/searchindex.js
index 72e93f186d..70344bb40e 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[56, "id1"]], "%": [[54, "id1"]], "&": [[44, "id1"]], "*": [[54, "id2"]], "+": [[54, "id3"]], "-": [[54, "id4"]], "/": [[54, "id5"]], "1. Format Your Code": [[41, "format-your-code"]], "1. Install Comet": [[62, "install-comet"], [72, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[41, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[62, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[56, "id1"]], "%": [[54, "id1"]], "&": [[44, "id1"]], "*": [[54, "id2"]], "+": [[54, "id3"]], "-": [[54, "id4"]], "/": [[54, "id5"]], "1. Format Your Code": [[41, "format-your-code"]], "1. Install Comet": [[62, "install-comet"], [72, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[41, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[62, "clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/datasources.html b/user-guide/latest/datasources.html
index 5da48be308..fcaf236c5a 100644
--- a/user-guide/latest/datasources.html
+++ b/user-guide/latest/datasources.html
@@ -815,6 +815,19 @@ converted into Arrow format, allowing the Comet pipeline to take over after that
<section id="hdfs">
<h3>HDFS<a class="headerlink" href="#hdfs" title="Link to this heading">#</a></h3>
<p>The Apache DataFusion Comet Rust-based reader seamlessly scans files from remote HDFS for <a class="reference internal" href="#supported-spark-data-sources">supported formats</a></p>
+<div class="admonition warning">
+<p class="admonition-title">Warning</p>
+<p>HDFS support is experimental and is not covered by continuous integration. Comet reads HDFS through
+<code class="docutils literal notranslate"><span class="pre">libhdfs</span></code>, which registers a thread-local destructor that detaches the calling thread from the JVM
+regardless of which component attached it
+(<a class="reference external" href="https://issues.apache.org/jira/browse/HDFS-16021">HDFS-16021</a>, still open upstream). Comet
+attaches its own worker threads, so a worker that has read from HDFS can crash the JVM with a
+<code class="docutils literal notranslate"><span class="pre">SIGSEGV</span></code> when it later exits
+(<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/5023">#5023</a>). The crash surfaces well after
+the HDFS read itself, typically while an unrelated query is running.</p>
+</div>
+<p>Native Iceberg scans do not support HDFS-backed tables; those scans fall back to Spark. See the
+<a class="reference internal" href="iceberg.html"><span class="std std-doc">Comet and Iceberg Guide</span></a>.</p>
</section>
<section id="building-comet-with-hdfs-support">
<h3>Building Comet with HDFS support<a class="headerlink" href="#building-comet-with-hdfs-support" title="Link to this heading">#</a></h3>
@@ -908,6 +921,12 @@ Input<span class="w"> </span><span class="o">[</span><span class="m">3</span><sp
</pre></div>
</div>
<p>Or use <code class="docutils literal notranslate"><span class="pre">spark-shell</span></code> with HDFS support as described <a class="reference internal" href="#building-comet-with-hdfs-support">above</a></p>
+<p>Comet also has a test suite that exercises a native scan through <code class="docutils literal notranslate"><span class="pre">libhdfs</span></code> against a fake Hadoop
+filesystem, so it needs no cluster. Because of the crash described above it is excluded from CI and
+run by hand:</p>
+<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>./mvnw<span class="w"> </span><span class="nb">test</span><span class="w"> </span>-Dtest<span class="o">=</span>none<span class="w"> </span>-Dsuites<span class="o">=</span><span class="s2">"org.apache.comet.parquet.ParquetReadFromFakeHadoopFsSuite"</span>
+</pre></div>
+</div>
</section>
</section>
<section id="s3">
diff --git a/user-guide/latest/iceberg.html b/user-guide/latest/iceberg.html
index 50dba1e9b3..5b42aaff50 100644
--- a/user-guide/latest/iceberg.html
+++ b/user-guide/latest/iceberg.html
@@ -861,9 +861,10 @@ Spark (no AES-192-GCM in the underlying crypto).</p></li>
<p><strong>Storage:</strong></p>
<ul class="simple">
<li><p>Local filesystem</p></li>
-<li><p>Hadoop Distributed File System (HDFS)</p></li>
<li><p>S3-compatible storage (AWS S3, MinIO)</p></li>
+<li><p>Google Cloud Storage (<code class="docutils literal notranslate"><span class="pre">gs</span></code>) and Alibaba Cloud OSS (<code class="docutils literal notranslate"><span class="pre">oss</span></code>)</p></li>
</ul>
+<p>HDFS-backed tables are not supported by the native Iceberg reader and fall back to Spark.</p>
</section>
<section id="rest-catalog">
<h3>REST Catalog<a class="headerlink" href="#rest-catalog" title="Link to this heading">#</a></h3>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org