You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/09/13 19:20:55 UTC

(datafusion-comet) branch asf-site updated: Publish built docs triggered by 451c99963206fa6bf0387239aa12887a16255516

This is an automated email from the ASF dual-hosted git repository.

github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git


The following commit(s) were added to refs/heads/asf-site by this push:
     new 2cf440f875 Publish built docs triggered by 451c99963206fa6bf0387239aa12887a16255516
2cf440f875 is described below

commit 2cf440f875182d2d339b0b9398972caf214ae3f3
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Sun Sep 13 19:20:51 2026 +0000

    Publish built docs triggered by 451c99963206fa6bf0387239aa12887a16255516
---
 _sources/contributor-guide/development.md.txt | 11 +++++++++++
 _sources/user-guide/latest/datasources.md.txt | 22 ++++++++++++++++++++++
 _sources/user-guide/latest/iceberg.md.txt     |  4 +++-
 contributor-guide/development.html            |  8 ++++++++
 searchindex.js                                |  2 +-
 user-guide/latest/datasources.html            | 19 +++++++++++++++++++
 user-guide/latest/iceberg.html                |  3 ++-
 7 files changed, 66 insertions(+), 3 deletions(-)

diff --git a/_sources/contributor-guide/development.md.txt b/_sources/contributor-guide/development.md.txt
index b04a3fdd42..3025e259e8 100644
--- a/_sources/contributor-guide/development.md.txt
+++ b/_sources/contributor-guide/development.md.txt
@@ -658,6 +658,17 @@ Choose the group that best matches the area your test covers:
 `*Suite.scala` files in the repository and verifies that each one appears in both workflow files.
 If any suite is missing, this check will fail and block the PR.
 
+A small number of suites are deliberately **not** run in CI, because they need infrastructure CI
+does not have or because running them there is not worth the cost. These are listed in the
+`ignore_list` in `dev/ci/check-suites.py`, and each one documents in its own scaladoc why it is
+excluded and how to run it. Run a manual suite with:
+
+```sh
+./mvnw test -Dtest=none -Dsuites="org.apache.comet.parquet.ParquetReadFromFakeHadoopFsSuite"
+```
+
+Only add a suite to that list with a good reason; the default is that a new suite runs in CI.
+
 The macOS suites only run in the merge queue by default. See
 [Continuous Integration](ci.md) for the two tiers and the labels that opt a pull request into a
 queue-only suite.
diff --git a/_sources/user-guide/latest/datasources.md.txt b/_sources/user-guide/latest/datasources.md.txt
index 84994c00c2..2f368a6b06 100644
--- a/_sources/user-guide/latest/datasources.md.txt
+++ b/_sources/user-guide/latest/datasources.md.txt
@@ -61,6 +61,20 @@ Comet supports most standard storage systems, such as local file system and obje
 
 The Apache DataFusion Comet Rust-based reader seamlessly scans files from remote HDFS for [supported formats](#supported-spark-data-sources)
 
+```{warning}
+HDFS support is experimental and is not covered by continuous integration. Comet reads HDFS through
+`libhdfs`, which registers a thread-local destructor that detaches the calling thread from the JVM
+regardless of which component attached it
+([HDFS-16021](https://issues.apache.org/jira/browse/HDFS-16021), still open upstream). Comet
+attaches its own worker threads, so a worker that has read from HDFS can crash the JVM with a
+`SIGSEGV` when it later exits
+([#5023](https://github.com/apache/datafusion-comet/issues/5023)). The crash surfaces well after
+the HDFS read itself, typically while an unrelated query is running.
+```
+
+Native Iceberg scans do not support HDFS-backed tables; those scans fall back to Spark. See the
+[Comet and Iceberg Guide](iceberg.md).
+
 ### Building Comet with HDFS support
 
 To build Comet with remote HDFS support it is required to have a JDK installed.
@@ -165,6 +179,14 @@ JAVA_HOME="/opt/homebrew/opt/openjdk@17" make release PROFILES="-Pspark-4.1" RUS
 
 Or use `spark-shell` with HDFS support as described [above](#building-comet-with-hdfs-support)
 
+Comet also has a test suite that exercises a native scan through `libhdfs` against a fake Hadoop
+filesystem, so it needs no cluster. Because of the crash described above it is excluded from CI and
+run by hand:
+
+```shell
+./mvnw test -Dtest=none -Dsuites="org.apache.comet.parquet.ParquetReadFromFakeHadoopFsSuite"
+```
+
 ## S3
 
 Comet's Parquet scan completely offloads data loading to Rust. It uses the
diff --git a/_sources/user-guide/latest/iceberg.md.txt b/_sources/user-guide/latest/iceberg.md.txt
index 9d2694544b..96e19ca11b 100644
--- a/_sources/user-guide/latest/iceberg.md.txt
+++ b/_sources/user-guide/latest/iceberg.md.txt
@@ -110,8 +110,10 @@ The native Iceberg reader supports the following features:
 **Storage:**
 
 - Local filesystem
-- Hadoop Distributed File System (HDFS)
 - S3-compatible storage (AWS S3, MinIO)
+- Google Cloud Storage (`gs`) and Alibaba Cloud OSS (`oss`)
+
+HDFS-backed tables are not supported by the native Iceberg reader and fall back to Spark.
 
 ### REST Catalog
 
diff --git a/contributor-guide/development.html b/contributor-guide/development.html
index 0a33c68fc7..4f6be74fe2 100644
--- a/contributor-guide/development.html
+++ b/contributor-guide/development.html
@@ -1183,6 +1183,14 @@ appropriate group. For example, if you add a new expression test suite, add it t
 <code class="docutils literal notranslate"><span class="pre">.github/workflows/ci.yml</span></code> runs <code class="docutils literal notranslate"><span class="pre">dev/ci/check-suites.py</span></code> on every pull request. It scans for all
 <code class="docutils literal notranslate"><span class="pre">*Suite.scala</span></code> files in the repository and verifies that each one appears in both workflow files.
 If any suite is missing, this check will fail and block the PR.</p>
+<p>A small number of suites are deliberately <strong>not</strong> run in CI, because they need infrastructure CI
+does not have or because running them there is not worth the cost. These are listed in the
+<code class="docutils literal notranslate"><span class="pre">ignore_list</span></code> in <code class="docutils literal notranslate"><span class="pre">dev/ci/check-suites.py</span></code>, and each one documents in its own scaladoc why it is
+excluded and how to run it. Run a manual suite with:</p>
+<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>./mvnw<span class="w"> </span><span class="nb">test</span><span class="w"> </span>-Dtest<span class="o">=</span>none<span class="w"> </span>-Dsuites<span class="o">=</span><span class="s2">"org.apache.comet.parquet.ParquetReadFromFakeHadoopFsSuite"</span>
+</pre></div>
+</div>
+<p>Only add a suite to that list with a good reason; the default is that a new suite runs in CI.</p>
 <p>The macOS suites only run in the merge queue by default. See
 <a class="reference internal" href="ci.html"><span class="std std-doc">Continuous Integration</span></a> for the two tiers and the labels that opt a pull request into a
 queue-only suite.</p>
diff --git a/searchindex.js b/searchindex.js
index 72e93f186d..70344bb40e 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[56, "id1"]], "%": [[54, "id1"]], "&": [[44, "id1"]], "*": [[54, "id2"]], "+": [[54, "id3"]], "-": [[54, "id4"]], "/": [[54, "id5"]], "1. Format Your Code": [[41, "format-your-code"]], "1. Install Comet": [[62, "install-comet"], [72, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[41, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[62, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[56, "id1"]], "%": [[54, "id1"]], "&": [[44, "id1"]], "*": [[54, "id2"]], "+": [[54, "id3"]], "-": [[54, "id4"]], "/": [[54, "id5"]], "1. Format Your Code": [[41, "format-your-code"]], "1. Install Comet": [[62, "install-comet"], [72, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[41, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[62, "clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/datasources.html b/user-guide/latest/datasources.html
index 5da48be308..fcaf236c5a 100644
--- a/user-guide/latest/datasources.html
+++ b/user-guide/latest/datasources.html
@@ -815,6 +815,19 @@ converted into Arrow format, allowing the Comet pipeline to take over after that
 <section id="hdfs">
 <h3>HDFS<a class="headerlink" href="#hdfs" title="Link to this heading">#</a></h3>
 <p>The Apache DataFusion Comet Rust-based reader seamlessly scans files from remote HDFS for <a class="reference internal" href="#supported-spark-data-sources">supported formats</a></p>
+<div class="admonition warning">
+<p class="admonition-title">Warning</p>
+<p>HDFS support is experimental and is not covered by continuous integration. Comet reads HDFS through
+<code class="docutils literal notranslate"><span class="pre">libhdfs</span></code>, which registers a thread-local destructor that detaches the calling thread from the JVM
+regardless of which component attached it
+(<a class="reference external" href="https://issues.apache.org/jira/browse/HDFS-16021">HDFS-16021</a>, still open upstream). Comet
+attaches its own worker threads, so a worker that has read from HDFS can crash the JVM with a
+<code class="docutils literal notranslate"><span class="pre">SIGSEGV</span></code> when it later exits
+(<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/5023">#5023</a>). The crash surfaces well after
+the HDFS read itself, typically while an unrelated query is running.</p>
+</div>
+<p>Native Iceberg scans do not support HDFS-backed tables; those scans fall back to Spark. See the
+<a class="reference internal" href="iceberg.html"><span class="std std-doc">Comet and Iceberg Guide</span></a>.</p>
 </section>
 <section id="building-comet-with-hdfs-support">
 <h3>Building Comet with HDFS support<a class="headerlink" href="#building-comet-with-hdfs-support" title="Link to this heading">#</a></h3>
@@ -908,6 +921,12 @@ Input<span class="w"> </span><span class="o">[</span><span class="m">3</span><sp
 </pre></div>
 </div>
 <p>Or use <code class="docutils literal notranslate"><span class="pre">spark-shell</span></code> with HDFS support as described <a class="reference internal" href="#building-comet-with-hdfs-support">above</a></p>
+<p>Comet also has a test suite that exercises a native scan through <code class="docutils literal notranslate"><span class="pre">libhdfs</span></code> against a fake Hadoop
+filesystem, so it needs no cluster. Because of the crash described above it is excluded from CI and
+run by hand:</p>
+<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>./mvnw<span class="w"> </span><span class="nb">test</span><span class="w"> </span>-Dtest<span class="o">=</span>none<span class="w"> </span>-Dsuites<span class="o">=</span><span class="s2">"org.apache.comet.parquet.ParquetReadFromFakeHadoopFsSuite"</span>
+</pre></div>
+</div>
 </section>
 </section>
 <section id="s3">
diff --git a/user-guide/latest/iceberg.html b/user-guide/latest/iceberg.html
index 50dba1e9b3..5b42aaff50 100644
--- a/user-guide/latest/iceberg.html
+++ b/user-guide/latest/iceberg.html
@@ -861,9 +861,10 @@ Spark (no AES-192-GCM in the underlying crypto).</p></li>
 <p><strong>Storage:</strong></p>
 <ul class="simple">
 <li><p>Local filesystem</p></li>
-<li><p>Hadoop Distributed File System (HDFS)</p></li>
 <li><p>S3-compatible storage (AWS S3, MinIO)</p></li>
+<li><p>Google Cloud Storage (<code class="docutils literal notranslate"><span class="pre">gs</span></code>) and Alibaba Cloud OSS (<code class="docutils literal notranslate"><span class="pre">oss</span></code>)</p></li>
 </ul>
+<p>HDFS-backed tables are not supported by the native Iceberg reader and fall back to Spark.</p>
 </section>
 <section id="rest-catalog">
 <h3>REST Catalog<a class="headerlink" href="#rest-catalog" title="Link to this heading">#</a></h3>


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org