You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/07/06 20:39:12 UTC
(datafusion-comet) branch asf-site updated: Publish built docs triggered by b49f6c671a4889fc837425b90df4bbaa3ce7d27c
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new b043a63082 Publish built docs triggered by b49f6c671a4889fc837425b90df4bbaa3ce7d27c
b043a63082 is described below
commit b043a63082882bb7bc3e916c94bbb07b86b0d327
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Mon Jul 6 20:39:07 2026 +0000
Publish built docs triggered by b49f6c671a4889fc837425b90df4bbaa3ce7d27c
---
_sources/contributor-guide/development.md.txt | 12 ++++++++----
contributor-guide/development.html | 12 ++++++++----
searchindex.js | 2 +-
3 files changed, 17 insertions(+), 9 deletions(-)
diff --git a/_sources/contributor-guide/development.md.txt b/_sources/contributor-guide/development.md.txt
index 3785358cf3..7e5c20c7ec 100644
--- a/_sources/contributor-guide/development.md.txt
+++ b/_sources/contributor-guide/development.md.txt
@@ -57,9 +57,11 @@ method on any worker thread, and may move it between threads across polls. Any s
in the operator struct or be shared via `Arc`.
**JNI calls work from any thread, but have overhead.** `JVMClasses::get_env()` calls
-`AttachCurrentThread`, which acquires JVM internal locks. The `AttachGuard` detaches the thread
-when dropped. Repeated attach/detach cycles on tokio workers add overhead, so avoid calling
-into the JVM on hot paths during stream execution.
+`AttachCurrentThread`, which acquires JVM internal locks. The attachment is cached in
+thread-local storage and is only released when the worker thread itself exits, not when the
+`AttachGuard` is dropped. This is why the tokio runtime has to be shut down (releasing its
+worker threads) for the JVM to be able to exit. Acquiring the JVM locks on each `get_env()`
+call adds overhead, so avoid calling into the JVM on hot paths during stream execution.
**Do not call `TaskContext.get()` from JVM callbacks during execution.** Spark's `TaskContext` is
a `ThreadLocal` on the executor task thread. JVM methods invoked from tokio worker threads will
@@ -84,7 +86,9 @@ to unwrap decryption keys during Parquet reads. It uses a stored `GlobalRef` and
### The tokio runtime
-The runtime is created once per executor JVM in a `Lazy<Runtime>` static:
+The runtime is stored in a `Mutex<Option<Runtime>>` static and created lazily on first use. It
+is torn down on plugin shutdown (via `release_runtime`) so that the tokio worker threads exit
+and the JVM can shut down cleanly:
- **Worker threads:** `num_cpus` by default, configurable via `COMET_WORKER_THREADS`
- **Max blocking threads:** 512 by default, configurable via `COMET_MAX_BLOCKING_THREADS`
diff --git a/contributor-guide/development.html b/contributor-guide/development.html
index 1daf31de0c..38f32eb339 100644
--- a/contributor-guide/development.html
+++ b/contributor-guide/development.html
@@ -577,9 +577,11 @@ task thread. All Spark tasks on an executor share one tokio runtime.</p>
method on any worker thread, and may move it between threads across polls. Any state must live
in the operator struct or be shared via <code class="docutils literal notranslate"><span class="pre">Arc</span></code>.</p>
<p><strong>JNI calls work from any thread, but have overhead.</strong> <code class="docutils literal notranslate"><span class="pre">JVMClasses::get_env()</span></code> calls
-<code class="docutils literal notranslate"><span class="pre">AttachCurrentThread</span></code>, which acquires JVM internal locks. The <code class="docutils literal notranslate"><span class="pre">AttachGuard</span></code> detaches the thread
-when dropped. Repeated attach/detach cycles on tokio workers add overhead, so avoid calling
-into the JVM on hot paths during stream execution.</p>
+<code class="docutils literal notranslate"><span class="pre">AttachCurrentThread</span></code>, which acquires JVM internal locks. The attachment is cached in
+thread-local storage and is only released when the worker thread itself exits, not when the
+<code class="docutils literal notranslate"><span class="pre">AttachGuard</span></code> is dropped. This is why the tokio runtime has to be shut down (releasing its
+worker threads) for the JVM to be able to exit. Acquiring the JVM locks on each <code class="docutils literal notranslate"><span class="pre">get_env()</span></code>
+call adds overhead, so avoid calling into the JVM on hot paths during stream execution.</p>
<p><strong>Do not call <code class="docutils literal notranslate"><span class="pre">TaskContext.get()</span></code> from JVM callbacks during execution.</strong> Spark’s <code class="docutils literal notranslate"><span class="pre">TaskContext</span></code> is
a <code class="docutils literal notranslate"><span class="pre">ThreadLocal</span></code> on the executor task thread. JVM methods invoked from tokio worker threads will
see <code class="docutils literal notranslate"><span class="pre">null</span></code>. If you need task metadata, capture it at construction time (in <code class="docutils literal notranslate"><span class="pre">createPlan</span></code> or
@@ -600,7 +602,9 @@ to unwrap decryption keys during Parquet reads. It uses a stored <code class="do
</section>
<section id="the-tokio-runtime">
<h3>The tokio runtime<a class="headerlink" href="#the-tokio-runtime" title="Link to this heading">#</a></h3>
-<p>The runtime is created once per executor JVM in a <code class="docutils literal notranslate"><span class="pre">Lazy<Runtime></span></code> static:</p>
+<p>The runtime is stored in a <code class="docutils literal notranslate"><span class="pre">Mutex<Option<Runtime>></span></code> static and created lazily on first use. It
+is torn down on plugin shutdown (via <code class="docutils literal notranslate"><span class="pre">release_runtime</span></code>) so that the tokio worker threads exit
+and the JVM can shut down cleanly:</p>
<ul class="simple">
<li><p><strong>Worker threads:</strong> <code class="docutils literal notranslate"><span class="pre">num_cpus</span></code> by default, configurable via <code class="docutils literal notranslate"><span class="pre">COMET_WORKER_THREADS</span></code></p></li>
<li><p><strong>Max blocking threads:</strong> 512 by default, configurable via <code class="docutils literal notranslate"><span class="pre">COMET_MAX_BLOCKING_THREADS</span></code></p></li>
diff --git a/searchindex.js b/searchindex.js
index 2daed86979..6a52d56e0c 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[50, "id1"]], "%": [[48, "id1"]], "&": [[38, "id1"]], "*": [[48, "id2"]], "+": [[48, "id3"]], "-": [[48, "id4"]], "/": [[48, "id5"]], "1. Format Your Code": [[35, "format-your-code"]], "1. Install Comet": [[56, "install-comet"], [65, "install-comet"]], "1. Native Operators (nativeExecs map)": [[24, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[35, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[56, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[50, "id1"]], "%": [[48, "id1"]], "&": [[38, "id1"]], "*": [[48, "id2"]], "+": [[48, "id3"]], "-": [[48, "id4"]], "/": [[48, "id5"]], "1. Format Your Code": [[35, "format-your-code"]], "1. Install Comet": [[56, "install-comet"], [65, "install-comet"]], "1. Native Operators (nativeExecs map)": [[24, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[35, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[56, "clone-iceberg-and-apply- [...]
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org