You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/05/05 15:44:08 UTC
(datafusion-comet) branch asf-site updated: Publish built docs triggered by 57181c19718ef7f64013ee0ccf73baebda75fead
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new f1d68ea9d Publish built docs triggered by 57181c19718ef7f64013ee0ccf73baebda75fead
f1d68ea9d is described below
commit f1d68ea9d50a694bf3110274e9f8691194f3e3c3
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Tue May 5 15:44:03 2026 +0000
Publish built docs triggered by 57181c19718ef7f64013ee0ccf73baebda75fead
---
_sources/contributor-guide/development.md.txt | 72 +++++++++++++++++++++++++++
contributor-guide/development.html | 63 +++++++++++++++++++++++
searchindex.js | 2 +-
3 files changed, 136 insertions(+), 1 deletion(-)
diff --git a/_sources/contributor-guide/development.md.txt b/_sources/contributor-guide/development.md.txt
index e1fbfee06..edca94ac4 100644
--- a/_sources/contributor-guide/development.md.txt
+++ b/_sources/contributor-guide/development.md.txt
@@ -276,6 +276,78 @@ Comet uses generated source files that are too large for IntelliJ's default size
by going to `Help -> Edit Custom Properties...`. For example, adding `idea.max.intellisense.filesize=16384` increases the file
size limit to 16 MB.
+#### Working with Spark 4.x profiles in IntelliJ IDEA
+
+Spark 4.x requires JDK 17 and Scala 2.13. When switching an existing IntelliJ project from the
+default Spark 3.x setup to a Spark 4.x profile, it is usually best to re-import the Maven project
+from a clean IntelliJ configuration:
+
+1. Close the IntelliJ project.
+2. Delete the `.idea` directory from the repository root.
+3. Make sure `protoc` is available. On macOS, install it with `brew install protobuf`; on Linux,
+ install `protobuf-compiler` or the equivalent package for your distribution.
+4. Make sure `JAVA_HOME` points to JDK 17. On macOS, you can select it with:
+
+ ```sh
+ export JAVA_HOME=$(/usr/libexec/java_home -v 17)
+ export PATH="$JAVA_HOME/bin:$PATH"
+ ```
+
+ On Apple Silicon, the selected JDK architecture must match the native Rust target. Verify that
+ `file "$JAVA_HOME/lib/server/libjvm.dylib"` reports `arm64`.
+
+5. Build the profile once from the command line so generated sources and native artifacts are in
+ place:
+
+ ```sh
+ PROFILES="-Pspark-4.0" make release
+ ```
+
+ The `spark-4.0` profile sets Scala 2.13 and Java 17 properties. If you need to be explicit, use
+ `PROFILES="-Pspark-4.0 -Pscala-2.13 -Pjdk17" make release`.
+
+ The Maven profile is named `jdk17` in this project.
+
+ If the native build previously used a different JDK, clear Cargo's cached JNI link path before
+ rebuilding:
+
+ ```sh
+ cd native && cargo clean
+ ```
+
+6. Open the repository root as a new project in IntelliJ IDEA.
+7. In the Maven tool window, enable the `spark-4.0` and `jdk17` profiles. Disable conflicting Spark
+ version profiles such as `spark-3.4` or `spark-3.5` if IntelliJ selected them.
+8. Re-import or reload the Maven project.
+9. Build the project in IntelliJ.
+
+When running tests from IntelliJ on JDK 17, add the same VM options that Maven uses through the
+root `pom.xml`'s `extraJavaTestArgs` property:
+
+```text
+-XX:+IgnoreUnrecognizedVMOptions
+--add-opens=java.base/java.lang=ALL-UNNAMED
+--add-opens=java.base/java.lang.invoke=ALL-UNNAMED
+--add-opens=java.base/java.lang.reflect=ALL-UNNAMED
+--add-opens=java.base/java.io=ALL-UNNAMED
+--add-opens=java.base/java.net=ALL-UNNAMED
+--add-opens=java.base/java.nio=ALL-UNNAMED
+--add-opens=java.base/java.util=ALL-UNNAMED
+--add-opens=java.base/java.util.concurrent=ALL-UNNAMED
+--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
+--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED
+--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
+--add-opens=java.base/sun.nio.cs=ALL-UNNAMED
+--add-opens=java.base/sun.security.action=ALL-UNNAMED
+--add-opens=java.base/sun.util.calendar=ALL-UNNAMED
+-Djdk.reflect.useDirectMethodHandle=false
+```
+
+If IntelliJ still resolves Spark 3.x dependencies or reports shim source errors after switching
+profiles, repeat the clean import steps above. Maven profile state is cached in the IntelliJ
+project files, and stale profile selections are a common cause of mixed Spark 3.x and Spark 4.x
+source roots.
+
### CLion
First make sure to install the Rust plugin in CLion or you can use the dedicated Rust IDE: RustRover.
diff --git a/contributor-guide/development.html b/contributor-guide/development.html
index 921a095cf..42e3cbcee 100644
--- a/contributor-guide/development.html
+++ b/contributor-guide/development.html
@@ -742,6 +742,69 @@ After that, you can open the project in IntelliJ IDEA. The IDE should automatica
<a class="reference external" href="https://intellij-support.jetbrains.com/hc/en-us/articles/206544869-Configuring-JVM-options-and-platform-properties">IntelliJ’s Platform Properties</a>
by going to <code class="docutils literal notranslate"><span class="pre">Help</span> <span class="pre">-></span> <span class="pre">Edit</span> <span class="pre">Custom</span> <span class="pre">Properties...</span></code>. For example, adding <code class="docutils literal notranslate"><span class="pre">idea.max.intellisense.filesize=16384</span></code> increases the file
size limit to 16 MB.</p>
+<section id="working-with-spark-4-x-profiles-in-intellij-idea">
+<h4>Working with Spark 4.x profiles in IntelliJ IDEA<a class="headerlink" href="#working-with-spark-4-x-profiles-in-intellij-idea" title="Link to this heading">#</a></h4>
+<p>Spark 4.x requires JDK 17 and Scala 2.13. When switching an existing IntelliJ project from the
+default Spark 3.x setup to a Spark 4.x profile, it is usually best to re-import the Maven project
+from a clean IntelliJ configuration:</p>
+<ol class="arabic">
+<li><p>Close the IntelliJ project.</p></li>
+<li><p>Delete the <code class="docutils literal notranslate"><span class="pre">.idea</span></code> directory from the repository root.</p></li>
+<li><p>Make sure <code class="docutils literal notranslate"><span class="pre">protoc</span></code> is available. On macOS, install it with <code class="docutils literal notranslate"><span class="pre">brew</span> <span class="pre">install</span> <span class="pre">protobuf</span></code>; on Linux,
+install <code class="docutils literal notranslate"><span class="pre">protobuf-compiler</span></code> or the equivalent package for your distribution.</p></li>
+<li><p>Make sure <code class="docutils literal notranslate"><span class="pre">JAVA_HOME</span></code> points to JDK 17. On macOS, you can select it with:</p>
+<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span><span class="nb">export</span><span class="w"> </span><span class="nv">JAVA_HOME</span><span class="o">=</span><span class="k">$(</span>/usr/libexec/java_home<span class="w"> </span>-v<span class="w"> </span><span class="m">17</span><span class="k">)</span>
+<span class="nb">export</span><span class="w"> </span><span class="nv">PATH</span><span class="o">=</span><span class="s2">"</span><span class="nv">$JAVA_HOME</span><span class="s2">/bin:</span><span class="nv">$PATH</span><span class="s2">"</span>
+</pre></div>
+</div>
+<p>On Apple Silicon, the selected JDK architecture must match the native Rust target. Verify that
+<code class="docutils literal notranslate"><span class="pre">file</span> <span class="pre">"$JAVA_HOME/lib/server/libjvm.dylib"</span></code> reports <code class="docutils literal notranslate"><span class="pre">arm64</span></code>.</p>
+</li>
+<li><p>Build the profile once from the command line so generated sources and native artifacts are in
+place:</p>
+<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span><span class="nv">PROFILES</span><span class="o">=</span><span class="s2">"-Pspark-4.0"</span><span class="w"> </span>make<span class="w"> </span>release
+</pre></div>
+</div>
+<p>The <code class="docutils literal notranslate"><span class="pre">spark-4.0</span></code> profile sets Scala 2.13 and Java 17 properties. If you need to be explicit, use
+<code class="docutils literal notranslate"><span class="pre">PROFILES="-Pspark-4.0</span> <span class="pre">-Pscala-2.13</span> <span class="pre">-Pjdk17"</span> <span class="pre">make</span> <span class="pre">release</span></code>.</p>
+<p>The Maven profile is named <code class="docutils literal notranslate"><span class="pre">jdk17</span></code> in this project.</p>
+<p>If the native build previously used a different JDK, clear Cargo’s cached JNI link path before
+rebuilding:</p>
+<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span><span class="nb">cd</span><span class="w"> </span>native<span class="w"> </span><span class="o">&&</span><span class="w"> </span>cargo<span class="w"> </span>clean
+</pre></div>
+</div>
+</li>
+<li><p>Open the repository root as a new project in IntelliJ IDEA.</p></li>
+<li><p>In the Maven tool window, enable the <code class="docutils literal notranslate"><span class="pre">spark-4.0</span></code> and <code class="docutils literal notranslate"><span class="pre">jdk17</span></code> profiles. Disable conflicting Spark
+version profiles such as <code class="docutils literal notranslate"><span class="pre">spark-3.4</span></code> or <code class="docutils literal notranslate"><span class="pre">spark-3.5</span></code> if IntelliJ selected them.</p></li>
+<li><p>Re-import or reload the Maven project.</p></li>
+<li><p>Build the project in IntelliJ.</p></li>
+</ol>
+<p>When running tests from IntelliJ on JDK 17, add the same VM options that Maven uses through the
+root <code class="docutils literal notranslate"><span class="pre">pom.xml</span></code>’s <code class="docutils literal notranslate"><span class="pre">extraJavaTestArgs</span></code> property:</p>
+<div class="highlight-text notranslate"><div class="highlight"><pre><span></span>-XX:+IgnoreUnrecognizedVMOptions
+--add-opens=java.base/java.lang=ALL-UNNAMED
+--add-opens=java.base/java.lang.invoke=ALL-UNNAMED
+--add-opens=java.base/java.lang.reflect=ALL-UNNAMED
+--add-opens=java.base/java.io=ALL-UNNAMED
+--add-opens=java.base/java.net=ALL-UNNAMED
+--add-opens=java.base/java.nio=ALL-UNNAMED
+--add-opens=java.base/java.util=ALL-UNNAMED
+--add-opens=java.base/java.util.concurrent=ALL-UNNAMED
+--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
+--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED
+--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
+--add-opens=java.base/sun.nio.cs=ALL-UNNAMED
+--add-opens=java.base/sun.security.action=ALL-UNNAMED
+--add-opens=java.base/sun.util.calendar=ALL-UNNAMED
+-Djdk.reflect.useDirectMethodHandle=false
+</pre></div>
+</div>
+<p>If IntelliJ still resolves Spark 3.x dependencies or reports shim source errors after switching
+profiles, repeat the clean import steps above. Maven profile state is cached in the IntelliJ
+project files, and stale profile selections are a common cause of mixed Spark 3.x and Spark 4.x
+source roots.</p>
+</section>
</section>
<section id="clion">
<h3>CLion<a class="headerlink" href="#clion" title="Link to this heading">#</a></h3>
diff --git a/searchindex.js b/searchindex.js
index 687282f2f..fcdbda511 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"1. Format Your Code": [[14, "format-your-code"]], "1. Install Comet": [[16, "install-comet"], [24, "install-comet"]], "1. Native Operators (nativeExecs map)": [[4, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[14, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[16, "clone-iceberg-and-apply-diff"]], "2. Clone Spark and Apply Diff": [[24, "clone-spark-and-apply-diff"]], "2. Sink Operators (sinks map)": [[4, "sink-operators-sinks-m [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"1. Format Your Code": [[14, "format-your-code"]], "1. Install Comet": [[16, "install-comet"], [24, "install-comet"]], "1. Native Operators (nativeExecs map)": [[4, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[14, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[16, "clone-iceberg-and-apply-diff"]], "2. Clone Spark and Apply Diff": [[24, "clone-spark-and-apply-diff"]], "2. Sink Operators (sinks map)": [[4, "sink-operators-sinks-m [...]
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org