You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2025/05/21 11:23:57 UTC
(datafusion-comet) branch asf-site updated: Publish built docs triggered by 8f14c72624b678ec9c7843877fe63ccd5ac80183
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new e9d970347 Publish built docs triggered by 8f14c72624b678ec9c7843877fe63ccd5ac80183
e9d970347 is described below
commit e9d9703474f78688a13ec46e168727ca49b45e35
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Wed May 21 11:23:54 2025 +0000
Publish built docs triggered by 8f14c72624b678ec9c7843877fe63ccd5ac80183
---
_images/img.png | Bin 0 -> 194412 bytes
_sources/contributor-guide/spark-sql-tests.md.txt | 10 +++++++-
contributor-guide/spark-sql-tests.html | 27 ++++++++++++++++++----
searchindex.js | 2 +-
4 files changed, 33 insertions(+), 6 deletions(-)
diff --git a/_images/img.png b/_images/img.png
new file mode 100644
index 000000000..e9d460eed
Binary files /dev/null and b/_images/img.png differ
diff --git a/_sources/contributor-guide/spark-sql-tests.md.txt b/_sources/contributor-guide/spark-sql-tests.md.txt
index 56e1262ad..2c74f36f1 100644
--- a/_sources/contributor-guide/spark-sql-tests.md.txt
+++ b/_sources/contributor-guide/spark-sql-tests.md.txt
@@ -65,7 +65,7 @@ ENABLE_COMET=true build/sbt "hive/testOnly * -- -l org.apache.spark.tags.Extende
ENABLE_COMET=true build/sbt "hive/testOnly * -- -n org.apache.spark.tags.ExtendedHiveTest"
ENABLE_COMET=true build/sbt "hive/testOnly * -- -n org.apache.spark.tags.SlowHiveTest"
```
-#### Steps to run individual test suites
+#### Steps to run individual test suites through SBT
1. Open SBT with Comet enabled
```sbt
ENABLE_COMET=true sbt -Dspark.test.includeSlowTests=true
@@ -74,6 +74,14 @@ ENABLE_COMET=true sbt -Dspark.test.includeSlowTests=true
```sbt
sql/testOnly org.apache.spark.sql.DynamicPartitionPruningV1SuiteAEOn -- -z "SPARK-35568"
```
+#### Steps to run individual test suites in IntelliJ IDE
+1. Add below configuration in VM Options for your test case (apache-spark repository)
+```sbt
+-Dspark.comet.enabled=true -Dspark.comet.debug.enabled=true -Dspark.plugins=org.apache.spark.CometPlugin -DXmx4096m -Dspark.executor.heartbeatInterval=20000 -Dspark.network.timeout=10000 --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED
+```
+2. Set `ENABLE_COMET=true` in environment variables
+
+3. After the above tests are configured, spark tests can be run with debugging enabled on spark/comet code. Note that Comet is added as a dependency and the classes are readonly while debugging from Spark. Any new changes to Comet are to be built and deployed locally through the command (`PROFILES="-Pspark-3.4" make release`)
## Creating a diff file for a new Spark version
diff --git a/contributor-guide/spark-sql-tests.html b/contributor-guide/spark-sql-tests.html
index 3c9f8bcb5..6005cdfc0 100644
--- a/contributor-guide/spark-sql-tests.html
+++ b/contributor-guide/spark-sql-tests.html
@@ -311,8 +311,13 @@ under the License.
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
- <a class="reference internal nav-link" href="#steps-to-run-individual-test-suites">
- Steps to run individual test suites
+ <a class="reference internal nav-link" href="#steps-to-run-individual-test-suites-through-sbt">
+ Steps to run individual test suites through SBT
+ </a>
+ </li>
+ <li class="toc-h3 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#steps-to-run-individual-test-suites-in-intellij-ide">
+ Steps to run individual test suites in IntelliJ IDE
</a>
</li>
</ul>
@@ -422,8 +427,8 @@ git<span class="w"> </span>apply<span class="w"> </span>../datafusion-comet/dev/
</pre></div>
</div>
</section>
-<section id="steps-to-run-individual-test-suites">
-<h3>Steps to run individual test suites<a class="headerlink" href="#steps-to-run-individual-test-suites" title="Link to this heading">¶</a></h3>
+<section id="steps-to-run-individual-test-suites-through-sbt">
+<h3>Steps to run individual test suites through SBT<a class="headerlink" href="#steps-to-run-individual-test-suites-through-sbt" title="Link to this heading">¶</a></h3>
<ol class="arabic simple">
<li><p>Open SBT with Comet enabled</p></li>
</ol>
@@ -437,6 +442,20 @@ git<span class="w"> </span>apply<span class="w"> </span>../datafusion-comet/dev/
</pre></div>
</div>
</section>
+<section id="steps-to-run-individual-test-suites-in-intellij-ide">
+<h3>Steps to run individual test suites in IntelliJ IDE<a class="headerlink" href="#steps-to-run-individual-test-suites-in-intellij-ide" title="Link to this heading">¶</a></h3>
+<ol class="arabic simple">
+<li><p>Add below configuration in VM Options for your test case (apache-spark repository)</p></li>
+</ol>
+<div class="highlight-sbt notranslate"><div class="highlight"><pre><span></span>-Dspark.comet.enabled=true -Dspark.comet.debug.enabled=true -Dspark.plugins=org.apache.spark.CometPlugin -DXmx4096m -Dspark.executor.heartbeatInterval=20000 -Dspark.network.timeout=10000 --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED
+</pre></div>
+</div>
+<ol class="arabic simple" start="2">
+<li><p>Set <code class="docutils literal notranslate"><span class="pre">ENABLE_COMET=true</span></code> in environment variables
+<img alt="img.png" src="../_images/img.png" /></p></li>
+<li><p>After the above tests are configured, spark tests can be run with debugging enabled on spark/comet code. Note that Comet is added as a dependency and the classes are readonly while debugging from Spark. Any new changes to Comet are to be built and deployed locally through the command (<code class="docutils literal notranslate"><span class="pre">PROFILES="-Pspark-3.4"</span> <span class="pre">make</span> <span class="pre">release</span></code>)</p></li>
+</ol>
+</section>
</section>
<section id="creating-a-diff-file-for-a-new-spark-version">
<h2>Creating a diff file for a new Spark version<a class="headerlink" href="#creating-a-diff-file-for-a-new-spark-version" title="Link to this heading">¶</a></h2>
diff --git a/searchindex.js b/searchindex.js
index 05253655c..ea98c5371 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"1. Install Comet": [[11, "install-comet"]], "2. Clone Spark and Apply Diff": [[11, "clone-spark-and-apply-diff"]], "3. Run Spark SQL Tests": [[11, "run-spark-sql-tests"]], "ANSI mode": [[14, "ansi-mode"]], "API Differences Between Spark Versions": [[0, "api-differences-between-spark-versions"]], "ASF Links": [[13, null]], "Accelerating Apache Iceberg Parquet Scans using Comet (Experimental)": [[19, null]], "Adding Spark-side Tests for the New Expression": [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"1. Install Comet": [[11, "install-comet"]], "2. Clone Spark and Apply Diff": [[11, "clone-spark-and-apply-diff"]], "3. Run Spark SQL Tests": [[11, "run-spark-sql-tests"]], "ANSI mode": [[14, "ansi-mode"]], "API Differences Between Spark Versions": [[0, "api-differences-between-spark-versions"]], "ASF Links": [[13, null]], "Accelerating Apache Iceberg Parquet Scans using Comet (Experimental)": [[19, null]], "Adding Spark-side Tests for the New Expression": [...]
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org