You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2025/09/17 15:23:20 UTC
[PR] feat: Add plan conversion statistics to extended explain info [datafusion-comet]
andygrove opened a new pull request, #2412:
URL: https://github.com/apache/datafusion-comet/pull/2412
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
N/A
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
I would like to make it easier to see what % of operators are accelerated by Comet.
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
Add some basic statistics to the "extended explain" output to show % of operators accelerated by Comet.
Example output:
```
BroadcastHashJoin
:- ColumnarToRow
: +- Scan parquet [COMET: Dynamic Partition Pruning is not supported]
: +- SubqueryBroadcast
: +- ReusedExchange
+- BroadcastExchange
+- CometColumnarToRow
+- CometFilter
+- CometScan [native_iceberg_compat] parquet
Comet accelerated 33% of eligible operators (sparkOperators=4, cometOperators=2, transitions=2, wrappers=1).
```
## How are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
Manually
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add plan conversion statistics to extended explain info [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #2412:
URL: https://github.com/apache/datafusion-comet/pull/2412#discussion_r2392444077
##########
spark/src/test/scala/org/apache/comet/exec/CometExecSuite.scala:
##########
@@ -118,6 +118,11 @@ class CometExecSuite extends CometTestBase {
val (_, cometPlan) = checkSparkAnswer(df)
val infos = new ExtendedExplainInfo().generateExtendedInfo(cometPlan)
assert(infos.contains("Dynamic Partition Pruning is not supported"))
+
+ withSQLConf(CometConf.COMET_EXPLAIN_VERBOSE_ENABLED.key -> "true") {
+ val extendedExplain = new ExtendedExplainInfo().generateExtendedInfo(cometPlan)
+ assert(extendedExplain.contains("Comet accelerated 33% of eligible operators"))
Review Comment:
I was wrong. There is a failure due to a different percentage. I will make the test less specific.
```
- DPP fallback *** FAILED *** (1 second, 553 milliseconds)
"BroadcastHashJoin
:- ColumnarToRow
: +- Scan parquet [COMET: Dynamic Partition Pruning is not supported]
: +- SubqueryBroadcast
: +- BroadcastExchange
: +- CometColumnarToRow
: +- CometFilter
: +- CometScan [native_iceberg_compat] parquet
+- BroadcastExchange
+- CometColumnarToRow
+- CometFilter
+- CometScan [native_iceberg_compat] parquet
Comet accelerated 4 out of 9 eligible operators (44%). Final plan contains 3 transitions between Spark and Comet." did not contain "Comet accelerated 33% of eligible operators" (CometExecSuite.scala:124)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add plan conversion statistics to extended explain info [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #2412:
URL: https://github.com/apache/datafusion-comet/pull/2412#discussion_r2356966422
##########
spark/src/test/scala/org/apache/comet/exec/CometExecSuite.scala:
##########
@@ -118,6 +118,11 @@ class CometExecSuite extends CometTestBase {
val (_, cometPlan) = checkSparkAnswer(df)
val infos = new ExtendedExplainInfo().generateExtendedInfo(cometPlan)
assert(infos.contains("Dynamic Partition Pruning is not supported"))
+
+ withSQLConf(CometConf.COMET_EXPLAIN_VERBOSE_ENABLED.key -> "true") {
+ val extendedExplain = new ExtendedExplainInfo().generateExtendedInfo(cometPlan)
+ assert(extendedExplain.contains("Comet accelerated 33% of eligible operators"))
Review Comment:
would be this number fluctuating?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add plan conversion statistics to extended explain info [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #2412:
URL: https://github.com/apache/datafusion-comet/pull/2412#discussion_r2391741279
##########
spark/src/test/scala/org/apache/comet/exec/CometExecSuite.scala:
##########
@@ -118,6 +118,11 @@ class CometExecSuite extends CometTestBase {
val (_, cometPlan) = checkSparkAnswer(df)
val infos = new ExtendedExplainInfo().generateExtendedInfo(cometPlan)
assert(infos.contains("Dynamic Partition Pruning is not supported"))
+
+ withSQLConf(CometConf.COMET_EXPLAIN_VERBOSE_ENABLED.key -> "true") {
+ val extendedExplain = new ExtendedExplainInfo().generateExtendedInfo(cometPlan)
+ assert(extendedExplain.contains("Comet accelerated 33% of eligible operators"))
Review Comment:
It is currently stable across all Spark versions that we test with.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add plan conversion statistics to extended explain info [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #2412:
URL: https://github.com/apache/datafusion-comet/pull/2412
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add plan conversion statistics to extended explain info [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #2412:
URL: https://github.com/apache/datafusion-comet/pull/2412#issuecomment-3304872465
```
Comet accelerated 33% of eligible operators (sparkOperators=4, cometOperators=2, transitions=2, wrappers=1).
```
From the user perspective it might be not very clear what is accelerated?
When seeing message first time, I thought all of them are accelerated?
```
(sparkOperators=4, cometOperators=2, transitions=2, wrappers=1).
```
which is not the case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add plan conversion statistics to extended explain info [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #2412:
URL: https://github.com/apache/datafusion-comet/pull/2412#issuecomment-3303846748
Subset of output from benchmarks
```
Reading query 16 using path /mnt/bigdata/tpch/queries//q16.sql
Comet accelerated 87% of eligible operators (sparkOperators=1, cometOperators=7, transitions=0, wrappers=0).
Comet accelerated 84% of eligible operators (sparkOperators=4, cometOperators=22, transitions=0, wrappers=0).
Comet accelerated 80% of eligible operators (sparkOperators=4, cometOperators=17, transitions=0, wrappers=0).
Comet accelerated 75% of eligible operators (sparkOperators=1, cometOperators=3, transitions=0, wrappers=0).
Comet accelerated 100% of eligible operators (sparkOperators=0, cometOperators=3, transitions=0, wrappers=0).
Comet accelerated 84% of eligible operators (sparkOperators=4, cometOperators=22, transitions=1, wrappers=0).
Comet accelerated 84% of eligible operators (sparkOperators=4, cometOperators=22, transitions=1, wrappers=0).
Comet accelerated 75% of eligible operators (sparkOperators=2, cometOperators=6, transitions=1, wrappers=0).
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add plan conversion statistics to extended explain info [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #2412:
URL: https://github.com/apache/datafusion-comet/pull/2412#issuecomment-3309471275
> ```
> Comet accelerated 33% of eligible operators (sparkOperators=4, cometOperators=2, transitions=2, wrappers=1).
> ```
>
> From the user perspective it might be not very clear what is accelerated? When seeing message first time, I thought all of them are accelerated?
>
> ```
> (sparkOperators=4, cometOperators=2, transitions=2, wrappers=1).
> ```
>
> which is not the case?
Thanks, I will add documentation as part of this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add plan conversion statistics to extended explain info [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #2412:
URL: https://github.com/apache/datafusion-comet/pull/2412#issuecomment-3357047457
Thanks for the reviews @parthchandra @comphead. I have addressed feedback and plan on merging this later today, unless you have any other feedback
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add plan conversion statistics to extended explain info [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #2412:
URL: https://github.com/apache/datafusion-comet/pull/2412#discussion_r2356968485
##########
spark/src/main/scala/org/apache/comet/ExtendedExplainInfo.scala:
##########
@@ -137,18 +161,37 @@ class ExtendedExplainInfo extends ExtendedExplainGenerator {
depth + 1,
lastChildren :+ false,
indent,
- outString)
+ outString,
+ planStats)
case _ =>
}
node.children.last match {
case c @ (_: TreeNode[_]) =>
- generateTreeString(getActualPlan(c), depth + 1, lastChildren :+ true, indent, outString)
+ generateTreeString(
+ getActualPlan(c),
+ depth + 1,
+ lastChildren :+ true,
+ indent,
+ outString,
+ planStats)
case _ =>
}
}
}
}
+class PlanStats {
Review Comment:
```suggestion
class CometCoverageStats {
```
?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add plan conversion statistics to extended explain info [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #2412:
URL: https://github.com/apache/datafusion-comet/pull/2412#discussion_r2361297496
##########
spark/src/main/scala/org/apache/comet/ExtendedExplainInfo.scala:
##########
@@ -137,18 +161,37 @@ class ExtendedExplainInfo extends ExtendedExplainGenerator {
depth + 1,
lastChildren :+ false,
indent,
- outString)
+ outString,
+ planStats)
case _ =>
}
node.children.last match {
case c @ (_: TreeNode[_]) =>
- generateTreeString(getActualPlan(c), depth + 1, lastChildren :+ true, indent, outString)
+ generateTreeString(
+ getActualPlan(c),
+ depth + 1,
+ lastChildren :+ true,
+ indent,
+ outString,
+ planStats)
case _ =>
}
}
}
}
+class PlanStats {
+ var sparkOperators: Int = 0
+ var cometOperators: Int = 0
+ var wrappers: Int = 0
+ var transitions: Int = 0
+
+ override def toString: String = {
+ s"sparkOperators=$sparkOperators, cometOperators=$cometOperators, " +
Review Comment:
Perhaps we could use a more verbose string here so that the meaning of these stats is a little more obvious
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add plan conversion statistics to extended explain info [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #2412:
URL: https://github.com/apache/datafusion-comet/pull/2412#discussion_r2389617165
##########
spark/src/main/scala/org/apache/comet/ExtendedExplainInfo.scala:
##########
@@ -82,9 +83,29 @@ class ExtendedExplainInfo extends ExtendedExplainGenerator {
// generates the extended info in a verbose manner, printing each node along with the
// extended information in a tree display
def generateVerboseExtendedInfo(plan: SparkPlan): String = {
+ val planStats = new CometCoverageStats()
val outString = new StringBuilder()
- generateTreeString(getActualPlan(plan), 0, Seq(), 0, outString)
- outString.toString()
+ generateTreeString(getActualPlan(plan), 0, Seq(), 0, outString, planStats)
+ val eligible = planStats.sparkOperators + planStats.cometOperators
+ val converted =
+ if (eligible == 0) 0.0 else planStats.cometOperators.toDouble / eligible * 100.0
+ val summary = s"Comet accelerated ${planStats.cometOperators} out of $eligible " +
+ s"eligible operators (${converted.toInt}%). " +
+ s"Final plan contains ${planStats.transitions} transitions."
+ s"${outString.toString()}\n$summary"
+ }
+
+ /** Get the coverage statistics without the full plan */
+ def generateCoverageInfo(plan: SparkPlan): String = {
+ val planStats = new CometCoverageStats()
+ val outString = new StringBuilder()
+ generateTreeString(getActualPlan(plan), 0, Seq(), 0, outString, planStats)
+ val eligible = planStats.sparkOperators + planStats.cometOperators
+ val converted =
+ if (eligible == 0) 0.0 else planStats.cometOperators.toDouble / eligible * 100.0
+ s"Comet accelerated ${planStats.cometOperators} out of $eligible " +
+ s"eligible operators (${converted.toInt}%). " +
+ s"Final plan contains ${planStats.transitions} transitions."
Review Comment:
```suggestion
s"Final plan contains ${planStats.transitions} transitions between Spark and Comet."
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add plan conversion statistics to extended explain info [datafusion-comet]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #2412:
URL: https://github.com/apache/datafusion-comet/pull/2412#issuecomment-3303607262
## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/2412?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `36 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 32.99%. Comparing base ([`f09f8af`](https://app.codecov.io/gh/apache/datafusion-comet/commit/f09f8af64c6599255e116a376f4f008f2fd63b43?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`8dca97d`](https://app.codecov.io/gh/apache/datafusion-comet/commit/8dca97da756f75d65dbf30c7c86f5e82ed324891?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 516 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion-comet/pull/2412?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...n/scala/org/apache/comet/ExtendedExplainInfo.scala](https://app.codecov.io/gh/apache/datafusion-comet/pull/2412?src=pr&el=tree&filepath=spark%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fcomet%2FExtendedExplainInfo.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9jb21ldC9FeHRlbmRlZEV4cGxhaW5JbmZvLnNjYWxh) | 0.00% | [36 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion-comet/pull/2412?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #2412 +/- ##
=============================================
- Coverage 56.12% 32.99% -23.14%
+ Complexity 976 736 -240
=============================================
Files 119 147 +28
Lines 11743 13498 +1755
Branches 2251 2343 +92
=============================================
- Hits 6591 4453 -2138
- Misses 4012 8250 +4238
+ Partials 1140 795 -345
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/datafusion-comet/pull/2412?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add plan conversion statistics to extended explain info [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #2412:
URL: https://github.com/apache/datafusion-comet/pull/2412#issuecomment-3348819831
Thanks for the reviews @comphead and @parthchandra. Could you take another look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add plan conversion statistics to extended explain info [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #2412:
URL: https://github.com/apache/datafusion-comet/pull/2412#issuecomment-3309884809
We could potentially add a method to get the plan stats independent of the explain string too. I can see this being useful in writing some tool to analyse queries/plans without executing them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add plan conversion statistics to extended explain info [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #2412:
URL: https://github.com/apache/datafusion-comet/pull/2412#discussion_r2356207171
##########
docs/source/user-guide/latest/installation.md:
##########
@@ -90,9 +90,10 @@ $SPARK_HOME/bin/spark-shell \
--conf spark.executor.extraClassPath=$COMET_JAR \
--conf spark.plugins=org.apache.spark.CometPlugin \
--conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \
- --conf spark.comet.explainFallback.enabled=true \
+ --conf spark.sql.extendedExplainProviders=org.apache.comet.ExtendedExplainInfo \
+ --conf spark.comet.explain.enabled=true \
--conf spark.memory.offHeap.enabled=true \
- --conf spark.memory.offHeap.size=16g
+ --conf spark.memory.offHeap.size=2g
Review Comment:
Unrelated change, but we don't need `16g` for this simple example
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org