You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2024/05/06 14:43:00 UTC
[I] Allow user to see reasons why query stage cannot be run natively [datafusion-comet]
andygrove opened a new issue, #385:
URL: https://github.com/apache/datafusion-comet/issues/385
### What is the problem the feature request solves?
As a user, I do not think I have a way to see why my query stage did not run natively.
We store information in the plan with the reasons why we don't support specific operators and expressions and this information is available programatically via `ExtendedExplainInfo.generateExtendedInfo` but this only seems to be accessed from tests.
My workaround for now is to add this code fragment at the end of `CometSparkSessionExtensions.CometExecRule#apply`:
```scala
if (!isCometNative(finalPlan)) {
val str = new ExtendedExplainInfo().generateExtendedInfo(finalPlan)
logInfo(s"Cannot run plan natively:\n\t- ${str.split("\n").mkString("\n\t- ")}")
}
```
Side note: It would be nice to store the reasons as `Set[String]` rather than newline-delimited string containing multiple reasons.
Example output:
```
Cannot run plan natively:
- LocalTableScan is not supported
- WriteFiles is not supported
- Execute InsertIntoHadoopFsRelationCommand is not supported
Cannot run plan natively:
- Execute CreateViewCommand is not supported
Cannot run plan natively:
- Comet does not guarantee correct results for cast from StringType to TimestampType with timezone Some(UTC) and evalMode LEGACY (Not all valid formats are supported). To enable all incompatible casts, set spark.comet.cast.allowIncompatible=true
```
It would be nice to enable this if a config is set. Perhaps `spark.comet.explain=VERBOSE`.
### Describe the potential solution
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Allow user to see reasons why query stage cannot be run natively [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on issue #385:
URL: https://github.com/apache/datafusion-comet/issues/385#issuecomment-2096496393
> Note we do want an ordered set as we want to display leaf node messages first.
This is not relevant to the fix in #386. This applies only to the method that rolls up all the explain tags into the final explain string.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Allow user to see reasons why query stage cannot be run natively [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #385:
URL: https://github.com/apache/datafusion-comet/issues/385#issuecomment-2096205801
@parthchandra Do you have any thoughts on the best solution for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Allow user to see reasons why query stage cannot be run natively [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #385: Allow user to see reasons why query stage cannot be run natively
URL: https://github.com/apache/datafusion-comet/issues/385
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Allow user to see reasons why query stage cannot be run natively [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #385:
URL: https://github.com/apache/datafusion-comet/issues/385#issuecomment-2096622643
Thanks @parthchandra. I wasn't aware of the Spark integration. That looks great. I will create a PR to add a log message to help us during development.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Allow user to see reasons why query stage cannot be run natively [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on issue #385:
URL: https://github.com/apache/datafusion-comet/issues/385#issuecomment-2096477397
This https://github.com/apache/spark/pull/45488 adds the capability to get the extended explain info into Spark (which was the original goal). So one can call `EXPLAIN` in spark SQL or in the Spark UI to see why comet was not (fully) enabled.
I should have added the log message you've added. That's a nice thing to have especially during development as it will hopefully encourage us to write user friendly messages :)
The only reason for not having a Set is because there are some expressions/operators that may have multiple reasons for not being enabled and I wasn't sure how to assign a set to the tag value. Otherwise having the explain messages in a Set is very useful since it eliminates duplicates. (Note we do want an ordered set as we want to display leaf node messages first.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org