You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2024/05/10 14:08:06 UTC
[I] bug: TPC-H q16 failed with "ColumnarToRow does not implement doExecuteBroadcast" [datafusion-comet]
andygrove opened a new issue, #408:
URL: https://github.com/apache/datafusion-comet/issues/408
### Describe the bug
I am benchmarking locally with small scale factor (100GB) and saw q16 fail.
Debug logging showed some reasons we can't support this query natively:
```
WARN CometSparkSessionExtensions$CometExecRule: Comet cannot execute this plan natively because:
- BuildRight is not supported
- Native shuffle is not enabled
```
The query failed at runtime:
```
org.apache.spark.SparkUnsupportedOperationException: ColumnarToRow does not implement doExecuteBroadcast.
```
I used these Comet configs:
```
command.append("--jars $COMET_JAR")
command.append("--conf spark.sql.extensions=org.apache.comet.CometSparkSessionExtensions")
command.append("--conf spark.comet.enabled=true")
command.append("--conf spark.comet.explainFallback.enabled=true")
command.append("--conf spark.comet.exec.enabled=true")
command.append("--conf spark.comet.exec.all.enabled=true")
command.append("--conf spark.comet.exec.all.expr.enabled=true")
command.append("--conf spark.comet.exec.shuffle.enabled=true")
command.append("--conf spark.comet.exec.broadcast.enabled=true")
command.append("--conf spark.comet.columnar.shuffle.enabled=true")
command.append("--conf spark.comet.cast.allowIncompatible=true")
command.append("--conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager")
command.append("--conf spark.driver.extraClassPath=$COMET_JAR")
command.append("--conf spark.executor.extraClassPath=$COMET_JAR")
```
### Steps to reproduce
_No response_
### Expected behavior
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] bug: TPC-H q16 failed with "ColumnarToRow does not implement doExecuteBroadcast" [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #408: bug: TPC-H q16 failed with "ColumnarToRow does not implement doExecuteBroadcast"
URL: https://github.com/apache/datafusion-comet/issues/408
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] bug: TPC-H q16 failed with "ColumnarToRow does not implement doExecuteBroadcast" [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on issue #408:
URL: https://github.com/apache/datafusion-comet/issues/408#issuecomment-2104818958
Oh, please disable `spark.comet.exec.broadcast.enabled`, which is only used to enforce enabling broadcast for invalid cases. I should make it internal config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] bug: TPC-H q16 failed with "ColumnarToRow does not implement doExecuteBroadcast" [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on issue #408:
URL: https://github.com/apache/datafusion-comet/issues/408#issuecomment-2104696252
I will take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] bug: TPC-H q16 failed with "ColumnarToRow does not implement doExecuteBroadcast" [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #408:
URL: https://github.com/apache/datafusion-comet/issues/408#issuecomment-2104912739
Thanks @viirya that resolves this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org