You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2024/05/23 20:06:12 UTC
[I] feat: Implement ANSI support for UnaryMinus [datafusion-comet]
andygrove opened a new issue, #465:
URL: https://github.com/apache/datafusion-comet/issues/465
### What is the problem the feature request solves?
Comet does not support ANSI mode for UnaryMinus.
## Create test data
```
val df = Seq(Int.MaxValue, Int.MinValue).toDF("a")
df.write.parquet("/tmp/int.parquet")
spark.read.parquet("/tmp/int.parquet").createTempView("t")
```
## Test with ANSI mode disabled
Behavior is correct with ANSI mode disabled:
```
scala> spark.conf.set("spark.sql.ansi.enabled", false)
scala> spark.conf.set("spark.comet.enabled", false)
scala> spark.sql("select a, -a from t").show
+-----------+-----------+
| a| (- a)|
+-----------+-----------+
| 2147483647|-2147483647|
|-2147483648|-2147483648|
+-----------+-----------+
scala> spark.conf.set("spark.comet.enabled", true)
scala> spark.sql("select a, -a from t").show
24/05/23 13:55:00 WARN CometSparkSessionExtensions$CometExecRule: Comet cannot execute some parts of this plan natively because CollectLimit is not supported
+-----------+-----------+
| a| (- a)|
+-----------+-----------+
| 2147483647|-2147483647|
|-2147483648|-2147483648|
+-----------+-----------+
```
## Test with ANSI mode enabled
With ANSI mode enabled, Spark throws an exception, but Comet does not.
```
spark.conf.set("spark.sql.ansi.enabled", true)
spark.conf.set("spark.comet.ansi.enabled", true)
scala> spark.conf.set("spark.comet.enabled", false)
scala> spark.sql("select a, -a from t").show
24/05/23 13:55:36 WARN CometSparkSessionExtensions$CometExecRule: Using Comet's experimental support for ANSI mode.
24/05/23 13:55:36 ERROR Executor: Exception in task 0.0 in stage 18.0 (TID 18)
org.apache.spark.SparkArithmeticException: [ARITHMETIC_OVERFLOW] integer overflow. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error.
scala> spark.conf.set("spark.comet.enabled", true)
scala> spark.sql("select a, -a from t").show
24/05/23 13:55:48 WARN CometSparkSessionExtensions$CometExecRule: Using Comet's experimental support for ANSI mode.
24/05/23 13:55:48 WARN CometSparkSessionExtensions$CometExecRule: Comet cannot execute some parts of this plan natively because CollectLimit is not supported
+-----------+-----------+
| a| (- a)|
+-----------+-----------+
| 2147483647|-2147483647|
|-2147483648|-2147483648|
+-----------+-----------+
```
### Describe the potential solution
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] feat: Implement ANSI support for UnaryMinus [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #465: feat: Implement ANSI support for UnaryMinus
URL: https://github.com/apache/datafusion-comet/issues/465
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] feat: Implement ANSI support for UnaryMinus [datafusion-comet]
Posted by "vaibhawvipul (via GitHub)" <gi...@apache.org>.
vaibhawvipul commented on issue #465:
URL: https://github.com/apache/datafusion-comet/issues/465#issuecomment-2128357365
I am working on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org