You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2024/05/23 20:06:12 UTC

[I] feat: Implement ANSI support for UnaryMinus [datafusion-comet]

andygrove opened a new issue, #465:
URL: https://github.com/apache/datafusion-comet/issues/465

   ### What is the problem the feature request solves?
   
   Comet does not support ANSI mode for UnaryMinus.
   
   ## Create test data
   
   ```
   val df = Seq(Int.MaxValue, Int.MinValue).toDF("a")
   df.write.parquet("/tmp/int.parquet")
   spark.read.parquet("/tmp/int.parquet").createTempView("t")
   ```
   
   ## Test with ANSI mode disabled
   
   Behavior is correct with ANSI mode disabled:
   
   ```
   scala> spark.conf.set("spark.sql.ansi.enabled", false)
   
   scala> spark.conf.set("spark.comet.enabled", false)
   
   scala> spark.sql("select a, -a from t").show
   +-----------+-----------+
   |          a|      (- a)|
   +-----------+-----------+
   | 2147483647|-2147483647|
   |-2147483648|-2147483648|
   +-----------+-----------+
   
   
   scala> spark.conf.set("spark.comet.enabled", true)
   
   scala> spark.sql("select a, -a from t").show
   24/05/23 13:55:00 WARN CometSparkSessionExtensions$CometExecRule: Comet cannot execute some parts of this plan natively because CollectLimit is not supported
   +-----------+-----------+
   |          a|      (- a)|
   +-----------+-----------+
   | 2147483647|-2147483647|
   |-2147483648|-2147483648|
   +-----------+-----------+
   ```
   
   ## Test with ANSI mode enabled
   
   With ANSI mode enabled, Spark throws an exception, but Comet does not.
   
   ```
   spark.conf.set("spark.sql.ansi.enabled", true)
   spark.conf.set("spark.comet.ansi.enabled", true)
   
   
   scala> spark.conf.set("spark.comet.enabled", false)
   
   scala> spark.sql("select a, -a from t").show
   24/05/23 13:55:36 WARN CometSparkSessionExtensions$CometExecRule: Using Comet's experimental support for ANSI mode.
   24/05/23 13:55:36 ERROR Executor: Exception in task 0.0 in stage 18.0 (TID 18)
   org.apache.spark.SparkArithmeticException: [ARITHMETIC_OVERFLOW] integer overflow. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error.
   
   
   scala> spark.conf.set("spark.comet.enabled", true)
   
   scala> spark.sql("select a, -a from t").show
   24/05/23 13:55:48 WARN CometSparkSessionExtensions$CometExecRule: Using Comet's experimental support for ANSI mode.
   24/05/23 13:55:48 WARN CometSparkSessionExtensions$CometExecRule: Comet cannot execute some parts of this plan natively because CollectLimit is not supported
   +-----------+-----------+
   |          a|      (- a)|
   +-----------+-----------+
   | 2147483647|-2147483647|
   |-2147483648|-2147483648|
   +-----------+-----------+
   ```
   
   
   ### Describe the potential solution
   
   _No response_
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] feat: Implement ANSI support for UnaryMinus [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #465: feat: Implement ANSI support for UnaryMinus
URL: https://github.com/apache/datafusion-comet/issues/465


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] feat: Implement ANSI support for UnaryMinus [datafusion-comet]

Posted by "vaibhawvipul (via GitHub)" <gi...@apache.org>.
vaibhawvipul commented on issue #465:
URL: https://github.com/apache/datafusion-comet/issues/465#issuecomment-2128357365

   I am working on this.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org