You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "HyukjinKwon (via GitHub)" <gi...@apache.org> on 2023/02/28 04:19:23 UTC
[GitHub] [spark] HyukjinKwon opened a new pull request, #40209: [SPARK-42427][SQL][TESTS][FOLLOW-UP] Disable ANSI for one more conv test case in MathFunctionsSuite
HyukjinKwon opened a new pull request, #40209:
URL: https://github.com/apache/spark/pull/40209
### What changes were proposed in this pull request?
This PR proposes to disable ANSI for one more conv test cases in `MathFunctionsSuite`. They are intentionally testing the behaviours when ANSI is disabled. This is another followup of https://github.com/apache/spark/pull/40117.
### Why are the changes needed?
To make the ANSI tests pass. It currently fails (https://github.com/apache/spark/actions/runs/4277973597/jobs/7447263656):
```
2023-02-28T02:34:53.0298317Z [0m[[0m[0minfo[0m] [0m[0m[32m- SPARK-36229 inconsistently behaviour where returned value is above the 64 char threshold (105 milliseconds)[0m[0m
2023-02-28T02:34:53.0631723Z 02:34:53.062 ERROR org.apache.spark.executor.Executor: Exception in task 0.0 in stage 146.0 (TID 268)
2023-02-28T02:34:53.0632672Z org.apache.spark.SparkArithmeticException: [ARITHMETIC_OVERFLOW] Overflow in function conv(). If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error.
2023-02-28T02:34:53.0633557Z at org.apache.spark.sql.errors.QueryExecutionErrors$.arithmeticOverflowError(QueryExecutionErrors.scala:643)
2023-02-28T02:34:53.0634361Z at org.apache.spark.sql.errors.QueryExecutionErrors$.overflowInConvError(QueryExecutionErrors.scala:315)
2023-02-28T02:34:53.0635124Z at org.apache.spark.sql.catalyst.util.NumberConverter$.encode(NumberConverter.scala:68)
2023-02-28T02:34:53.0711747Z at org.apache.spark.sql.catalyst.util.NumberConverter$.convert(NumberConverter.scala:158)
2023-02-28T02:34:53.0712298Z at org.apache.spark.sql.catalyst.util.NumberConverter.convert(NumberConverter.scala)
2023-02-28T02:34:53.0712925Z at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(generated.java:38)
2023-02-28T02:34:53.0713547Z at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
2023-02-28T02:34:53.0714098Z at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:760)
2023-02-28T02:34:53.0714552Z at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
2023-02-28T02:34:53.0715094Z at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
2023-02-28T02:34:53.0715466Z at org.apache.spark.util.Iterators$.size(Iterators.scala:29)
2023-02-28T02:34:53.0715829Z at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1944)
2023-02-28T02:34:53.0716195Z at org.apache.spark.rdd.RDD.$anonfun$count$1(RDD.scala:1266)
2023-02-28T02:34:53.0716555Z at org.apache.spark.rdd.RDD.$anonfun$count$1$adapted(RDD.scala:1266)
2023-02-28T02:34:53.0716963Z at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2303)
2023-02-28T02:34:53.0717400Z at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:92)
2023-02-28T02:34:53.0717857Z at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
2023-02-28T02:34:53.0718332Z at org.apache.spark.scheduler.Task.run(Task.scala:139)
2023-02-28T02:34:53.0718743Z at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:554)
2023-02-28T02:34:53.0719152Z at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1520)
2023-02-28T02:34:53.0719548Z at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:557)
2023-02-28T02:34:53.0720001Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2023-02-28T02:34:53.0720481Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2023-02-28T02:34:53.0720848Z at java.lang.Thread.run(Thread.java:750)
2023-02-28T02:34:53.0721492Z 02:34:53.065 WARN org.apache.spark.scheduler.TaskSetManager: Lost task 0.0 in stage 146.0 (TID 268) (localhost executor driver): org.apache.spark.SparkArithmeticException: [ARITHMETIC_OVERFLOW] Overflow in function conv(). If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error.
2023-02-28T02:34:53.0722264Z at org.apache.spark.sql.errors.QueryExecutionErrors$.arithmeticOverflowError(QueryExecutionErrors.scala:643)
2023-02-28T02:34:53.0722821Z at org.apache.spark.sql.errors.QueryExecutionErrors$.overflowInConvError(QueryExecutionErrors.scala:315)
2023-02-28T02:34:53.0723337Z at org.apache.spark.sql.catalyst.util.NumberConverter$.encode(NumberConverter.scala:68)
2023-02-28T02:34:53.0723963Z at org.apache.spark.sql.catalyst.util.NumberConverter$.convert(NumberConverter.scala:158)
2023-02-28T02:34:53.0724474Z at org.apache.spark.sql.catalyst.util.NumberConverter.convert(NumberConverter.scala)
2023-02-28T02:34:53.0725128Z at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(generated.java:38)
2023-02-28T02:34:53.0725826Z at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
2023-02-28T02:34:53.0726376Z at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:760)
2023-02-28T02:34:53.0726827Z at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
2023-02-28T02:34:53.0727189Z at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
2023-02-28T02:34:53.0727556Z at org.apache.spark.util.Iterators$.size(Iterators.scala:29)
2023-02-28T02:34:53.0727931Z at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1944)
2023-02-28T02:34:53.0728346Z at org.apache.spark.rdd.RDD.$anonfun$count$1(RDD.scala:1266)
2023-02-28T02:34:53.0728701Z at org.apache.spark.rdd.RDD.$anonfun$count$1$adapted(RDD.scala:1266)
2023-02-28T02:34:53.0729096Z at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2303)
2023-02-28T02:34:53.0729523Z at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:92)
2023-02-28T02:34:53.0729966Z at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
```
### Does this PR introduce _any_ user-facing change?
No, test-only.
### How was this patch tested?
Fixed unittests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #40209: [SPARK-42427][SQL][TESTS][FOLLOW-UP] Disable ANSI for one more conv test case in MathFunctionsSuite
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #40209:
URL: https://github.com/apache/spark/pull/40209#issuecomment-1447553959
cc @gengliangwang sorry for many followups .. I don't know why I missed this ... this is the last followup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #40209: [SPARK-42427][SQL][TESTS][FOLLOW-UP] Disable ANSI for one more conv test case in MathFunctionsSuite
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #40209:
URL: https://github.com/apache/spark/pull/40209#issuecomment-1447757856
Merged to master and branch-3.4.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #40209: [SPARK-42427][SQL][TESTS][FOLLOW-UP] Disable ANSI for one more conv test case in MathFunctionsSuite
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #40209: [SPARK-42427][SQL][TESTS][FOLLOW-UP] Disable ANSI for one more conv test case in MathFunctionsSuite
URL: https://github.com/apache/spark/pull/40209
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #40209: [SPARK-42427][SQL][TESTS][FOLLOW-UP] Disable ANSI for one more conv test case in MathFunctionsSuite
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on PR #40209:
URL: https://github.com/apache/spark/pull/40209#issuecomment-1447601513
@HyukjinKwon I also verified MathFunctionsSuite can pass with ANSI Enabled
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org