You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/09/07 09:51:39 UTC
[GitHub] [spark] LuciferYang opened a new pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
LuciferYang opened a new pull request #29660:
URL: https://github.com/apache/spark/pull/29660
### What changes were proposed in this pull request?
The purpose of this pr is to partial resolve [SPARK-32808](https://issues.apache.org/jira/browse/SPARK-32808), total of 26 failed test cases were fixed, the related suite as follow:
- `StreamingAggregationSuite` related test cases (2 FAILED -> Pass)
- `GeneratorFunctionSuite` related test cases (2 FAILED -> Pass)
- `UDFSuite` related test cases (2 FAILED -> Pass)
- `SQLQueryTestSuite` related test cases (5 FAILED -> Pass)
- `WholeStageCodegenSuite` related test cases (1 FAILED -> Pass)
- `DataFrameSuite` related test cases (3 FAILED -> Pass)
- `OrcV1QuerySuite\OrcV2QuerySuite` related test cases (4 FAILED -> Pass)
- `ExpressionsSchemaSuite` related test cases (1 FAILED -> Pass)
- `DataFrameStatSuite` related test cases (1 FAILED -> Pass)
- `JsonV1Suite\JsonV2Suite\JsonLegacyTimeParserSuite` related test cases (6 FAILED -> Pass)
The main change of this pr as following:
- Fix Scala 2.13 compilation problems in `ShuffleBlockFetcherIterator` and `Analyzer`
- Specified `Seq` to `scala.collection.Seq` in `objects.scala` and `GenericArrayData` because internal use `Seq` maybe `mutable.ArraySeq` and not easy to call `.toSeq`
- Should specified `Seq` to `scala.collection.Seq` when we call `Row.getAs[Seq]` and `Row.get(i).asInstanceOf[Seq]` because the data maybe `mutable.ArraySeq` but `Seq` is `immutable.Seq` in Scala 2.13
- Use a compatible way to let `+` and `-` method of `Decimal` having the same behavior in Scala 2.12 and Scala 2.13
- Call `toList` in `RelationalGroupedDataset.toDF` method when `groupingExprs` is `Stream` type because `Stream` can't serialize in Scala 2.13
### Why are the changes needed?
We need to support a Scala 2.13 build.
### Does this PR introduce _any_ user-facing change?
Should specified `Seq` to `scala.collection.Seq` when we call `Row.getAs[Seq]` and `Row.get(i).asInstanceOf[Seq]` because the data maybe `mutable.ArraySeq` but the `Seq` is `immutable.Seq` in Scala 2.13
### How was this patch tested?
- Scala 2.12: Pass the Jenkins or GitHub Action
- Scala 2.13: Do the following:
```
dev/change-scala-version.sh 2.13
mvn clean install -DskipTests -pl sql/core -Pscala-2.13 -am
mvn test -pl sql/core -Pscala-2.13
```
**Before**
```
Tests: succeeded 8166, failed 319, canceled 1, ignored 52, pending 0
*** 319 TESTS FAILED ***
```
**After**
```
Tests: succeeded 8192, failed 293, canceled 1, ignored 52, pending 0
*** 293 TESTS FAILED ***
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-689270118
Address 9185a95 re-sync master and local test core module ,all test passed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-689022386
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-688270733
`add executors default profile *** FAILED *** (82 milliseconds)` failed in GItHub Action, but successful in local test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-689326251
org.apache.spark.sql.hive.thriftserver.CliSuite.* failed because db exists.......
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-689326251
org.apache.spark.sql.hive.thriftserver.CliSuite.* failed because `Database clitestdb already exists`.......
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-689471539
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-689560756
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-689313938
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128430/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-688214289
**[Test build #128352 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128352/testReport)** for PR 29660 at commit [`1fa24b9`](https://github.com/apache/spark/commit/1fa24b9e2ccd33ce65c55464125e392290fcf719).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-689021063
**[Test build #128409 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128409/testReport)** for PR 29660 at commit [`454b53c`](https://github.com/apache/spark/commit/454b53cb20952b01ecc3e9a8e4a186c9324aac79).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class ExecutorDecommissionInfo(message: String, workerHost: Option[String] = None)`
* ` throw new AnalysisException(s\"Can not load class '$className' when registering \" +`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-688214503
cc @srowen , this pr is try to pass some test case of `sql/core` module in Scala 2.13, other failures may related to `CostBasedJoinReorder`, I will check it and feedback later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #29660:
URL: https://github.com/apache/spark/pull/29660#discussion_r484332008
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/GenericArrayData.scala
##########
@@ -38,7 +40,9 @@ class GenericArrayData(val array: Array[Any]) extends ArrayData {
def this(primitiveArray: Array[Boolean]) = this(primitiveArray.toSeq)
def this(seqOrArray: Any) = this(seqOrArray match {
- case seq: Seq[Any] => seq.toArray
+ // Specified this as`scala.collection.Seq` because seqOrArray can be
+ // `mutable.ArraySeq` in Scala 2.13
+ case seq: scala.collection.Seq[Any] => seq.toArray
Review comment:
The entrance is
https://github.com/apache/spark/blob/04f7f6dac0b9177e11482cca4e7ebf7b7564e45f/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ArrayData.scala#L28-L38
not easy to call `toSeq`, so I changed it here
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-689313928
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-689578790
Merged to master. I have left the JIRA open though.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-688295918
**[Test build #128352 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128352/testReport)** for PR 29660 at commit [`1fa24b9`](https://github.com/apache/spark/commit/1fa24b9e2ccd33ce65c55464125e392290fcf719).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-689560756
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-689471539
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #29660:
URL: https://github.com/apache/spark/pull/29660#discussion_r484491139
##########
File path: sql/core/src/test/resources/sql-functions/sql-expression-schema.md
##########
@@ -2,7 +2,7 @@
## Summary
- Number of queries: 339
- Number of expressions that missing example: 34
- - Expressions missing examples: and,string,tinyint,double,smallint,date,decimal,boolean,float,binary,bigint,int,timestamp,struct,cume_dist,dense_rank,input_file_block_length,input_file_block_start,input_file_name,lag,lead,monotonically_increasing_id,ntile,!,not,or,percent_rank,rank,row_number,spark_partition_id,version,window,positive,count_min_sketch
+ - Expressions missing examples: and,bigint,binary,boolean,date,decimal,double,float,int,smallint,string,timestamp,tinyint,struct,cume_dist,dense_rank,input_file_block_length,input_file_block_start,input_file_name,lag,lead,monotonically_increasing_id,ntile,!,not,or,percent_rank,rank,row_number,spark_partition_id,version,window,positive,count_min_sketch
Review comment:
Is this change part of the PR on purpose?
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -478,15 +478,15 @@ final class Decimal extends Ordered[Decimal] with Serializable {
if (decimalVal.eq(null) && that.decimalVal.eq(null) && scale == that.scale) {
Decimal(longVal + that.longVal, Math.max(precision, that.precision), scale)
} else {
- Decimal(toBigDecimal + that.toBigDecimal)
Review comment:
I don't think we want to set a MathContext here anyway?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-688296965
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-688877757
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-689471142
**[Test build #128446 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128446/testReport)** for PR 29660 at commit [`9185a95`](https://github.com/apache/spark/commit/9185a95c29bc532e9f0aea5bdd4f2185c5093642).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-689270118
Address 9185a95 re-sync master
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-689262234
local test `mvn clean test -pl core -DwildcardSuites=org.apache.spark.scheduler.BarrierTaskContextSuite -Dtest=none` all passed.
```
Discovery starting.
Discovery completed in 3 seconds, 740 milliseconds.
Run starting. Expected test count is: 11
BarrierTaskContextSuite:
- global sync by barrier() call
- share messages with allGather() call
- throw exception if we attempt to synchronize with different blocking calls
- successively sync with allGather and barrier
- support multiple barrier() call within a single task
- throw exception on barrier() call timeout
- throw exception if barrier() call doesn't happen on every task
- throw exception if the number of barrier() calls are not the same on every task
- barrier task killed, no interrupt
- barrier task killed, interrupt
- SPARK-31485: barrier stage should fail if only partial tasks are launched
Run completed in 4 minutes, 51 seconds.
Total number of tests run: 11
Suites: completed 2, aborted 0
Tests: succeeded 11, failed 0, canceled 0, ignored 0, pending 0
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-688875856
**[Test build #128409 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128409/testReport)** for PR 29660 at commit [`454b53c`](https://github.com/apache/spark/commit/454b53cb20952b01ecc3e9a8e4a186c9324aac79).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-688634013
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #29660:
URL: https://github.com/apache/spark/pull/29660#discussion_r484333977
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -478,15 +478,15 @@ final class Decimal extends Ordered[Decimal] with Serializable {
if (decimalVal.eq(null) && that.decimalVal.eq(null) && scale == that.scale) {
Decimal(longVal + that.longVal, Math.max(precision, that.precision), scale)
} else {
- Decimal(toBigDecimal + that.toBigDecimal)
Review comment:
In Scala 2.13, `+` method is
```
def + (that: BigDecimal): BigDecimal = new BigDecimal(this.bigDecimal.add(that.bigDecimal, mc), mc)
```
and in Scala 2.12 `+` method is
```
def + (that: BigDecimal): BigDecimal = new BigDecimal(this.bigDecimal add that.bigDecimal, mc)
```
There are some differences in accuracy.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-689270118
Address 9185a95 re-sync with master
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #29660:
URL: https://github.com/apache/spark/pull/29660#discussion_r484503571
##########
File path: sql/core/src/test/resources/sql-functions/sql-expression-schema.md
##########
@@ -2,7 +2,7 @@
## Summary
- Number of queries: 339
- Number of expressions that missing example: 34
- - Expressions missing examples: and,string,tinyint,double,smallint,date,decimal,boolean,float,binary,bigint,int,timestamp,struct,cume_dist,dense_rank,input_file_block_length,input_file_block_start,input_file_name,lag,lead,monotonically_increasing_id,ntile,!,not,or,percent_rank,rank,row_number,spark_partition_id,version,window,positive,count_min_sketch
+ - Expressions missing examples: and,bigint,binary,boolean,date,decimal,double,float,int,smallint,string,timestamp,tinyint,struct,cume_dist,dense_rank,input_file_block_length,input_file_block_start,input_file_name,lag,lead,monotonically_increasing_id,ntile,!,not,or,percent_rank,rank,row_number,spark_partition_id,version,window,positive,count_min_sketch
Review comment:
> Add a manual sort to classFunsMap in ExpressionsSchemaSuite because Iterable.groupBy in Scala 2.13 has different result with TraversableLike.groupBy in Scala 2.12
Yes, Because the change of `ExpressionsSchemaSuite`, this file need regenerate.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-689562247
@srowen Jenkins and GitHub Action all passed ~
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-689022386
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-688877757
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-688379117
> cc @srowen , this pr is try to pass some test case of sql/core module in Scala 2.13, other failures may related to CostBasedJoinReorder, I will check it and feedback later.
There are other reasons. I'm working on it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-688215095
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-689451710
local test `org.apache.spark.sql.hive.thriftserver.CliSuite`, all 28 case succeeded
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-688214289
**[Test build #128352 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128352/testReport)** for PR 29660 at commit [`1fa24b9`](https://github.com/apache/spark/commit/1fa24b9e2ccd33ce65c55464125e392290fcf719).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-688215095
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-689271310
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-688875856
**[Test build #128409 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128409/testReport)** for PR 29660 at commit [`454b53c`](https://github.com/apache/spark/commit/454b53cb20952b01ecc3e9a8e4a186c9324aac79).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang removed a comment on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
LuciferYang removed a comment on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-689328928
Jenkins retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-688633651
**[Test build #128381 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128381/testReport)** for PR 29660 at commit [`454b53c`](https://github.com/apache/spark/commit/454b53cb20952b01ecc3e9a8e4a186c9324aac79).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-689022395
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128409/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen closed pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
srowen closed pull request #29660:
URL: https://github.com/apache/spark/pull/29660
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-689270979
**[Test build #128430 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128430/testReport)** for PR 29660 at commit [`9185a95`](https://github.com/apache/spark/commit/9185a95c29bc532e9f0aea5bdd4f2185c5093642).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #29660:
URL: https://github.com/apache/spark/pull/29660#discussion_r484509296
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -478,15 +478,15 @@ final class Decimal extends Ordered[Decimal] with Serializable {
if (decimalVal.eq(null) && that.decimalVal.eq(null) && scale == that.scale) {
Decimal(longVal + that.longVal, Math.max(precision, that.precision), scale)
} else {
- Decimal(toBigDecimal + that.toBigDecimal)
Review comment:
I think the change is OK here, because we actually do not want to modify the rounding, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-688664642
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-689262234
local test `mvn clean test -pl core -DwildcardSuites=org.apache.spark.scheduler.BarrierTaskContextSuite -Dtest=none` all passed.
Discovery starting.
Discovery completed in 3 seconds, 740 milliseconds.
Run starting. Expected test count is: 11
BarrierTaskContextSuite:
- global sync by barrier() call
- share messages with allGather() call
- throw exception if we attempt to synchronize with different blocking calls
- successively sync with allGather and barrier
- support multiple barrier() call within a single task
- throw exception on barrier() call timeout
- throw exception if barrier() call doesn't happen on every task
- throw exception if the number of barrier() calls are not the same on every task
- barrier task killed, no interrupt
- barrier task killed, interrupt
- SPARK-31485: barrier stage should fail if only partial tasks are launched
Run completed in 4 minutes, 51 seconds.
Total number of tests run: 11
Suites: completed 2, aborted 0
Tests: succeeded 11, failed 0, canceled 0, ignored 0, pending 0
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-688634013
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-689581400
thx @srowen @xuanyuanking
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-689471142
**[Test build #128446 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128446/testReport)** for PR 29660 at commit [`9185a95`](https://github.com/apache/spark/commit/9185a95c29bc532e9f0aea5bdd4f2185c5093642).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #29660:
URL: https://github.com/apache/spark/pull/29660#discussion_r484506518
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -478,15 +478,15 @@ final class Decimal extends Ordered[Decimal] with Serializable {
if (decimalVal.eq(null) && that.decimalVal.eq(null) && scale == that.scale) {
Decimal(longVal + that.longVal, Math.max(precision, that.precision), scale)
} else {
- Decimal(toBigDecimal + that.toBigDecimal)
Review comment:
Do you mean we need change to use methods with MathContext ? Like `BigDecimal add(BigDecimal augend, MathContext mc) `?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-688632207
Address 454b53c merge upstream master and resolve conflict file
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-688426646
Do you want to add more changes here? we can merge it whenever it gets big and continue in another PR if desired.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #29660:
URL: https://github.com/apache/spark/pull/29660#discussion_r484512144
##########
File path: sql/core/src/test/resources/sql-functions/sql-expression-schema.md
##########
@@ -2,7 +2,7 @@
## Summary
- Number of queries: 339
- Number of expressions that missing example: 34
- - Expressions missing examples: and,string,tinyint,double,smallint,date,decimal,boolean,float,binary,bigint,int,timestamp,struct,cume_dist,dense_rank,input_file_block_length,input_file_block_start,input_file_name,lag,lead,monotonically_increasing_id,ntile,!,not,or,percent_rank,rank,row_number,spark_partition_id,version,window,positive,count_min_sketch
+ - Expressions missing examples: and,bigint,binary,boolean,date,decimal,double,float,int,smallint,string,timestamp,tinyint,struct,cume_dist,dense_rank,input_file_block_length,input_file_block_start,input_file_name,lag,lead,monotonically_increasing_id,ntile,!,not,or,percent_rank,rank,row_number,spark_partition_id,version,window,positive,count_min_sketch
Review comment:
Code changed from
```
val classFunsMap = funInfos.groupBy(_.getClassName).toSeq.sortBy(_._1)
```
to
```
val classFunsMap = funInfos.groupBy(_.getClassName).toSeq.sortBy(_._1).map {
case (className, infos) => (className, infos.sortBy(_.getName))
}
```
in `ExpressionsSchemaSuite`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-688296965
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #29660:
URL: https://github.com/apache/spark/pull/29660#discussion_r484512144
##########
File path: sql/core/src/test/resources/sql-functions/sql-expression-schema.md
##########
@@ -2,7 +2,7 @@
## Summary
- Number of queries: 339
- Number of expressions that missing example: 34
- - Expressions missing examples: and,string,tinyint,double,smallint,date,decimal,boolean,float,binary,bigint,int,timestamp,struct,cume_dist,dense_rank,input_file_block_length,input_file_block_start,input_file_name,lag,lead,monotonically_increasing_id,ntile,!,not,or,percent_rank,rank,row_number,spark_partition_id,version,window,positive,count_min_sketch
+ - Expressions missing examples: and,bigint,binary,boolean,date,decimal,double,float,int,smallint,string,timestamp,tinyint,struct,cume_dist,dense_rank,input_file_block_length,input_file_block_start,input_file_name,lag,lead,monotonically_increasing_id,ntile,!,not,or,percent_rank,rank,row_number,spark_partition_id,version,window,positive,count_min_sketch
Review comment:
Code change from
```
val classFunsMap = funInfos.groupBy(_.getClassName).toSeq.sortBy(_._1)
```
to
```
val classFunsMap = funInfos.groupBy(_.getClassName).toSeq.sortBy(_._1).map {
case (className, infos) => (className, infos.sortBy(_.getName))
}
```
in `ExpressionsSchemaSuite`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-688429955
@srowen Maybe we can merge this first, other failures are related to the 'PlanStabilitySuite' and I will continue to fix these in another pr.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-689313326
**[Test build #128430 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128430/testReport)** for PR 29660 at commit [`9185a95`](https://github.com/apache/spark/commit/9185a95c29bc532e9f0aea5bdd4f2185c5093642).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-689270979
**[Test build #128430 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128430/testReport)** for PR 29660 at commit [`9185a95`](https://github.com/apache/spark/commit/9185a95c29bc532e9f0aea5bdd4f2185c5093642).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-688664177
**[Test build #128381 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128381/testReport)** for PR 29660 at commit [`454b53c`](https://github.com/apache/spark/commit/454b53cb20952b01ecc3e9a8e4a186c9324aac79).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class ExecutorDecommissionInfo(message: String, workerHost: Option[String] = None)`
* ` throw new AnalysisException(s\"Can not load class '$className' when registering \" +`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-688664642
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-688664652
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128381/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-689270118
Address 9185a95 re-sync master. EDIT: and local test core module ,all test passed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #29660:
URL: https://github.com/apache/spark/pull/29660#discussion_r484506887
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -478,15 +478,15 @@ final class Decimal extends Ordered[Decimal] with Serializable {
if (decimalVal.eq(null) && that.decimalVal.eq(null) && scale == that.scale) {
Decimal(longVal + that.longVal, Math.max(precision, that.precision), scale)
} else {
- Decimal(toBigDecimal + that.toBigDecimal)
Review comment:
Sorry , I think I don't fully understand this comments ....
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-688633651
**[Test build #128381 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128381/testReport)** for PR 29660 at commit [`454b53c`](https://github.com/apache/spark/commit/454b53cb20952b01ecc3e9a8e4a186c9324aac79).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-689559022
**[Test build #128446 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128446/testReport)** for PR 29660 at commit [`9185a95`](https://github.com/apache/spark/commit/9185a95c29bc532e9f0aea5bdd4f2185c5093642).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-689313928
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-688869611
Jenkins retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-689328928
Jenkins retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #29660:
URL: https://github.com/apache/spark/pull/29660#discussion_r484511613
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -478,15 +478,15 @@ final class Decimal extends Ordered[Decimal] with Serializable {
if (decimalVal.eq(null) && that.decimalVal.eq(null) && scale == that.scale) {
Decimal(longVal + that.longVal, Math.max(precision, that.precision), scale)
} else {
- Decimal(toBigDecimal + that.toBigDecimal)
Review comment:
Yes, you are right ~
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-689271310
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on pull request #29660: [SPARK-32808][SQL] Fix some test cases of `sql/core` module in scala 2.13
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on pull request #29660:
URL: https://github.com/apache/spark/pull/29660#issuecomment-689468148
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org