You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/06/21 13:05:35 UTC
[GitHub] [spark] LuciferYang opened a new pull request, #36942: [SPARK-39545][SQL] Override `concat` method for `ExpressionSet` in Scala 2.13 to improve the performance
LuciferYang opened a new pull request, #36942:
URL: https://github.com/apache/spark/pull/36942
### What changes were proposed in this pull request?
`ExpressionSet ++` method in the master branch a little slower than the branch-3.3 with Scala-2.13, so this pr override `concat` method for `ExpressionSet` in Scala 2.13.
### Why are the changes needed?
Improve the performance
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Pass GA
- Manual test 1:
microbench as follows and run with Scala 2.13:
```scala
val valuesPerIteration = 100000
val benchmark = new Benchmark("Test ExpressionSet ++ ", valuesPerIteration, output = output)
val aUpper = AttributeReference("A", IntegerType)(exprId = ExprId(1))
val initialSet = ExpressionSet(aUpper + 1 :: Rand(0) :: Nil)
val setToAddWithSameDeterministicExpression = ExpressionSet(aUpper + 1 :: Rand(0) :: Nil)
benchmark.addCase("Test ++") { _: Int =>
for (_ <- 0L until valuesPerIteration) {
initialSet ++ setToAddWithSameDeterministicExpression
}
}
benchmark.run()
```
**branch-3.3 result:**
```
OpenJDK 64-Bit Server VM 1.8.0_312-b07 on Linux 4.14.0_1-0-0-45
Intel(R) Xeon(R) Gold 6XXXC CPU @ 2.60GHz
Test ExpressionSet ++ : Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Test ++ 14 16 4 7.2 139.1 1.0X
```
**master result before this pr:**
```
OpenJDK 64-Bit Server VM 1.8.0_312-b07 on Linux 4.14.0_1-0-0-45
Intel(R) Xeon(R) Gold 6271C CPU @ 2.60GHz
Test ExpressionSet ++ : Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Test ++ 16 19 5 6.1 163.9 1.0X
```
**master result after this pr:**
```
Running benchmark: Test ExpressionSet ++
Running case: Test ++
Stopped after 154 iterations, 2010 ms
OpenJDK 64-Bit Server VM 1.8.0_312-b07 on Linux 4.14.0_1-0-0-45
Intel(R) Xeon(R) Gold 6271C CPU @ 2.60GHz
Test ExpressionSet ++ : Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Test ++ 12 13 3 8.6 115.7 1.0X
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #36942: [SPARK-39545][SQL] Override `concat` method for `ExpressionSet` in Scala 2.13 to improve the performance
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on PR #36942:
URL: https://github.com/apache/spark/pull/36942#issuecomment-1163820993
late LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #36942: [SPARK-39545][SQL] Override `concat` method for `ExpressionSet` in Scala 2.13 to improve the performance
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #36942:
URL: https://github.com/apache/spark/pull/36942#issuecomment-1163977592
thanks all ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #36942: [SPARK-39545][SQL] Override `concat` method for `ExpressionSet` in Scala 2.13 to improve the performance
Posted by GitBox <gi...@apache.org>.
srowen commented on PR #36942:
URL: https://github.com/apache/spark/pull/36942#issuecomment-1163770025
Merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen closed pull request #36942: [SPARK-39545][SQL] Override `concat` method for `ExpressionSet` in Scala 2.13 to improve the performance
Posted by GitBox <gi...@apache.org>.
srowen closed pull request #36942: [SPARK-39545][SQL] Override `concat` method for `ExpressionSet` in Scala 2.13 to improve the performance
URL: https://github.com/apache/spark/pull/36942
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org