You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/05/17 05:14:12 UTC
[GitHub] [spark] LuciferYang commented on pull request #36571: [SPARK-39202][SQL] Introduce a `putByteArrays` method to `WritableColumnVector` to support setting multiple duplicate `byte[]`
LuciferYang commented on PR #36571:
URL: https://github.com/apache/spark/pull/36571#issuecomment-1128419927
For `ColumnVectorUtils.populate` method:
```scala
def testPopulate(valuesPerIteration: Int, length: Int): Unit = {
val batchSize = 4096
val onHeapColumnVector = new OnHeapColumnVector(batchSize, StringType)
val offHeapColumnVector = new OffHeapColumnVector(batchSize, StringType)
val benchmark = new Benchmark(
s"Test ColumnVectorUtils.populate, row length = $length",
valuesPerIteration * batchSize,
output = output)
val builder = new UTF8StringBuilder()
builder.append(RandomStringUtils.random(length))
val row = InternalRow(builder.build())
benchmark.addCase("OnHeapColumnVector") { _: Int =>
for (_ <- 0L until valuesPerIteration) {
onHeapColumnVector.reset()
ColumnVectorUtils.populate(onHeapColumnVector, row, 0)
}
}
benchmark.addCase("OffHeapColumnVector") { _: Int =>
for (_ <- 0L until valuesPerIteration) {
offHeapColumnVector.reset()
ColumnVectorUtils.populate(offHeapColumnVector, row, 0)
}
}
benchmark.run()
}
override def runBenchmarkSuite(mainArgs: Array[String]): Unit = {
val valuesPerIteration = 100000
Seq(1, 5, 10, 15, 20).foreach { length =>
testPopulate(valuesPerIteration, length)
}
}
```
**Before**
```
OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1022-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Test ColumnVectorUtils.populate, row length = 1: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------------
OnHeapColumnVector 3381 3404 32 121.2 8.3 1.0X
OffHeapColumnVector 3931 3968 53 104.2 9.6 0.9X
OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1022-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Test ColumnVectorUtils.populate, row length = 5: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------------
OnHeapColumnVector 4700 4767 96 87.2 11.5 1.0X
OffHeapColumnVector 5258 5356 139 77.9 12.8 0.9X
OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1022-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Test ColumnVectorUtils.populate, row length = 10: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------------------------------------------
OnHeapColumnVector 4920 4934 19 83.2 12.0 1.0X
OffHeapColumnVector 5007 5017 14 81.8 12.2 1.0X
OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1022-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Test ColumnVectorUtils.populate, row length = 15: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------------------------------------------
OnHeapColumnVector 5227 5255 40 78.4 12.8 1.0X
OffHeapColumnVector 5626 5731 148 72.8 13.7 0.9X
OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1022-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Test ColumnVectorUtils.populate, row length = 20: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------------------------------------------
OnHeapColumnVector 5226 5263 53 78.4 12.8 1.0X
OffHeapColumnVector 5526 5699 244 74.1 13.5 0.9X
```
**After**
```
OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1022-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Test ColumnVectorUtils.populate, row length = 1: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------------
OnHeapColumnVector 3734 3742 11 109.7 9.1 1.0X
OffHeapColumnVector 3683 3683 0 111.2 9.0 1.0X
OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1022-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Test ColumnVectorUtils.populate, row length = 5: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------------
OnHeapColumnVector 4085 4088 4 100.3 10.0 1.0X
OffHeapColumnVector 4770 4771 2 85.9 11.6 0.9X
OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1022-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Test ColumnVectorUtils.populate, row length = 10: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------------------------------------------
OnHeapColumnVector 4788 4789 1 85.5 11.7 1.0X
OffHeapColumnVector 4387 4387 0 93.4 10.7 1.1X
OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1022-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Test ColumnVectorUtils.populate, row length = 15: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------------------------------------------
OnHeapColumnVector 4669 4669 0 87.7 11.4 1.0X
OffHeapColumnVector 5197 5198 1 78.8 12.7 0.9X
OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1022-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Test ColumnVectorUtils.populate, row length = 20: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------------------------------------------
OnHeapColumnVector 4769 4769 0 85.9 11.6 1.0X
OffHeapColumnVector 5441 5441 1 75.3 13.3 0.9X
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org