You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/08/11 07:25:05 UTC
[GitHub] [spark] LuciferYang commented on pull request #37480: [SPARK-40046][SQL] Use Jackson instead of json4s to serialize `RocksDBMetrics`
LuciferYang commented on PR #37480:
URL: https://github.com/apache/spark/pull/37480#issuecomment-1211638145
Although this is a minor change, in order not to have a negative impact, I simply compared the performance as follows:
**Bench Code**
```
val valuesPerIteration = 100000
val jsonString =
"""{"numCommittedKeys":1,"numUncommittedKeys":1,"totalMemUsageBytes":3976,
|"writeBatchMemUsageBytes":17,"totalSSTFilesBytes":1146,
|"nativeOpsHistograms":{"get":{"sum":7,"avg":3.5,"stddev":0.5,
|"median":3.0,"p95":3.9,"p99":3.98,"count":2},"put":{"sum":17927,
|"avg":17927.0,"stddev":0.0,"median":17927.0,"p95":17927.0,
|"p99":17927.0,"count":1},"compaction":{"sum":0,"avg":0.0,
|"stddev":0.0,"median":0.0,"p95":0.0,"p99":0.0,"count":0}},
|"lastCommitLatencyMs":{"fileSync":595,"writeBatch":17,
|"flush":60,"pause":0,"checkpoint":64,"compact":0},"filesCopied":1,
|"bytesCopied":1146,"filesReused":0,"zipFileBytesUncompressed":6973,
|"nativeOpsMetrics":{"writerStallDuration":0,"totalBytesReadThroughIterator":0,
|"readBlockCacheHitCount":0,"totalBytesWrittenByCompaction":0,
|"readBlockCacheMissCount":0,"totalBytesReadByCompaction":0,
|"totalBytesWritten":17,"totalBytesRead":0}}""".stripMargin
val metrics: RocksDBMetrics = new ObjectMapper().registerModule(DefaultScalaModule)
.readValue(jsonString, classOf[RocksDBMetrics])
val benchmark = new Benchmark("Test RocksDBMetrics to Json",
valuesPerIteration, output = output)
benchmark.addCase("Use Json4s") { _: Int =>
for (_ <- 0L until valuesPerIteration) {
metrics.json
}
}
val mapper = new ObjectMapper().registerModule(DefaultScalaModule)
benchmark.addCase("Use Jackson") { _: Int =>
for (_ <- 0L until valuesPerIteration) {
mapper.writeValueAsString(metrics)
}
}
benchmark.run()
```
**Java8**
```
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1014-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Test RocksDBMetrics to Json: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Json4s 4078 4091 17 0.0 40785.0 1.0X
Use Jackson 384 386 1 0.3 3837.3 10.6X
```
**Java11**
```
OpenJDK 64-Bit Server VM 11.0.16+8-LTS on Linux 5.15.0-1014-azure
Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Test RocksDBMetrics to Json: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Json4s 3361 3362 2 0.0 33606.9 1.0X
Use Jackson 361 362 1 0.3 3610.3 9.3X
```
**Java17**
```
OpenJDK 64-Bit Server VM 17.0.4+8-LTS on Linux 5.15.0-1014-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Test RocksDBMetrics to Json: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Use Json4s 3787 3792 7 0.0 37870.2 1.0X
Use Jackson 362 367 5 0.3 3618.8 10.5X
```
Jackson looks better, though that's not the key point
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org