You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2019/02/01 09:50:46 UTC
[GitHub] liupc edited a comment on issue #23602:
[SPARK-26674][CORE]Consolidate CompositeByteBuf when reading large frame
liupc edited a comment on issue #23602: [SPARK-26674][CORE]Consolidate CompositeByteBuf when reading large frame
URL: https://github.com/apache/spark/pull/23602#issuecomment-459666639
I have done some benchmark tests on my local machine, seems it can save large memory with small time cost -- ~ 200 milis for 50% saving from a 1GB CompositeByteBuf
--------------------------------------------------------------------------------------------------------
Java HotSpot(TM) 64-Bit Server VM 1.8.0_111-b14 on Linux 4.15.0-43-generic
Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz
[test consolidate 100 buffers each with 10m, 50% used for 1 loop]
Allocating 5242880 bytes
Time cost with 1 loop for consolidating: 223 millis
[test consolidate 100 buffers each with 10m, 100% used for 1 loop]
Allocating 10485760 bytes
Time cost with 1 loop for consolidating: 451 millis
[test consolidate 100 buffers each with 10m, 50% used for 10 loop]
Allocating 5242880 bytes
Time cost with 10 loop for consolidating: 1924 millis
[test consolidate 100 buffers each with 10m, 100% used for 10 loop]
Allocating 10485760 bytes
Time cost with 10 loop for consolidating: 3787 millis
[test consolidate 100 buffers each with 10m, 50% used for 50 loop]
Allocating 5242880 bytes
Time cost with 50 loop for consolidating: 8870 millis
[test consolidate 100 buffers each with 10m, 100% used for 50 loop]
Allocating 10485760 bytes
Time cost with 50 loop for consolidating: 17472 millis
[test consolidate 20 buffers each with 50m, 50% used for 1 loop]
Allocating 26214400 bytes
Time cost with 1 loop for consolidating: 184 millis
[test consolidate 20 buffers each with 50m, 100% used for 1 loop]
Allocating 52428800 bytes
Time cost with 1 loop for consolidating: 367 millis
[test consolidate 20 buffers each with 50m, 50% used for 10 loop]
Allocating 26214400 bytes
Time cost with 10 loop for consolidating: 1847 millis
[test consolidate 20 buffers each with 50m, 100% used for 10 loop]
Allocating 52428800 bytes
Time cost with 10 loop for consolidating: 3638 millis
[test consolidate 20 buffers each with 50m, 50% used for 50 loop]
Allocating 26214400 bytes
Time cost with 50 loop for consolidating: 9126 millis
[test consolidate 20 buffers each with 50m, 100% used for 50 loop]
Allocating 52428800 bytes
Time cost with 50 loop for consolidating: 19391 millis
[test consolidate 10 buffers each with 100m, 50% used for 1 loop]
Allocating 52428800 bytes
Time cost with 1 loop for consolidating: 211 millis
[test consolidate 10 buffers each with 100m, 100% used for 1 loop]
Allocating 104857600 bytes
Time cost with 1 loop for consolidating: 400 millis
[test consolidate 10 buffers each with 100m, 50% used for 10 loop]
Allocating 52428800 bytes
Time cost with 10 loop for consolidating: 1954 millis
[test consolidate 10 buffers each with 100m, 100% used for 10 loop]
Allocating 104857600 bytes
Time cost with 10 loop for consolidating: 3846 millis
[test consolidate 10 buffers each with 100m, 50% used for 50 loop]
Allocating 52428800 bytes
Time cost with 50 loop for consolidating: 9747 millis
[test consolidate 10 buffers each with 100m, 100% used for 50 loop]
Allocating 104857600 bytes
Time cost with 50 loop for consolidating: 19542 millis
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org