You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@commons.apache.org by "Michael Osipov (Jira)" <ji...@apache.org> on 2020/01/14 13:18:00 UTC
[jira] [Commented] (IO-650) Improve IOUtils performance by
increasing DEFAULT_BUFFER_SIZE
[ https://issues.apache.org/jira/browse/IO-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17015081#comment-17015081 ]
Michael Osipov commented on IO-650:
-----------------------------------
I can confirm the first test on the following machine:
{noformat}
$ sysctl -a | grep model
hw.model: Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz
$ uname -a
FreeBSD deblndw011x.ad001.siemens.net 12.1-STABLE FreeBSD 12.1-STABLE #4 r356437: Tue Jan 7 14:17:23 CET 2020 root@deblndw011x.ad001.siemens.net:/usr/obj/usr/src/amd64.amd64/sys/DEBLNDW011X amd64
{noformat}
with this output:
{noformat}
42726365 65077381
41092613 56539813
41340653 57771915
41260906 57100291
43628662 56817779
41679902 56969041
39059266 53804538
39170168 53087482
38555712 52766964
38928121 55344044
38839020 52901258
39127235 52106839
38814114 52770678
38421275 53190439
38819153 53332602
39053075 52326426
38799316 53072601
38743910 52785995
38653971 53132402
39314262 53287530
38647202 52809558
38805353 53193302
{noformat}
> Improve IOUtils performance by increasing DEFAULT_BUFFER_SIZE
> -------------------------------------------------------------
>
> Key: IO-650
> URL: https://issues.apache.org/jira/browse/IO-650
> Project: Commons IO
> Issue Type: Improvement
> Affects Versions: 1.0
> Reporter: Brett Lounsbury
> Priority: Major
> Fix For: 2.6
>
>
> IOUtils has a 4096B default buffer size that is used by copy() methods (and will be used for contentEquals methods when IO-649 is pulled). This number should be updated to 8192 for a few reasons:
> # It has a big improvement in performance in my micro-benchmark. I tested both copy() and contentEquals() with 4K and 8K buffers. This was done on a Late 2019 Macbook Pro (2.8GHz i7) with a 128MB file loaded into the OS buffer cache. See below for the test harness used. Past 8K performance does improve but it begins to experience some diminishing returns and could lead to excessive memory allocation.
> # It mirrors the default buffer size of java.io.Buffered* classes. This makes buffer sizing consistent regardless of if it is being done internally in the method or externally via a Buffered*. These classes are used internally in IOUtils as well so the buffer size is not unreasonable.
>
> For copy():
> |*Metric*|*8K Buffer Millisecond*|*4K Buffer Millisecond*|*Performance Improvement*|
> |*AVG*|44.2853417|64.2600679|0.31084197|
> |*P50*|42.692406|62.371984|0.31551951|
> |*P90*|49.5538826|68.4303876|0.27584975|
> |*P99*|62.8831473|89.759114|0.29942326|
> |*P100*|102.563615|177.143364|0.42101351|
>
> For contentEquals() with IO-649:
> |*Metric*|*8K Buffer Millisecond*|*4K Buffer Millisecond*|*Performance Improvement*|
> |*AVG*|81.5009567|128.497828|0.36574059|
> |*P50*|78.517749|124.191476|0.36776861|
> |*P90*|89.9172708|136.779763|0.34261276|
> |*P99*|125.814333|183.881989|0.31578762|
> |*P100*|308.936585|559.611217|0.44794426|
> {color:#000080}public static void {color}main(String[] args) {color:#000080}throws {color}Exception {
> NullOutputStream nos = NullOutputStream.{color:#660e7a}NULL_OUTPUT_STREAM{color};
> {color:#000080}for {color}({color:#000080}int {color}i = {color:#0000ff}0{color}; i < {color:#0000ff}1000{color}; i++) {
> InputStream fis = {color:#000080}new {color}FileInputStream({color:#000080}new {color}File({color:#008000}"/tmp/random_data"{color}));
> {color:#000080}long {color}start = System.nanoTime();
> IOUtils.copy(fis, nos);
> {color:#000080}long {color}defaultCopyTime = System.nanoTime() - start;
> fis = {color:#000080}new {color}FileInputStream({color:#000080}new {color}File({color:#008000}"/tmp/random_data"{color}));
> start = System.nanoTime();
> IOUtils.copy(fis, nos, {color:#0000ff}8192{color});
> {color:#000080}long {color}bufferSizeSpecifiedCopyTime = System.nanoTime() - start;
> System.{color:#660e7a}out{color}.println(bufferSizeSpecifiedCopyTime + {color:#008000}"{color}{color:#000080}\t{color}{color:#008000}" {color}+ defaultCopyTime);
> }
> }
> {color:#000080}public static void {color}main(String[] args) {color:#000080}throws {color}Exception {
> {color:#000080} for {color}({color:#000080}int {color}i = {color:#0000ff}0{color}; i < {color:#0000ff}1000{color}; i++) {
> InputStream fis = {color:#000080}new {color}FileInputStream({color:#000080}new {color}File({color:#008000}"/tmp/random_data"{color}));
> InputStream fis2 = {color:#000080}new {color}FileInputStream({color:#000080}new {color}File({color:#008000}"/tmp/random_data"{color}));
> {color:#000080}long {color}start = System.nanoTime();
> IOUtils.contentEquals(fis, fis2);
> {color:#000080}long {color}defaultContentEqualsTime = System.nanoTime() - start;
> fis = {color:#000080}new {color}FileInputStream({color:#000080}new {color}File({color:#008000}"/tmp/random_data"{color}));
> fis2 = {color:#000080}new {color}FileInputStream({color:#000080}new {color}File({color:#008000}"/tmp/random_data"{color}));
> start = System.nanoTime();
> IOUtils.contentEquals(fis, fis2, {color:#0000ff}8192{color});
> {color:#000080}long {color}bufferSizeSpecifiedContentEqualsTime = System.nanoTime() - start;
> System.{color:#660e7a}out{color}.println(bufferSizeSpecifiedContentEqualsTime + {color:#008000}"{color}{color:#000080}\t{color}{color:#008000}" {color}+ defaultContentEqualsTime);
> }
> }
--
This message was sent by Atlassian Jira
(v8.3.4#803005)