You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@commons.apache.org by "Brett Lounsbury (Jira)" <ji...@apache.org> on 2020/01/13 22:06:00 UTC
[jira] [Updated] (IO-650) Improve IOUtils performance by increasing
DEFAULT_BUFFER_SIZE
[ https://issues.apache.org/jira/browse/IO-650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brett Lounsbury updated IO-650:
-------------------------------
Description:
IOUtils has a 4096B default buffer size that is used by copy() methods (and will be used for contentEquals methods when IO-649 is pulled). This number should be updated to 8192 for a few reasons:
# It has a big improvement in performance in my micro-benchmark. I tested both copy() and contentEquals() with 4K and 8K buffers. This was done on a Late 2019 Macbook Pro (2.8GHz i7) with a 128MB file loaded into the OS buffer cache. See below for the test harness used. Past 8K performance does improve but it begins to experience some diminishing returns and could lead to excessive memory allocation.
# It mirrors the default buffer size of java.io.Buffered* classes. This makes buffer sizing consistent regardless of if it is being done internally in the method or externally via a Buffered*. These classes are used internally in IOUtils as well so the buffer size is not unreasonable.
For copy():
|*Metric*|*8K Buffer Millisecond*|*4K Buffer Millisecond*|*Performance Improvement*|
|*AVG*|44.2853417|64.2600679|0.31084197|
|*P50*|42.692406|62.371984|0.31551951|
|*P90*|49.5538826|68.4303876|0.27584975|
|*P99*|62.8831473|89.759114|0.29942326|
|*P100*|102.563615|177.143364|0.42101351|
For contentEquals() with IO-649:
|*Metric*|*8K Buffer Millisecond*|*4K Buffer Millisecond*|*Performance Improvement*|
|*AVG*|81.5009567|128.497828|0.36574059|
|*P50*|78.517749|124.191476|0.36776861|
|*P90*|89.9172708|136.779763|0.34261276|
|*P99*|125.814333|183.881989|0.31578762|
|*P100*|308.936585|559.611217|0.44794426|
```
{color:#000080}public static void {color}main(String[] args) {color:#000080}throws {color}Exception {
NullOutputStream nos = NullOutputStream.{color:#660e7a}NULL_OUTPUT_STREAM{color};
{color:#000080}for {color}({color:#000080}int {color}i = {color:#0000ff}0{color}; i < {color:#0000ff}1000{color}; i++) {
InputStream fis = {color:#000080}new {color}FileInputStream({color:#000080}new {color}File({color:#008000}"/tmp/random_data"{color}));
{color:#000080}long {color}start = System.nanoTime();
IOUtils.copy(fis, nos);
{color:#000080}long {color}defaultCopyTime = System.nanoTime() - start;
fis = {color:#000080}new {color}FileInputStream({color:#000080}new {color}File({color:#008000}"/tmp/random_data"{color}));
start = System.nanoTime();
IOUtils.copy(fis, nos, {color:#0000ff}8192{color});
{color:#000080}long {color}bufferSizeSpecifiedCopyTime = System.nanoTime() - start;
System.{color:#660e7a}out{color}.println(bufferSizeSpecifiedCopyTime + {color:#008000}"{color}{color:#000080}\t{color}{color:#008000}" {color}+ defaultCopyTime);
}
}```
```
{color:#000080}public static void {color}main(String[] args) {color:#000080}throws {color}Exception {
{color:#000080} for {color}({color:#000080}int {color}i = {color:#0000ff}0{color}; i < {color:#0000ff}1000{color}; i++) {
InputStream fis = {color:#000080}new {color}FileInputStream({color:#000080}new {color}File({color:#008000}"/tmp/random_data"{color}));
InputStream fis2 = {color:#000080}new {color}FileInputStream({color:#000080}new {color}File({color:#008000}"/tmp/random_data"{color}));
{color:#000080}long {color}start = System.nanoTime();
IOUtils.contentEquals(fis, fis2)
{color:#000080}long {color}defaultContentEqualsTime = System.nanoTime() - start;
fis = {color:#000080}new {color}FileInputStream({color:#000080}new {color}File({color:#008000}"/tmp/random_data"{color}));
fis2 = {color:#000080}new {color}FileInputStream({color:#000080}new {color}File({color:#008000}"/tmp/random_data"{color}));
start = System.nanoTime();
IOUtils.contentEquals(fis, fis2, {color:#0000ff}8192{color});
{color:#000080}long {color}bufferSizeSpecifiedContentEqualsTime = System.nanoTime() - start;
System.{color:#660e7a}out{color}.println(bufferSizeSpecifiedContentEqualsTime + {color:#008000}"{color}{color:#000080}\t{color}{color:#008000}" {color}+ defaultContentEqualsTime);
}
}```
was:
IOUtils has a 4096B default buffer size that is used by copy() methods (and will be used for contentEquals methods when IO-649 is pulled). This number should be updated to 8192 for a few reasons:
# It has a big improvement in performance in my micro-benchmark. I tested both copy() and contentEquals() with 4K and 8K buffers. This was done on a Late 2019 Macbook Pro (2.8GHz i7) with a 128MB file loaded into the OS buffer cache. See below for the test harness used. Past 8K performance does improve but it begins to experience some diminishing returns and could lead to excessive memory allocation.
# It mirrors the default buffer size of java.io.Buffered* classes. These classes are used internally in IOUtils as well so the buffer size is not unreasonable.
For copy():
|*Metric*|*8K Buffer Millisecond*|*4K Buffer Millisecond*|*Performance Improvement*|
|*AVG*|44.2853417|64.2600679|0.31084197|
|*P50*|42.692406|62.371984|0.31551951|
|*P90*|49.5538826|68.4303876|0.27584975|
|*P99*|62.8831473|89.759114|0.29942326|
|*P100*|102.563615|177.143364|0.42101351|
For contentEquals() with IO-649:
|*Metric*|*8K Buffer Millisecond*|*4K Buffer Millisecond*|*Performance Improvement*|
|*AVG*|81.5009567|128.497828|0.36574059|
|*P50*|78.517749|124.191476|0.36776861|
|*P90*|89.9172708|136.779763|0.34261276|
|*P99*|125.814333|183.881989|0.31578762|
|*P100*|308.936585|559.611217|0.44794426|
```
{color:#000080}public static void {color}main(String[] args) {color:#000080}throws {color}Exception {
NullOutputStream nos = NullOutputStream.{color:#660e7a}NULL_OUTPUT_STREAM{color};
{color:#000080}for {color}({color:#000080}int {color}i = {color:#0000ff}0{color}; i < {color:#0000ff}1000{color}; i++) {
InputStream fis = {color:#000080}new {color}FileInputStream({color:#000080}new {color}File({color:#008000}"/tmp/random_data"{color}));
{color:#000080}long {color}start = System.nanoTime();
IOUtils.copy(fis, nos);
{color:#000080}long {color}defaultCopyTime = System.nanoTime() - start;
fis = {color:#000080}new {color}FileInputStream({color:#000080}new {color}File({color:#008000}"/tmp/random_data"{color}));
start = System.nanoTime();
IOUtils.copy(fis, nos, {color:#0000ff}8192{color});
{color:#000080}long {color}bufferSizeSpecifiedCopyTime = System.nanoTime() - start;
System.{color:#660e7a}out{color}.println(bufferSizeSpecifiedCopyTime + {color:#008000}"{color}{color:#000080}\t{color}{color:#008000}" {color}+ defaultCopyTime);
}
}```
```
{color:#000080}public static void {color}main(String[] args) {color:#000080}throws {color}Exception {
{color:#000080} for {color}({color:#000080}int {color}i = {color:#0000ff}0{color}; i < {color:#0000ff}1000{color}; i++) {
InputStream fis = {color:#000080}new {color}FileInputStream({color:#000080}new {color}File({color:#008000}"/tmp/random_data"{color}));
InputStream fis2 = {color:#000080}new {color}FileInputStream({color:#000080}new {color}File({color:#008000}"/tmp/random_data"{color}));
{color:#000080}long {color}start = System.nanoTime();
IOUtils.contentEquals(fis, fis2)
{color:#000080}long {color}defaultContentEqualsTime = System.nanoTime() - start;
fis = {color:#000080}new {color}FileInputStream({color:#000080}new {color}File({color:#008000}"/tmp/random_data"{color}));
fis2 = {color:#000080}new {color}FileInputStream({color:#000080}new {color}File({color:#008000}"/tmp/random_data"{color}));
start = System.nanoTime();
IOUtils.contentEquals(fis, fis2, {color:#0000ff}8192{color});
{color:#000080}long {color}bufferSizeSpecifiedContentEqualsTime = System.nanoTime() - start;
System.{color:#660e7a}out{color}.println(bufferSizeSpecifiedContentEqualsTime + {color:#008000}"{color}{color:#000080}\t{color}{color:#008000}" {color}+ defaultContentEqualsTime);
}
}```
> Improve IOUtils performance by increasing DEFAULT_BUFFER_SIZE
> -------------------------------------------------------------
>
> Key: IO-650
> URL: https://issues.apache.org/jira/browse/IO-650
> Project: Commons IO
> Issue Type: Improvement
> Affects Versions: 1.0
> Reporter: Brett Lounsbury
> Priority: Major
> Fix For: 2.6
>
>
> IOUtils has a 4096B default buffer size that is used by copy() methods (and will be used for contentEquals methods when IO-649 is pulled). This number should be updated to 8192 for a few reasons:
> # It has a big improvement in performance in my micro-benchmark. I tested both copy() and contentEquals() with 4K and 8K buffers. This was done on a Late 2019 Macbook Pro (2.8GHz i7) with a 128MB file loaded into the OS buffer cache. See below for the test harness used. Past 8K performance does improve but it begins to experience some diminishing returns and could lead to excessive memory allocation.
> # It mirrors the default buffer size of java.io.Buffered* classes. This makes buffer sizing consistent regardless of if it is being done internally in the method or externally via a Buffered*. These classes are used internally in IOUtils as well so the buffer size is not unreasonable.
>
> For copy():
> |*Metric*|*8K Buffer Millisecond*|*4K Buffer Millisecond*|*Performance Improvement*|
> |*AVG*|44.2853417|64.2600679|0.31084197|
> |*P50*|42.692406|62.371984|0.31551951|
> |*P90*|49.5538826|68.4303876|0.27584975|
> |*P99*|62.8831473|89.759114|0.29942326|
> |*P100*|102.563615|177.143364|0.42101351|
>
> For contentEquals() with IO-649:
> |*Metric*|*8K Buffer Millisecond*|*4K Buffer Millisecond*|*Performance Improvement*|
> |*AVG*|81.5009567|128.497828|0.36574059|
> |*P50*|78.517749|124.191476|0.36776861|
> |*P90*|89.9172708|136.779763|0.34261276|
> |*P99*|125.814333|183.881989|0.31578762|
> |*P100*|308.936585|559.611217|0.44794426|
>
> ```
> {color:#000080}public static void {color}main(String[] args) {color:#000080}throws {color}Exception {
> NullOutputStream nos = NullOutputStream.{color:#660e7a}NULL_OUTPUT_STREAM{color};
> {color:#000080}for {color}({color:#000080}int {color}i = {color:#0000ff}0{color}; i < {color:#0000ff}1000{color}; i++) {
> InputStream fis = {color:#000080}new {color}FileInputStream({color:#000080}new {color}File({color:#008000}"/tmp/random_data"{color}));
> {color:#000080}long {color}start = System.nanoTime();
> IOUtils.copy(fis, nos);
> {color:#000080}long {color}defaultCopyTime = System.nanoTime() - start;
> fis = {color:#000080}new {color}FileInputStream({color:#000080}new {color}File({color:#008000}"/tmp/random_data"{color}));
> start = System.nanoTime();
> IOUtils.copy(fis, nos, {color:#0000ff}8192{color});
> {color:#000080}long {color}bufferSizeSpecifiedCopyTime = System.nanoTime() - start;
> System.{color:#660e7a}out{color}.println(bufferSizeSpecifiedCopyTime + {color:#008000}"{color}{color:#000080}\t{color}{color:#008000}" {color}+ defaultCopyTime);
> }
> }```
>
> ```
> {color:#000080}public static void {color}main(String[] args) {color:#000080}throws {color}Exception {
> {color:#000080} for {color}({color:#000080}int {color}i = {color:#0000ff}0{color}; i < {color:#0000ff}1000{color}; i++) {
> InputStream fis = {color:#000080}new {color}FileInputStream({color:#000080}new {color}File({color:#008000}"/tmp/random_data"{color}));
> InputStream fis2 = {color:#000080}new {color}FileInputStream({color:#000080}new {color}File({color:#008000}"/tmp/random_data"{color}));
> {color:#000080}long {color}start = System.nanoTime();
> IOUtils.contentEquals(fis, fis2)
> {color:#000080}long {color}defaultContentEqualsTime = System.nanoTime() - start;
> fis = {color:#000080}new {color}FileInputStream({color:#000080}new {color}File({color:#008000}"/tmp/random_data"{color}));
> fis2 = {color:#000080}new {color}FileInputStream({color:#000080}new {color}File({color:#008000}"/tmp/random_data"{color}));
> start = System.nanoTime();
> IOUtils.contentEquals(fis, fis2, {color:#0000ff}8192{color});
> {color:#000080}long {color}bufferSizeSpecifiedContentEqualsTime = System.nanoTime() - start;
> System.{color:#660e7a}out{color}.println(bufferSizeSpecifiedContentEqualsTime + {color:#008000}"{color}{color:#000080}\t{color}{color:#008000}" {color}+ defaultContentEqualsTime);
> }
> }```
--
This message was sent by Atlassian Jira
(v8.3.4#803005)