You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ant.apache.org by pkures <gi...@git.apache.org> on 2015/04/22 13:05:58 UTC
[GitHub] ant pull request: increase buffer size to improve transfer speed
GitHub user pkures opened a pull request:
https://github.com/apache/ant/pull/8
increase buffer size to improve transfer speed
1024 byte buffer size is insufficient for transfering large files over faster (> 100Mbit) networks. We are transfering files tens or hundreds MB in size and transfer speed was 1MB/s. After increasing the buffer size too 100KB we are getting transfer speeds up to 60MB/s. Please consider increasing the BUFFER_SIZE in the next Ant release. Thanks, Peter
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pkures/ant patch-2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/ant/pull/8.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #8
----
commit 815a36f52f2e1a2211b58d7eab31cdcb97709dee
Author: pkures <pk...@gmail.com>
Date: 2015-04-22T11:05:25Z
increase buffer size to improve transfer speed
1024 byte buffer size is insufficient for transfering large files over faster (> 100Mbit) networks. We are transfering files tens or hundreds MB in size and transfer speed was 1MB/s. After increasing the buffer size too 100KB we are getting transfer speeds up to 60MB/s. Please consider increasing the BUFFER_SIZE in the next Ant release. Thanks, Peter
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
RE: [GitHub] ant pull request: increase buffer size to improve
transfer speed
Posted by Loren Kratzke <LK...@blueorigin.com>.
Just my 2 cents...
In the year 2015, a 1K buffer is quite small. It is a bottleneck. I personally default to using 64K buffers unless I know for a fact that I am going to be transferring more than 1GB in which case I bump to a 1M or larger buffer.
64K or 128K was a lot of RAM back in 1982. I was there. In 1982 a 1K buffer was about the right size when you had 128K of RAM. These days 1K is not useful for anything. Peoples time is far more valuable and far less plentiful than this tiny amount of RAM.
Nobody will notice if they have 63K less free RAM for 30 seconds on a machine with 4GB, but they will notice when their files copy/transfer 60x faster as pkures has demonstrated. 1 second vs 1 minute? Increase the buffer size. Make it an option to have a larger (or smaller) buffer, but the default size should be a sensible default.
L.K.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
[GitHub] ant pull request: increase buffer size to improve transfer speed
Posted by bodewig <gi...@git.apache.org>.
Github user bodewig commented on the pull request:
https://github.com/apache/ant/pull/8#issuecomment-95677851
I think this is something we should make configurable via a task attribute rather than hardcode new default values.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
[GitHub] ant pull request: increase buffer size to improve transfer speed
Posted by ajmas <gi...@git.apache.org>.
Github user ajmas commented on a diff in the pull request:
https://github.com/apache/ant/pull/8#discussion_r28864296
--- Diff: src/main/org/apache/tools/ant/taskdefs/optional/ssh/ScpToMessage.java ---
@@ -36,7 +36,7 @@
public class ScpToMessage extends AbstractSshMessage {
private static final int HUNDRED_KILOBYTES = 102400;
- private static final int BUFFER_SIZE = 1024;
+ private static final int BUFFER_SIZE = 100*1024;
--- End diff --
Any reason you don't make BUFFER = HUNDRED_KILOBYTES, since that constant is defined just above?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
[GitHub] ant pull request: increase buffer size to improve transfer speed
Posted by ajmas <gi...@git.apache.org>.
Github user ajmas commented on the pull request:
https://github.com/apache/ant/pull/8#issuecomment-95704421
May want to look at this for a perspective: http://stackoverflow.com/questions/236861/how-do-you-determine-the-ideal-buffer-size-when-using-fileinputstream while
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
[GitHub] ant pull request: increase buffer size to improve transfer speed
Posted by pkures <gi...@git.apache.org>.
Github user pkures commented on a diff in the pull request:
https://github.com/apache/ant/pull/8#discussion_r28864813
--- Diff: src/main/org/apache/tools/ant/taskdefs/optional/ssh/ScpToMessage.java ---
@@ -36,7 +36,7 @@
public class ScpToMessage extends AbstractSshMessage {
private static final int HUNDRED_KILOBYTES = 102400;
- private static final int BUFFER_SIZE = 1024;
+ private static final int BUFFER_SIZE = 100*1024;
--- End diff --
Yes it's possible, but I didn't want to tie them together. The HUNDRED_KILOBYTES constant is used as a progress display threshold and IMHO should be named PROGRESS_DISPLAY_THRESHOLD or something like that, but that is left for another patch ;-) I would then increase the value of PROGRESS_DISPLAY_THRESHOLD to at least 1MB, on local network 100KB file is quite small and progress display is really unnecessary.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
[GitHub] ant pull request: increase buffer size to improve transfer speed
Posted by pkures <gi...@git.apache.org>.
Github user pkures commented on the pull request:
https://github.com/apache/ant/pull/8#issuecomment-95700146
Also Get task uses exactly sam buffer size when downloading files, probably for the same reasons
private static final int BIG_BUFFER_SIZE = 100 * 1024;
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
[GitHub] ant pull request: increase buffer size to improve transfer speed
Posted by bodewig <gi...@git.apache.org>.
Github user bodewig commented on the pull request:
https://github.com/apache/ant/pull/8#issuecomment-95691366
github pull requests are an option that we haven't used heavily, but it is fine. The "canonical aproach" would have been a Bugzilla issue with attachment (that's what we've been telling users for years now).
Also, if you want to get involved with Ant development (welcome!), please join the dev mailing list.
It's more about network packet size than memory. Maybe it's a sign of me being old but 100k feels a bit too big. Are you feeling up to the task of adding an attribute to the scp task?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
[GitHub] ant pull request: increase buffer size to improve transfer speed
Posted by ajmas <gi...@git.apache.org>.
Github user ajmas commented on the pull request:
https://github.com/apache/ant/pull/8#issuecomment-95706031
Last comment got cut short. How did you come about the 100k value? Has this been tested on networks with smaller through put? Consider the nature of data packet sizes. Not saying a bad value, though just trying to play it safe.
Making it configurable would certainly be good. Even wondering whether there are algorithms out there for auto tuning the buffer size?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
[GitHub] ant pull request: increase buffer size to improve transfer speed
Posted by pkures <gi...@git.apache.org>.
Github user pkures commented on the pull request:
https://github.com/apache/ant/pull/8#issuecomment-95852323
Cool, thank you.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
[GitHub] ant pull request: increase buffer size to improve transfer speed
Posted by pkures <gi...@git.apache.org>.
Github user pkures commented on the pull request:
https://github.com/apache/ant/pull/8#issuecomment-95697171
Thank you for tips how to do it more conventionally in the future. I will try to look at the code over the weekend and maybe add the attribtue to configure buffer size if I have the time to do it.
Are you proposing to leave the default at 1KB and only allow to change it by attribute ?
When you are receiving the data over 1Gbit you need either large number of small buffers or one larger buffer. You will not handle ten thousand 1 KB packets/s from Java easiliy, I started programming on computers that had much less memory than 100 KB so I can understand your concerns :) But again 1MB/s on such fast network is also not optimal and our current machines have gigabytes of memory.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
[GitHub] ant pull request: increase buffer size to improve transfer speed
Posted by bodewig <gi...@git.apache.org>.
Github user bodewig commented on the pull request:
https://github.com/apache/ant/pull/8#issuecomment-95794438
I appreciate the offer of doing more research, but it's not necessary.
The argument about the <ftp> task made me look into the other tasks we use for transferring files over the network - <get>also uses 100k. I'm going to merge this and #9 int the next days (unless anybody else beats me to it).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
[GitHub] ant pull request: increase buffer size to improve transfer speed
Posted by pkures <gi...@git.apache.org>.
Github user pkures commented on the pull request:
https://github.com/apache/ant/pull/8#issuecomment-95680061
You are right, that it should be configurable, but the default value really should be something more sensible than 1KB. There is no reason to use such small buffer, memory consumption is really not an issue here and increasing it will help most users on LANs and over internet (if they don't use dial-up connection ;-). Also, as this is my first attempt at contributing towards a major open source project I afraid to propose larger changes than changing the default value :) Also, thank you for responding. I really wasn't sure if this is the right way to propose modifications to Ant codebase.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
[GitHub] ant pull request: increase buffer size to improve transfer speed
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/ant/pull/8
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
[GitHub] ant pull request: increase buffer size to improve transfer speed
Posted by pkures <gi...@git.apache.org>.
Github user pkures commented on the pull request:
https://github.com/apache/ant/pull/8#issuecomment-95718851
I was thinking along the lines of ... the transfer speed is 1MB/s but theoretically on 1 Gbit network you should see up to 100MB/s transfer speed ideally, so I tried to increase 100 times. It's about how often Java gets a chance to process InputStream data and how many of those 1K packets arrive between OS context switches.
Increasing the buffer size from 1K to 100K increased the transfer speed to 75MB/s (very large file), so the buffer size really does matter a lot. I can look how much the InputStream read method reads in one call on average - this should tell us how large the buffer needs to be for gigabit network. I can also test over our 15Mbit link, but I'am afraid that's slowest connection I have available except maybe cellular internet, hmmm, maybe I can test that also.
InputStream.read - Reads up to len bytes of data from the input stream into an array of bytes.
In my experience it means that on slow network the only risk is that the excessive buffer space will be wasted for the duration of the transfer. As it will not wait for the buffer to fill.
But this is really getting little bit complicated for my time constraints :-) Maybe just adding the attribute to override the defaul buffer size is not a bad option after all.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org