You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by "Boxer, Aaron" <Aa...@uhn.ca> on 2014/01/16 05:33:31 UTC
ZeroCopyPut mystery
Hello,
When I use ZeroCopyPut with hard drive (spinning disk ) disk files, I get a network transfer rate of about 250 MBPS for first time read of files; 850 MBPS if the files are already in the Windows OS file cache. Synchronous put gives me about 30 MBPS.
But, when I am dealing with files on DVD, I get a network transfer rate of 5 MBPS for first time read of files, and 850 MBPS if files are in the OS file cache.
Synchronous put gives me about 30 MBPS.
So, in the case of first read of DVD files, synchronous client is superior. Synchronous should be faster, because it can read and send at the same time.
Does this make sense? It seems a little strange to me.
Thanks,
Aaron
This e-mail may contain confidential and/or privileged information for the sole use of the intended recipient.
Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited.
If you have received this e-mail in error, please contact the sender and delete all copies.
Opinions, conclusions or other information contained in this e-mail may not be that of the organization.
RE: ZeroCopyPut mystery
Posted by "Boxer, Aaron" <Aa...@uhn.ca>.
> If it is only slower for DVD copies, that seems more like an issue with DVD caching.
>
> Are you sure that the synch and asynch tests both started from exactly the same environment?
>
> What do you mean by environment? Both tests read the same DVD, after flushing the Windows OS file cache.
I meant the phsyical environment.
There are other potential hardware caches. For example the DVD drive.
> But, the clients are, of course, set up differently. Would you like me to post how I am configuring the clients?
>
What about the DVD cache?
Did you reset the drive?
Did you repeat the tests multiple times?
Sometimes with async first, other times with sync first?
Yes, this is completely repeatable. Each run, I clear the OS file cache, and each time, asynch is about the same slow rate
compared to synch.
>
>
>
>> Perhaps it has to do with buffer size, thread count, ..... ?
>>
>> -----Original Message-----
>> From: Oleg Kalnichevski [mailto:olegk@apache.org]
>> Sent: Thursday, January 16, 2014 11:15 AM
>> To: HttpClient User Discussion
>> Subject: Re: ZeroCopyPut mystery
>>
>> On Thu, 2014-01-16 at 04:33 +0000, Boxer, Aaron wrote:
>>> Hello,
>>>
>>> When I use ZeroCopyPut with hard drive (spinning disk ) disk files, I get a network transfer rate of about 250 MBPS for first time read of files; 850 MBPS if the files are already in the Windows OS file cache. Synchronous put gives me about 30 MBPS.
>>>
>>> But, when I am dealing with files on DVD, I get a network transfer rate of 5 MBPS for first time read of files, and 850 MBPS if files are in the OS file cache.
>>> Synchronous put gives me about 30 MBPS.
>>>
>>> So, in the case of first read of DVD files, synchronous client is superior. Synchronous should be faster, because it can read and send at the same time.
>>>
>>> Does this make sense? It seems a little strange to me.
>>>
>>
>> Aaron,
>>
>> Zero copy data transfer is very platform specific. Java merely provides a common APIs, which however can have radically different implementations on different platforms and therefore can have different performance characteristics.
>>
>> I guess these numbers have more to do with Windows than with HttpAsyncClient.
>>
>> Oleg
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>
>>
>> This e-mail may contain confidential and/or privileged information for the sole use of the intended recipient.
>> Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited.
>> If you have received this e-mail in error, please contact the sender and delete all copies.
>> Opinions, conclusions or other information contained in this e-mail may not be that of the organization.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> This e-mail may contain confidential and/or privileged information for the sole use of the intended recipient.
> Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited.
> If you have received this e-mail in error, please contact the sender and delete all copies.
> Opinions, conclusions or other information contained in this e-mail may not be that of the organization.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
This e-mail may contain confidential and/or privileged information for the sole use of the intended recipient.
Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited.
If you have received this e-mail in error, please contact the sender and delete all copies.
Opinions, conclusions or other information contained in this e-mail may not be that of the organization.
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: ZeroCopyPut mystery
Posted by sebb <se...@gmail.com>.
On 16 January 2014 16:54, Boxer, Aaron <Aa...@uhn.ca> wrote:
> On 16 January 2014 16:35, Boxer, Aaron <Aa...@uhn.ca> wrote:
>> Thanks, Oleg. I am not trying to criticize the project in any way;
>> just wondering if anyone has insight into why, on Windows, Java NIO is so much slower than stream IO.
>
> If it is only slower for DVD copies, that seems more like an issue with DVD caching.
>
> Are you sure that the synch and asynch tests both started from exactly the same environment?
>
> What do you mean by environment? Both tests read the same DVD, after flushing the Windows OS file cache.
I meant the phsyical environment.
There are other potential hardware caches. For example the DVD drive.
> But, the clients are, of course, set up differently. Would you like me to post how I am configuring the clients?
>
What about the DVD cache?
Did you reset the drive?
Did you repeat the tests multiple times?
Sometimes with async first, other times with sync first?
>
>
>
>> Perhaps it has to do with buffer size, thread count, ..... ?
>>
>> -----Original Message-----
>> From: Oleg Kalnichevski [mailto:olegk@apache.org]
>> Sent: Thursday, January 16, 2014 11:15 AM
>> To: HttpClient User Discussion
>> Subject: Re: ZeroCopyPut mystery
>>
>> On Thu, 2014-01-16 at 04:33 +0000, Boxer, Aaron wrote:
>>> Hello,
>>>
>>> When I use ZeroCopyPut with hard drive (spinning disk ) disk files, I get a network transfer rate of about 250 MBPS for first time read of files; 850 MBPS if the files are already in the Windows OS file cache. Synchronous put gives me about 30 MBPS.
>>>
>>> But, when I am dealing with files on DVD, I get a network transfer rate of 5 MBPS for first time read of files, and 850 MBPS if files are in the OS file cache.
>>> Synchronous put gives me about 30 MBPS.
>>>
>>> So, in the case of first read of DVD files, synchronous client is superior. Synchronous should be faster, because it can read and send at the same time.
>>>
>>> Does this make sense? It seems a little strange to me.
>>>
>>
>> Aaron,
>>
>> Zero copy data transfer is very platform specific. Java merely provides a common APIs, which however can have radically different implementations on different platforms and therefore can have different performance characteristics.
>>
>> I guess these numbers have more to do with Windows than with HttpAsyncClient.
>>
>> Oleg
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>
>>
>> This e-mail may contain confidential and/or privileged information for the sole use of the intended recipient.
>> Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited.
>> If you have received this e-mail in error, please contact the sender and delete all copies.
>> Opinions, conclusions or other information contained in this e-mail may not be that of the organization.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> This e-mail may contain confidential and/or privileged information for the sole use of the intended recipient.
> Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited.
> If you have received this e-mail in error, please contact the sender and delete all copies.
> Opinions, conclusions or other information contained in this e-mail may not be that of the organization.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: ZeroCopyPut mystery
Posted by "Boxer, Aaron" <Aa...@uhn.ca>.
On 16 January 2014 16:35, Boxer, Aaron <Aa...@uhn.ca> wrote:
> Thanks, Oleg. I am not trying to criticize the project in any way;
> just wondering if anyone has insight into why, on Windows, Java NIO is so much slower than stream IO.
If it is only slower for DVD copies, that seems more like an issue with DVD caching.
Are you sure that the synch and asynch tests both started from exactly the same environment?
What do you mean by environment? Both tests read the same DVD, after flushing the Windows OS file cache.
But, the clients are, of course, set up differently. Would you like me to post how I am configuring the clients?
> Perhaps it has to do with buffer size, thread count, ..... ?
>
> -----Original Message-----
> From: Oleg Kalnichevski [mailto:olegk@apache.org]
> Sent: Thursday, January 16, 2014 11:15 AM
> To: HttpClient User Discussion
> Subject: Re: ZeroCopyPut mystery
>
> On Thu, 2014-01-16 at 04:33 +0000, Boxer, Aaron wrote:
>> Hello,
>>
>> When I use ZeroCopyPut with hard drive (spinning disk ) disk files, I get a network transfer rate of about 250 MBPS for first time read of files; 850 MBPS if the files are already in the Windows OS file cache. Synchronous put gives me about 30 MBPS.
>>
>> But, when I am dealing with files on DVD, I get a network transfer rate of 5 MBPS for first time read of files, and 850 MBPS if files are in the OS file cache.
>> Synchronous put gives me about 30 MBPS.
>>
>> So, in the case of first read of DVD files, synchronous client is superior. Synchronous should be faster, because it can read and send at the same time.
>>
>> Does this make sense? It seems a little strange to me.
>>
>
> Aaron,
>
> Zero copy data transfer is very platform specific. Java merely provides a common APIs, which however can have radically different implementations on different platforms and therefore can have different performance characteristics.
>
> I guess these numbers have more to do with Windows than with HttpAsyncClient.
>
> Oleg
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> This e-mail may contain confidential and/or privileged information for the sole use of the intended recipient.
> Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited.
> If you have received this e-mail in error, please contact the sender and delete all copies.
> Opinions, conclusions or other information contained in this e-mail may not be that of the organization.
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
This e-mail may contain confidential and/or privileged information for the sole use of the intended recipient.
Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited.
If you have received this e-mail in error, please contact the sender and delete all copies.
Opinions, conclusions or other information contained in this e-mail may not be that of the organization.
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: ZeroCopyPut mystery
Posted by sebb <se...@gmail.com>.
On 16 January 2014 16:35, Boxer, Aaron <Aa...@uhn.ca> wrote:
> Thanks, Oleg. I am not trying to criticize the project in any way; just wondering if anyone has insight into why, on Windows,
> Java NIO is so much slower than stream IO.
If it is only slower for DVD copies, that seems more like an issue
with DVD caching.
Are you sure that the synch and asynch tests both started from exactly
the same environment?
> Perhaps it has to do with buffer size, thread count, ..... ?
>
> -----Original Message-----
> From: Oleg Kalnichevski [mailto:olegk@apache.org]
> Sent: Thursday, January 16, 2014 11:15 AM
> To: HttpClient User Discussion
> Subject: Re: ZeroCopyPut mystery
>
> On Thu, 2014-01-16 at 04:33 +0000, Boxer, Aaron wrote:
>> Hello,
>>
>> When I use ZeroCopyPut with hard drive (spinning disk ) disk files, I get a network transfer rate of about 250 MBPS for first time read of files; 850 MBPS if the files are already in the Windows OS file cache. Synchronous put gives me about 30 MBPS.
>>
>> But, when I am dealing with files on DVD, I get a network transfer rate of 5 MBPS for first time read of files, and 850 MBPS if files are in the OS file cache.
>> Synchronous put gives me about 30 MBPS.
>>
>> So, in the case of first read of DVD files, synchronous client is superior. Synchronous should be faster, because it can read and send at the same time.
>>
>> Does this make sense? It seems a little strange to me.
>>
>
> Aaron,
>
> Zero copy data transfer is very platform specific. Java merely provides a common APIs, which however can have radically different implementations on different platforms and therefore can have different performance characteristics.
>
> I guess these numbers have more to do with Windows than with HttpAsyncClient.
>
> Oleg
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> This e-mail may contain confidential and/or privileged information for the sole use of the intended recipient.
> Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited.
> If you have received this e-mail in error, please contact the sender and delete all copies.
> Opinions, conclusions or other information contained in this e-mail may not be that of the organization.
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Password exposure in logs
Posted by Bob Anderson <mu...@gmail.com>.
Hi,
I know that the password is exposed in clear text on some versions of the http client in debug. Can you tell me what versions? Also it appears this also happens on some errors. Can you send an info on this. Any areas where password are exposed in the clear I need information on.
Thanks,
Bob
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: ZeroCopyPut mystery
Posted by "Boxer, Aaron" <Aa...@uhn.ca>.
I know this is an old thread, but I wanted to relate the final scheme I came up with for optimal uploading of DVD files on windows,
in case anyone else has to deal with this issue:
1) issue Powershell command from Java to recursively list all files in a root directory, and copy each file to dev null, as follows:
String cmd = "powershell.exe \"Get-Childitem -Recurse " + rootDirectory +
" | where {!$_.PsIsContainer} | %{ Get-Content $_.FullName > $null; echo $_.FullName;} \" ";
2) Notice the echo command at the end; now that the file is in the windows file cache, I can read this echoed name in java and issue an asynchronous zero copy put on the file via
httpasyncclient (this is now very fast, since the file is in the cache)
So, this method pipelines the reads and puts, so that put time can be ignored, and time to upload is just the time to read the files from DVD.
Cheers,
Aaron
-----Original Message-----
From: Boxer, Aaron
Sent: Friday, January 17, 2014 3:26 PM
To: HttpClient User Discussion
Subject: RE: ZeroCopyPut mystery
On Fri, 2014-01-17 at 16:28 +0000, Boxer, Aaron wrote:
> I made a simple test: I wrote a few lines of Java using File channel and transferTo(...) method to read files from my DVD and write to the hard drive. With this simple test, the transferTo(...) method gets about 4 MB /S transfer rate to disk.
>
> Then, I put some timing code into the LengthDelimitedEncoder.transfer(...) method, and ran my application, which copies files from DVD
> to a socket. And the transferTo(...) from file to socket was getting about 0.7 MB /s . This is a 6 X degradation in performance.
> And, if you recall, if the files are already in the OS file cache, I get about 100 MB/S, so the socket is not slowing me down.
>
I wanted to share the solution I came up with that is pretty optimal, in my opinion:
1) sequentially read files from DVD, from FileChannel into Direct ByteBuffer - once read, these files are now in the Windows OS file cache
2) after every 50 files read (or certain amount of MB), I send these files out over the asynch client( good transfer rate; asynchronous execution)
I do this until all files are read.
The total cost to me is just the cost of reading the DVD, which I can't really avoid. So, this is pretty optimal.
This e-mail may contain confidential and/or privileged information for the sole use of the intended recipient.
Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited.
If you have received this e-mail in error, please contact the sender and delete all copies.
Opinions, conclusions or other information contained in this e-mail may not be that of the organization.
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
This e-mail may contain confidential and/or privileged information for the sole use of the intended recipient.
Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited.
If you have received this e-mail in error, please contact the sender and delete all copies.
Opinions, conclusions or other information contained in this e-mail may not be that of the organization.
RE: ZeroCopyPut mystery
Posted by "Boxer, Aaron" <Aa...@uhn.ca>.
On Fri, 2014-01-17 at 16:28 +0000, Boxer, Aaron wrote:
> I made a simple test: I wrote a few lines of Java using File channel and transferTo(...) method to read files from my DVD and write to the hard drive. With this simple test, the transferTo(...) method gets about 4 MB /S transfer rate to disk.
>
> Then, I put some timing code into the LengthDelimitedEncoder.transfer(...) method, and ran my application, which copies files from DVD
> to a socket. And the transferTo(...) from file to socket was getting about 0.7 MB /s . This is a 6 X degradation in performance.
> And, if you recall, if the files are already in the OS file cache, I get about 100 MB/S, so the socket is not slowing me down.
>
I wanted to share the solution I came up with that is pretty optimal, in my opinion:
1) sequentially read files from DVD, from FileChannel into Direct ByteBuffer - once read, these files are now in the Windows OS file cache
2) after every 50 files read (or certain amount of MB), I send these files out over the asynch client( good transfer rate; asynchronous execution)
I do this until all files are read.
The total cost to me is just the cost of reading the DVD, which I can't really avoid. So, this is pretty optimal.
This e-mail may contain confidential and/or privileged information for the sole use of the intended recipient.
Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited.
If you have received this e-mail in error, please contact the sender and delete all copies.
Opinions, conclusions or other information contained in this e-mail may not be that of the organization.
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: ZeroCopyPut mystery
Posted by "Boxer, Aaron" <Aa...@uhn.ca>.
On Fri, 2014-01-17 at 16:28 +0000, Boxer, Aaron wrote:
> I made a simple test: I wrote a few lines of Java using File channel and transferTo(...) method to read files from my DVD and write to the hard drive. With this simple test, the transferTo(...) method gets about 4 MB /S transfer rate to disk.
>
> Then, I put some timing code into the LengthDelimitedEncoder.transfer(...) method, and ran my application, which copies files from DVD
> to a socket. And the transferTo(...) from file to socket was getting about 0.7 MB /s . This is a 6 X degradation in performance.
> And, if you recall, if the files are already in the OS file cache, I get about 100 MB/S, so the socket is not slowing me down.
>
> I have done a little more testing: transferTo from DVD to local socket gets 3-4 MB/S transfer rate.
> I tried using a thread pool, and performance degrades (not surprisingly) as thread count rises from 1.
>
> Since DVD seeks are very slow, multiple threads trying to read at the same time will kill performance.
>
> But, I am not sure this is the issue here, because I have set
>
> connManager.setMaxTotal(1);
> connManager.setDefaultMaxPerRoute(1);
>
>
> So, there should only be one thread doing the transfer (is this true?)
>
Actually HttpAsyncClient uses multiple i/o dispatch threads, one per CPU core. I suppose even with a cap of 1 total connection multiple threads might end up trying to access the same physical device.
You can force HttpAsyncClient to use only one i/o dispatch thread by using a custom IOReactorConfig
Thanks, Oleg. Here is my reactor config:
IOReactorConfig ioReactorConfig = IOReactorConfig.custom()
.setIoThreadCount(1)
.setConnectTimeout(30000)
.setSoTimeout(30000)
.setTcpNoDelay(true)
.build();
And the problem is still there with a thread count of 1.
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
This e-mail may contain confidential and/or privileged information for the sole use of the intended recipient.
Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited.
If you have received this e-mail in error, please contact the sender and delete all copies.
Opinions, conclusions or other information contained in this e-mail may not be that of the organization.
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: ZeroCopyPut mystery
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Fri, 2014-01-17 at 16:28 +0000, Boxer, Aaron wrote:
> I made a simple test: I wrote a few lines of Java using File channel and transferTo(...) method to read files from my DVD and write to the hard drive. With this simple test, the transferTo(...) method gets about 4 MB /S transfer rate to disk.
>
> Then, I put some timing code into the LengthDelimitedEncoder.transfer(...) method, and ran my application, which copies files from DVD
> to a socket. And the transferTo(...) from file to socket was getting about 0.7 MB /s . This is a 6 X degradation in performance.
> And, if you recall, if the files are already in the OS file cache, I get about 100 MB/S, so the socket is not slowing me down.
>
> I have done a little more testing: transferTo from DVD to local socket gets 3-4 MB/S transfer rate.
> I tried using a thread pool, and performance degrades (not surprisingly) as thread count rises from 1.
>
> Since DVD seeks are very slow, multiple threads trying to read at the same time will kill performance.
>
> But, I am not sure this is the issue here, because I have set
>
> connManager.setMaxTotal(1);
> connManager.setDefaultMaxPerRoute(1);
>
>
> So, there should only be one thread doing the transfer (is this true?)
>
Actually HttpAsyncClient uses multiple i/o dispatch threads, one per CPU
core. I suppose even with a cap of 1 total connection multiple threads
might end up trying to access the same physical device.
You can force HttpAsyncClient to use only one i/o dispatch thread by
using a custom IOReactorConfig
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: ZeroCopyPut mystery
Posted by "Boxer, Aaron" <Aa...@uhn.ca>.
I made a simple test: I wrote a few lines of Java using File channel and transferTo(...) method to read files from my DVD and write to the hard drive. With this simple test, the transferTo(...) method gets about 4 MB /S transfer rate to disk.
Then, I put some timing code into the LengthDelimitedEncoder.transfer(...) method, and ran my application, which copies files from DVD
to a socket. And the transferTo(...) from file to socket was getting about 0.7 MB /s . This is a 6 X degradation in performance.
And, if you recall, if the files are already in the OS file cache, I get about 100 MB/S, so the socket is not slowing me down.
I have done a little more testing: transferTo from DVD to local socket gets 3-4 MB/S transfer rate.
I tried using a thread pool, and performance degrades (not surprisingly) as thread count rises from 1.
Since DVD seeks are very slow, multiple threads trying to read at the same time will kill performance.
But, I am not sure this is the issue here, because I have set
connManager.setMaxTotal(1);
connManager.setDefaultMaxPerRoute(1);
So, there should only be one thread doing the transfer (is this true?)
Anyways, I guess I will switch back to the synchronous client for my application.
Aaron
This e-mail may contain confidential and/or privileged information for the sole use of the intended recipient.
Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited.
If you have received this e-mail in error, please contact the sender and delete all copies.
Opinions, conclusions or other information contained in this e-mail may not be that of the organization.
B KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKCB [ X ܚX KK[XZ[
Y[
]\ \ ][ X ܚX P˘\X K ܙ B ܈Y][ۘ[ [X[ K[XZ[
Y[
]\ \ Z[ ˘\X K ܙ B B
This e-mail may contain confidential and/or privileged information for the sole use of the intended recipient.
Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited.
If you have received this e-mail in error, please contact the sender and delete all copies.
Opinions, conclusions or other information contained in this e-mail may not be that of the organization.
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: ZeroCopyPut mystery
Posted by "Boxer, Aaron" <Aa...@uhn.ca>.
"Boxer, Aaron" <Aa...@uhn.ca> wrote:
>Thanks, Oleg. I am not trying to criticize the project in any way; just
>wondering if anyone has insight into why, on Windows, Java NIO is so
>much slower than stream IO.
>
>Perhaps it has to do with buffer size, thread count, ..... ?
>
You are welcome to criticize. I am merely trying to say this might be not the best list for this question.
Thanks, Oleg. I do think there is something in the library, or how I am configuring the library, that is slowing me down:
I made a simple test: I wrote a few lines of Java using File channel and transferTo(...) method to read files from my DVD
and write to the hard drive. With this simple test, the transferTo(...) method gets about 4 MB /S transfer rate to disk.
Then, I put some timing code into the LengthDelimitedEncoder.transfer(...) method, and ran my application, which copies files from DVD
to a socket. And the transferTo(...) from file to socket was getting about 0.7 MB /s . This is a 6 X degradation in performance.
And, if you recall, if the files are already in the OS file cache, I get about 100 MB/S, so the socket is not slowing me down.
So, it is in the library somewhere. Any advice on how to debug this would be very much appreciated.
Cheers,
Aaron
This e-mail may contain confidential and/or privileged information for the sole use of the intended recipient.
Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited.
If you have received this e-mail in error, please contact the sender and delete all copies.
Opinions, conclusions or other information contained in this e-mail may not be that of the organization.
RE: ZeroCopyPut mystery
Posted by Oleg Kalnichevski <ol...@ok2consulting.com>.
"Boxer, Aaron" <Aa...@uhn.ca> wrote:
>Thanks, Oleg. I am not trying to criticize the project in any way; just
>wondering if anyone has insight into why, on Windows,
>Java NIO is so much slower than stream IO.
>
>Perhaps it has to do with buffer size, thread count, ..... ?
>
You are welcome to criticize. I am merely trying to say this might be not the best list for this question.
Oleg
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: ZeroCopyPut mystery
Posted by "Boxer, Aaron" <Aa...@uhn.ca>.
Thanks, Oleg. I am not trying to criticize the project in any way; just wondering if anyone has insight into why, on Windows,
Java NIO is so much slower than stream IO.
Perhaps it has to do with buffer size, thread count, ..... ?
-----Original Message-----
From: Oleg Kalnichevski [mailto:olegk@apache.org]
Sent: Thursday, January 16, 2014 11:15 AM
To: HttpClient User Discussion
Subject: Re: ZeroCopyPut mystery
On Thu, 2014-01-16 at 04:33 +0000, Boxer, Aaron wrote:
> Hello,
>
> When I use ZeroCopyPut with hard drive (spinning disk ) disk files, I get a network transfer rate of about 250 MBPS for first time read of files; 850 MBPS if the files are already in the Windows OS file cache. Synchronous put gives me about 30 MBPS.
>
> But, when I am dealing with files on DVD, I get a network transfer rate of 5 MBPS for first time read of files, and 850 MBPS if files are in the OS file cache.
> Synchronous put gives me about 30 MBPS.
>
> So, in the case of first read of DVD files, synchronous client is superior. Synchronous should be faster, because it can read and send at the same time.
>
> Does this make sense? It seems a little strange to me.
>
Aaron,
Zero copy data transfer is very platform specific. Java merely provides a common APIs, which however can have radically different implementations on different platforms and therefore can have different performance characteristics.
I guess these numbers have more to do with Windows than with HttpAsyncClient.
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
This e-mail may contain confidential and/or privileged information for the sole use of the intended recipient.
Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited.
If you have received this e-mail in error, please contact the sender and delete all copies.
Opinions, conclusions or other information contained in this e-mail may not be that of the organization.
Re: ZeroCopyPut mystery
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Thu, 2014-01-16 at 04:33 +0000, Boxer, Aaron wrote:
> Hello,
>
> When I use ZeroCopyPut with hard drive (spinning disk ) disk files, I get a network transfer rate of about 250 MBPS for first time read of files; 850 MBPS if the files are already in the Windows OS file cache. Synchronous put gives me about 30 MBPS.
>
> But, when I am dealing with files on DVD, I get a network transfer rate of 5 MBPS for first time read of files, and 850 MBPS if files are in the OS file cache.
> Synchronous put gives me about 30 MBPS.
>
> So, in the case of first read of DVD files, synchronous client is superior. Synchronous should be faster, because it can read and send at the same time.
>
> Does this make sense? It seems a little strange to me.
>
Aaron,
Zero copy data transfer is very platform specific. Java merely provides
a common APIs, which however can have radically different
implementations on different platforms and therefore can have different
performance characteristics.
I guess these numbers have more to do with Windows than with
HttpAsyncClient.
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org