You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by "Boxer, Aaron" <Aa...@uhn.ca> on 2014/05/05 21:15:16 UTC

RE: ZeroCopyPut mystery

I know this is an old thread, but I wanted to relate the final scheme I came up with for optimal uploading of DVD files on windows,
in case anyone else has to deal with this issue:

1) issue Powershell command from Java to recursively list all files in a root directory, and copy each file to dev null, as follows: 

   String cmd =     "powershell.exe  \"Get-Childitem -Recurse " +  rootDirectory +
                            "   | where {!$_.PsIsContainer} | %{ Get-Content $_.FullName > $null; echo $_.FullName;} \" ";

2) Notice the echo command at the end; now that the file is in the windows file cache, I can read this echoed name in java and issue an asynchronous zero copy put on the file via
   httpasyncclient (this is now very fast, since the file is in the cache)

So, this method pipelines the reads and puts, so that put time can be ignored, and time to upload is just the time to read the files from DVD.

Cheers,
Aaron




-----Original Message-----
From: Boxer, Aaron 
Sent: Friday, January 17, 2014 3:26 PM
To: HttpClient User Discussion
Subject: RE: ZeroCopyPut mystery


On Fri, 2014-01-17 at 16:28 +0000, Boxer, Aaron wrote:
>  I made a simple test: I wrote a few lines of Java using  File channel and transferTo(...) method to read files from my DVD and write to the hard drive.  With this simple test,  the transferTo(...) method gets  about 4 MB /S transfer rate to disk.
> 
> Then, I put some timing code into the LengthDelimitedEncoder.transfer(...) method, and ran my  application, which copies files from DVD
> to a socket.  And the transferTo(...) from file to socket was getting about 0.7 MB /s .   This is a 6 X degradation in performance.  
> And, if you recall, if the files are already in the OS file cache, I get about 100 MB/S, so the socket is not slowing me down.
> 


I wanted to share the solution I came up with that is pretty optimal, in my opinion:

1) sequentially read files from DVD, from FileChannel into Direct ByteBuffer - once read,  these files are now in the Windows OS file cache
2) after every 50 files read (or certain amount of MB), I send these files out over the asynch client( good transfer rate; asynchronous execution)

I do this until all files are read.

The total cost to me is just the cost of reading the DVD, which I can't really avoid. So, this is pretty  optimal.










This e-mail may contain confidential and/or privileged information for the sole use of the intended recipient. 
Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. 
If you have received this e-mail in error, please contact the sender and delete all copies. 
Opinions, conclusions or other information contained in this e-mail may not be that of the organization.

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org


This e-mail may contain confidential and/or privileged information for the sole use of the intended recipient. 
Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. 
If you have received this e-mail in error, please contact the sender and delete all copies. 
Opinions, conclusions or other information contained in this e-mail may not be that of the organization.

Password exposure in logs

Posted by Bob Anderson <mu...@gmail.com>.
Hi,

  I know that the password is exposed in clear text on some versions of the http client in debug. Can you tell me what versions?  Also it appears this also happens on some errors.  Can you send an info on this.  Any areas where password are exposed in the clear I need information on.

Thanks,

 Bob
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org