You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@maven.apache.org by Michael Osipov <mi...@apache.org> on 2017/01/03 20:18:27 UTC
Re: [VOTE] Releasing Wagon 2.11
I have made some concluding tests on my machine at work because my home
machine is so fast that you don't see a difference if you change something.
My tests are two-fold: improve HugeFileDownloadTest and manually
reconstruct with pure Jetty and HttpClient (same versions).
First of all sparse files work with pure Java within tens of milliseconds:
Path tempDirectory = Files.createTempDirectory("jetty");
final ByteBuffer buf = ByteBuffer.allocate(4).putInt(2);
buf.rewind();
final OpenOption[] options = { StandardOpenOption.WRITE,
StandardOpenOption.CREATE_NEW , StandardOpenOption.SPARSE };
final Path hugeFile = tempDirectory.resolve("hugefile.txt");
try (final SeekableByteChannel channel =
Files.newByteChannel(hugeFile, options);) {
channel.position(HUGE_FILE_SIZE);
channel.write(buf);
}
Handmade tests are always faster than Wagon tests (all times approx.):
Wagon:
Create file: 80 s to 90 s
Download w/ content length: 3,5 min (optimized)
Download w/ chunks: 2,5 min
Handmade:
Create file: 60 s to 70 s (with NIO.2 and without sparse)
Download w/ content length: 2,5 min
Download w/ chunks: < 1,5 min
There is a similar issue: http://stackoverflow.com/q/9031311/696632
I started to play with buffer sizes in Wagon, the anonymous inner class
and DefaultServlet. It seems to me that Jetty's IO class
buffer size of 64 KiB -- which is used by the DefaultServlet -- makes
downloads slower in this use case.
I have turned the anonymous inner HttpServlet to 8 KiB as well as
Wagon's internal buffer to 8 KiB (default 4 KiB has been tried too).
Unfortunately, the results sometimes vary but the smaller buffer most of
the time wins against Jetty's IO. I will commit a changed version to
that test and raise to 800 seconds. This should do it, in my opinion.
There is nothing we can do anymore with spinning disks and the
non-available sparse files on Windows. Note that RandomAccessFile
creates them on Unix.
Additionally, I tried to download with curl from Jetty, it is a bit
faster than the handmade version. Personally, there must be some
misconfiguration in the HttpClient if it performs slower than my cheap
hand-written code.
To finalize this. I ran the tests on an old Pentium 4 2,4 GHz, 2 GiB RAM
with FreeBSD 11.0-STABLE:
2017-01-02T21:39:53.542 [main] INFO
org.apache.maven.wagon.providers.http.HugeFileDownloadTest - Creating
test file
2017-01-02T21:39:53.549 [main] INFO
org.apache.maven.wagon.providers.http.HugeFileDownloadTest - Test file
created
2017-01-02T21:39:53.552 [main] INFO org.eclipse.jetty.server.Server -
jetty-8.1.22.v20160922
2017-01-02T21:39:53.556 [main] INFO
org.eclipse.jetty.server.AbstractConnector - Started
SelectChannelConnector@0.0.0.0:20435
http://localhost:20435 - Session: Opened
2017-01-02T21:39:53.569 [main] INFO
org.apache.maven.wagon.providers.http.HugeFileDownloadTest - Fetching
'hugefile.txt' in chunks
2017-01-02T21:44:51.315 [main] INFO
org.apache.maven.wagon.providers.http.HugeFileDownloadTest - The file
was successfully fetched
http://localhost:20435 - Session: Disconnecting
http://localhost:20435 - Session: Disconnected
2017-01-02T21:44:51.329 [main] INFO
org.eclipse.jetty.server.handler.ContextHandler - stopped
o.e.j.s.ServletContextHandler{/,file:/usr/home/mosipov/Projekte/maven-wagon/wagon-providers/wagon-http/target/}
2017-01-02T21:44:51.868 [main] INFO
org.apache.maven.wagon.providers.http.HugeFileDownloadTest - Creating
test file
2017-01-02T21:44:51.893 [main] INFO
org.apache.maven.wagon.providers.http.HugeFileDownloadTest - Test file
created
2017-01-02T21:44:51.894 [main] INFO org.eclipse.jetty.server.Server -
jetty-8.1.22.v20160922
2017-01-02T21:44:51.905 [main] INFO
org.eclipse.jetty.server.AbstractConnector - Started
SelectChannelConnector@0.0.0.0:23105
http://localhost:23105 - Session: Opened
2017-01-02T21:44:52.019 [main] INFO
org.apache.maven.wagon.providers.http.HugeFileDownloadTest - Fetching
'hugefile.txt' with content length
2017-01-02T21:48:03.017 [main] INFO
org.apache.maven.wagon.providers.http.HugeFileDownloadTest - The file
was successfully fetched
http://localhost:23105 - Session: Disconnecting
http://localhost:23105 - Session: Disconnected
2017-01-02T21:48:03.030 [main] INFO
org.eclipse.jetty.server.handler.ContextHandler - stopped
o.e.j.s.ServletContextHandler{/,file:/usr/home/mosipov/Projekte/maven-wagon/wagon-providers/wagon-http/target/}
For such an old hardware, this is still decent.
Michael
Am 2016-12-31 um 21:20 schrieb Guillaume Boué:
> Thanks for the analysis! Agree with dropping fsutil then; I committed
> the addition of the logs with it just so that we can have concrete
> numbers for comparison (the last build indicates there was no permission
> issues in using it, otherwise it wouldn't have timed out but just failed
> to find the file). NIO.2 requires Java 7 though, so looks like a good
> reason to try and update to it (and Jetty 9 in the process, as Olivier
> suggested), but maybe not for this version. That also makes me wonder if
> the NIO.2 SPARSE option doesn't implictly require special privileges on
> Windows (like it does for creating symbolic links).
>
> In any case, the result of the Jenkins build with regard to the commit
> are here
> https://builds.apache.org/job/maven-wagon-jetty8-windows/org.apache.maven.wagon$wagon-http/3/consoleFull.
> The 4 Go test file was created instantly (30 milliseconds), and it timed
> out performing the first download; so the creation of the file isn't an
> issue. I was following the growing size of the download in the target
> directory as it happened and things didn't appear to work slowly.
>
> The tests for wagon-http started at 11:47:28 and timed out after
> 11:53:04 when doing the first download for HugeFileDownloadTest; this
> makes up for the 400 seconds (roughly 6-7 minutes) of the Surefire
> timeout. Another breakdown is seen here (oddly without
> HugeFileDownloadTest):
>
> https://builds.apache.org/job/maven-wagon-jetty8-windows/org.apache.maven.wagon$wagon-http/3/testReport/org.apache.maven.wagon.providers.http/
>
>
> All of this is actually caused by HttpsWagonTest,
> HttpsWagonPreemptiveTest and HugeFileDownloadTest taking each a bit more
> than 2 minutes. I don't think we can cut
> <https://builds.apache.org/job/maven-wagon-jetty8-windows/org.apache.maven.wagon$wagon-http/3/testReport/org.apache.maven.wagon.providers.http/HttpsWagonTest/><https://builds.apache.org/job/maven-wagon-jetty8-windows/org.apache.maven.wagon$wagon-http/3/testReport/org.apache.maven.wagon.providers.http/HttpsWagonPreemptiveTest/>HugeFileDownloadTest
> below 2 minutes (with the sparse file created instantly, I see no room
> for improvements here), so focus should actually be made on those 2
> other tests.
>
> Guillaume
>
>
> Le 31/12/2016 à 16:51, Michael Osipov a écrit :
>> Am 2016-12-31 um 12:13 schrieb Guillaume Boué:
>>> Do you think I can add a dummy log before the creation of the test file
>>> (and add the timestamps to the logs of wagon-http), to see how much time
>>> it takes on the Windows Server 2012? I'd like to see the breakdown of
>>> what takes time on the Jenkins machine, perhaps there is nothing we can
>>> do better (except increase the timeout to 500 or so...).
>>
>> Let me share new findings before I get to the answers. I tested this
>> branch on my machine at work which is part of an Active Directory
>> domain, I am local admin. It is a regular, 2-core i5 machine with 12
>> GiB RAM and medium-size HDD:
>>
>> 1. fsutil failed, domain policy requires elevated permissions to run
>> fsutil in command prompt and you even see it because stderr/stdout of
>> fsutil is swalled. I tried manually.
>> 2. I have even raised the timeout to 800 seconds, it still fails.
>>
>> Given this, I'd wouldn't really use fsutil. Rather NIO2 with
>> StandardOption.SPARSE. This might work faster.
>>
>> To your questions: go head and use SLF4J, it is already available.
>> Print the log messages and you'll see how much creation really consumes.
>> I'd test this ony my machine at home and work.
>>
>> Michael
>>
>>
>>> Le 30/12/2016 à 22:46, Michael Osipov a écrit :
>>>> I just pushed another commit to the branch with your changes.
>>>> The job does not really work on Windows [1], it simply takes too long
>>>> to complete on the Windows Server 2012.
>>>>
>>>> [1]
>>>> https://builds.apache.org/job/maven-wagon-jetty8-windows/2/org.apache.maven.wagon$wagon-http/console
>>>>
>>>>
>>>>
>>>> Am 2016-12-29 um 19:10 schrieb Guillaume Boué:
>>>>> I have a Toshiba SSHD (MQ02ABD100H). I think the issue is that the
>>>>> Java
>>>>> code should create a sparse file to have things faster. Using
>>>>> setLength
>>>>> on a random access file probably does it depending on the OS and
>>>>> type of
>>>>> drive, but it isn't creating one in my situation.
>>>>>
>>>>> When run on Ubuntu, creating the test file is done practically
>>>>> instantly
>>>>> whereas it takes 60 seconds on my Windows machine. Downloading takes
>>>>> around 30 seconds in Ubuntu, whereas it takes 150 seconds in Windows.
>>>>>
>>>>> I changed the method creating the test file (makeHugeFile) to:
>>>>>
>>>>> if ( Os.isFamily( Os.FAMILY_WINDOWS ) )
>>>>> {
>>>>> Process p = new ProcessBuilder( "fsutil", "file",
>>>>> "createnew", hugeFile.getAbsolutePath(),
>>>>> String.valueOf(
>>>>> HUGE_FILE_SIZE ) ).start();
>>>>> p.waitFor();
>>>>> }
>>>>> else
>>>>> {
>>>>> RandomAccessFile ra = new RandomAccessFile(
>>>>> hugeFile.getPath(), "rw" );
>>>>> ra.setLength( HUGE_FILE_SIZE + 1 );
>>>>> ra.seek( HUGE_FILE_SIZE );
>>>>> ra.write( 1 );
>>>>> ra.close();
>>>>> }
>>>>>
>>>>> This matches with the timings I get on Ubuntu (both for the
>>>>> creation of
>>>>> the file, and the downloading via Wagon). I ran the build several
>>>>> times
>>>>> with this change without any timeout issues. Can you test this
>>>>> change on
>>>>> your side?
>>>>>
>>>>> Side-note: I added "ra.close();" in the code above, because the random
>>>>> access file wasn't closed otherwise, and then the file wasn't getting
>>>>> deleted properly.
>>>>>
>>>>> Guillaume
>>>>>
>>>>> Le 29/12/2016 à 15:50, Michael Osipov a écrit :
>>>>>> Am 2016-12-29 um 12:24 schrieb Guillaume Boué:
>>>>>>> I ran them at least 10 times, and there was the timeout issue each
>>>>>>> time.
>>>>>>> Yes, the timeout is Surefire waiting for the forked VM. I applied
>>>>>>> the
>>>>>>> patch (I had done similar tries also) but I still have the issue on
>>>>>>> Windows 10.
>>>>>>>
>>>>>>> You'll find the logs attached. It seems that the download is really
>>>>>>> advancing (inside target, I can see the file huge<numbers>txt slowly
>>>>>>> growing), but is just taking a long time. Additionally, the test
>>>>>>> class
>>>>>>> HugeFileDownloadTest is composed of 2 tests, downloading the same
>>>>>>> file 2
>>>>>>> times with different parameters: from the logs, it seems one of
>>>>>>> them is
>>>>>>> succeeding (taking 140 seconds of the 400 in total), and then it
>>>>>>> times
>>>>>>> out performing the second.
>>>>>>>
>>>>>>> Also, part of the issue is probably with the time it takes to
>>>>>>> create the
>>>>>>> 4 Go test file that is later downloaded through Wagon (maybe it
>>>>>>> should
>>>>>>> be done by calling the fsutil command when the test runs on
>>>>>>> Windows).
>>>>>>> I'll try to do more timings when running the tests with Ubuntu
>>>>>>> and see
>>>>>>> where the difference is.
>>>>>>
>>>>>> Thanks for the analysis. I think the cheapest improvement we can
>>>>>> do is
>>>>>> to create the file only once (setUp(), tearDown(). It is created for
>>>>>> each test again. I do not think that this is really necessary.
>>>>>>
>>>>>> What type of disks do you have? This test passes on my machine within
>>>>>> 70 seconds on a Samsung SSD.
>>>>>>
>>>>>> I will apply the patch to the branch because those fixes are
>>>>>> necessary
>>>>>> anyway.
>>>>>>
>>>>>> Please keep me updated.
>>>>>>
>>>>>> Michael
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>>>>> For additional commands, e-mail: dev-help@maven.apache.org
>>>>>>
>>>>>
>>>>>
>>>>> ---
>>>>> L'absence de virus dans ce courrier électronique a été vérifiée par le
>>>>> logiciel antivirus Avast.
>>>>> https://www.avast.com/antivirus
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>>>> For additional commands, e-mail: dev-help@maven.apache.org
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>>> For additional commands, e-mail: dev-help@maven.apache.org
>>>>
>>>
>>>
>>> ---
>>> L'absence de virus dans ce courrier électronique a été vérifiée par le
>>> logiciel antivirus Avast.
>>> https://www.avast.com/antivirus
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>> For additional commands, e-mail: dev-help@maven.apache.org
>>>
>>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> For additional commands, e-mail: dev-help@maven.apache.org
>>
>
>
>
> ---
> L'absence de virus dans ce courrier électronique a été vérifiée par le
> logiciel antivirus Avast.
> https://www.avast.com/antivirus
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org