You are viewing a plain text version of this content. The canonical link for it is here.
Posted to ivy-user@ant.apache.org by testn <te...@doramail.com> on 2007/06/12 14:28:46 UTC
ivy:retrieve performance
Hi,
I have a couple comments regarding ivy:retrieve.
1. It seems that when I turn "sync" option on, ivy:retrieve will copy files
from .ivy folder to lib folder everytime even though files in lib folder are
updated. It should have an option to check file size and timestamp to ensure
that the file is updated.
2. The buffer in FileUtils.java is too small. It's set at 8192. It seems to
be much better for me to set it much larger. This is due to the fact that it
needs to read and write simultaneously. The bigger the buffer is, the
smaller number of time, HD header has to move. For me, 65536 seems to
perform much better but I haven't tried other numbers.
--
View this message in context: http://www.nabble.com/ivy%3Aretrieve-performance-tf3907253.html#a11077846
Sent from the ivy-user mailing list archive at Nabble.com.
Re: ivy:retrieve performance
Posted by Xavier Hanin <xa...@gmail.com>.
On 6/14/07, Buck, Robert <rb...@verisign.com> wrote:
>
> Folks,
>
> We benchmarked a number of JDK IO API's for an internal project. To
> neutralize any questions regarding NIO vs Old IO, please take a look at
> the attached diagram. These rates will be largely identical on both
> Linux and Windows. Blue line NIO, red line old io.
Thanks for sharing this with us!
If you do not get the attached JPEG file, let me know. I can send it
> directly to you if you so request.
I can't see the attached file, could you please send it to me directly. TIA.
Xavier
Cheers,
>
> Bob
>
> > -----Original Message-----
> > From: Xavier Hanin [mailto:xavier.hanin@gmail.com]
> > Sent: Thursday, June 14, 2007 6:22 AM
> > To: ivy-user@incubator.apache.org
> > Subject: Re: ivy:retrieve performance
> >
> > On 6/14/07, Gilles Scokart <gs...@gmail.com> wrote:
> > >
> > > > 2. The buffer in FileUtils.java is too small. It's set at
> > 8192. It
> > > > seems to
> > > > > be much better for me to set it much larger. This is due to the
> > > > > fact
> > > > that
> > > > > it
> > > > > needs to read and write simultaneously. The bigger the
> > buffer is,
> > > > > the smaller number of time, HD header has to move. For
> > me, 65536
> > > > > seems to perform much better but I haven't tried other numbers.
> > > >
> > > >
> > > > I'd like to get more feedback on this. One use case is
> > not the other.
> > > This
> > > > size has been borrowed from Ant copy mechanism. Maybe
> > what we could
> > > > do
> > > is
> > > > make this configurable, so that one could adapt to its
> > needs. Or try
> > > > to guess a good size depending on the size (when it's possible to
> > > > get an
> > > idea
> > > > of the size before copying).
> > > >
> > > > Xavier
> > > >
> > >
> > >
> > > Couldn't we use the nio for that? (See
> > > http://www.javalobby.org/java/forums/t17036.html)
> >
> >
> > According to comments 10 and 11 NIO have bad performance for
> > large files on linux, and input stream with byte buffer is
> > pretty close to NIO for small files (see comment 13
> > conclusion). So I'm not sure switching to NIO would indeed
> > help a lot. According to the tests in the thread you pointed
> > using a 64kB buffer seems to be a good choice (which confirms
> > testn tests), at least for large files. OTOH the last
> > conclusion (comment 17) is different.
> > So I don't really know what to think about that. We should
> > make some tests on several platforms and jvms to draw
> > conclusion ourself, but it takes time.
> >
> >
> > Xavier
> >
> > Gilles
> > >
> > >
> > >
> > >
> >
> >
> > --
> > Xavier Hanin - Independent Java Consultant Manage your
> > dependencies with Ivy!
> > http://incubator.apache.org/ivy/
> >
>
--
Xavier Hanin - Independent Java Consultant
Manage your dependencies with Ivy!
http://incubator.apache.org/ivy/
RE: ivy:retrieve performance
Posted by Bourzeix,
Hervé <He...@Genesys.com>.
I resolved my issue, I deleted my ivy cache and I supsect a latest.integration in one of the dependency chain.
thx anyway
-----Original Message-----
From: Gilles Scokart [mailto:gscokart@gmail.com]
Sent: samedi 16 juin 2007 20:57
To: ivy-user@incubator.apache.org
Subject: Re: ivy:retrieve performance
Did they have exactly the same organisation/module?
Yes
Did you put a defaultConflictManager in your settings?
Did they have the same artefact name?
Yes
Gilles
2007/6/15, Bourzeix, Hervé <He...@genesys.com>:
> Hi,
> I got something really strange :
>
> In the listed conflict, I got a module
> A by Company selected : (2.2.1 , 1.0 ) evicted: None
>
> How it is possible ? Using the ivy:retrieve I got 2 versions in my lib directory, which is really really bad for me.
>
> thx
>
> Hervé
>
>
>
--
Gilles SCOKART
Re: ivy:retrieve performance
Posted by Gilles Scokart <gs...@gmail.com>.
Did they have exactly the same organisation/module?
Did you put a defaultConflictManager in your settings?
Did they have the same artefact name?
Gilles
2007/6/15, Bourzeix, Hervé <He...@genesys.com>:
> Hi,
> I got something really strange :
>
> In the listed conflict, I got a module
> A by Company selected : (2.2.1 , 1.0 ) evicted: None
>
> How it is possible ? Using the ivy:retrieve I got 2 versions in my lib directory, which is really really bad for me.
>
> thx
>
> Hervé
>
>
>
--
Gilles SCOKART
Re: ivy:retrieve performance
Posted by testn <te...@doramail.com>.
IVY-551 has been opened for this.
Thanks,
Xavier Hanin wrote:
>
> On 6/21/07, testn <te...@doramail.com> wrote:
>>
>>
>> Hi Xavier,
>
>
> Hi, but this question is not only for me, Ivy is a community project where
> the community takes decisions.
>
> Any plan to put it in alpha2? It doesn't look like it requires a lot of
> work
>> if we just want to change the buffer size. nio may be a long shot.
>
>
> I'm +1 to increase the buffer to 64kB. I'm +1 to open an issue for this,
> and
> another one to later further improve this with NIO. What do others think?
>
> Xavier
>
> Thanks
>>
>>
>> Xavier Hanin wrote:
>> >
>> > Great, thanks a lot for these data!
>> >
>> > Xavier
>> >
>> > On 6/14/07, Buck, Robert <rb...@verisign.com> wrote:
>> >>
>> >> What the performance test did is:
>> >>
>> >> 1. increase by powers of 2 a file size variable
>> >>
>> >> 2. for each file size, create/open a new file
>> >>
>> >> 3. write specified number of bytes to file
>> >>
>> >> 4. close file
>> >>
>> >> Tests were performed on two systems:
>> >>
>> >> 1. IBM ThinkPad T42 (slow) laptop, stock hardware, 1.5GB RAM
>> >>
>> >> 2. Dell Dual CPU, 5150, 64-bit Red Hat 4; (2) SAS 15k 146GB disks in
>> >> RAID 1 configuration; 16 GB RAM; this is production grade hardware.
>> >>
>> >> Both sytems produced similar profiles (shape), however, on the
>> >> production hardware with Linux (ext2, I believe) the numbers were
>> >> substantially greater (makes sense when you consider the fast disks I
>> >> have; at 3GB/sec rates, these SAS drives scream).
>> >>
>> >> In both cases buffers were configured for both Old IO and New IO. With
>> >> NIO direct buffers were used. In both cases, I determined the ideal
>> >> buffer sizes based upon experimentation to be around 32kb - 64 kb.
>> Going
>> >> to buffer sizes less than, or greater than, this range reduced overall
>> >> throughput.
>> >>
>> >> -Bob
>> >>
>> >> > -----Original Message-----
>> >> > From: Xavier Hanin [mailto:xavier.hanin@gmail.com]
>> >> > Sent: Thursday, June 14, 2007 8:38 AM
>> >> > To: ivy-user@incubator.apache.org
>> >> > Subject: Re: ivy:retrieve performance
>> >> >
>> >> > On 6/14/07, Buck, Robert <rb...@verisign.com> wrote:
>> >> > >
>> >> > > Folks,
>> >> > >
>> >> > > We benchmarked a number of JDK IO API's for an internal project.
>> To
>> >> > > neutralize any questions regarding NIO vs Old IO, please
>> >> > take a look
>> >> > > at the attached diagram. These rates will be largely
>> >> > identical on both
>> >> > > Linux and Windows. Blue line NIO, red line old io.
>> >> >
>> >> >
>> >> > I've uploaded the image here:
>> >> > http://www.flickr.com/photos/41363014@N00/548163404/
>> >> >
>> >> > Could you give a little bit more details about how you did
>> >> > the tests: jvm used, size of buffer used for old IO, ...
>> >> >
>> >> > According to your tests it seems that NIO should be preferred
>> >> > in any case, it wasn't what some users in the javalobby
>> >> > thread seemed to say. So I wonder what makes the difference.
>> >> >
>> >> > Xavier
>> >> >
>> >> > If you do not get the attached JPEG file, let me know. I can send it
>> >> > > directly to you if you so request.
>> >> > >
>> >> > > Cheers,
>> >> > >
>> >> > > Bob
>> >> > >
>> >> > > > -----Original Message-----
>> >> > > > From: Xavier Hanin [mailto:xavier.hanin@gmail.com]
>> >> > > > Sent: Thursday, June 14, 2007 6:22 AM
>> >> > > > To: ivy-user@incubator.apache.org
>> >> > > > Subject: Re: ivy:retrieve performance
>> >> > > >
>> >> > > > On 6/14/07, Gilles Scokart <gs...@gmail.com> wrote:
>> >> > > > >
>> >> > > > > > 2. The buffer in FileUtils.java is too small. It's set at
>> >> > > > 8192. It
>> >> > > > > > seems to
>> >> > > > > > > be much better for me to set it much larger. This is due
>> to
>> >> > > > > > > the fact
>> >> > > > > > that
>> >> > > > > > > it
>> >> > > > > > > needs to read and write simultaneously. The bigger the
>> >> > > > buffer is,
>> >> > > > > > > the smaller number of time, HD header has to move. For
>> >> > > > me, 65536
>> >> > > > > > > seems to perform much better but I haven't tried
>> >> > other numbers.
>> >> > > > > >
>> >> > > > > >
>> >> > > > > > I'd like to get more feedback on this. One use case is
>> >> > > > not the other.
>> >> > > > > This
>> >> > > > > > size has been borrowed from Ant copy mechanism. Maybe
>> >> > > > what we could
>> >> > > > > > do
>> >> > > > > is
>> >> > > > > > make this configurable, so that one could adapt to its
>> >> > > > needs. Or try
>> >> > > > > > to guess a good size depending on the size (when it's
>> >> > possible
>> >> > > > > > to get an
>> >> > > > > idea
>> >> > > > > > of the size before copying).
>> >> > > > > >
>> >> > > > > > Xavier
>> >> > > > > >
>> >> > > > >
>> >> > > > >
>> >> > > > > Couldn't we use the nio for that? (See
>> >> > > > > http://www.javalobby.org/java/forums/t17036.html)
>> >> > > >
>> >> > > >
>> >> > > > According to comments 10 and 11 NIO have bad performance
>> >> > for large
>> >> > > > files on linux, and input stream with byte buffer is
>> >> > pretty close to
>> >> > > > NIO for small files (see comment 13 conclusion). So I'm not sure
>> >> > > > switching to NIO would indeed help a lot. According to
>> >> > the tests in
>> >> > > > the thread you pointed using a 64kB buffer seems to be a
>> >> > good choice
>> >> > > > (which confirms testn tests), at least for large files. OTOH
>> the
>> >> > > > last conclusion (comment 17) is different.
>> >> > > > So I don't really know what to think about that. We
>> >> > should make some
>> >> > > > tests on several platforms and jvms to draw conclusion
>> >> > ourself, but
>> >> > > > it takes time.
>> >> > > >
>> >> > > >
>> >> > > > Xavier
>> >> > > >
>> >> > > > Gilles
>> >> > > > >
>> >> > > > >
>> >> > > > >
>> >> > > > >
>> >> > > >
>> >> > > >
>> >> > > > --
>> >> > > > Xavier Hanin - Independent Java Consultant Manage your
>> >> > dependencies
>> >> > > > with Ivy!
>> >> > > > http://incubator.apache.org/ivy/
>> >> > > >
>> >> > >
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Xavier Hanin - Independent Java Consultant Manage your
>> >> > dependencies with Ivy!
>> >> > http://incubator.apache.org/ivy/
>> >> >
>> >>
>> >
>> >
>> >
>> > --
>> > Xavier Hanin - Independent Java Consultant
>> > Manage your dependencies with Ivy!
>> > http://incubator.apache.org/ivy/
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/ivy%3Aretrieve-performance-tf3907253.html#a11230918
>> Sent from the ivy-user mailing list archive at Nabble.com.
>>
>>
>
>
> --
> Xavier Hanin - Independent Java Consultant
> Manage your dependencies with Ivy!
> http://incubator.apache.org/ivy/
>
>
--
View this message in context: http://www.nabble.com/ivy%3Aretrieve-performance-tf3907253.html#a11387196
Sent from the ivy-user mailing list archive at Nabble.com.
Re: ivy:retrieve performance
Posted by Xavier Hanin <xa...@gmail.com>.
On 6/21/07, testn <te...@doramail.com> wrote:
>
>
> Hi Xavier,
Hi, but this question is not only for me, Ivy is a community project where
the community takes decisions.
Any plan to put it in alpha2? It doesn't look like it requires a lot of work
> if we just want to change the buffer size. nio may be a long shot.
I'm +1 to increase the buffer to 64kB. I'm +1 to open an issue for this, and
another one to later further improve this with NIO. What do others think?
Xavier
Thanks
>
>
> Xavier Hanin wrote:
> >
> > Great, thanks a lot for these data!
> >
> > Xavier
> >
> > On 6/14/07, Buck, Robert <rb...@verisign.com> wrote:
> >>
> >> What the performance test did is:
> >>
> >> 1. increase by powers of 2 a file size variable
> >>
> >> 2. for each file size, create/open a new file
> >>
> >> 3. write specified number of bytes to file
> >>
> >> 4. close file
> >>
> >> Tests were performed on two systems:
> >>
> >> 1. IBM ThinkPad T42 (slow) laptop, stock hardware, 1.5GB RAM
> >>
> >> 2. Dell Dual CPU, 5150, 64-bit Red Hat 4; (2) SAS 15k 146GB disks in
> >> RAID 1 configuration; 16 GB RAM; this is production grade hardware.
> >>
> >> Both sytems produced similar profiles (shape), however, on the
> >> production hardware with Linux (ext2, I believe) the numbers were
> >> substantially greater (makes sense when you consider the fast disks I
> >> have; at 3GB/sec rates, these SAS drives scream).
> >>
> >> In both cases buffers were configured for both Old IO and New IO. With
> >> NIO direct buffers were used. In both cases, I determined the ideal
> >> buffer sizes based upon experimentation to be around 32kb - 64 kb.
> Going
> >> to buffer sizes less than, or greater than, this range reduced overall
> >> throughput.
> >>
> >> -Bob
> >>
> >> > -----Original Message-----
> >> > From: Xavier Hanin [mailto:xavier.hanin@gmail.com]
> >> > Sent: Thursday, June 14, 2007 8:38 AM
> >> > To: ivy-user@incubator.apache.org
> >> > Subject: Re: ivy:retrieve performance
> >> >
> >> > On 6/14/07, Buck, Robert <rb...@verisign.com> wrote:
> >> > >
> >> > > Folks,
> >> > >
> >> > > We benchmarked a number of JDK IO API's for an internal project. To
> >> > > neutralize any questions regarding NIO vs Old IO, please
> >> > take a look
> >> > > at the attached diagram. These rates will be largely
> >> > identical on both
> >> > > Linux and Windows. Blue line NIO, red line old io.
> >> >
> >> >
> >> > I've uploaded the image here:
> >> > http://www.flickr.com/photos/41363014@N00/548163404/
> >> >
> >> > Could you give a little bit more details about how you did
> >> > the tests: jvm used, size of buffer used for old IO, ...
> >> >
> >> > According to your tests it seems that NIO should be preferred
> >> > in any case, it wasn't what some users in the javalobby
> >> > thread seemed to say. So I wonder what makes the difference.
> >> >
> >> > Xavier
> >> >
> >> > If you do not get the attached JPEG file, let me know. I can send it
> >> > > directly to you if you so request.
> >> > >
> >> > > Cheers,
> >> > >
> >> > > Bob
> >> > >
> >> > > > -----Original Message-----
> >> > > > From: Xavier Hanin [mailto:xavier.hanin@gmail.com]
> >> > > > Sent: Thursday, June 14, 2007 6:22 AM
> >> > > > To: ivy-user@incubator.apache.org
> >> > > > Subject: Re: ivy:retrieve performance
> >> > > >
> >> > > > On 6/14/07, Gilles Scokart <gs...@gmail.com> wrote:
> >> > > > >
> >> > > > > > 2. The buffer in FileUtils.java is too small. It's set at
> >> > > > 8192. It
> >> > > > > > seems to
> >> > > > > > > be much better for me to set it much larger. This is due to
> >> > > > > > > the fact
> >> > > > > > that
> >> > > > > > > it
> >> > > > > > > needs to read and write simultaneously. The bigger the
> >> > > > buffer is,
> >> > > > > > > the smaller number of time, HD header has to move. For
> >> > > > me, 65536
> >> > > > > > > seems to perform much better but I haven't tried
> >> > other numbers.
> >> > > > > >
> >> > > > > >
> >> > > > > > I'd like to get more feedback on this. One use case is
> >> > > > not the other.
> >> > > > > This
> >> > > > > > size has been borrowed from Ant copy mechanism. Maybe
> >> > > > what we could
> >> > > > > > do
> >> > > > > is
> >> > > > > > make this configurable, so that one could adapt to its
> >> > > > needs. Or try
> >> > > > > > to guess a good size depending on the size (when it's
> >> > possible
> >> > > > > > to get an
> >> > > > > idea
> >> > > > > > of the size before copying).
> >> > > > > >
> >> > > > > > Xavier
> >> > > > > >
> >> > > > >
> >> > > > >
> >> > > > > Couldn't we use the nio for that? (See
> >> > > > > http://www.javalobby.org/java/forums/t17036.html)
> >> > > >
> >> > > >
> >> > > > According to comments 10 and 11 NIO have bad performance
> >> > for large
> >> > > > files on linux, and input stream with byte buffer is
> >> > pretty close to
> >> > > > NIO for small files (see comment 13 conclusion). So I'm not sure
> >> > > > switching to NIO would indeed help a lot. According to
> >> > the tests in
> >> > > > the thread you pointed using a 64kB buffer seems to be a
> >> > good choice
> >> > > > (which confirms testn tests), at least for large files. OTOH the
> >> > > > last conclusion (comment 17) is different.
> >> > > > So I don't really know what to think about that. We
> >> > should make some
> >> > > > tests on several platforms and jvms to draw conclusion
> >> > ourself, but
> >> > > > it takes time.
> >> > > >
> >> > > >
> >> > > > Xavier
> >> > > >
> >> > > > Gilles
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > > > Xavier Hanin - Independent Java Consultant Manage your
> >> > dependencies
> >> > > > with Ivy!
> >> > > > http://incubator.apache.org/ivy/
> >> > > >
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Xavier Hanin - Independent Java Consultant Manage your
> >> > dependencies with Ivy!
> >> > http://incubator.apache.org/ivy/
> >> >
> >>
> >
> >
> >
> > --
> > Xavier Hanin - Independent Java Consultant
> > Manage your dependencies with Ivy!
> > http://incubator.apache.org/ivy/
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/ivy%3Aretrieve-performance-tf3907253.html#a11230918
> Sent from the ivy-user mailing list archive at Nabble.com.
>
>
--
Xavier Hanin - Independent Java Consultant
Manage your dependencies with Ivy!
http://incubator.apache.org/ivy/
Re: ivy:retrieve performance
Posted by testn <te...@doramail.com>.
Hi Xavier,
Any plan to put it in alpha2? It doesn't look like it requires a lot of work
if we just want to change the buffer size. nio may be a long shot.
Thanks
Xavier Hanin wrote:
>
> Great, thanks a lot for these data!
>
> Xavier
>
> On 6/14/07, Buck, Robert <rb...@verisign.com> wrote:
>>
>> What the performance test did is:
>>
>> 1. increase by powers of 2 a file size variable
>>
>> 2. for each file size, create/open a new file
>>
>> 3. write specified number of bytes to file
>>
>> 4. close file
>>
>> Tests were performed on two systems:
>>
>> 1. IBM ThinkPad T42 (slow) laptop, stock hardware, 1.5GB RAM
>>
>> 2. Dell Dual CPU, 5150, 64-bit Red Hat 4; (2) SAS 15k 146GB disks in
>> RAID 1 configuration; 16 GB RAM; this is production grade hardware.
>>
>> Both sytems produced similar profiles (shape), however, on the
>> production hardware with Linux (ext2, I believe) the numbers were
>> substantially greater (makes sense when you consider the fast disks I
>> have; at 3GB/sec rates, these SAS drives scream).
>>
>> In both cases buffers were configured for both Old IO and New IO. With
>> NIO direct buffers were used. In both cases, I determined the ideal
>> buffer sizes based upon experimentation to be around 32kb - 64 kb. Going
>> to buffer sizes less than, or greater than, this range reduced overall
>> throughput.
>>
>> -Bob
>>
>> > -----Original Message-----
>> > From: Xavier Hanin [mailto:xavier.hanin@gmail.com]
>> > Sent: Thursday, June 14, 2007 8:38 AM
>> > To: ivy-user@incubator.apache.org
>> > Subject: Re: ivy:retrieve performance
>> >
>> > On 6/14/07, Buck, Robert <rb...@verisign.com> wrote:
>> > >
>> > > Folks,
>> > >
>> > > We benchmarked a number of JDK IO API's for an internal project. To
>> > > neutralize any questions regarding NIO vs Old IO, please
>> > take a look
>> > > at the attached diagram. These rates will be largely
>> > identical on both
>> > > Linux and Windows. Blue line NIO, red line old io.
>> >
>> >
>> > I've uploaded the image here:
>> > http://www.flickr.com/photos/41363014@N00/548163404/
>> >
>> > Could you give a little bit more details about how you did
>> > the tests: jvm used, size of buffer used for old IO, ...
>> >
>> > According to your tests it seems that NIO should be preferred
>> > in any case, it wasn't what some users in the javalobby
>> > thread seemed to say. So I wonder what makes the difference.
>> >
>> > Xavier
>> >
>> > If you do not get the attached JPEG file, let me know. I can send it
>> > > directly to you if you so request.
>> > >
>> > > Cheers,
>> > >
>> > > Bob
>> > >
>> > > > -----Original Message-----
>> > > > From: Xavier Hanin [mailto:xavier.hanin@gmail.com]
>> > > > Sent: Thursday, June 14, 2007 6:22 AM
>> > > > To: ivy-user@incubator.apache.org
>> > > > Subject: Re: ivy:retrieve performance
>> > > >
>> > > > On 6/14/07, Gilles Scokart <gs...@gmail.com> wrote:
>> > > > >
>> > > > > > 2. The buffer in FileUtils.java is too small. It's set at
>> > > > 8192. It
>> > > > > > seems to
>> > > > > > > be much better for me to set it much larger. This is due to
>> > > > > > > the fact
>> > > > > > that
>> > > > > > > it
>> > > > > > > needs to read and write simultaneously. The bigger the
>> > > > buffer is,
>> > > > > > > the smaller number of time, HD header has to move. For
>> > > > me, 65536
>> > > > > > > seems to perform much better but I haven't tried
>> > other numbers.
>> > > > > >
>> > > > > >
>> > > > > > I'd like to get more feedback on this. One use case is
>> > > > not the other.
>> > > > > This
>> > > > > > size has been borrowed from Ant copy mechanism. Maybe
>> > > > what we could
>> > > > > > do
>> > > > > is
>> > > > > > make this configurable, so that one could adapt to its
>> > > > needs. Or try
>> > > > > > to guess a good size depending on the size (when it's
>> > possible
>> > > > > > to get an
>> > > > > idea
>> > > > > > of the size before copying).
>> > > > > >
>> > > > > > Xavier
>> > > > > >
>> > > > >
>> > > > >
>> > > > > Couldn't we use the nio for that? (See
>> > > > > http://www.javalobby.org/java/forums/t17036.html)
>> > > >
>> > > >
>> > > > According to comments 10 and 11 NIO have bad performance
>> > for large
>> > > > files on linux, and input stream with byte buffer is
>> > pretty close to
>> > > > NIO for small files (see comment 13 conclusion). So I'm not sure
>> > > > switching to NIO would indeed help a lot. According to
>> > the tests in
>> > > > the thread you pointed using a 64kB buffer seems to be a
>> > good choice
>> > > > (which confirms testn tests), at least for large files. OTOH the
>> > > > last conclusion (comment 17) is different.
>> > > > So I don't really know what to think about that. We
>> > should make some
>> > > > tests on several platforms and jvms to draw conclusion
>> > ourself, but
>> > > > it takes time.
>> > > >
>> > > >
>> > > > Xavier
>> > > >
>> > > > Gilles
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > >
>> > > >
>> > > > --
>> > > > Xavier Hanin - Independent Java Consultant Manage your
>> > dependencies
>> > > > with Ivy!
>> > > > http://incubator.apache.org/ivy/
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > Xavier Hanin - Independent Java Consultant Manage your
>> > dependencies with Ivy!
>> > http://incubator.apache.org/ivy/
>> >
>>
>
>
>
> --
> Xavier Hanin - Independent Java Consultant
> Manage your dependencies with Ivy!
> http://incubator.apache.org/ivy/
>
>
--
View this message in context: http://www.nabble.com/ivy%3Aretrieve-performance-tf3907253.html#a11230918
Sent from the ivy-user mailing list archive at Nabble.com.
RE: ivy:retrieve performance
Posted by Bourzeix,
Hervé <He...@Genesys.com>.
Hi,
I got something really strange :
In the listed conflict, I got a module
A by Company selected : (2.2.1 , 1.0 ) evicted: None
How it is possible ? Using the ivy:retrieve I got 2 versions in my lib directory, which is really really bad for me.
thx
Hervé
Re: ivy:retrieve performance
Posted by Xavier Hanin <xa...@gmail.com>.
Great, thanks a lot for these data!
Xavier
On 6/14/07, Buck, Robert <rb...@verisign.com> wrote:
>
> What the performance test did is:
>
> 1. increase by powers of 2 a file size variable
>
> 2. for each file size, create/open a new file
>
> 3. write specified number of bytes to file
>
> 4. close file
>
> Tests were performed on two systems:
>
> 1. IBM ThinkPad T42 (slow) laptop, stock hardware, 1.5GB RAM
>
> 2. Dell Dual CPU, 5150, 64-bit Red Hat 4; (2) SAS 15k 146GB disks in
> RAID 1 configuration; 16 GB RAM; this is production grade hardware.
>
> Both sytems produced similar profiles (shape), however, on the
> production hardware with Linux (ext2, I believe) the numbers were
> substantially greater (makes sense when you consider the fast disks I
> have; at 3GB/sec rates, these SAS drives scream).
>
> In both cases buffers were configured for both Old IO and New IO. With
> NIO direct buffers were used. In both cases, I determined the ideal
> buffer sizes based upon experimentation to be around 32kb - 64 kb. Going
> to buffer sizes less than, or greater than, this range reduced overall
> throughput.
>
> -Bob
>
> > -----Original Message-----
> > From: Xavier Hanin [mailto:xavier.hanin@gmail.com]
> > Sent: Thursday, June 14, 2007 8:38 AM
> > To: ivy-user@incubator.apache.org
> > Subject: Re: ivy:retrieve performance
> >
> > On 6/14/07, Buck, Robert <rb...@verisign.com> wrote:
> > >
> > > Folks,
> > >
> > > We benchmarked a number of JDK IO API's for an internal project. To
> > > neutralize any questions regarding NIO vs Old IO, please
> > take a look
> > > at the attached diagram. These rates will be largely
> > identical on both
> > > Linux and Windows. Blue line NIO, red line old io.
> >
> >
> > I've uploaded the image here:
> > http://www.flickr.com/photos/41363014@N00/548163404/
> >
> > Could you give a little bit more details about how you did
> > the tests: jvm used, size of buffer used for old IO, ...
> >
> > According to your tests it seems that NIO should be preferred
> > in any case, it wasn't what some users in the javalobby
> > thread seemed to say. So I wonder what makes the difference.
> >
> > Xavier
> >
> > If you do not get the attached JPEG file, let me know. I can send it
> > > directly to you if you so request.
> > >
> > > Cheers,
> > >
> > > Bob
> > >
> > > > -----Original Message-----
> > > > From: Xavier Hanin [mailto:xavier.hanin@gmail.com]
> > > > Sent: Thursday, June 14, 2007 6:22 AM
> > > > To: ivy-user@incubator.apache.org
> > > > Subject: Re: ivy:retrieve performance
> > > >
> > > > On 6/14/07, Gilles Scokart <gs...@gmail.com> wrote:
> > > > >
> > > > > > 2. The buffer in FileUtils.java is too small. It's set at
> > > > 8192. It
> > > > > > seems to
> > > > > > > be much better for me to set it much larger. This is due to
> > > > > > > the fact
> > > > > > that
> > > > > > > it
> > > > > > > needs to read and write simultaneously. The bigger the
> > > > buffer is,
> > > > > > > the smaller number of time, HD header has to move. For
> > > > me, 65536
> > > > > > > seems to perform much better but I haven't tried
> > other numbers.
> > > > > >
> > > > > >
> > > > > > I'd like to get more feedback on this. One use case is
> > > > not the other.
> > > > > This
> > > > > > size has been borrowed from Ant copy mechanism. Maybe
> > > > what we could
> > > > > > do
> > > > > is
> > > > > > make this configurable, so that one could adapt to its
> > > > needs. Or try
> > > > > > to guess a good size depending on the size (when it's
> > possible
> > > > > > to get an
> > > > > idea
> > > > > > of the size before copying).
> > > > > >
> > > > > > Xavier
> > > > > >
> > > > >
> > > > >
> > > > > Couldn't we use the nio for that? (See
> > > > > http://www.javalobby.org/java/forums/t17036.html)
> > > >
> > > >
> > > > According to comments 10 and 11 NIO have bad performance
> > for large
> > > > files on linux, and input stream with byte buffer is
> > pretty close to
> > > > NIO for small files (see comment 13 conclusion). So I'm not sure
> > > > switching to NIO would indeed help a lot. According to
> > the tests in
> > > > the thread you pointed using a 64kB buffer seems to be a
> > good choice
> > > > (which confirms testn tests), at least for large files. OTOH the
> > > > last conclusion (comment 17) is different.
> > > > So I don't really know what to think about that. We
> > should make some
> > > > tests on several platforms and jvms to draw conclusion
> > ourself, but
> > > > it takes time.
> > > >
> > > >
> > > > Xavier
> > > >
> > > > Gilles
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Xavier Hanin - Independent Java Consultant Manage your
> > dependencies
> > > > with Ivy!
> > > > http://incubator.apache.org/ivy/
> > > >
> > >
> >
> >
> >
> > --
> > Xavier Hanin - Independent Java Consultant Manage your
> > dependencies with Ivy!
> > http://incubator.apache.org/ivy/
> >
>
--
Xavier Hanin - Independent Java Consultant
Manage your dependencies with Ivy!
http://incubator.apache.org/ivy/
RE: ivy:retrieve performance
Posted by "Buck, Robert" <rb...@verisign.com>.
What the performance test did is:
1. increase by powers of 2 a file size variable
2. for each file size, create/open a new file
3. write specified number of bytes to file
4. close file
Tests were performed on two systems:
1. IBM ThinkPad T42 (slow) laptop, stock hardware, 1.5GB RAM
2. Dell Dual CPU, 5150, 64-bit Red Hat 4; (2) SAS 15k 146GB disks in
RAID 1 configuration; 16 GB RAM; this is production grade hardware.
Both sytems produced similar profiles (shape), however, on the
production hardware with Linux (ext2, I believe) the numbers were
substantially greater (makes sense when you consider the fast disks I
have; at 3GB/sec rates, these SAS drives scream).
In both cases buffers were configured for both Old IO and New IO. With
NIO direct buffers were used. In both cases, I determined the ideal
buffer sizes based upon experimentation to be around 32kb - 64 kb. Going
to buffer sizes less than, or greater than, this range reduced overall
throughput.
-Bob
> -----Original Message-----
> From: Xavier Hanin [mailto:xavier.hanin@gmail.com]
> Sent: Thursday, June 14, 2007 8:38 AM
> To: ivy-user@incubator.apache.org
> Subject: Re: ivy:retrieve performance
>
> On 6/14/07, Buck, Robert <rb...@verisign.com> wrote:
> >
> > Folks,
> >
> > We benchmarked a number of JDK IO API's for an internal project. To
> > neutralize any questions regarding NIO vs Old IO, please
> take a look
> > at the attached diagram. These rates will be largely
> identical on both
> > Linux and Windows. Blue line NIO, red line old io.
>
>
> I've uploaded the image here:
> http://www.flickr.com/photos/41363014@N00/548163404/
>
> Could you give a little bit more details about how you did
> the tests: jvm used, size of buffer used for old IO, ...
>
> According to your tests it seems that NIO should be preferred
> in any case, it wasn't what some users in the javalobby
> thread seemed to say. So I wonder what makes the difference.
>
> Xavier
>
> If you do not get the attached JPEG file, let me know. I can send it
> > directly to you if you so request.
> >
> > Cheers,
> >
> > Bob
> >
> > > -----Original Message-----
> > > From: Xavier Hanin [mailto:xavier.hanin@gmail.com]
> > > Sent: Thursday, June 14, 2007 6:22 AM
> > > To: ivy-user@incubator.apache.org
> > > Subject: Re: ivy:retrieve performance
> > >
> > > On 6/14/07, Gilles Scokart <gs...@gmail.com> wrote:
> > > >
> > > > > 2. The buffer in FileUtils.java is too small. It's set at
> > > 8192. It
> > > > > seems to
> > > > > > be much better for me to set it much larger. This is due to
> > > > > > the fact
> > > > > that
> > > > > > it
> > > > > > needs to read and write simultaneously. The bigger the
> > > buffer is,
> > > > > > the smaller number of time, HD header has to move. For
> > > me, 65536
> > > > > > seems to perform much better but I haven't tried
> other numbers.
> > > > >
> > > > >
> > > > > I'd like to get more feedback on this. One use case is
> > > not the other.
> > > > This
> > > > > size has been borrowed from Ant copy mechanism. Maybe
> > > what we could
> > > > > do
> > > > is
> > > > > make this configurable, so that one could adapt to its
> > > needs. Or try
> > > > > to guess a good size depending on the size (when it's
> possible
> > > > > to get an
> > > > idea
> > > > > of the size before copying).
> > > > >
> > > > > Xavier
> > > > >
> > > >
> > > >
> > > > Couldn't we use the nio for that? (See
> > > > http://www.javalobby.org/java/forums/t17036.html)
> > >
> > >
> > > According to comments 10 and 11 NIO have bad performance
> for large
> > > files on linux, and input stream with byte buffer is
> pretty close to
> > > NIO for small files (see comment 13 conclusion). So I'm not sure
> > > switching to NIO would indeed help a lot. According to
> the tests in
> > > the thread you pointed using a 64kB buffer seems to be a
> good choice
> > > (which confirms testn tests), at least for large files. OTOH the
> > > last conclusion (comment 17) is different.
> > > So I don't really know what to think about that. We
> should make some
> > > tests on several platforms and jvms to draw conclusion
> ourself, but
> > > it takes time.
> > >
> > >
> > > Xavier
> > >
> > > Gilles
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > Xavier Hanin - Independent Java Consultant Manage your
> dependencies
> > > with Ivy!
> > > http://incubator.apache.org/ivy/
> > >
> >
>
>
>
> --
> Xavier Hanin - Independent Java Consultant Manage your
> dependencies with Ivy!
> http://incubator.apache.org/ivy/
>
Re: ivy:retrieve performance
Posted by Xavier Hanin <xa...@gmail.com>.
On 6/14/07, Buck, Robert <rb...@verisign.com> wrote:
>
> Folks,
>
> We benchmarked a number of JDK IO API's for an internal project. To
> neutralize any questions regarding NIO vs Old IO, please take a look at
> the attached diagram. These rates will be largely identical on both
> Linux and Windows. Blue line NIO, red line old io.
I've uploaded the image here:
http://www.flickr.com/photos/41363014@N00/548163404/
Could you give a little bit more details about how you did the tests: jvm
used, size of buffer used for old IO, ...
According to your tests it seems that NIO should be preferred in any case,
it wasn't what some users in the javalobby thread seemed to say. So I wonder
what makes the difference.
Xavier
If you do not get the attached JPEG file, let me know. I can send it
> directly to you if you so request.
>
> Cheers,
>
> Bob
>
> > -----Original Message-----
> > From: Xavier Hanin [mailto:xavier.hanin@gmail.com]
> > Sent: Thursday, June 14, 2007 6:22 AM
> > To: ivy-user@incubator.apache.org
> > Subject: Re: ivy:retrieve performance
> >
> > On 6/14/07, Gilles Scokart <gs...@gmail.com> wrote:
> > >
> > > > 2. The buffer in FileUtils.java is too small. It's set at
> > 8192. It
> > > > seems to
> > > > > be much better for me to set it much larger. This is due to the
> > > > > fact
> > > > that
> > > > > it
> > > > > needs to read and write simultaneously. The bigger the
> > buffer is,
> > > > > the smaller number of time, HD header has to move. For
> > me, 65536
> > > > > seems to perform much better but I haven't tried other numbers.
> > > >
> > > >
> > > > I'd like to get more feedback on this. One use case is
> > not the other.
> > > This
> > > > size has been borrowed from Ant copy mechanism. Maybe
> > what we could
> > > > do
> > > is
> > > > make this configurable, so that one could adapt to its
> > needs. Or try
> > > > to guess a good size depending on the size (when it's possible to
> > > > get an
> > > idea
> > > > of the size before copying).
> > > >
> > > > Xavier
> > > >
> > >
> > >
> > > Couldn't we use the nio for that? (See
> > > http://www.javalobby.org/java/forums/t17036.html)
> >
> >
> > According to comments 10 and 11 NIO have bad performance for
> > large files on linux, and input stream with byte buffer is
> > pretty close to NIO for small files (see comment 13
> > conclusion). So I'm not sure switching to NIO would indeed
> > help a lot. According to the tests in the thread you pointed
> > using a 64kB buffer seems to be a good choice (which confirms
> > testn tests), at least for large files. OTOH the last
> > conclusion (comment 17) is different.
> > So I don't really know what to think about that. We should
> > make some tests on several platforms and jvms to draw
> > conclusion ourself, but it takes time.
> >
> >
> > Xavier
> >
> > Gilles
> > >
> > >
> > >
> > >
> >
> >
> > --
> > Xavier Hanin - Independent Java Consultant Manage your
> > dependencies with Ivy!
> > http://incubator.apache.org/ivy/
> >
>
--
Xavier Hanin - Independent Java Consultant
Manage your dependencies with Ivy!
http://incubator.apache.org/ivy/
Re: ivy:retrieve performance
Posted by testn <te...@doramail.com>.
I use ivy 2.0 alpha 1. I will open an issue. BTW, 8192 is the magic number in
BufferedReader/BufferedWriter library. It came from the old day where you
don't have a lot of memory. I think it's now different. We have much more
memory available. We should kick it up a notch.
testn wrote:
>
> Hi Hanin,
>
> I'm pretty sure. I fired up filemon to check it. I use <ivy:retrieve
> sync="true" /> and it seems to copy all the file everytime.
>
> 10:11:55 AM java.exe:6380 READ
> C:\workspace\repo\.ivy\cache\org.springframework\spring-core\jars\spring-core-2.0.3.jar
> SUCCESS Offset: 163840 Length: 8192
> 10:11:55 AM java.exe:6380 WRITE
> C:\workspace\mystuff\lib\spring-core-2.0.3.jar SUCCESS Offset: 163840
> Length: 8192
>
> Does it have anything to do if I don't have md5 and sha1 files?
>
> Thanks
>
>
>
> Xavier Hanin wrote:
>>
>> On 6/14/07, Gilles Scokart <gs...@gmail.com> wrote:
>>>
>>> > 2. The buffer in FileUtils.java is too small. It's set at 8192. It
>>> seems
>>> > to
>>> > > be much better for me to set it much larger. This is due to the fact
>>> > that
>>> > > it
>>> > > needs to read and write simultaneously. The bigger the buffer is,
>>> the
>>> > > smaller number of time, HD header has to move. For me, 65536 seems
>>> to
>>> > > perform much better but I haven't tried other numbers.
>>> >
>>> >
>>> > I'd like to get more feedback on this. One use case is not the other.
>>> This
>>> > size has been borrowed from Ant copy mechanism. Maybe what we could do
>>> is
>>> > make this configurable, so that one could adapt to its needs. Or try
>>> to
>>> > guess a good size depending on the size (when it's possible to get an
>>> idea
>>> > of the size before copying).
>>> >
>>> > Xavier
>>> >
>>>
>>>
>>> Couldn't we use the nio for that? (See
>>> http://www.javalobby.org/java/forums/t17036.html)
>>
>>
>> According to comments 10 and 11 NIO have bad performance for large files
>> on
>> linux, and input stream with byte buffer is pretty close to NIO for small
>> files (see comment 13 conclusion). So I'm not sure switching to NIO would
>> indeed help a lot. According to the tests in the thread you pointed using
>> a
>> 64kB buffer seems to be a good choice (which confirms testn tests), at
>> least for large files. OTOH the last conclusion (comment 17) is
>> different.
>> So I don't really know what to think about that. We should make some
>> tests
>> on several platforms and jvms to draw conclusion ourself, but it takes
>> time.
>>
>>
>> Xavier
>>
>> Gilles
>>>
>>>
>>>
>>>
>>
>>
>> --
>> Xavier Hanin - Independent Java Consultant
>> Manage your dependencies with Ivy!
>> http://incubator.apache.org/ivy/
>>
>>
>
>
--
View this message in context: http://www.nabble.com/ivy%3Aretrieve-performance-tf3907253.html#a11121501
Sent from the ivy-user mailing list archive at Nabble.com.
Re: ivy:retrieve performance
Posted by testn <te...@doramail.com>.
Hanin,
I just found the problem. My setting accidentally contains
checkUpToDate="false". That's why it tries to copy the file every time. I
will close down the issue.
Thanks,
testn wrote:
>
> It actually happens with both sync="true" and sync="false". The files are
> copied every time but the modified timestamp will be reset to the original
> timestamp in .ivy folder.
>
>
>
> Xavier Hanin wrote:
>>
>> On 6/14/07, testn <te...@doramail.com> wrote:
>>>
>>>
>>> Hi Hanin,
>>>
>>> I'm pretty sure. I fired up filemon to check it. I use <ivy:retrieve
>>> sync="true" /> and it seems to copy all the file everytime.
>>>
>>> 10:11:55 AM java.exe:6380 READ
>>> C:\workspace\repo\.ivy\cache\org.springframework\spring-core\jars\spring-
>>> core-2.0.3.jar
>>> SUCCESS Offset: 163840 Length: 8192
>>> 10:11:55 AM java.exe:6380 WRITE
>>> C:\workspace\mystuff\lib\spring-core-2.0.3.jar SUCCESS Offset: 163840
>>> Length: 8192
>>>
>>> Does it have anything to do if I don't have md5 and sha1 files?
>>
>>
>> No, it shouldn't. It seems to be a bug so. Which version of Ivy do you
>> use?
>> Could you open a JIRA issue?
>>
>> Xavier
>>
>> Thanks
>>>
>>>
>>>
>>> Xavier Hanin wrote:
>>> >
>>> > On 6/14/07, Gilles Scokart <gs...@gmail.com> wrote:
>>> >>
>>> >> > 2. The buffer in FileUtils.java is too small. It's set at 8192. It
>>> >> seems
>>> >> > to
>>> >> > > be much better for me to set it much larger. This is due to the
>>> fact
>>> >> > that
>>> >> > > it
>>> >> > > needs to read and write simultaneously. The bigger the buffer is,
>>> the
>>> >> > > smaller number of time, HD header has to move. For me, 65536
>>> seems
>>> to
>>> >> > > perform much better but I haven't tried other numbers.
>>> >> >
>>> >> >
>>> >> > I'd like to get more feedback on this. One use case is not the
>>> other.
>>> >> This
>>> >> > size has been borrowed from Ant copy mechanism. Maybe what we could
>>> do
>>> >> is
>>> >> > make this configurable, so that one could adapt to its needs. Or
>>> try
>>> to
>>> >> > guess a good size depending on the size (when it's possible to get
>>> an
>>> >> idea
>>> >> > of the size before copying).
>>> >> >
>>> >> > Xavier
>>> >> >
>>> >>
>>> >>
>>> >> Couldn't we use the nio for that? (See
>>> >> http://www.javalobby.org/java/forums/t17036.html)
>>> >
>>> >
>>> > According to comments 10 and 11 NIO have bad performance for large
>>> files
>>> > on
>>> > linux, and input stream with byte buffer is pretty close to NIO for
>>> small
>>> > files (see comment 13 conclusion). So I'm not sure switching to NIO
>>> would
>>> > indeed help a lot. According to the tests in the thread you pointed
>>> using
>>> > a
>>> > 64kB buffer seems to be a good choice (which confirms testn tests),
>>> at
>>> > least for large files. OTOH the last conclusion (comment 17) is
>>> different.
>>> > So I don't really know what to think about that. We should make some
>>> tests
>>> > on several platforms and jvms to draw conclusion ourself, but it takes
>>> > time.
>>> >
>>> >
>>> > Xavier
>>> >
>>> > Gilles
>>> >>
>>> >>
>>> >>
>>> >>
>>> >
>>> >
>>> > --
>>> > Xavier Hanin - Independent Java Consultant
>>> > Manage your dependencies with Ivy!
>>> > http://incubator.apache.org/ivy/
>>> >
>>> >
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/ivy%3Aretrieve-performance-tf3907253.html#a11121249
>>> Sent from the ivy-user mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>> --
>> Xavier Hanin - Independent Java Consultant
>> Manage your dependencies with Ivy!
>> http://incubator.apache.org/ivy/
>>
>>
>
>
--
View this message in context: http://www.nabble.com/ivy%3Aretrieve-performance-tf3907253.html#a11121635
Sent from the ivy-user mailing list archive at Nabble.com.
Re: ivy:retrieve performance
Posted by testn <te...@doramail.com>.
It actually happens with both sync="true" and sync="false". The files are
copied every time but the modified timestamp will be reset to the original
timestamp in .ivy folder.
Xavier Hanin wrote:
>
> On 6/14/07, testn <te...@doramail.com> wrote:
>>
>>
>> Hi Hanin,
>>
>> I'm pretty sure. I fired up filemon to check it. I use <ivy:retrieve
>> sync="true" /> and it seems to copy all the file everytime.
>>
>> 10:11:55 AM java.exe:6380 READ
>> C:\workspace\repo\.ivy\cache\org.springframework\spring-core\jars\spring-
>> core-2.0.3.jar
>> SUCCESS Offset: 163840 Length: 8192
>> 10:11:55 AM java.exe:6380 WRITE
>> C:\workspace\mystuff\lib\spring-core-2.0.3.jar SUCCESS Offset: 163840
>> Length: 8192
>>
>> Does it have anything to do if I don't have md5 and sha1 files?
>
>
> No, it shouldn't. It seems to be a bug so. Which version of Ivy do you
> use?
> Could you open a JIRA issue?
>
> Xavier
>
> Thanks
>>
>>
>>
>> Xavier Hanin wrote:
>> >
>> > On 6/14/07, Gilles Scokart <gs...@gmail.com> wrote:
>> >>
>> >> > 2. The buffer in FileUtils.java is too small. It's set at 8192. It
>> >> seems
>> >> > to
>> >> > > be much better for me to set it much larger. This is due to the
>> fact
>> >> > that
>> >> > > it
>> >> > > needs to read and write simultaneously. The bigger the buffer is,
>> the
>> >> > > smaller number of time, HD header has to move. For me, 65536 seems
>> to
>> >> > > perform much better but I haven't tried other numbers.
>> >> >
>> >> >
>> >> > I'd like to get more feedback on this. One use case is not the
>> other.
>> >> This
>> >> > size has been borrowed from Ant copy mechanism. Maybe what we could
>> do
>> >> is
>> >> > make this configurable, so that one could adapt to its needs. Or try
>> to
>> >> > guess a good size depending on the size (when it's possible to get
>> an
>> >> idea
>> >> > of the size before copying).
>> >> >
>> >> > Xavier
>> >> >
>> >>
>> >>
>> >> Couldn't we use the nio for that? (See
>> >> http://www.javalobby.org/java/forums/t17036.html)
>> >
>> >
>> > According to comments 10 and 11 NIO have bad performance for large
>> files
>> > on
>> > linux, and input stream with byte buffer is pretty close to NIO for
>> small
>> > files (see comment 13 conclusion). So I'm not sure switching to NIO
>> would
>> > indeed help a lot. According to the tests in the thread you pointed
>> using
>> > a
>> > 64kB buffer seems to be a good choice (which confirms testn tests), at
>> > least for large files. OTOH the last conclusion (comment 17) is
>> different.
>> > So I don't really know what to think about that. We should make some
>> tests
>> > on several platforms and jvms to draw conclusion ourself, but it takes
>> > time.
>> >
>> >
>> > Xavier
>> >
>> > Gilles
>> >>
>> >>
>> >>
>> >>
>> >
>> >
>> > --
>> > Xavier Hanin - Independent Java Consultant
>> > Manage your dependencies with Ivy!
>> > http://incubator.apache.org/ivy/
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/ivy%3Aretrieve-performance-tf3907253.html#a11121249
>> Sent from the ivy-user mailing list archive at Nabble.com.
>>
>>
>
>
> --
> Xavier Hanin - Independent Java Consultant
> Manage your dependencies with Ivy!
> http://incubator.apache.org/ivy/
>
>
--
View this message in context: http://www.nabble.com/ivy%3Aretrieve-performance-tf3907253.html#a11121509
Sent from the ivy-user mailing list archive at Nabble.com.
Re: ivy:retrieve performance
Posted by Xavier Hanin <xa...@gmail.com>.
On 6/14/07, testn <te...@doramail.com> wrote:
>
>
> Hi Hanin,
>
> I'm pretty sure. I fired up filemon to check it. I use <ivy:retrieve
> sync="true" /> and it seems to copy all the file everytime.
>
> 10:11:55 AM java.exe:6380 READ
> C:\workspace\repo\.ivy\cache\org.springframework\spring-core\jars\spring-
> core-2.0.3.jar
> SUCCESS Offset: 163840 Length: 8192
> 10:11:55 AM java.exe:6380 WRITE
> C:\workspace\mystuff\lib\spring-core-2.0.3.jar SUCCESS Offset: 163840
> Length: 8192
>
> Does it have anything to do if I don't have md5 and sha1 files?
No, it shouldn't. It seems to be a bug so. Which version of Ivy do you use?
Could you open a JIRA issue?
Xavier
Thanks
>
>
>
> Xavier Hanin wrote:
> >
> > On 6/14/07, Gilles Scokart <gs...@gmail.com> wrote:
> >>
> >> > 2. The buffer in FileUtils.java is too small. It's set at 8192. It
> >> seems
> >> > to
> >> > > be much better for me to set it much larger. This is due to the
> fact
> >> > that
> >> > > it
> >> > > needs to read and write simultaneously. The bigger the buffer is,
> the
> >> > > smaller number of time, HD header has to move. For me, 65536 seems
> to
> >> > > perform much better but I haven't tried other numbers.
> >> >
> >> >
> >> > I'd like to get more feedback on this. One use case is not the other.
> >> This
> >> > size has been borrowed from Ant copy mechanism. Maybe what we could
> do
> >> is
> >> > make this configurable, so that one could adapt to its needs. Or try
> to
> >> > guess a good size depending on the size (when it's possible to get an
> >> idea
> >> > of the size before copying).
> >> >
> >> > Xavier
> >> >
> >>
> >>
> >> Couldn't we use the nio for that? (See
> >> http://www.javalobby.org/java/forums/t17036.html)
> >
> >
> > According to comments 10 and 11 NIO have bad performance for large files
> > on
> > linux, and input stream with byte buffer is pretty close to NIO for
> small
> > files (see comment 13 conclusion). So I'm not sure switching to NIO
> would
> > indeed help a lot. According to the tests in the thread you pointed
> using
> > a
> > 64kB buffer seems to be a good choice (which confirms testn tests), at
> > least for large files. OTOH the last conclusion (comment 17) is
> different.
> > So I don't really know what to think about that. We should make some
> tests
> > on several platforms and jvms to draw conclusion ourself, but it takes
> > time.
> >
> >
> > Xavier
> >
> > Gilles
> >>
> >>
> >>
> >>
> >
> >
> > --
> > Xavier Hanin - Independent Java Consultant
> > Manage your dependencies with Ivy!
> > http://incubator.apache.org/ivy/
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/ivy%3Aretrieve-performance-tf3907253.html#a11121249
> Sent from the ivy-user mailing list archive at Nabble.com.
>
>
--
Xavier Hanin - Independent Java Consultant
Manage your dependencies with Ivy!
http://incubator.apache.org/ivy/
Re: ivy:retrieve performance
Posted by testn <te...@doramail.com>.
Hi Hanin,
I'm pretty sure. I fired up filemon to check it. I use <ivy:retrieve
sync="true" /> and it seems to copy all the file everytime.
10:11:55 AM java.exe:6380 READ
C:\workspace\repo\.ivy\cache\org.springframework\spring-core\jars\spring-core-2.0.3.jar
SUCCESS Offset: 163840 Length: 8192
10:11:55 AM java.exe:6380 WRITE
C:\workspace\mystuff\lib\spring-core-2.0.3.jar SUCCESS Offset: 163840
Length: 8192
Does it have anything to do if I don't have md5 and sha1 files?
Thanks
Xavier Hanin wrote:
>
> On 6/14/07, Gilles Scokart <gs...@gmail.com> wrote:
>>
>> > 2. The buffer in FileUtils.java is too small. It's set at 8192. It
>> seems
>> > to
>> > > be much better for me to set it much larger. This is due to the fact
>> > that
>> > > it
>> > > needs to read and write simultaneously. The bigger the buffer is, the
>> > > smaller number of time, HD header has to move. For me, 65536 seems to
>> > > perform much better but I haven't tried other numbers.
>> >
>> >
>> > I'd like to get more feedback on this. One use case is not the other.
>> This
>> > size has been borrowed from Ant copy mechanism. Maybe what we could do
>> is
>> > make this configurable, so that one could adapt to its needs. Or try to
>> > guess a good size depending on the size (when it's possible to get an
>> idea
>> > of the size before copying).
>> >
>> > Xavier
>> >
>>
>>
>> Couldn't we use the nio for that? (See
>> http://www.javalobby.org/java/forums/t17036.html)
>
>
> According to comments 10 and 11 NIO have bad performance for large files
> on
> linux, and input stream with byte buffer is pretty close to NIO for small
> files (see comment 13 conclusion). So I'm not sure switching to NIO would
> indeed help a lot. According to the tests in the thread you pointed using
> a
> 64kB buffer seems to be a good choice (which confirms testn tests), at
> least for large files. OTOH the last conclusion (comment 17) is different.
> So I don't really know what to think about that. We should make some tests
> on several platforms and jvms to draw conclusion ourself, but it takes
> time.
>
>
> Xavier
>
> Gilles
>>
>>
>>
>>
>
>
> --
> Xavier Hanin - Independent Java Consultant
> Manage your dependencies with Ivy!
> http://incubator.apache.org/ivy/
>
>
--
View this message in context: http://www.nabble.com/ivy%3Aretrieve-performance-tf3907253.html#a11121249
Sent from the ivy-user mailing list archive at Nabble.com.
RE: ivy:retrieve performance
Posted by "Buck, Robert" <rb...@verisign.com>.
Oh, to add one more comment. The performance should not be an issue with
NIO if the code follows a few commonly documented strategies.
What is an issue are the gaps in Sun's implementation of NIO. We, at
VeriSign, discovered a number of issues in NIO where Sun failed to
complete the NIO code base. There are calls, that when made to NFS for
instance, will literally cause an exception to be thrown indicating
"SuchAndSuch API is not presently supported on NFS" -- this is where the
real frustration with NIO is in my own mind. We have found some places
where we MAY have to literally write some code in C++, write a JNI
interface, and use our own IO logic instead of Sun's.
Summary: For the simple case NIO is probably fine. If you try to do
something slightly on the edge, you may run into all sorts of problems
with NIO.
-Bob
> -----Original Message-----
> From: Buck, Robert [mailto:rbuck@verisign.com]
> Sent: Thursday, June 14, 2007 6:46 AM
> To: ivy-user@incubator.apache.org
> Subject: RE: ivy:retrieve performance
>
> Folks,
>
> We benchmarked a number of JDK IO API's for an internal
> project. To neutralize any questions regarding NIO vs Old IO,
> please take a look at the attached diagram. These rates will
> be largely identical on both Linux and Windows. Blue line
> NIO, red line old io.
>
> If you do not get the attached JPEG file, let me know. I can
> send it directly to you if you so request.
>
> Cheers,
>
> Bob
>
> > -----Original Message-----
> > From: Xavier Hanin [mailto:xavier.hanin@gmail.com]
> > Sent: Thursday, June 14, 2007 6:22 AM
> > To: ivy-user@incubator.apache.org
> > Subject: Re: ivy:retrieve performance
> >
> > On 6/14/07, Gilles Scokart <gs...@gmail.com> wrote:
> > >
> > > > 2. The buffer in FileUtils.java is too small. It's set at
> > 8192. It
> > > > seems to
> > > > > be much better for me to set it much larger. This is
> due to the
> > > > > fact
> > > > that
> > > > > it
> > > > > needs to read and write simultaneously. The bigger the
> > buffer is,
> > > > > the smaller number of time, HD header has to move. For
> > me, 65536
> > > > > seems to perform much better but I haven't tried
> other numbers.
> > > >
> > > >
> > > > I'd like to get more feedback on this. One use case is
> > not the other.
> > > This
> > > > size has been borrowed from Ant copy mechanism. Maybe
> > what we could
> > > > do
> > > is
> > > > make this configurable, so that one could adapt to its
> > needs. Or try
> > > > to guess a good size depending on the size (when it's
> possible to
> > > > get an
> > > idea
> > > > of the size before copying).
> > > >
> > > > Xavier
> > > >
> > >
> > >
> > > Couldn't we use the nio for that? (See
> > > http://www.javalobby.org/java/forums/t17036.html)
> >
> >
> > According to comments 10 and 11 NIO have bad performance for large
> > files on linux, and input stream with byte buffer is pretty
> close to
> > NIO for small files (see comment 13 conclusion). So I'm not sure
> > switching to NIO would indeed help a lot. According to the tests in
> > the thread you pointed using a 64kB buffer seems to be a
> good choice
> > (which confirms testn tests), at least for large files.
> OTOH the last
> > conclusion (comment 17) is different.
> > So I don't really know what to think about that. We should
> make some
> > tests on several platforms and jvms to draw conclusion
> ourself, but it
> > takes time.
> >
> >
> > Xavier
> >
> > Gilles
> > >
> > >
> > >
> > >
> >
> >
> > --
> > Xavier Hanin - Independent Java Consultant Manage your
> > dependencies with Ivy!
> > http://incubator.apache.org/ivy/
> >
>
RE: ivy:retrieve performance
Posted by "Buck, Robert" <rb...@verisign.com>.
Folks,
We benchmarked a number of JDK IO API's for an internal project. To
neutralize any questions regarding NIO vs Old IO, please take a look at
the attached diagram. These rates will be largely identical on both
Linux and Windows. Blue line NIO, red line old io.
If you do not get the attached JPEG file, let me know. I can send it
directly to you if you so request.
Cheers,
Bob
> -----Original Message-----
> From: Xavier Hanin [mailto:xavier.hanin@gmail.com]
> Sent: Thursday, June 14, 2007 6:22 AM
> To: ivy-user@incubator.apache.org
> Subject: Re: ivy:retrieve performance
>
> On 6/14/07, Gilles Scokart <gs...@gmail.com> wrote:
> >
> > > 2. The buffer in FileUtils.java is too small. It's set at
> 8192. It
> > > seems to
> > > > be much better for me to set it much larger. This is due to the
> > > > fact
> > > that
> > > > it
> > > > needs to read and write simultaneously. The bigger the
> buffer is,
> > > > the smaller number of time, HD header has to move. For
> me, 65536
> > > > seems to perform much better but I haven't tried other numbers.
> > >
> > >
> > > I'd like to get more feedback on this. One use case is
> not the other.
> > This
> > > size has been borrowed from Ant copy mechanism. Maybe
> what we could
> > > do
> > is
> > > make this configurable, so that one could adapt to its
> needs. Or try
> > > to guess a good size depending on the size (when it's possible to
> > > get an
> > idea
> > > of the size before copying).
> > >
> > > Xavier
> > >
> >
> >
> > Couldn't we use the nio for that? (See
> > http://www.javalobby.org/java/forums/t17036.html)
>
>
> According to comments 10 and 11 NIO have bad performance for
> large files on linux, and input stream with byte buffer is
> pretty close to NIO for small files (see comment 13
> conclusion). So I'm not sure switching to NIO would indeed
> help a lot. According to the tests in the thread you pointed
> using a 64kB buffer seems to be a good choice (which confirms
> testn tests), at least for large files. OTOH the last
> conclusion (comment 17) is different.
> So I don't really know what to think about that. We should
> make some tests on several platforms and jvms to draw
> conclusion ourself, but it takes time.
>
>
> Xavier
>
> Gilles
> >
> >
> >
> >
>
>
> --
> Xavier Hanin - Independent Java Consultant Manage your
> dependencies with Ivy!
> http://incubator.apache.org/ivy/
>
Re: ivy:retrieve performance
Posted by Xavier Hanin <xa...@gmail.com>.
On 6/14/07, Gilles Scokart <gs...@gmail.com> wrote:
>
> > 2. The buffer in FileUtils.java is too small. It's set at 8192. It seems
> > to
> > > be much better for me to set it much larger. This is due to the fact
> > that
> > > it
> > > needs to read and write simultaneously. The bigger the buffer is, the
> > > smaller number of time, HD header has to move. For me, 65536 seems to
> > > perform much better but I haven't tried other numbers.
> >
> >
> > I'd like to get more feedback on this. One use case is not the other.
> This
> > size has been borrowed from Ant copy mechanism. Maybe what we could do
> is
> > make this configurable, so that one could adapt to its needs. Or try to
> > guess a good size depending on the size (when it's possible to get an
> idea
> > of the size before copying).
> >
> > Xavier
> >
>
>
> Couldn't we use the nio for that? (See
> http://www.javalobby.org/java/forums/t17036.html)
According to comments 10 and 11 NIO have bad performance for large files on
linux, and input stream with byte buffer is pretty close to NIO for small
files (see comment 13 conclusion). So I'm not sure switching to NIO would
indeed help a lot. According to the tests in the thread you pointed using a
64kB buffer seems to be a good choice (which confirms testn tests), at
least for large files. OTOH the last conclusion (comment 17) is different.
So I don't really know what to think about that. We should make some tests
on several platforms and jvms to draw conclusion ourself, but it takes time.
Xavier
Gilles
>
>
>
>
--
Xavier Hanin - Independent Java Consultant
Manage your dependencies with Ivy!
http://incubator.apache.org/ivy/
RE: ivy:retrieve performance
Posted by Gilles Scokart <gs...@gmail.com>.
> 2. The buffer in FileUtils.java is too small. It's set at 8192. It seems
> to
> > be much better for me to set it much larger. This is due to the fact
> that
> > it
> > needs to read and write simultaneously. The bigger the buffer is, the
> > smaller number of time, HD header has to move. For me, 65536 seems to
> > perform much better but I haven't tried other numbers.
>
>
> I'd like to get more feedback on this. One use case is not the other. This
> size has been borrowed from Ant copy mechanism. Maybe what we could do is
> make this configurable, so that one could adapt to its needs. Or try to
> guess a good size depending on the size (when it's possible to get an idea
> of the size before copying).
>
> Xavier
>
Couldn't we use the nio for that? (See
http://www.javalobby.org/java/forums/t17036.html)
Gilles
Re: ivy:retrieve performance
Posted by Xavier Hanin <xa...@gmail.com>.
On 6/12/07, testn <te...@doramail.com> wrote:
>
>
> Hi,
>
> I have a couple comments regarding ivy:retrieve.
> 1. It seems that when I turn "sync" option on, ivy:retrieve will copy
> files
> from .ivy folder to lib folder everytime even though files in lib folder
> are
> updated. It should have an option to check file size and timestamp to
> ensure
> that the file is updated.
This is strange, which version of Ivy do you use? The sync option shouldn't
change the up to date check before copying files from cache to lib dir. Are
you sure the dates haven't changed?
2. The buffer in FileUtils.java is too small. It's set at 8192. It seems to
> be much better for me to set it much larger. This is due to the fact that
> it
> needs to read and write simultaneously. The bigger the buffer is, the
> smaller number of time, HD header has to move. For me, 65536 seems to
> perform much better but I haven't tried other numbers.
I'd like to get more feedback on this. One use case is not the other. This
size has been borrowed from Ant copy mechanism. Maybe what we could do is
make this configurable, so that one could adapt to its needs. Or try to
guess a good size depending on the size (when it's possible to get an idea
of the size before copying).
Xavier
--
> View this message in context:
> http://www.nabble.com/ivy%3Aretrieve-performance-tf3907253.html#a11077846
> Sent from the ivy-user mailing list archive at Nabble.com.
>
>
--
Xavier Hanin - Independent Java Consultant
Manage your dependencies with Ivy!
http://incubator.apache.org/ivy/
Re: ivy:retrieve performance
Posted by testn <te...@doramail.com>.
Any take on this? The performance seems to be sluggish when there are a lot
of big libraries to download and sync
Thanks,
--
View this message in context: http://www.nabble.com/ivy%3Aretrieve-performance-tf3907253.html#a11118311
Sent from the ivy-user mailing list archive at Nabble.com.