You are viewing a plain text version of this content. The canonical link for it is here.
Posted to ivy-user@ant.apache.org by JamesPoli <Ja...@sas.com> on 2007/10/09 02:04:09 UTC
RE: problem with work files created in cache by ivy:resolve (ivy
1.4)
As Jim Adams had stated we'd moved our cache to a tmp location to get around
this problem. Another solution I've been working on is using a lock file to
protect from having two or more resovles use the cache at the same time.
To "guard" the creation/deletion of the lock file I'm using a Java FileLock
- it acts like a Java object monitor that is visible across JVM's. It
appears to work but for multiple processes (15) using resolve. Could there
be something I'm missing? Should I also lock the publish? I'm thinking if
I lock too many ivy tasks I'll end up defeating the whole purpose I'm doing
it, performance gains by using a single cache.
Thanks in advance, Jim :super:
Jim Adams-6 wrote:
>
> We have something similar but our only recourse was to put the ivy cache
> in a tmp location and clear it after every build. Ivy can't handle
> multiple processes accessing the cache at the same time.
>
>> -----Original Message-----
>> From: Gregoire Henry [mailto:ghynxmail@gmail.com]
>> Sent: Friday, July 27, 2007 11:21 AM
>> To: ivy-user@incubator.apache.org
>> Subject: problem with work files created in cache by
>> ivy:resolve (ivy 1.4)
>>
>> hello
>>
>> I have a problem with the files created by ivy in the cache
>> during the resolve phase ..
>> Ivy create some files nammed [org]-[module]-[conf].xml in the
>> cache , should'nt those files be created in a temporary
>> location outside of the
>> cache, and preferably in project specific directory ?
>>
>> My use case is the following
>>
>> We have a Continuous Build Server who can build at the same
>> time several different build from the same project ....
>> In my build i create a clean.resolve task for deleting the
>> [org]-[module]-[conf].xml files at the beginnig of each build
>> ... as we can have multiple build at the same time for the
>> same project I sometime have some build failled because of
>> the deletion of this files by another starting build ...
>>
>> I still have created different cache location for my
>> different project , but this use case should drive me to
>> create a different cache for each build !!!
>>
>> Have you a solution for that ?
>> would it be possible to have an ivy.temp.dir properties ?
>> Is there a different implementation in 2.0.0 ?
>>
>>
>> regards
>>
>> Greg
>>
>>
>>
>>
>
>
%-|%-|:confused::super::super::super::super::super::super::super:
--
View this message in context: http://www.nabble.com/problem-with-work-files-created-in-cache-by-ivy%3Aresolve-%28ivy-1.4%29-tf4158540.html#a13107058
Sent from the ivy-user mailing list archive at Nabble.com.
Re: problem with work files created in cache by ivy:resolve (ivy
1.4)
Posted by Steve Loughran <st...@apache.org>.
Buck, Robert wrote:
> Hi,
>
> I am new to this thread. I don't mean to butt in, but I have one
> question regarding it, and this may be slightly unrelated...
>
> I see mention use of the FileLock class. When I saw a reference to this,
> I was immediately concerned.
>
> IVY itself does NOT use FileLock, correct? (I hope not)
>
> The reason I ask is that at VeriSign we discovered that FileLock does
> not work on any of the NFS devices we have. The class itself throws an
> exception, whose localized message effectively states, "FileLock is
> unsupported on NFS devices.".
to put it differently: NFS has historically not supported locking, be a
stateless UDP-based protocol from the early eighties, designed to
support diskless 68K workstations on 1 mbit LAN.
> Additionally, there are many File oriented
> classes in NIO that either: (a) do not work at all, or (b) are a LOT
> slower than old IO for IO intensive applications.
Is this documented somewhere?
>
> So this is all to say, please, only use NIO when you have NFS, NetAPP,
> or EMC, hardware to test on, and you can prove with complete certainty
> the individual API works. You may be surprised to find that the more
> crucial parts of NIO is effectively broken.
1. do they provide VMWare images of their OS?
2. netapp servers do support file locking with other protocols (e.g
samba)(*). So perhaps the issue is choice of NFS rather than modern
alternatives....
(*) Though they have other defects, like the #of files in a directory is
limited to 65K.
testing over NFS is something that could be done between two linux
images in a vmware cluster. testing on EMC hardware? Unlikely, unless
EMC want to donate kit to apache, the way others do.
-steve
--
Steve Loughran http://www.1060.org/blogxter/publish/5
Author: Ant in Action http://antbook.org/
RE: problem with work files created in cache by ivy:resolve (ivy 1.4)
Posted by James Poli <Ja...@sas.com>.
Xavier and Robert,
Sorry I haven't opened an issue yet. Robert thanks for the information
on filelocks. I knew they
had problems on various platforms/hardware so I'd tested using them with
several use cases to verify
they worked where we needed them before I got too far into
implementation. FWIW, I did have to change
the use of the filelock on Linux from blocking to non-blocking after
noticing a blocked lock took
several minutes to see a release! BTW, I'm doing all the filelocking
via an ANT task I wrote before
and after the ivy:resolve task.
Jim
-----Original Message-----
From: Xavier Hanin [mailto:xavier.hanin@gmail.com]
Sent: Wednesday, October 10, 2007 1:46 PM
To: ivy-user@incubator.apache.org
Subject: Re: problem with work files created in cache by ivy:resolve
(ivy 1.4)
On 10/10/07, Buck, Robert <rb...@verisign.com> wrote:
>
> Hi,
>
> I am new to this thread. I don't mean to butt in, but I have one
> question regarding it, and this may be slightly unrelated...
>
> I see mention use of the FileLock class. When I saw a reference to
this,
> I was immediately concerned.
>
> IVY itself does NOT use FileLock, correct? (I hope not)
Correct.
The reason I ask is that at VeriSign we discovered that FileLock does
> not work on any of the NFS devices we have. The class itself throws an
> exception, whose localized message effectively states, "FileLock is
> unsupported on NFS devices.". Additionally, there are many File
oriented
> classes in NIO that either: (a) do not work at all, or (b) are a LOT
> slower than old IO for IO intensive applications.
>
> So this is all to say, please, only use NIO when you have NFS, NetAPP,
> or EMC, hardware to test on, and you can prove with complete certainty
> the individual API works. You may be surprised to find that the more
> crucial parts of NIO is effectively broken.
Thanks for your experience feedback, we will take that into account when
we
will address this issue. Maybe you could add this comment to the issue
too
(not sure if James already opened it).
Xavier
Thanks in advance, no response necessary,
>
> Bob
>
> -----Original Message-----
> From: Xavier Hanin [mailto:xavier.hanin@gmail.com]
> Sent: Wednesday, October 10, 2007 1:38 AM
> To: ivy-user@incubator.apache.org
> Subject: Re: problem with work files created in cache by ivy:resolve
> (ivy 1.4)
>
> On 10/9/07, James Poli <Ja...@sas.com> wrote:
> >
> > Xavier,
> >
> > Thanks for the reply. Yes I can open an issue. Question: if I lock
> > the entire resolve should I also lock the publish? In other words
can
>
> > the Ivy resolve task be updating the module resolution results while
> > an Ivy publish (in another process) is trying to use them?
>
>
> Yes, you need to lock the publish od the same module, because Ivy
> doesn't update its resolution data transactionnally.
>
> When you open the issue, you can link it to IVY-605, which points to
> similar problem.
>
> Xavier
>
> Thanks again for any help,
> > Jim
> >
> > -----Original Message-----
> > From: Xavier Hanin [mailto:xavier.hanin@gmail.com]
> > Sent: Tuesday, October 09, 2007 1:20 AM
> > To: ivy-user@incubator.apache.org
> > Subject: Re: problem with work files created in cache by ivy:resolve
> > (ivy 1.4)
> >
> > On 10/9/07, JamesPoli <Ja...@sas.com> wrote:
> > >
> > >
> > > As Jim Adams had stated we'd moved our cache to a tmp location to
> > > get around this problem. Another solution I've been working on is
> > > using a lock
> > file
> > > to
> > > protect from having two or more resovles use the cache at the
same
> > time.
> > > To "guard" the creation/deletion of the lock file I'm using a Java
> > > FileLock
> > > - it acts like a Java object monitor that is visible across JVM's.
> > > It appears to work but for multiple processes (15) using resolve.
> > > Could there be something I'm missing? Should I also lock the
> > > publish? I'm
> > thinking
> > > if
> > > I lock too many ivy tasks I'll end up defeating the whole purpose
> > > I'm doing it, performance gains by using a single cache.
> >
> >
> > Indeed, using a lock file can defeat the whole purpose of using a
> > single cache. The main problem is that we handle two different
things
> > in the
> > cache:
> > cache of downloaded data (ivy files and artifacts), and cache of a
> > module resolution results. The former is what is interesting to
share
> > and less a problem to lock, the latter is not really interesting to
> > share and more a problem to lock. Once again we see that the cache
> > management in Ivy needs improvement, it's one other vote for
IVY-399.
> > But this could even deserve a new issue: make Ivy able to use a
> > repository cache shared by multiple processes. Could you create such
> > an issue?
> >
> > Xavier
> >
> > Thanks in advance, Jim :super:
> > >
> > >
> > > Jim Adams-6 wrote:
> > > >
> > > > We have something similar but our only recourse was to put the
ivy
> > cache
> > > > in a tmp location and clear it after every build. Ivy can't
handle
>
> > > > multiple processes accessing the cache at the same time.
> > > >
> > > >> -----Original Message-----
> > > >> From: Gregoire Henry [mailto:ghynxmail@gmail.com]
> > > >> Sent: Friday, July 27, 2007 11:21 AM
> > > >> To: ivy-user@incubator.apache.org
> > > >> Subject: problem with work files created in cache by
ivy:resolve
> > > >> (ivy 1.4)
> > > >>
> > > >> hello
> > > >>
> > > >> I have a problem with the files created by ivy in the cache
> > > >> during the resolve phase ..
> > > >> Ivy create some files nammed [org]-[module]-[conf].xml in the
> > > >> cache , should'nt those files be created in a temporary
location
> > > >> outside of the
> > > >> cache, and preferably in project specific directory ?
> > > >>
> > > >> My use case is the following
> > > >>
> > > >> We have a Continuous Build Server who can build at the same
time
> > > >> several different build from the same project ....
> > > >> In my build i create a clean.resolve task for deleting the
> > > >> [org]-[module]-[conf].xml files at the beginnig of each build
...
>
> > > >> as we can have multiple build at the same time for the same
> > > >> project I sometime have some build failled because of the
> > > >> deletion of this files by another starting build ...
> > > >>
> > > >> I still have created different cache location for my different
> > > >> project , but this use case should drive me to create a
different
>
> > > >> cache for each build !!!
> > > >>
> > > >> Have you a solution for that ?
> > > >> would it be possible to have an ivy.temp.dir properties ?
> > > >> Is there a different implementation in 2.0.0 ?
> > > >>
> > > >>
> > > >> regards
> > > >>
> > > >> Greg
> > > >>
> > > >>
> > > >>
> > > >>
> > > >
> > > >
> > > %-|%-|:confused::super::super::super::super::super::super::super:
> > > --
> > > View this message in context:
> > >
> >
http://www.nabble.com/problem-with-work-files-created-in-cache-by-ivy%
> > 3A
> > resolve-%28ivy-1.4%29-tf4158540.html#a13107058
> > > Sent from the ivy-user mailing list archive at Nabble.com.
> > >
> > >
> >
> >
> > --
> > Xavier Hanin - Independent Java Consultant http://xhab.blogspot.com/
> > http://incubator.apache.org/ivy/ http://www.xoocode.org/
> >
>
>
>
> --
> Xavier Hanin - Independent Java Consultant http://xhab.blogspot.com/
> http://incubator.apache.org/ivy/ http://www.xoocode.org/
>
--
Xavier Hanin - Independent Java Consultant
http://xhab.blogspot.com/
http://incubator.apache.org/ivy/
http://www.xoocode.org/
Re: problem with work files created in cache by ivy:resolve (ivy 1.4)
Posted by Xavier Hanin <xa...@gmail.com>.
On 10/10/07, Buck, Robert <rb...@verisign.com> wrote:
>
> Hi,
>
> I am new to this thread. I don't mean to butt in, but I have one
> question regarding it, and this may be slightly unrelated...
>
> I see mention use of the FileLock class. When I saw a reference to this,
> I was immediately concerned.
>
> IVY itself does NOT use FileLock, correct? (I hope not)
Correct.
The reason I ask is that at VeriSign we discovered that FileLock does
> not work on any of the NFS devices we have. The class itself throws an
> exception, whose localized message effectively states, "FileLock is
> unsupported on NFS devices.". Additionally, there are many File oriented
> classes in NIO that either: (a) do not work at all, or (b) are a LOT
> slower than old IO for IO intensive applications.
>
> So this is all to say, please, only use NIO when you have NFS, NetAPP,
> or EMC, hardware to test on, and you can prove with complete certainty
> the individual API works. You may be surprised to find that the more
> crucial parts of NIO is effectively broken.
Thanks for your experience feedback, we will take that into account when we
will address this issue. Maybe you could add this comment to the issue too
(not sure if James already opened it).
Xavier
Thanks in advance, no response necessary,
>
> Bob
>
> -----Original Message-----
> From: Xavier Hanin [mailto:xavier.hanin@gmail.com]
> Sent: Wednesday, October 10, 2007 1:38 AM
> To: ivy-user@incubator.apache.org
> Subject: Re: problem with work files created in cache by ivy:resolve
> (ivy 1.4)
>
> On 10/9/07, James Poli <Ja...@sas.com> wrote:
> >
> > Xavier,
> >
> > Thanks for the reply. Yes I can open an issue. Question: if I lock
> > the entire resolve should I also lock the publish? In other words can
>
> > the Ivy resolve task be updating the module resolution results while
> > an Ivy publish (in another process) is trying to use them?
>
>
> Yes, you need to lock the publish od the same module, because Ivy
> doesn't update its resolution data transactionnally.
>
> When you open the issue, you can link it to IVY-605, which points to
> similar problem.
>
> Xavier
>
> Thanks again for any help,
> > Jim
> >
> > -----Original Message-----
> > From: Xavier Hanin [mailto:xavier.hanin@gmail.com]
> > Sent: Tuesday, October 09, 2007 1:20 AM
> > To: ivy-user@incubator.apache.org
> > Subject: Re: problem with work files created in cache by ivy:resolve
> > (ivy 1.4)
> >
> > On 10/9/07, JamesPoli <Ja...@sas.com> wrote:
> > >
> > >
> > > As Jim Adams had stated we'd moved our cache to a tmp location to
> > > get around this problem. Another solution I've been working on is
> > > using a lock
> > file
> > > to
> > > protect from having two or more resovles use the cache at the same
> > time.
> > > To "guard" the creation/deletion of the lock file I'm using a Java
> > > FileLock
> > > - it acts like a Java object monitor that is visible across JVM's.
> > > It appears to work but for multiple processes (15) using resolve.
> > > Could there be something I'm missing? Should I also lock the
> > > publish? I'm
> > thinking
> > > if
> > > I lock too many ivy tasks I'll end up defeating the whole purpose
> > > I'm doing it, performance gains by using a single cache.
> >
> >
> > Indeed, using a lock file can defeat the whole purpose of using a
> > single cache. The main problem is that we handle two different things
> > in the
> > cache:
> > cache of downloaded data (ivy files and artifacts), and cache of a
> > module resolution results. The former is what is interesting to share
> > and less a problem to lock, the latter is not really interesting to
> > share and more a problem to lock. Once again we see that the cache
> > management in Ivy needs improvement, it's one other vote for IVY-399.
> > But this could even deserve a new issue: make Ivy able to use a
> > repository cache shared by multiple processes. Could you create such
> > an issue?
> >
> > Xavier
> >
> > Thanks in advance, Jim :super:
> > >
> > >
> > > Jim Adams-6 wrote:
> > > >
> > > > We have something similar but our only recourse was to put the ivy
> > cache
> > > > in a tmp location and clear it after every build. Ivy can't handle
>
> > > > multiple processes accessing the cache at the same time.
> > > >
> > > >> -----Original Message-----
> > > >> From: Gregoire Henry [mailto:ghynxmail@gmail.com]
> > > >> Sent: Friday, July 27, 2007 11:21 AM
> > > >> To: ivy-user@incubator.apache.org
> > > >> Subject: problem with work files created in cache by ivy:resolve
> > > >> (ivy 1.4)
> > > >>
> > > >> hello
> > > >>
> > > >> I have a problem with the files created by ivy in the cache
> > > >> during the resolve phase ..
> > > >> Ivy create some files nammed [org]-[module]-[conf].xml in the
> > > >> cache , should'nt those files be created in a temporary location
> > > >> outside of the
> > > >> cache, and preferably in project specific directory ?
> > > >>
> > > >> My use case is the following
> > > >>
> > > >> We have a Continuous Build Server who can build at the same time
> > > >> several different build from the same project ....
> > > >> In my build i create a clean.resolve task for deleting the
> > > >> [org]-[module]-[conf].xml files at the beginnig of each build ...
>
> > > >> as we can have multiple build at the same time for the same
> > > >> project I sometime have some build failled because of the
> > > >> deletion of this files by another starting build ...
> > > >>
> > > >> I still have created different cache location for my different
> > > >> project , but this use case should drive me to create a different
>
> > > >> cache for each build !!!
> > > >>
> > > >> Have you a solution for that ?
> > > >> would it be possible to have an ivy.temp.dir properties ?
> > > >> Is there a different implementation in 2.0.0 ?
> > > >>
> > > >>
> > > >> regards
> > > >>
> > > >> Greg
> > > >>
> > > >>
> > > >>
> > > >>
> > > >
> > > >
> > > %-|%-|:confused::super::super::super::super::super::super::super:
> > > --
> > > View this message in context:
> > >
> > http://www.nabble.com/problem-with-work-files-created-in-cache-by-ivy%
> > 3A
> > resolve-%28ivy-1.4%29-tf4158540.html#a13107058
> > > Sent from the ivy-user mailing list archive at Nabble.com.
> > >
> > >
> >
> >
> > --
> > Xavier Hanin - Independent Java Consultant http://xhab.blogspot.com/
> > http://incubator.apache.org/ivy/ http://www.xoocode.org/
> >
>
>
>
> --
> Xavier Hanin - Independent Java Consultant http://xhab.blogspot.com/
> http://incubator.apache.org/ivy/ http://www.xoocode.org/
>
--
Xavier Hanin - Independent Java Consultant
http://xhab.blogspot.com/
http://incubator.apache.org/ivy/
http://www.xoocode.org/
RE: problem with work files created in cache by ivy:resolve (ivy 1.4)
Posted by "Buck, Robert" <rb...@verisign.com>.
Hi,
I am new to this thread. I don't mean to butt in, but I have one
question regarding it, and this may be slightly unrelated...
I see mention use of the FileLock class. When I saw a reference to this,
I was immediately concerned.
IVY itself does NOT use FileLock, correct? (I hope not)
The reason I ask is that at VeriSign we discovered that FileLock does
not work on any of the NFS devices we have. The class itself throws an
exception, whose localized message effectively states, "FileLock is
unsupported on NFS devices.". Additionally, there are many File oriented
classes in NIO that either: (a) do not work at all, or (b) are a LOT
slower than old IO for IO intensive applications.
So this is all to say, please, only use NIO when you have NFS, NetAPP,
or EMC, hardware to test on, and you can prove with complete certainty
the individual API works. You may be surprised to find that the more
crucial parts of NIO is effectively broken.
Thanks in advance, no response necessary,
Bob
-----Original Message-----
From: Xavier Hanin [mailto:xavier.hanin@gmail.com]
Sent: Wednesday, October 10, 2007 1:38 AM
To: ivy-user@incubator.apache.org
Subject: Re: problem with work files created in cache by ivy:resolve
(ivy 1.4)
On 10/9/07, James Poli <Ja...@sas.com> wrote:
>
> Xavier,
>
> Thanks for the reply. Yes I can open an issue. Question: if I lock
> the entire resolve should I also lock the publish? In other words can
> the Ivy resolve task be updating the module resolution results while
> an Ivy publish (in another process) is trying to use them?
Yes, you need to lock the publish od the same module, because Ivy
doesn't update its resolution data transactionnally.
When you open the issue, you can link it to IVY-605, which points to
similar problem.
Xavier
Thanks again for any help,
> Jim
>
> -----Original Message-----
> From: Xavier Hanin [mailto:xavier.hanin@gmail.com]
> Sent: Tuesday, October 09, 2007 1:20 AM
> To: ivy-user@incubator.apache.org
> Subject: Re: problem with work files created in cache by ivy:resolve
> (ivy 1.4)
>
> On 10/9/07, JamesPoli <Ja...@sas.com> wrote:
> >
> >
> > As Jim Adams had stated we'd moved our cache to a tmp location to
> > get around this problem. Another solution I've been working on is
> > using a lock
> file
> > to
> > protect from having two or more resovles use the cache at the same
> time.
> > To "guard" the creation/deletion of the lock file I'm using a Java
> > FileLock
> > - it acts like a Java object monitor that is visible across JVM's.
> > It appears to work but for multiple processes (15) using resolve.
> > Could there be something I'm missing? Should I also lock the
> > publish? I'm
> thinking
> > if
> > I lock too many ivy tasks I'll end up defeating the whole purpose
> > I'm doing it, performance gains by using a single cache.
>
>
> Indeed, using a lock file can defeat the whole purpose of using a
> single cache. The main problem is that we handle two different things
> in the
> cache:
> cache of downloaded data (ivy files and artifacts), and cache of a
> module resolution results. The former is what is interesting to share
> and less a problem to lock, the latter is not really interesting to
> share and more a problem to lock. Once again we see that the cache
> management in Ivy needs improvement, it's one other vote for IVY-399.
> But this could even deserve a new issue: make Ivy able to use a
> repository cache shared by multiple processes. Could you create such
> an issue?
>
> Xavier
>
> Thanks in advance, Jim :super:
> >
> >
> > Jim Adams-6 wrote:
> > >
> > > We have something similar but our only recourse was to put the ivy
> cache
> > > in a tmp location and clear it after every build. Ivy can't handle
> > > multiple processes accessing the cache at the same time.
> > >
> > >> -----Original Message-----
> > >> From: Gregoire Henry [mailto:ghynxmail@gmail.com]
> > >> Sent: Friday, July 27, 2007 11:21 AM
> > >> To: ivy-user@incubator.apache.org
> > >> Subject: problem with work files created in cache by ivy:resolve
> > >> (ivy 1.4)
> > >>
> > >> hello
> > >>
> > >> I have a problem with the files created by ivy in the cache
> > >> during the resolve phase ..
> > >> Ivy create some files nammed [org]-[module]-[conf].xml in the
> > >> cache , should'nt those files be created in a temporary location
> > >> outside of the
> > >> cache, and preferably in project specific directory ?
> > >>
> > >> My use case is the following
> > >>
> > >> We have a Continuous Build Server who can build at the same time
> > >> several different build from the same project ....
> > >> In my build i create a clean.resolve task for deleting the
> > >> [org]-[module]-[conf].xml files at the beginnig of each build ...
> > >> as we can have multiple build at the same time for the same
> > >> project I sometime have some build failled because of the
> > >> deletion of this files by another starting build ...
> > >>
> > >> I still have created different cache location for my different
> > >> project , but this use case should drive me to create a different
> > >> cache for each build !!!
> > >>
> > >> Have you a solution for that ?
> > >> would it be possible to have an ivy.temp.dir properties ?
> > >> Is there a different implementation in 2.0.0 ?
> > >>
> > >>
> > >> regards
> > >>
> > >> Greg
> > >>
> > >>
> > >>
> > >>
> > >
> > >
> > %-|%-|:confused::super::super::super::super::super::super::super:
> > --
> > View this message in context:
> >
> http://www.nabble.com/problem-with-work-files-created-in-cache-by-ivy%
> 3A
> resolve-%28ivy-1.4%29-tf4158540.html#a13107058
> > Sent from the ivy-user mailing list archive at Nabble.com.
> >
> >
>
>
> --
> Xavier Hanin - Independent Java Consultant http://xhab.blogspot.com/
> http://incubator.apache.org/ivy/ http://www.xoocode.org/
>
--
Xavier Hanin - Independent Java Consultant http://xhab.blogspot.com/
http://incubator.apache.org/ivy/ http://www.xoocode.org/
Re: problem with work files created in cache by ivy:resolve (ivy 1.4)
Posted by Xavier Hanin <xa...@gmail.com>.
On 10/9/07, James Poli <Ja...@sas.com> wrote:
>
> Xavier,
>
> Thanks for the reply. Yes I can open an issue. Question: if I lock the
> entire resolve should I also
> lock the publish? In other words can the Ivy resolve task be updating
> the module resolution results while an Ivy publish (in another process)
> is trying to use them?
Yes, you need to lock the publish od the same module, because Ivy doesn't
update its resolution data transactionnally.
When you open the issue, you can link it to IVY-605, which points to similar
problem.
Xavier
Thanks again for any help,
> Jim
>
> -----Original Message-----
> From: Xavier Hanin [mailto:xavier.hanin@gmail.com]
> Sent: Tuesday, October 09, 2007 1:20 AM
> To: ivy-user@incubator.apache.org
> Subject: Re: problem with work files created in cache by ivy:resolve
> (ivy 1.4)
>
> On 10/9/07, JamesPoli <Ja...@sas.com> wrote:
> >
> >
> > As Jim Adams had stated we'd moved our cache to a tmp location to get
> > around
> > this problem. Another solution I've been working on is using a lock
> file
> > to
> > protect from having two or more resovles use the cache at the same
> time.
> > To "guard" the creation/deletion of the lock file I'm using a Java
> > FileLock
> > - it acts like a Java object monitor that is visible across JVM's. It
> > appears to work but for multiple processes (15) using resolve. Could
> > there
> > be something I'm missing? Should I also lock the publish? I'm
> thinking
> > if
> > I lock too many ivy tasks I'll end up defeating the whole purpose I'm
> > doing
> > it, performance gains by using a single cache.
>
>
> Indeed, using a lock file can defeat the whole purpose of using a single
> cache. The main problem is that we handle two different things in the
> cache:
> cache of downloaded data (ivy files and artifacts), and cache of a
> module
> resolution results. The former is what is interesting to share and less
> a
> problem to lock, the latter is not really interesting to share and more
> a
> problem to lock. Once again we see that the cache management in Ivy
> needs
> improvement, it's one other vote for IVY-399. But this could even
> deserve a
> new issue: make Ivy able to use a repository cache shared by multiple
> processes. Could you create such an issue?
>
> Xavier
>
> Thanks in advance, Jim :super:
> >
> >
> > Jim Adams-6 wrote:
> > >
> > > We have something similar but our only recourse was to put the ivy
> cache
> > > in a tmp location and clear it after every build. Ivy can't handle
> > > multiple processes accessing the cache at the same time.
> > >
> > >> -----Original Message-----
> > >> From: Gregoire Henry [mailto:ghynxmail@gmail.com]
> > >> Sent: Friday, July 27, 2007 11:21 AM
> > >> To: ivy-user@incubator.apache.org
> > >> Subject: problem with work files created in cache by
> > >> ivy:resolve (ivy 1.4)
> > >>
> > >> hello
> > >>
> > >> I have a problem with the files created by ivy in the cache
> > >> during the resolve phase ..
> > >> Ivy create some files nammed [org]-[module]-[conf].xml in the
> > >> cache , should'nt those files be created in a temporary
> > >> location outside of the
> > >> cache, and preferably in project specific directory ?
> > >>
> > >> My use case is the following
> > >>
> > >> We have a Continuous Build Server who can build at the same
> > >> time several different build from the same project ....
> > >> In my build i create a clean.resolve task for deleting the
> > >> [org]-[module]-[conf].xml files at the beginnig of each build
> > >> ... as we can have multiple build at the same time for the
> > >> same project I sometime have some build failled because of
> > >> the deletion of this files by another starting build ...
> > >>
> > >> I still have created different cache location for my
> > >> different project , but this use case should drive me to
> > >> create a different cache for each build !!!
> > >>
> > >> Have you a solution for that ?
> > >> would it be possible to have an ivy.temp.dir properties ?
> > >> Is there a different implementation in 2.0.0 ?
> > >>
> > >>
> > >> regards
> > >>
> > >> Greg
> > >>
> > >>
> > >>
> > >>
> > >
> > >
> > %-|%-|:confused::super::super::super::super::super::super::super:
> > --
> > View this message in context:
> >
> http://www.nabble.com/problem-with-work-files-created-in-cache-by-ivy%3A
> resolve-%28ivy-1.4%29-tf4158540.html#a13107058
> > Sent from the ivy-user mailing list archive at Nabble.com.
> >
> >
>
>
> --
> Xavier Hanin - Independent Java Consultant
> http://xhab.blogspot.com/
> http://incubator.apache.org/ivy/
> http://www.xoocode.org/
>
--
Xavier Hanin - Independent Java Consultant
http://xhab.blogspot.com/
http://incubator.apache.org/ivy/
http://www.xoocode.org/
RE: problem with work files created in cache by ivy:resolve (ivy 1.4)
Posted by James Poli <Ja...@sas.com>.
Xavier,
Thanks for the reply. Yes I can open an issue. Question: if I lock the
entire resolve should I also
lock the publish? In other words can the Ivy resolve task be updating
the module resolution results while an Ivy publish (in another process)
is trying to use them?
Thanks again for any help,
Jim
-----Original Message-----
From: Xavier Hanin [mailto:xavier.hanin@gmail.com]
Sent: Tuesday, October 09, 2007 1:20 AM
To: ivy-user@incubator.apache.org
Subject: Re: problem with work files created in cache by ivy:resolve
(ivy 1.4)
On 10/9/07, JamesPoli <Ja...@sas.com> wrote:
>
>
> As Jim Adams had stated we'd moved our cache to a tmp location to get
> around
> this problem. Another solution I've been working on is using a lock
file
> to
> protect from having two or more resovles use the cache at the same
time.
> To "guard" the creation/deletion of the lock file I'm using a Java
> FileLock
> - it acts like a Java object monitor that is visible across JVM's. It
> appears to work but for multiple processes (15) using resolve. Could
> there
> be something I'm missing? Should I also lock the publish? I'm
thinking
> if
> I lock too many ivy tasks I'll end up defeating the whole purpose I'm
> doing
> it, performance gains by using a single cache.
Indeed, using a lock file can defeat the whole purpose of using a single
cache. The main problem is that we handle two different things in the
cache:
cache of downloaded data (ivy files and artifacts), and cache of a
module
resolution results. The former is what is interesting to share and less
a
problem to lock, the latter is not really interesting to share and more
a
problem to lock. Once again we see that the cache management in Ivy
needs
improvement, it's one other vote for IVY-399. But this could even
deserve a
new issue: make Ivy able to use a repository cache shared by multiple
processes. Could you create such an issue?
Xavier
Thanks in advance, Jim :super:
>
>
> Jim Adams-6 wrote:
> >
> > We have something similar but our only recourse was to put the ivy
cache
> > in a tmp location and clear it after every build. Ivy can't handle
> > multiple processes accessing the cache at the same time.
> >
> >> -----Original Message-----
> >> From: Gregoire Henry [mailto:ghynxmail@gmail.com]
> >> Sent: Friday, July 27, 2007 11:21 AM
> >> To: ivy-user@incubator.apache.org
> >> Subject: problem with work files created in cache by
> >> ivy:resolve (ivy 1.4)
> >>
> >> hello
> >>
> >> I have a problem with the files created by ivy in the cache
> >> during the resolve phase ..
> >> Ivy create some files nammed [org]-[module]-[conf].xml in the
> >> cache , should'nt those files be created in a temporary
> >> location outside of the
> >> cache, and preferably in project specific directory ?
> >>
> >> My use case is the following
> >>
> >> We have a Continuous Build Server who can build at the same
> >> time several different build from the same project ....
> >> In my build i create a clean.resolve task for deleting the
> >> [org]-[module]-[conf].xml files at the beginnig of each build
> >> ... as we can have multiple build at the same time for the
> >> same project I sometime have some build failled because of
> >> the deletion of this files by another starting build ...
> >>
> >> I still have created different cache location for my
> >> different project , but this use case should drive me to
> >> create a different cache for each build !!!
> >>
> >> Have you a solution for that ?
> >> would it be possible to have an ivy.temp.dir properties ?
> >> Is there a different implementation in 2.0.0 ?
> >>
> >>
> >> regards
> >>
> >> Greg
> >>
> >>
> >>
> >>
> >
> >
> %-|%-|:confused::super::super::super::super::super::super::super:
> --
> View this message in context:
>
http://www.nabble.com/problem-with-work-files-created-in-cache-by-ivy%3A
resolve-%28ivy-1.4%29-tf4158540.html#a13107058
> Sent from the ivy-user mailing list archive at Nabble.com.
>
>
--
Xavier Hanin - Independent Java Consultant
http://xhab.blogspot.com/
http://incubator.apache.org/ivy/
http://www.xoocode.org/
Re: problem with work files created in cache by ivy:resolve (ivy 1.4)
Posted by Xavier Hanin <xa...@gmail.com>.
On 10/9/07, JamesPoli <Ja...@sas.com> wrote:
>
>
> As Jim Adams had stated we'd moved our cache to a tmp location to get
> around
> this problem. Another solution I've been working on is using a lock file
> to
> protect from having two or more resovles use the cache at the same time.
> To "guard" the creation/deletion of the lock file I'm using a Java
> FileLock
> - it acts like a Java object monitor that is visible across JVM's. It
> appears to work but for multiple processes (15) using resolve. Could
> there
> be something I'm missing? Should I also lock the publish? I'm thinking
> if
> I lock too many ivy tasks I'll end up defeating the whole purpose I'm
> doing
> it, performance gains by using a single cache.
Indeed, using a lock file can defeat the whole purpose of using a single
cache. The main problem is that we handle two different things in the cache:
cache of downloaded data (ivy files and artifacts), and cache of a module
resolution results. The former is what is interesting to share and less a
problem to lock, the latter is not really interesting to share and more a
problem to lock. Once again we see that the cache management in Ivy needs
improvement, it's one other vote for IVY-399. But this could even deserve a
new issue: make Ivy able to use a repository cache shared by multiple
processes. Could you create such an issue?
Xavier
Thanks in advance, Jim :super:
>
>
> Jim Adams-6 wrote:
> >
> > We have something similar but our only recourse was to put the ivy cache
> > in a tmp location and clear it after every build. Ivy can't handle
> > multiple processes accessing the cache at the same time.
> >
> >> -----Original Message-----
> >> From: Gregoire Henry [mailto:ghynxmail@gmail.com]
> >> Sent: Friday, July 27, 2007 11:21 AM
> >> To: ivy-user@incubator.apache.org
> >> Subject: problem with work files created in cache by
> >> ivy:resolve (ivy 1.4)
> >>
> >> hello
> >>
> >> I have a problem with the files created by ivy in the cache
> >> during the resolve phase ..
> >> Ivy create some files nammed [org]-[module]-[conf].xml in the
> >> cache , should'nt those files be created in a temporary
> >> location outside of the
> >> cache, and preferably in project specific directory ?
> >>
> >> My use case is the following
> >>
> >> We have a Continuous Build Server who can build at the same
> >> time several different build from the same project ....
> >> In my build i create a clean.resolve task for deleting the
> >> [org]-[module]-[conf].xml files at the beginnig of each build
> >> ... as we can have multiple build at the same time for the
> >> same project I sometime have some build failled because of
> >> the deletion of this files by another starting build ...
> >>
> >> I still have created different cache location for my
> >> different project , but this use case should drive me to
> >> create a different cache for each build !!!
> >>
> >> Have you a solution for that ?
> >> would it be possible to have an ivy.temp.dir properties ?
> >> Is there a different implementation in 2.0.0 ?
> >>
> >>
> >> regards
> >>
> >> Greg
> >>
> >>
> >>
> >>
> >
> >
> %-|%-|:confused::super::super::super::super::super::super::super:
> --
> View this message in context:
> http://www.nabble.com/problem-with-work-files-created-in-cache-by-ivy%3Aresolve-%28ivy-1.4%29-tf4158540.html#a13107058
> Sent from the ivy-user mailing list archive at Nabble.com.
>
>
--
Xavier Hanin - Independent Java Consultant
http://xhab.blogspot.com/
http://incubator.apache.org/ivy/
http://www.xoocode.org/