You are viewing a plain text version of this content. The canonical link for it is here.
Posted to ivy-user@ant.apache.org by "Ivan \"Rambius\" Ivanov" <ra...@yahoo.com> on 2007/11/05 22:32:56 UTC
Problem when running ivy in threads
Hello,
I am using Ivy 1.4.1 on Windows XP SP2. I have developed some CruiseControl plugins similar to ivycruise and I may contribute them to ivy in the future, but for now I have the following problem:
My CruiseControl plugin checks the ivy repository if there is a new revision of a project's dependencies since the last successful build and if there is a new CruiseControl build is triggered. In order to check which are the project's dependencies, I parse (using Ivy API) the project's ivy descriptor. So far so good, but I noticed that some of the ivy descriptors get locked on Windows environment. After some debugging it turned out that the problem with the locked files come from the fact that we run our cruisecontrol builds on multiple threads, see [1].
I tried the reproduce the problem with some simple test case and I invented the following:
import java.io.File;
import java.net.URL;
import fr.jayasoft.ivy.Ivy;
import fr.jayasoft.ivy.ModuleDescriptor;
import fr.jayasoft.ivy.parser.ModuleDescriptorParserRegistry;
public class Main {
public static void main(String[] args) throws Exception {
String ivyDescriptor = args[0]; // location of any ivy.xml
Ivy ivy = new Ivy();
ModuleDescriptorParserRegistry mdpr = ModuleDescriptorParserRegistry.getInstance();
File ivyFile = new File(ivyDescriptor);
URL ivyFileURL = ivyFile.toURL();
ModuleDescriptor md = mdpr.parseDescriptor(ivy, ivyFileURL, true);
System.out.println("module name is " + md.getModuleRevisionId().getName());
Thread.sleep(60 * 1000);// sleep for one minute
}
}
After the line
ModuleDescriptor md = mdpr.parseDescriptor(ivy, ivyFileURL, true);
I would expect that all the streams opened to the ivyDescriptor file should be closed. However, during the 1 minute thread sleep, I am not able, for example, to rename the file in Windows Explorer, because it says it is being used by another program. After one minute I can rename the file.
One my hypothesis is that the Ant API is not thread-safe. Is it true or not and can you give me some advice on how to prevent ivy descriptors from being locked when I parse them with the code above?
Regards
Rambius
[1] http://cruisecontrol.sourceforge.net/main/configxml.html#threads
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
Re: Problem when running ivy in threads
Posted by "Ivan \"Rambius\" Ivanov" <ra...@yahoo.com>.
Hello Giles,
Thank you for your response.
--- Gilles Scokart <gs...@gmail.com> wrote:
> You might be right, but in your example, you only
> have a single thread. So
> it is rather a problem of file handle not closed.
Yes, you are right. After I switched to 1 thread in
CruiseControl there are still locked files.
> Did you tried with a 2.0 version. I think that some
> missing close have been
> added a few months ago.
Yes, I did. I tested the use case with the following
code:
import java.io.File;
import java.net.URL;
import org.apache.ivy.Ivy;
import
org.apache.ivy.core.module.descriptor.ModuleDescriptor;
import
org.apache.ivy.plugins.parser.ModuleDescriptorParserRegistry;
public class Main {
public static void main(String[] args) throws
Exception {
String ivyDescriptor = args[0];// location of
any ivy.xml
Ivy ivy = new Ivy();
ivy.configureDefault();
ModuleDescriptorParserRegistry mdpr =
ModuleDescriptorParserRegistry.getInstance();
File ivyFile = new File(ivyDescriptor);
URL ivyFileURL = ivyFile.toURL();
ModuleDescriptor md =
mdpr.parseDescriptor(ivy.getSettings(), ivyFileURL,
true);
System.out.println("module name is " +
md.getModuleRevisionId().getName());
Thread.sleep(60 * 1000);
}
}
When I executed this code with the latest ivy from svn
the file was not locked.
> By the way, what are the difference between your
> plugin and ivyCruise?
> Did you restart from scratch or did you updated it?
I used ivycruise only as an example for my plugins. I
started from scratch. IvyCruise makes lots of
assumptions which are not true for my project:
-) It assumes the names of the ivy modules are the
same as the names of the cruisecontrol projects - I am
building different branches of one project and I
append the branch name to the project name in
CruiseControl
-) IvyCruise assumes that all ivy projects are built
in the same CruiseControl instance - I have 2
CruiseControl instances publishing to one central ivy
repository. I have cases like 2 depenendencies of a
project are built by one of the CruiseControl servers,
and 3 dependencies by the other. In this case I can
query only the Ivy repository to find out of there is
a new revision of a given dependency.
> Maybe you have some usefull info.
I have also some working code. I will discuss it with
my boss if I can contribute it back. For now, here is
my algorithm. I am using a sourcecontrol plugin [1]
and a publisher plugin
-) After a successful publication to the ivy
repository of an ivy module built by CruiseControl,
the CC publisher copies the delivered ivy.xml to
$CRUISECONTROL_HOME/logs/${project.name}/ivy.xml
-) When a new build is started the ivy.xml of the
previous build is taken (from
$CRUISECONTROL_HOME/logs/${project.name}/ivy.xml) and
parsed. Then the current ivy.xml is again parsed and
all the dependencies whose names start with "latest"
are taken and compared with the same dependencies from
the old ivy.xml. If a difference is found, it is
considered a modification.
> PS: note that <thread> of cruise control only
> control the number of build in
> parallel, but every project has his own thread used
> when invoking your
> modification set plugin. So, even if you set thread
> to 1, you will still
> have your modificationset executed in multithread.
Yes, I checked CruiseControl code and found this out.
Regards
Rambius
Tangra Mega Rock: http://www.radiotangra.com/
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
Re: Problem when running ivy in threads
Posted by Gilles Scokart <gs...@gmail.com>.
You might be right, but in your example, you only have a single thread. So
it is rather a problem of file handle not closed.
Did you tried with a 2.0 version. I think that some missing close have been
added a few months ago.
By the way, what are the difference bteween your plugin and ivyCruise? Did
you restart from scratch or did you updated it? I would like to make a
setup using a <ivybuilder>. Maybe you have some usefull info.
PS: note that <thread> of cruise control only control the number of build in
parallel, but every project has his own thread used when invoking your
modification set plugin. So, even if you set thread to 1, you will still
have your modificationset executed in multithread.
Gilles
2007/11/5, Ivan Rambius Ivanov <ra...@yahoo.com>:
>
> Hello,
>
> I am using Ivy 1.4.1 on Windows XP SP2. I have developed some
> CruiseControl plugins similar to ivycruise and I may contribute them to ivy
> in the future, but for now I have the following problem:
>
> My CruiseControl plugin checks the ivy repository if there is a new
> revision of a project's dependencies since the last successful build and if
> there is a new CruiseControl build is triggered. In order to check which are
> the project's dependencies, I parse (using Ivy API) the project's ivy
> descriptor. So far so good, but I noticed that some of the ivy descriptors
> get locked on Windows environment. After some debugging it turned out that
> the problem with the locked files come from the fact that we run our
> cruisecontrol builds on multiple threads, see [1].
>
> I tried the reproduce the problem with some simple test case and I
> invented the following:
>
> import java.io.File;
> import java.net.URL;
>
> import fr.jayasoft.ivy.Ivy;
> import fr.jayasoft.ivy.ModuleDescriptor;
> import fr.jayasoft.ivy.parser.ModuleDescriptorParserRegistry;
>
> public class Main {
>
> public static void main(String[] args) throws Exception {
> String ivyDescriptor = args[0]; // location of any ivy.xml
> Ivy ivy = new Ivy();
>
> ModuleDescriptorParserRegistry mdpr =
> ModuleDescriptorParserRegistry.getInstance();
> File ivyFile = new File(ivyDescriptor);
> URL ivyFileURL = ivyFile.toURL();
> ModuleDescriptor md = mdpr.parseDescriptor(ivy, ivyFileURL, true);
> System.out.println("module name is " + md.getModuleRevisionId
> ().getName());
>
> Thread.sleep(60 * 1000);// sleep for one minute
> }
> }
>
> After the line
>
> ModuleDescriptor md = mdpr.parseDescriptor(ivy, ivyFileURL, true);
>
> I would expect that all the streams opened to the ivyDescriptor file
> should be closed. However, during the 1 minute thread sleep, I am not able,
> for example, to rename the file in Windows Explorer, because it says it is
> being used by another program. After one minute I can rename the file.
>
> One my hypothesis is that the Ant API is not thread-safe. Is it true or
> not and can you give me some advice on how to prevent ivy descriptors from
> being locked when I parse them with the code above?
>
> Regards
> Rambius
>
>
> [1] http://cruisecontrol.sourceforge.net/main/configxml.html#threads
>
>
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam? Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com
>
--
Gilles SCOKART