You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@geronimo.apache.org by Vamsavardhana Reddy <c1...@gmail.com> on 2008/01/23 11:23:35 UTC
Offline Deployer leaving behind temporary files
I see the following in Deployer.java
// todo jar url handling with Sun's VM on Windows leaves a lock
on the module file *preventing rebuilds*
// to address this we use a gross hack and copy the file to a
temporary directory
// the lock on the file will prevent that being deleted properly
until the URLJarFile has
// been GC'ed.
boolean cleanup = true;
try {
tmpDir = File.createTempFile("geronimo-deployer",
".tmpdir");
tmpDir.delete();
tmpDir.mkdir();
tmpFile = new File(tmpDir, moduleFile.getName());
DeploymentUtil.copyFile(moduleFile, tmpFile);
moduleFile = tmpFile;
Can someone explain the "preventing rebuilds" part in the above? It is
followed by code that creates a temporary copy of the module archive that
should be cleaned up by DeployerReaper which does not delete these files in
case of offline deployment. In online deployment also, the files may be
left behind if the DeployerReaper does not get a chance to run after the
files are added to pendingDeletionIndex. Incase of offline deployment
DeployerReaper does not get a chance at all as the java process terminates
immediately. I have tried deleteOnExit() as well with offline deployment,
but the files won't just go away. I am wondering if the reason this is
introduced in the first place is applicable to 2.x. If not, we can get rid
of this code.
++Vamsi
Re: Offline Deployer leaving behind temporary files
Posted by Vamsavardhana Reddy <c1...@gmail.com>.
I have changed DeploymentUtil.toTempFile(URL url) and
DeploymentUtil.readAll(URL
url) to use JarFile.getInputStream() instead of URL.openStream() when the
protocol is "jar". But, the other code the uses URL.openStream() still
leaves locks on the jar files which prevent immediate deletion of temp
files. The idea that I have is to change DeploymentUtil.createJarURL() such
that when the URL is for a file inside a jar file, rewrite the content to a
temp file and return the URL to that file.
The code will look something like the following:
String urlString = "jar:" + new File(jarFile.getName()).toURL()
+ "!/" + path;
if(jarUrlRewrite) {
// To prevent the lockout of archive, instead of returning a
jar url, write the content to a
// temp file and return the url of that file.
File tempFile = null;
try {
tempFile = toTempFile(new URL(urlString));
} catch (IOException e) {
// The JarEntry does not exist!
// Return url of a file that does not exist.
try {
tempFile = createTempFile();
tempFile.delete();
} catch (IOException ignored) {
}
}
return tempFile.toURL();
} else {
return new URL(urlString);
}
And the jarUrlRewrite value (false by default, which means no change from
current behavior) can be controlled by a system property "
org.apache.geronimo.deployment.util.DeploymentUtil.jarUrlRewrite" which the
users can set either using GERONIMO_OPTS environment variable or by using
the SystemProperties GBean.
When jarUrlRewrite is true, more temporary files are created, but the
Deployer will be able to delete the temporary files immediately and it won't
be needing to use DeployerReaper. On the other hand, each temp file created
will also be calling deleteOnExit() which may be holding on to 1K of memory
until the JVM terminates (if I recall correctly). Since DeployerReaper is
doing a fairly good job in case of online deployement, I suggest we set
jarUrlRewrite system property to true for offline-deployer and let users
decide what it should be for online deployer.
Comments? Suggestions? Anything I am missing.
++Vamsi
On Jan 24, 2008 12:13 AM, Vamsavardhana Reddy <c1...@gmail.com> wrote:
> I have found the culprit. It is the URLs that we use to read content from
> an archive, for e.g., META-INF/application.xml from an ear file. The
> deployer is creating a JarFile and closing the JarFile after the deployment
> operation is completed. JarFile.close() closes all the InputStreams
> obtained from the JarFile and releases any locks.
>
> For e.g., in EARConfigBuilder.getEarPlan(), we have code like the
> following:
>
> URL applicationXmlUrl = DeploymentUtil.createJarURL(earFile,
> "META-INF/application.xml");
> specDD = DeploymentUtil.readAll (applicationXmlUrl);
>
> After this code is executed the earFile is locked until the JVM
> terminates. If you replace the above with something similar to
>
> InputStream inp = earFile.getInputStream(earFile.getJarEntry
> ("META-INF/application.xml"));
> BufferedReader br = new BufferedReader(new
> InputStreamReader(inp));
> String line;
> while((line = br.readLine()) != null) {
> specDD += line;
> }
>
> the earFile is no longer locked once earFile.close() is called and can be
> deleted anytime. This is what is observed on Win XP.
>
> ++Vamsi
>
>
> On Jan 23, 2008 8:52 PM, Kevan Miller <ke...@gmail.com> wrote:
>
> >
> > On Jan 23, 2008, at 10:02 AM, Vamsavardhana Reddy wrote:
> >
> > > Kevan,
> > >
> > > I am testing this with an ear file. So, the EARConfiBuilder should
> > > be reading this file. I guess it is the same with other builders as
> > > well. The JarFileClassLoader has the following comment
> > >
> > > * Note: This implementation currently does not work reliably on
> > > windows, since the jar URL handler included with the Sun JavaVM
> > > * holds a read lock on the JarFile, and this lock is not released
> > > when the jar url is dereferenced. To fix this a
> > > * replacement for the jar url handler must be written.
> > >
> > > BTW, I am running G 2.0.3-SNAPSHOT on Win XP.
> >
> > I wouldn't trust that comment. Dain's commit was addressing this very
> > problem (at least within the server runtime:
> > http://svn.apache.org/viewvc?view=rev&revision=399522
> >
> > You need to check to see what type of ClassLoader is being used. If
> > it's not JarFileClassLoader, then we understand the problem. If it is
> > JarFileClassLoader, then maybe we aren't calling destroy. If we doing
> > all of these things, then obviously I'm wrong again... ;-)
> >
> > --kevan
> >
>
>
Re: Offline Deployer leaving behind temporary files
Posted by Vamsavardhana Reddy <c1...@gmail.com>.
I have found the culprit. It is the URLs that we use to read content from
an archive, for e.g., META-INF/application.xml from an ear file. The
deployer is creating a JarFile and closing the JarFile after the deployment
operation is completed. JarFile.close() closes all the InputStreams
obtained from the JarFile and releases any locks.
For e.g., in EARConfigBuilder.getEarPlan(), we have code like the following:
URL applicationXmlUrl = DeploymentUtil.createJarURL(earFile,
"META-INF/application.xml");
specDD = DeploymentUtil.readAll(applicationXmlUrl);
After this code is executed the earFile is locked until the JVM terminates.
If you replace the above with something similar to
InputStream inp = earFile.getInputStream(earFile.getJarEntry
("META-INF/application.xml"));
BufferedReader br = new BufferedReader(new InputStreamReader(inp));
String line;
while((line = br.readLine()) != null) {
specDD += line;
}
the earFile is no longer locked once earFile.close() is called and can be
deleted anytime. This is what is observed on Win XP.
++Vamsi
On Jan 23, 2008 8:52 PM, Kevan Miller <ke...@gmail.com> wrote:
>
> On Jan 23, 2008, at 10:02 AM, Vamsavardhana Reddy wrote:
>
> > Kevan,
> >
> > I am testing this with an ear file. So, the EARConfiBuilder should
> > be reading this file. I guess it is the same with other builders as
> > well. The JarFileClassLoader has the following comment
> >
> > * Note: This implementation currently does not work reliably on
> > windows, since the jar URL handler included with the Sun JavaVM
> > * holds a read lock on the JarFile, and this lock is not released
> > when the jar url is dereferenced. To fix this a
> > * replacement for the jar url handler must be written.
> >
> > BTW, I am running G 2.0.3-SNAPSHOT on Win XP.
>
> I wouldn't trust that comment. Dain's commit was addressing this very
> problem (at least within the server runtime:
> http://svn.apache.org/viewvc?view=rev&revision=399522
>
> You need to check to see what type of ClassLoader is being used. If
> it's not JarFileClassLoader, then we understand the problem. If it is
> JarFileClassLoader, then maybe we aren't calling destroy. If we doing
> all of these things, then obviously I'm wrong again... ;-)
>
> --kevan
>
Re: Offline Deployer leaving behind temporary files
Posted by Kevan Miller <ke...@gmail.com>.
On Jan 23, 2008, at 10:02 AM, Vamsavardhana Reddy wrote:
> Kevan,
>
> I am testing this with an ear file. So, the EARConfiBuilder should
> be reading this file. I guess it is the same with other builders as
> well. The JarFileClassLoader has the following comment
>
> * Note: This implementation currently does not work reliably on
> windows, since the jar URL handler included with the Sun JavaVM
> * holds a read lock on the JarFile, and this lock is not released
> when the jar url is dereferenced. To fix this a
> * replacement for the jar url handler must be written.
>
> BTW, I am running G 2.0.3-SNAPSHOT on Win XP.
I wouldn't trust that comment. Dain's commit was addressing this very
problem (at least within the server runtime:
http://svn.apache.org/viewvc?view=rev&revision=399522
You need to check to see what type of ClassLoader is being used. If
it's not JarFileClassLoader, then we understand the problem. If it is
JarFileClassLoader, then maybe we aren't calling destroy. If we doing
all of these things, then obviously I'm wrong again... ;-)
--kevan
Re: Offline Deployer leaving behind temporary files
Posted by Vamsavardhana Reddy <c1...@gmail.com>.
Kevan,
I am testing this with an ear file. So, the EARConfiBuilder should be
reading this file. I guess it is the same with other builders as well. The
JarFileClassLoader has the following comment
* Note: This implementation currently does not work reliably on windows,
since the jar URL handler included with the Sun JavaVM
* holds a read lock on the JarFile, and this lock is not released when the
jar url is dereferenced. To fix this a
* replacement for the jar url handler must be written.
BTW, I am running G 2.0.3-SNAPSHOT on Win XP.
++Vamsi
On Jan 23, 2008 7:48 PM, Kevan Miller <ke...@gmail.com> wrote:
>
> On Jan 23, 2008, at 5:23 AM, Vamsavardhana Reddy wrote:
>
> I see the following in Deployer.java
>
> // todo jar url handling with Sun's VM on Windows leaves a
> lock on the module file *preventing rebuilds*
> // to address this we use a gross hack and copy the file to a
> temporary directory
> // the lock on the file will prevent that being deleted
> properly until the URLJarFile has
> // been GC'ed.
> boolean cleanup = true;
> try {
> tmpDir = File.createTempFile("geronimo-deployer",
> ".tmpdir");
> tmpDir.delete();
> tmpDir.mkdir();
> tmpFile = new File(tmpDir, moduleFile.getName());
> DeploymentUtil.copyFile(moduleFile, tmpFile);
> moduleFile = tmpFile;
>
> Can someone explain the "preventing rebuilds" part in the above? It is
> followed by code that creates a temporary copy of the module archive that
> should be cleaned up by DeployerReaper which does not delete these files in
> case of offline deployment. In online deployment also, the files may be
> left behind if the DeployerReaper does not get a chance to run after the
> files are added to pendingDeletionIndex. Incase of offline deployment
> DeployerReaper does not get a chance at all as the java process terminates
> immediately. I have tried deleteOnExit() as well with offline deployment,
> but the files won't just go away. I am wondering if the reason this is
> introduced in the first place is applicable to 2.x. If not, we can get
> rid of this code.
>
>
> Hi Vamsi,
> Well, the fact that the files are locked indicates a problem, I think. Can
> you tell who's reading the files? I thought we would be using
> org.apache.geronimo.kernel.classloader.JarFileClassLoader and would thus
> avoid the Windows file locking problem.
>
> Hmm, perhaps we're not calling JarFileClassLoader.destroy()? This should
> free up the file locks.
>
> --kevan
>
Re: Offline Deployer leaving behind temporary files
Posted by Kevan Miller <ke...@gmail.com>.
On Jan 23, 2008, at 5:23 AM, Vamsavardhana Reddy wrote:
> I see the following in Deployer.java
>
> // todo jar url handling with Sun's VM on Windows leaves
> a lock on the module file preventing rebuilds
> // to address this we use a gross hack and copy the file
> to a temporary directory
> // the lock on the file will prevent that being deleted
> properly until the URLJarFile has
> // been GC'ed.
> boolean cleanup = true;
> try {
> tmpDir = File.createTempFile("geronimo-deployer",
> ".tmpdir");
> tmpDir.delete();
> tmpDir.mkdir();
> tmpFile = new File(tmpDir, moduleFile.getName());
> DeploymentUtil.copyFile(moduleFile, tmpFile);
> moduleFile = tmpFile;
>
> Can someone explain the "preventing rebuilds" part in the above? It
> is followed by code that creates a temporary copy of the module
> archive that should be cleaned up by DeployerReaper which does not
> delete these files in case of offline deployment. In online
> deployment also, the files may be left behind if the DeployerReaper
> does not get a chance to run after the files are added to
> pendingDeletionIndex. Incase of offline deployment DeployerReaper
> does not get a chance at all as the java process terminates
> immediately. I have tried deleteOnExit() as well with offline
> deployment, but the files won't just go away. I am wondering if the
> reason this is introduced in the first place is applicable to 2.x.
> If not, we can get rid of this code.
Hi Vamsi,
Well, the fact that the files are locked indicates a problem, I think.
Can you tell who's reading the files? I thought we would be using
org.apache.geronimo.kernel.classloader.JarFileClassLoader and would
thus avoid the Windows file locking problem.
Hmm, perhaps we're not calling JarFileClassLoader.destroy()? This
should free up the file locks.
--kevan