You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@axis.apache.org by "Talbott, Thomas" <Th...@serialssolutions.com> on 2008/08/29 20:10:32 UTC
Hot update: Too Many Open Files
Hey all,
Our project just started utilizing Axis2 for our web service framework and
we are running into an issue with Hot Update. We recognize that this is not
recommended for production, but would like to use it in our staging
environment that will be managed through a continuous integration platform.
The problem is after a few days, we started getting the following when the
aar file was updated:
8/27|13:52:57 [Timer-3] INFO :ServiceDeployer - Deploying Web service:
version-1.4.aar - file:/var/tomcat5/webapps/axis2/WEB-INF/services/ve
rsion-1.4.aar
8/27|13:52:57 [Timer-3] WARN :Utils - Exception extracting jars into
temporary directory : java.io.IOException: Too many open files : switch
ing to alternate class loading mechanism
8/27|13:52:59 [Timer-3] ERROR:ServiceDeployer - The tc_web_service_1.aar
service, which is not valid, caused The following error occurred du
ring schema generation: Class Not found :
com.sersol.fedsearch.dta.server.service.TCWSServer
org.apache.axis2.deployment.DeploymentException: The following error
occurred during schema generation: Class Not found : com.sersol.fedsearc
We are currently running Axis2 1.4 under Tomcat 5.5.26 on CentOS 5.2 (Linux
version 2.6.18-92.1.6.el5 (mockbuild@builder15.centos.org) (gcc version
4.1.2 20071124 (Red Hat 4.1.2-42)) #1 SMP Wed Jun 25 13:45:47 EDT 2008).
If I look at the open files, the majority are under the work folder in the
Tomcat directory structure. Example:
/var/cache/tomcat5/work/Catalina/localhost/axis2/_axis2/axis21612jaxen-1.1.1
.jar
Every time I do an update, the number of open files under the '_axis2'
directory grows:
# lsof | grep _axis2 | wc -l
232
# lsof | grep _axis2 | wc -l
424
# lsof | grep _axis2 | wc -l
616
I don't see anyone else reporting this issue. Is this a problem running
under Tomcat?
If this has been reported elsewhere that I have missed, please direct me to
that discussion. Otherwise, if you have any insight, I'd love to hear it.
Thanks,
-Tom
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-user-help@ws.apache.org
Re: Hot update: Too Many Open Files
Posted by "Talbott, Thomas" <Th...@serialssolutions.com>.
Hello Martin,
Forgive me, but I am having a hard time figuring out what your are
responding to. Must be the long weekend...
What exactly do you intend these to be applied to? The temporary files?
Personally, I still think we are dealing with a leak of class loaders
(aren't being cleaned up when they are no longer in use). In the case of
Steve, it is likely a similar issue. The only difference is that when he
shuts down, there is still no cleanup of the class loaders.
-Tom
On 8/29/08 6:36 PM, "Martin Gainty" <mg...@hotmail.com> wrote:
> set sticky bit off so other users (other than root) can access
> chmod chmod [OPTION]... MODE[,MODE]... FILE...
> http://linux.die.net/man/1/chmod
>
> chown --from=CURRENT_OWNER:CURRENT_GROUP file
> http://linux.die.net/man/1/chown
>
> if the file is located on linux extended file use
> chattr [ -RV ] [ -v version ] [ mode ] files...
> http://linux.die.net/man/1/chattr
>
> Martin
>
> ______________________________________________
> Disclaimer and confidentiality note
> Everything in this e-mail and any attachments relates to the official business
> of Sender. This transmission is of a confidential nature and Sender does not
> endorse distribution to any party other than intended recipient. Sender does
> not necessarily endorse content contained within this transmission.
>
>
>> From: Thomas.Talbott@serialssolutions.com
>> To: axis-user@ws.apache.org
>> Date: Fri, 29 Aug 2008 21:04:25 -0400
>> Subject: Re: Hot update: Too Many Open Files
>>
>> Hello Steve,
>>
>> We are not running under VMWare. I would say that our situation is similar,
>> but not the same. In your previous messages you say:
>>
>> ³A new set gets generated each time I restart Tomcat. On my Windows XP
>> system, these are deleted each time Tomcat stops, but not on our Linux
>> (CentOS) systems.²
>>
>> In our case, we are trying to deploy without restarting Tomcat. We have the
>> following set in axis2.xml:
>>
>> <parameter name="hotdeployment">true</parameter>
>> <parameter name="hotupdate">true</parameter>
>>
>> When we copy the updated aar file into the WEB-INF/services directory of the
>> axis2 webapp, temporary files are created in the work directory. The next
>> time we update aar file, more files are created in the work directory. When
>> I shutdown tomcat, the files ARE cleaned up. But, if I don't shutdown
>> tomcat, file handles are left open for every one of these temporary files!
>> So, soon, we run out of file handles.
>>
>> As I mentioned in my original message, we want to do this for staging and we
>> are not looking for this to be a production solution. As far as I can tell,
>> we should be able to shutdown tomcat, deploy the aar, and restart tomcat in
>> production without the problems you are seeing. We may just need to resort
>> to the same solution in production.
>>
>> It would seem, given our joint experience, that there is an issue with the
>> the temporary files used for deployment and update. Given what Deepal says:
>>
>> "Yes , Axis2 creates temp files from your services and modules , and
>> create a class loader from that. In that way we can ensure better
>> performance. This help us a lot when we have service aar or mar with
>> third party library inside it."
>>
>> It would appear that there is an issue with the class loader that is
>> created. In the "update" scenario, I wonder if the class loaders are
>> leaking (old class loaders not going away) after a service is updated.
>> Current evidence for this is that my files do eventually get cleaned up and
>> I assume that the class loader is doing this once it finally goes away.
>> But, there may be some management process that is responsible for this. I
>> have not looked into the code.
>>
>> Anyone with insight into this?
>>
>> Thanks,
>> -Tom
>>
>> On 8/29/08 2:22 PM, "SGruverman@intellicare.com"
>> <SG...@intellicare.com> wrote:
>>
>>> Thomas,
>>>
>>> This is very similar to two threads I started, one in May ("cached archive
>>> files not deleted") and one in July ("Axis2 work files not deleted").
>>>
>>> I did not here reports from anyone else experiencing this. Particularly
>>> interesting because our environment is similar:
>>> Tomcat 5.5.17
>>> CentOS 4.3
>>> Axis2 1.3
>>>
>>> Are you by any chance running your OS under VMWare??
>>>
>>> Anyway, the solution (or workaround) is to deploy your web services and
>>> modules as an exploded directory structure rather than an AAR or MAR
>>> archive. One of the posts in the second thread has more details.
>>>
>>> - Steve
>>>
>>> ______________________________________________
>>> Steve Gruverman, Programmer
>>> IntelliCare, Inc. | A Medco Health Solutions Company
>>>
>>> 500 Southborough Drive | South Portland ME 04106
>>>
>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: axis-user-unsubscribe@ws.apache.org
>> For additional commands, e-mail: axis-user-help@ws.apache.org
>>
>
>
>
> Be the filmmaker you always wanted to be—learn how to burn a DVD with
> Windows®. Make your smash hit
> <http://clk.atdmt.com/MRT/go/108588797/direct/01/>
RE: Hot update: Too Many Open Files
Posted by Martin Gainty <mg...@hotmail.com>.
set sticky bit off so other users (other than root) can access
chmod chmod [OPTION]... MODE[,MODE]... FILE...
http://linux.die.net/man/1/chmod
chown --from=CURRENT_OWNER:CURRENT_GROUP file
http://linux.die.net/man/1/chown
if the file is located on linux extended file use
chattr [ -RV ] [ -v version ] [ mode ] files...
http://linux.die.net/man/1/chattr
Martin ______________________________________________ Disclaimer and confidentiality note Everything in this e-mail and any attachments relates to the official business of Sender. This transmission is of a confidential nature and Sender does not endorse distribution to any party other than intended recipient. Sender does not necessarily endorse content contained within this transmission. > From: Thomas.Talbott@serialssolutions.com> To: axis-user@ws.apache.org> Date: Fri, 29 Aug 2008 21:04:25 -0400> Subject: Re: Hot update: Too Many Open Files> > Hello Steve,> > We are not running under VMWare. I would say that our situation is similar,> but not the same. In your previous messages you say:> > ³A new set gets generated each time I restart Tomcat. On my Windows XP> system, these are deleted each time Tomcat stops, but not on our Linux> (CentOS) systems.²> > In our case, we are trying to deploy without restarting Tomcat. We have the> following set in axis2.xml:> > <parameter name="hotdeployment">true</parameter>> <parameter name="hotupdate">true</parameter>> > When we copy the updated aar file into the WEB-INF/services directory of the> axis2 webapp, temporary files are created in the work directory. The next> time we update aar file, more files are created in the work directory. When> I shutdown tomcat, the files ARE cleaned up. But, if I don't shutdown> tomcat, file handles are left open for every one of these temporary files!> So, soon, we run out of file handles.> > As I mentioned in my original message, we want to do this for staging and we> are not looking for this to be a production solution. As far as I can tell,> we should be able to shutdown tomcat, deploy the aar, and restart tomcat in> production without the problems you are seeing. We may just need to resort> to the same solution in production.> > It would seem, given our joint experience, that there is an issue with the> the temporary files used for deployment and update. Given what Deepal says:> > "Yes , Axis2 creates temp files from your services and modules , and> create a class loader from that. In that way we can ensure better> performance. This help us a lot when we have service aar or mar with> third party library inside it."> > It would appear that there is an issue with the class loader that is> created. In the "update" scenario, I wonder if the class loaders are> leaking (old class loaders not going away) after a service is updated.> Current evidence for this is that my files do eventually get cleaned up and> I assume that the class loader is doing this once it finally goes away.> But, there may be some management process that is responsible for this. I> have not looked into the code.> > Anyone with insight into this?> > Thanks,> -Tom> > On 8/29/08 2:22 PM, "SGruverman@intellicare.com"> <SG...@intellicare.com> wrote:> > > Thomas,> >> > This is very similar to two threads I started, one in May ("cached archive> > files not deleted") and one in July ("Axis2 work files not deleted").> >> > I did not here reports from anyone else experiencing this. Particularly> > interesting because our environment is similar:> > Tomcat 5.5.17> > CentOS 4.3> > Axis2 1.3> >> > Are you by any chance running your OS under VMWare??> >> > Anyway, the solution (or workaround) is to deploy your web services and> > modules as an exploded directory structure rather than an AAR or MAR> > archive. One of the posts in the second thread has more details.> >> > - Steve> >> > ______________________________________________> > Steve Gruverman, Programmer> > IntelliCare, Inc. | A Medco Health Solutions Company> >> > 500 Southborough Drive | South Portland ME 04106> >> >> >> > > ---------------------------------------------------------------------> To unsubscribe, e-mail: axis-user-unsubscribe@ws.apache.org> For additional commands, e-mail: axis-user-help@ws.apache.org>
_________________________________________________________________
Be the filmmaker you always wanted to be—learn how to burn a DVD with Windows®.
http://clk.atdmt.com/MRT/go/108588797/direct/01/
Re: Hot update: Too Many Open Files
Posted by "Talbott, Thomas" <Th...@serialssolutions.com>.
Hello Steve,
We are not running under VMWare. I would say that our situation is similar,
but not the same. In your previous messages you say:
³A new set gets generated each time I restart Tomcat. On my Windows XP
system, these are deleted each time Tomcat stops, but not on our Linux
(CentOS) systems.²
In our case, we are trying to deploy without restarting Tomcat. We have the
following set in axis2.xml:
<parameter name="hotdeployment">true</parameter>
<parameter name="hotupdate">true</parameter>
When we copy the updated aar file into the WEB-INF/services directory of the
axis2 webapp, temporary files are created in the work directory. The next
time we update aar file, more files are created in the work directory. When
I shutdown tomcat, the files ARE cleaned up. But, if I don't shutdown
tomcat, file handles are left open for every one of these temporary files!
So, soon, we run out of file handles.
As I mentioned in my original message, we want to do this for staging and we
are not looking for this to be a production solution. As far as I can tell,
we should be able to shutdown tomcat, deploy the aar, and restart tomcat in
production without the problems you are seeing. We may just need to resort
to the same solution in production.
It would seem, given our joint experience, that there is an issue with the
the temporary files used for deployment and update. Given what Deepal says:
"Yes , Axis2 creates temp files from your services and modules , and
create a class loader from that. In that way we can ensure better
performance. This help us a lot when we have service aar or mar with
third party library inside it."
It would appear that there is an issue with the class loader that is
created. In the "update" scenario, I wonder if the class loaders are
leaking (old class loaders not going away) after a service is updated.
Current evidence for this is that my files do eventually get cleaned up and
I assume that the class loader is doing this once it finally goes away.
But, there may be some management process that is responsible for this. I
have not looked into the code.
Anyone with insight into this?
Thanks,
-Tom
On 8/29/08 2:22 PM, "SGruverman@intellicare.com"
<SG...@intellicare.com> wrote:
> Thomas,
>
> This is very similar to two threads I started, one in May ("cached archive
> files not deleted") and one in July ("Axis2 work files not deleted").
>
> I did not here reports from anyone else experiencing this. Particularly
> interesting because our environment is similar:
> Tomcat 5.5.17
> CentOS 4.3
> Axis2 1.3
>
> Are you by any chance running your OS under VMWare??
>
> Anyway, the solution (or workaround) is to deploy your web services and
> modules as an exploded directory structure rather than an AAR or MAR
> archive. One of the posts in the second thread has more details.
>
> - Steve
>
> ______________________________________________
> Steve Gruverman, Programmer
> IntelliCare, Inc. | A Medco Health Solutions Company
>
> 500 Southborough Drive | South Portland ME 04106
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-user-help@ws.apache.org
Re: Hot update: Too Many Open Files
Posted by SG...@intellicare.com.
Thomas,
This is very similar to two threads I started, one in May ("cached archive
files not deleted") and one in July ("Axis2 work files not deleted").
I did not here reports from anyone else experiencing this. Particularly
interesting because our environment is similar:
Tomcat 5.5.17
CentOS 4.3
Axis2 1.3
Are you by any chance running your OS under VMWare??
Anyway, the solution (or workaround) is to deploy your web services and
modules as an exploded directory structure rather than an AAR or MAR
archive. One of the posts in the second thread has more details.
- Steve
______________________________________________
Steve Gruverman, Programmer
IntelliCare, Inc. | A Medco Health Solutions Company
500 Southborough Drive | South Portland ME 04106
"Talbott, Thomas" <Th...@serialssolutions.com> wrote on 08/29/2008
02:10:32 PM:
> [image removed]
>
> Hot update: Too Many Open Files
>
> Talbott, Thomas
>
> to:
>
> Axis2
>
> 08/29/2008 02:11 PM
>
> Please respond to axis-user@ws.apache.org
>
> Hey all,
>
> Our project just started utilizing Axis2 for our web service framework
and
> we are running into an issue with Hot Update. We recognize that this is
not
> recommended for production, but would like to use it in our staging
> environment that will be managed through a continuous integration
platform.
>
> The problem is after a few days, we started getting the following when
the
> aar file was updated:
>
> 8/27|13:52:57 [Timer-3] INFO :ServiceDeployer - Deploying Web service:
> version-1.4.aar - file:/var/tomcat5/webapps/axis2/WEB-INF/services/ve
> rsion-1.4.aar
> 8/27|13:52:57 [Timer-3] WARN :Utils - Exception extracting jars into
> temporary directory : java.io.IOException: Too many open files : switch
> ing to alternate class loading mechanism
> 8/27|13:52:59 [Timer-3] ERROR:ServiceDeployer - The tc_web_service_1.aar
> service, which is not valid, caused The following error occurred du
> ring schema generation: Class Not found :
> com.sersol.fedsearch.dta.server.service.TCWSServer
> org.apache.axis2.deployment.DeploymentException: The following error
> occurred during schema generation: Class Not found : com.sersol.fedsearc
>
> We are currently running Axis2 1.4 under Tomcat 5.5.26 on CentOS 5.2
(Linux
> version 2.6.18-92.1.6.el5 (mockbuild@builder15.centos.org) (gcc version
> 4.1.2 20071124 (Red Hat 4.1.2-42)) #1 SMP Wed Jun 25 13:45:47 EDT 2008).
>
> If I look at the open files, the majority are under the work folder in
the
> Tomcat directory structure. Example:
>
>
/var/cache/tomcat5/work/Catalina/localhost/axis2/_axis2/axis21612jaxen-1.1.1
> .jar
>
> Every time I do an update, the number of open files under the '_axis2'
> directory grows:
>
> # lsof | grep _axis2 | wc -l
> 232
> # lsof | grep _axis2 | wc -l
> 424
> # lsof | grep _axis2 | wc -l
> 616
>
> I don't see anyone else reporting this issue. Is this a problem running
> under Tomcat?
>
> If this has been reported elsewhere that I have missed, please direct me
to
> that discussion. Otherwise, if you have any insight, I'd love to hear
it.
>
> Thanks,
> -Tom
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-user-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-user-help@ws.apache.org
>
>
> --
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: axis-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-user-help@ws.apache.org