You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Kirill Ilyukhin <ki...@gmail.com> on 2013/02/22 06:09:36 UTC
Webapp reload corrupts Tomcat
Hi!
I have 5 servers with the following configuration:
Windows Server 2008 R2 64-bit,
Oracle JDK 1.7.0_11-b21 64-bit,
Tomcat 7.0.35 running as a service.
A few webapps are running perfectly well under the Tomcat until one of
them is reloaded.
Reload of a webapp (either by pressing [reload] on manager page or by
changing web.xml file) makes Tomcat extremely unstable.
The first (pre-production) server sporadically fails with absolutely
no traces in log files or Windows event logs: does not accept
connections, responds with HTTP 404 or does not respond at all.
Other 4 (production) servers also become unstable but in a different
manner - a lot of HTTP 500 responses (logs are also clean). Plus on
some of these servers I have a lot (less than HTTP 500 responses
though) of NPEs:
-------------
SEVERE: Exception Processing /path/here
java.lang.NullPointerException
at org.apache.catalina.core.StandardWrapper.servletSecurityAnnotationScan(StandardWrapper.java:1216)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:461)
at org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:305)
at org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:83)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:1822)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
-------------
These exception occurs for any webapp, not only the one being reloaded.
All the above happens irregularly, quite often on the pre-production
server, only once on production servers. Tomcat service restart fixes
the problem. Updating Tomcat to 7.0.37 did not help.
I cannot reproduce this on any of developers hosts with the same
versions of JDK and Tomcat.
Any ideas?
Thanks a lot in advance,
Kirill
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Webapp reload corrupts Tomcat
Posted by Technical Support <te...@brainstalk.com>.
Hi,
Im beginner in Linux. Can anyone help me develop my skills in Linux?
Thanks
Chak Teylor
On Fri, Feb 22, 2013 at 9:32 PM, Daniel Mikusa <dm...@vmware.com> wrote:
> On Feb 22, 2013, at 12:09 AM, Kirill Ilyukhin wrote:
>
> > Hi!
> >
> > I have 5 servers with the following configuration:
> > Windows Server 2008 R2 64-bit,
> > Oracle JDK 1.7.0_11-b21 64-bit,
> > Tomcat 7.0.35 running as a service.
> > A few webapps are running perfectly well under the Tomcat until one of
> > them is reloaded.
> >
> > Reload of a webapp (either by pressing [reload] on manager page or by
> > changing web.xml file) makes Tomcat extremely unstable.
> > The first (pre-production) server sporadically fails with absolutely
> > no traces in log files or Windows event logs:
>
> These three symptoms can often mean different things. For example, if
> it's not accepting connections then you can't really get a 404 error. Can
> you be more specific about when you see them and the status of your Tomcat
> server when you see them.
>
> Some troubleshooting thoughts...
>
> > does not accept connections,
>
> 1.) Is the process still running? Did the JVM crash? Is there a crash
> file? hs_err_podXXXX.log?
> 2.) Is the socket still listening? Have you changed any firewall rules?
> 3.) Try "telnet <port>". What happens?
> 4.) Try connecting to Tomcat with jconsole or jvisualvm? Can you? If so
> look at the connector mbean. What is its state?
> 5.) Take some thread dumps.
>
> > responds with HTTP 404 or
>
> Assuming the URL that you entered is valid, this sounds like your
> application may have failed to reload. Can you turn up the log level in
> your application to see what it is doing?
>
> > does not respond at all.
>
> Assuming that you mean it accepts your connection and request, but never
> processes and returns a response. In this case, take some thread dumps
> after you have sent the response. That should show you what is going on
> inside the JVM.
>
> > Other 4 (production) servers also become unstable but in a different
> > manner - a lot of HTTP 500 responses (logs are also clean).
>
> It would be very odd to see nothing in the logs after a 500 error. Have
> you looked at all of the log files, in particular
> "localhost-YYYY-MM-DD.log"? That file should contain a stack trace after a
> 500 error. Also, what do the see in your browser when you get the 500
> error? The default error page should print the stack trace for the 500
> error.
>
> > Plus on
> > some of these servers I have a lot (less than HTTP 500 responses
> > though) of NPEs:
> > -------------
> > SEVERE: Exception Processing /path/here
> > java.lang.NullPointerException
> > at
> org.apache.catalina.core.StandardWrapper.servletSecurityAnnotationScan(StandardWrapper.java:1216)
> > at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:461)
> > at
> org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:305)
> > at
> org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:83)
> > at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> > at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
> > at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
> > at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> > at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
> > at
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
> > at
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
> > at
> org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:1822)
> > at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> > at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> > at java.lang.Thread.run(Thread.java:722)
> > -------------
> > These exception occurs for any webapp, not only the one being reloaded.
>
> What's the specific version of Tomcat where you saw this error occur? Is
> it 7.0.37? or was this on an older version?
>
> >
> > All the above happens irregularly, quite often on the pre-production
> > server, only once on production servers. Tomcat service restart fixes
> > the problem. Updating Tomcat to 7.0.37 did not help.
> > I cannot reproduce this on any of developers hosts with the same
> > versions of JDK and Tomcat.
> >
> > Any ideas?
>
> 1.) An easy temporary fix would be to not reload the application in
> production. If you need to redeploy or reload the application, stop the
> server, undeploy, redeploy and start your server.
>
> 2.) Include your Tomcat configuration, minus comments.
>
> 3.) See troubleshooting steps above.
>
> Dan
>
> >
> >
> > Thanks a lot in advance,
> > Kirill
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Webapp reload corrupts Tomcat
Posted by Daniel Mikusa <dm...@vmware.com>.
On Feb 24, 2013, at 9:27 PM, Kirill Ilyukhin wrote:
>
> On 25.02.2013 2:10, Daniel Mikusa wrote:
>> On Feb 24, 2013, at 11:48 AM, Kirill Ilyukhin wrote:
>>
>>> On Feb 22, 2013, at 22:32 , Daniel Mikusa <dm...@vmware.com> wrote:
>>>
>>>> On Feb 22, 2013, at 12:09 AM, Kirill Ilyukhin wrote:
>>>>
>>>>> Hi!
>>>>>
>>>>> I have 5 servers with the following configuration:
>>>>> Windows Server 2008 R2 64-bit,
>>>>> Oracle JDK 1.7.0_11-b21 64-bit,
>>>>> Tomcat 7.0.35 running as a service.
>>>>> A few webapps are running perfectly well under the Tomcat until one of
>>>>> them is reloaded.
>>>>>
>>>>> Reload of a webapp (either by pressing [reload] on manager page or by
>>>>> changing web.xml file) makes Tomcat extremely unstable.
>>>>> The first (pre-production) server sporadically fails with absolutely
>>>>> no traces in log files or Windows event logs:
>>>> These three symptoms can often mean different things. For example, if it's not accepting connections then you can't really get a 404 error. Can you be more specific about when you see them and the status of your Tomcat server when you see them.
>>>>
>>>> Some troubleshooting thoughts...
>>>>
>>>>> does not accept connections,
>>>> 1.) Is the process still running? Did the JVM crash? Is there a crash file? hs_err_podXXXX.log?
>>>> 2.) Is the socket still listening? Have you changed any firewall rules?
>>>> 3.) Try "telnet <port>". What happens?
>>>> 4.) Try connecting to Tomcat with jconsole or jvisualvm? Can you? If so look at the connector mbean. What is its state?
>>>> 5.) Take some thread dumps.
>>>>
>>>>> responds with HTTP 404 or
>>>> Assuming the URL that you entered is valid, this sounds like your application may have failed to reload. Can you turn up the log level in your application to see what it is doing?
>>>>
>>>>> does not respond at all.
>>>> Assuming that you mean it accepts your connection and request, but never processes and returns a response. In this case, take some thread dumps after you have sent the response. That should show you what is going on inside the JVM.
>>> Here is more details.
>>> All these was perfectly working on JDK 1.6 and Tomcat 7 a few builds earlier.
>>> A webapp is serving real-time data with updates. Open the page in a browser, it downloads an HTML (a static file), then a bunch of CSS and Javascripts (also static files), and starts fetching the data and updates (from a servlet) - this makes an indefinite sequence of HTTP requests.
>>> When I open the page after the webapp is reloaded, some of this requests (including the static files) fail with either 404 code or no response or not accepted connection. Requests are being failed quite randomly. If I reload the page, also a lot of fails, but slightly different - e.g. I might get the file which was 404 earlier. The data requests also fail randomly, some of them are successful, which means that the webapp is loaded.
>>> So, answering to the questions above: the process is definitely running, socket is definitely listening, telneting Tomcat gives random results, JConsole shows nothing uncommon.
>> Can you attach the access log that shows these requests?
> I do not have the access logs at my disposal to attach them. From what I remember, the access logs were clean - no 404 codes, every request is responded. In other words if you look from the server side (logs, JConsole, Windows logs) , everything is fine.
>>
>> Also, a couple more follow up questions…
>>
>> 1.) When this occurs does it only occur for the application that you reload? or does it happen for all the apps that you have deployed to the Tomcat server?
> It happens to all the webapps on the server, even "pre-installed" ones (manager and ROOT).
>> 2.) Is this application specific, in other words does this only occur when you reload one particular application? or does reloading any application on the Tomcat server cause the problem?
> It is random. One of the application, when reloaded, causes the problem almost for sure. Others can also cause the problem, but less frequently.
>>
>> 3.) Can you replicate this problem in a testing environment?
> No, I can not.
>>
>>>>> Other 4 (production) servers also become unstable but in a different
>>>>> manner - a lot of HTTP 500 responses (logs are also clean).
>>>> It would be very odd to see nothing in the logs after a 500 error. Have you looked at all of the log files, in particular "localhost-YYYY-MM-DD.log"? That file should contain a stack trace after a 500 error. Also, what do the see in your browser when you get the 500 error? The default error page should print the stack trace for the 500 error.
>>> That is what I am talking about - quite odd behaviour. Nothing in log files, no body with HTTP 500 response in a browser.
>>>
>>>>> Plus on
>>>>> some of these servers I have a lot (less than HTTP 500 responses
>>>>> though) of NPEs:
>>>>> -------------
>>>>> SEVERE: Exception Processing /path/here
>>>>> java.lang.NullPointerException
>>>>> at org.apache.catalina.core.StandardWrapper.servletSecurityAnnotationScan(StandardWrapper.java:1216)
>>>>> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:461)
>>>>> at org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:305)
>>>>> at org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:83)
>>>>> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>>>>> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>>>>> at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
>>>>> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>>>>> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
>>>>> at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
>>>>> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
>>>>> at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:1822)
>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>>> at java.lang.Thread.run(Thread.java:722)
>>>>> -------------
>>>>> These exception occurs for any webapp, not only the one being reloaded.
>>>> What's the specific version of Tomcat where you saw this error occur? Is it 7.0.37? or was this on an older version?
>>> This was on 7.0.35
>>>
>>>>> All the above happens irregularly, quite often on the pre-production
>>>>> server, only once on production servers. Tomcat service restart fixes
>>>>> the problem. Updating Tomcat to 7.0.37 did not help.
>>>>> I cannot reproduce this on any of developers hosts with the same
>>>>> versions of JDK and Tomcat.
>>>>>
>>>>> Any ideas?
>>>> 1.) An easy temporary fix would be to not reload the application in production. If you need to redeploy or reload the application, stop the server, undeploy, redeploy and start your server.
>>>> 2.) Include your Tomcat configuration, minus comments.
>>> Please see below:
>>> <Server port="8005" shutdown="SHUTDOWN">
>>> <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
>>> <Listener className="org.apache.catalina.core.JasperListener" />
>>> <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
>>> <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
>>> <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />
>>> <GlobalNamingResources>
>>> <Resource name="UserDatabase" auth="Container"
>>> type="org.apache.catalina.UserDatabase"
>>> description="User database that can be updated and saved"
>>> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
>>> pathname="conf/tomcat-users.xml" />
>>> <!-- A couple of custom resources here -->
>>> </GlobalNamingResources>
>>> <Service name="Catalina">
>>> <Connector port="80" protocol="HTTP/1.1"
>>> asyncTimeout="20000"
>>> URIEncoding="utf-8"
>>> acceptorThreadCount="2"
>>> compressableMimeType="text/html,text/xml,text/plain,text/x-json,application/javascript,application/json,text/css"
>>> compression="on"
>> You might want to disable compression and see if that makes a difference. I can't say that this is specifically causing the problem, but it would allow you to rule it out as a possibility.
> Thanks, I will try disabling compression, but, as It might not help, that is not enough for me. Servers are located in a data centre with limited access and I should have some strong ideas of how to fix the problem (or detect its cause) when I go there next time.
Some other thoughts for troubleshooting this…
1.) Take a heap dump and a couple thread dumps. Perform the reload. Confirm the problem. Take another heap dump and some more thread dumps. Compare to see what has changed and if you see anything odd.
2.) Setup a packet capture like Wireshark or tcpdump. Capture some packets prior to the reload, capture some after the reload. Compare and look for a difference in the requests and responses.
3.) If you haven't already, update to the latest Java 7. You could be hitting a bug.
> As I have said, everything worked fine with JDK 1.6. Can this be because of some unexpected behaviour of Oracle JDK 1.7 classloader for win64? Can it be because class files of the most of the apps were build with JDK 1.6 and now running on Java 7? Can it be because some apps may leak threads and ThreadLocals and Java 6 JVM could handle it but Java 7 JVM cannot?
Hard to say exactly. What I can say is that there are tons of people running Tomcat on Java 7 without issue, so I would suggest focusing on anything that might be different or non-standard about the environment where you are seeing the issue. For example, can you replicate the problem with fresh Tomcat installation (i.e. download, unzip, deploy your app)? Can you replicate the problem without deploying your apps (i.e. with just standard TC apps deployed)?
Hope that helps
Dan
>
> Thanks,
> Kirill
>
>>
>> Dan
>>
>>
>>> connectionTimeout="20000"
>>> maxThreads="1024"
>>> processorCache="512"
>>> redirectPort="8443" />
>>> <Engine name="Catalina" defaultHost="localhost">
>>> <Realm className="org.apache.catalina.realm.LockOutRealm">
>>> <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
>>> resourceName="UserDatabase"/>
>>> </Realm>
>>> <Host name="localhost" appBase="webapps"
>>> unpackWARs="true" autoDeploy="true">
>>> <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
>>> prefix="localhost_access_log." suffix=".txt"
>>> pattern="%h %l %u %t "%r" %s %b %Dms" resolveHosts="false"/>
>>> </Host>
>>> </Engine>
>>> </Service>
>>> </Server>
>>>
>>>
>>> Best regards,
>>> Kirill
>>>
>>>
>>>> 3.) See troubleshooting steps above.
>>>>
>>>> Dan
>>>>
>>>>>
>>>>> Thanks a lot in advance,
>>>>> Kirill
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Webapp reload corrupts Tomcat
Posted by Kirill Ilyukhin <ki...@gmail.com>.
On 25.02.2013 2:10, Daniel Mikusa wrote:
> On Feb 24, 2013, at 11:48 AM, Kirill Ilyukhin wrote:
>
>> On Feb 22, 2013, at 22:32 , Daniel Mikusa <dm...@vmware.com> wrote:
>>
>>> On Feb 22, 2013, at 12:09 AM, Kirill Ilyukhin wrote:
>>>
>>>> Hi!
>>>>
>>>> I have 5 servers with the following configuration:
>>>> Windows Server 2008 R2 64-bit,
>>>> Oracle JDK 1.7.0_11-b21 64-bit,
>>>> Tomcat 7.0.35 running as a service.
>>>> A few webapps are running perfectly well under the Tomcat until one of
>>>> them is reloaded.
>>>>
>>>> Reload of a webapp (either by pressing [reload] on manager page or by
>>>> changing web.xml file) makes Tomcat extremely unstable.
>>>> The first (pre-production) server sporadically fails with absolutely
>>>> no traces in log files or Windows event logs:
>>> These three symptoms can often mean different things. For example, if it's not accepting connections then you can't really get a 404 error. Can you be more specific about when you see them and the status of your Tomcat server when you see them.
>>>
>>> Some troubleshooting thoughts...
>>>
>>>> does not accept connections,
>>> 1.) Is the process still running? Did the JVM crash? Is there a crash file? hs_err_podXXXX.log?
>>> 2.) Is the socket still listening? Have you changed any firewall rules?
>>> 3.) Try "telnet <port>". What happens?
>>> 4.) Try connecting to Tomcat with jconsole or jvisualvm? Can you? If so look at the connector mbean. What is its state?
>>> 5.) Take some thread dumps.
>>>
>>>> responds with HTTP 404 or
>>> Assuming the URL that you entered is valid, this sounds like your application may have failed to reload. Can you turn up the log level in your application to see what it is doing?
>>>
>>>> does not respond at all.
>>> Assuming that you mean it accepts your connection and request, but never processes and returns a response. In this case, take some thread dumps after you have sent the response. That should show you what is going on inside the JVM.
>> Here is more details.
>> All these was perfectly working on JDK 1.6 and Tomcat 7 a few builds earlier.
>> A webapp is serving real-time data with updates. Open the page in a browser, it downloads an HTML (a static file), then a bunch of CSS and Javascripts (also static files), and starts fetching the data and updates (from a servlet) - this makes an indefinite sequence of HTTP requests.
>> When I open the page after the webapp is reloaded, some of this requests (including the static files) fail with either 404 code or no response or not accepted connection. Requests are being failed quite randomly. If I reload the page, also a lot of fails, but slightly different - e.g. I might get the file which was 404 earlier. The data requests also fail randomly, some of them are successful, which means that the webapp is loaded.
>> So, answering to the questions above: the process is definitely running, socket is definitely listening, telneting Tomcat gives random results, JConsole shows nothing uncommon.
> Can you attach the access log that shows these requests?
I do not have the access logs at my disposal to attach them. From what I
remember, the access logs were clean - no 404 codes, every request is
responded. In other words if you look from the server side (logs,
JConsole, Windows logs) , everything is fine.
>
>
> Also, a couple more follow up questions…
>
> 1.) When this occurs does it only occur for the application that you reload? or does it happen for all the apps that you have deployed to the Tomcat server?
It happens to all the webapps on the server, even "pre-installed" ones
(manager and ROOT).
> 2.) Is this application specific, in other words does this only occur when you reload one particular application? or does reloading any application on the Tomcat server cause the problem?
It is random. One of the application, when reloaded, causes the problem
almost for sure. Others can also cause the problem, but less frequently.
>
> 3.) Can you replicate this problem in a testing environment?
No, I can not.
>
>>>> Other 4 (production) servers also become unstable but in a different
>>>> manner - a lot of HTTP 500 responses (logs are also clean).
>>> It would be very odd to see nothing in the logs after a 500 error. Have you looked at all of the log files, in particular "localhost-YYYY-MM-DD.log"? That file should contain a stack trace after a 500 error. Also, what do the see in your browser when you get the 500 error? The default error page should print the stack trace for the 500 error.
>> That is what I am talking about - quite odd behaviour. Nothing in log files, no body with HTTP 500 response in a browser.
>>
>>>> Plus on
>>>> some of these servers I have a lot (less than HTTP 500 responses
>>>> though) of NPEs:
>>>> -------------
>>>> SEVERE: Exception Processing /path/here
>>>> java.lang.NullPointerException
>>>> at org.apache.catalina.core.StandardWrapper.servletSecurityAnnotationScan(StandardWrapper.java:1216)
>>>> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:461)
>>>> at org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:305)
>>>> at org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:83)
>>>> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>>>> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>>>> at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
>>>> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>>>> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
>>>> at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
>>>> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
>>>> at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:1822)
>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>> at java.lang.Thread.run(Thread.java:722)
>>>> -------------
>>>> These exception occurs for any webapp, not only the one being reloaded.
>>> What's the specific version of Tomcat where you saw this error occur? Is it 7.0.37? or was this on an older version?
>> This was on 7.0.35
>>
>>>> All the above happens irregularly, quite often on the pre-production
>>>> server, only once on production servers. Tomcat service restart fixes
>>>> the problem. Updating Tomcat to 7.0.37 did not help.
>>>> I cannot reproduce this on any of developers hosts with the same
>>>> versions of JDK and Tomcat.
>>>>
>>>> Any ideas?
>>> 1.) An easy temporary fix would be to not reload the application in production. If you need to redeploy or reload the application, stop the server, undeploy, redeploy and start your server.
>>> 2.) Include your Tomcat configuration, minus comments.
>> Please see below:
>> <Server port="8005" shutdown="SHUTDOWN">
>> <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
>> <Listener className="org.apache.catalina.core.JasperListener" />
>> <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
>> <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
>> <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />
>> <GlobalNamingResources>
>> <Resource name="UserDatabase" auth="Container"
>> type="org.apache.catalina.UserDatabase"
>> description="User database that can be updated and saved"
>> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
>> pathname="conf/tomcat-users.xml" />
>> <!-- A couple of custom resources here -->
>> </GlobalNamingResources>
>> <Service name="Catalina">
>> <Connector port="80" protocol="HTTP/1.1"
>> asyncTimeout="20000"
>> URIEncoding="utf-8"
>> acceptorThreadCount="2"
>> compressableMimeType="text/html,text/xml,text/plain,text/x-json,application/javascript,application/json,text/css"
>> compression="on"
> You might want to disable compression and see if that makes a difference. I can't say that this is specifically causing the problem, but it would allow you to rule it out as a possibility.
Thanks, I will try disabling compression, but, as It might not help,
that is not enough for me. Servers are located in a data centre with
limited access and I should have some strong ideas of how to fix the
problem (or detect its cause) when I go there next time.
As I have said, everything worked fine with JDK 1.6. Can this be because
of some unexpected behaviour of Oracle JDK 1.7 classloader for win64?
Can it be because class files of the most of the apps were build with
JDK 1.6 and now running on Java 7? Can it be because some apps may leak
threads and ThreadLocals and Java 6 JVM could handle it but Java 7 JVM
cannot?
Thanks,
Kirill
>
> Dan
>
>
>> connectionTimeout="20000"
>> maxThreads="1024"
>> processorCache="512"
>> redirectPort="8443" />
>> <Engine name="Catalina" defaultHost="localhost">
>> <Realm className="org.apache.catalina.realm.LockOutRealm">
>> <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
>> resourceName="UserDatabase"/>
>> </Realm>
>> <Host name="localhost" appBase="webapps"
>> unpackWARs="true" autoDeploy="true">
>> <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
>> prefix="localhost_access_log." suffix=".txt"
>> pattern="%h %l %u %t "%r" %s %b %Dms" resolveHosts="false"/>
>> </Host>
>> </Engine>
>> </Service>
>> </Server>
>>
>>
>> Best regards,
>> Kirill
>>
>>
>>> 3.) See troubleshooting steps above.
>>>
>>> Dan
>>>
>>>>
>>>> Thanks a lot in advance,
>>>> Kirill
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Webapp reload corrupts Tomcat
Posted by Daniel Mikusa <dm...@vmware.com>.
On Feb 24, 2013, at 11:48 AM, Kirill Ilyukhin wrote:
> On Feb 22, 2013, at 22:32 , Daniel Mikusa <dm...@vmware.com> wrote:
>
>> On Feb 22, 2013, at 12:09 AM, Kirill Ilyukhin wrote:
>>
>>> Hi!
>>>
>>> I have 5 servers with the following configuration:
>>> Windows Server 2008 R2 64-bit,
>>> Oracle JDK 1.7.0_11-b21 64-bit,
>>> Tomcat 7.0.35 running as a service.
>>> A few webapps are running perfectly well under the Tomcat until one of
>>> them is reloaded.
>>>
>>> Reload of a webapp (either by pressing [reload] on manager page or by
>>> changing web.xml file) makes Tomcat extremely unstable.
>>> The first (pre-production) server sporadically fails with absolutely
>>> no traces in log files or Windows event logs:
>>
>> These three symptoms can often mean different things. For example, if it's not accepting connections then you can't really get a 404 error. Can you be more specific about when you see them and the status of your Tomcat server when you see them.
>>
>> Some troubleshooting thoughts...
>>
>>> does not accept connections,
>>
>> 1.) Is the process still running? Did the JVM crash? Is there a crash file? hs_err_podXXXX.log?
>> 2.) Is the socket still listening? Have you changed any firewall rules?
>> 3.) Try "telnet <port>". What happens?
>> 4.) Try connecting to Tomcat with jconsole or jvisualvm? Can you? If so look at the connector mbean. What is its state?
>> 5.) Take some thread dumps.
>>
>>> responds with HTTP 404 or
>>
>> Assuming the URL that you entered is valid, this sounds like your application may have failed to reload. Can you turn up the log level in your application to see what it is doing?
>>
>>> does not respond at all.
>>
>> Assuming that you mean it accepts your connection and request, but never processes and returns a response. In this case, take some thread dumps after you have sent the response. That should show you what is going on inside the JVM.
>
> Here is more details.
> All these was perfectly working on JDK 1.6 and Tomcat 7 a few builds earlier.
> A webapp is serving real-time data with updates. Open the page in a browser, it downloads an HTML (a static file), then a bunch of CSS and Javascripts (also static files), and starts fetching the data and updates (from a servlet) - this makes an indefinite sequence of HTTP requests.
> When I open the page after the webapp is reloaded, some of this requests (including the static files) fail with either 404 code or no response or not accepted connection. Requests are being failed quite randomly. If I reload the page, also a lot of fails, but slightly different - e.g. I might get the file which was 404 earlier. The data requests also fail randomly, some of them are successful, which means that the webapp is loaded.
> So, answering to the questions above: the process is definitely running, socket is definitely listening, telneting Tomcat gives random results, JConsole shows nothing uncommon.
Can you attach the access log that shows these requests?
Also, a couple more follow up questions…
1.) When this occurs does it only occur for the application that you reload? or does it happen for all the apps that you have deployed to the Tomcat server?
2.) Is this application specific, in other words does this only occur when you reload one particular application? or does reloading any application on the Tomcat server cause the problem?
3.) Can you replicate this problem in a testing environment?
>
>>
>>> Other 4 (production) servers also become unstable but in a different
>>> manner - a lot of HTTP 500 responses (logs are also clean).
>>
>> It would be very odd to see nothing in the logs after a 500 error. Have you looked at all of the log files, in particular "localhost-YYYY-MM-DD.log"? That file should contain a stack trace after a 500 error. Also, what do the see in your browser when you get the 500 error? The default error page should print the stack trace for the 500 error.
>
> That is what I am talking about - quite odd behaviour. Nothing in log files, no body with HTTP 500 response in a browser.
>
>>
>>> Plus on
>>> some of these servers I have a lot (less than HTTP 500 responses
>>> though) of NPEs:
>>> -------------
>>> SEVERE: Exception Processing /path/here
>>> java.lang.NullPointerException
>>> at org.apache.catalina.core.StandardWrapper.servletSecurityAnnotationScan(StandardWrapper.java:1216)
>>> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:461)
>>> at org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:305)
>>> at org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:83)
>>> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>>> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>>> at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
>>> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>>> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
>>> at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
>>> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
>>> at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:1822)
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>> at java.lang.Thread.run(Thread.java:722)
>>> -------------
>>> These exception occurs for any webapp, not only the one being reloaded.
>>
>> What's the specific version of Tomcat where you saw this error occur? Is it 7.0.37? or was this on an older version?
>
> This was on 7.0.35
>
>>
>>>
>>> All the above happens irregularly, quite often on the pre-production
>>> server, only once on production servers. Tomcat service restart fixes
>>> the problem. Updating Tomcat to 7.0.37 did not help.
>>> I cannot reproduce this on any of developers hosts with the same
>>> versions of JDK and Tomcat.
>>>
>>> Any ideas?
>>
>> 1.) An easy temporary fix would be to not reload the application in production. If you need to redeploy or reload the application, stop the server, undeploy, redeploy and start your server.
>
>>
>> 2.) Include your Tomcat configuration, minus comments.
> Please see below:
> <Server port="8005" shutdown="SHUTDOWN">
> <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
> <Listener className="org.apache.catalina.core.JasperListener" />
> <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
> <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
> <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />
> <GlobalNamingResources>
> <Resource name="UserDatabase" auth="Container"
> type="org.apache.catalina.UserDatabase"
> description="User database that can be updated and saved"
> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
> pathname="conf/tomcat-users.xml" />
> <!-- A couple of custom resources here -->
> </GlobalNamingResources>
> <Service name="Catalina">
> <Connector port="80" protocol="HTTP/1.1"
> asyncTimeout="20000"
> URIEncoding="utf-8"
> acceptorThreadCount="2"
> compressableMimeType="text/html,text/xml,text/plain,text/x-json,application/javascript,application/json,text/css"
> compression="on"
You might want to disable compression and see if that makes a difference. I can't say that this is specifically causing the problem, but it would allow you to rule it out as a possibility.
Dan
> connectionTimeout="20000"
> maxThreads="1024"
> processorCache="512"
> redirectPort="8443" />
> <Engine name="Catalina" defaultHost="localhost">
> <Realm className="org.apache.catalina.realm.LockOutRealm">
> <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
> resourceName="UserDatabase"/>
> </Realm>
> <Host name="localhost" appBase="webapps"
> unpackWARs="true" autoDeploy="true">
> <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
> prefix="localhost_access_log." suffix=".txt"
> pattern="%h %l %u %t "%r" %s %b %Dms" resolveHosts="false"/>
> </Host>
> </Engine>
> </Service>
> </Server>
>
>
> Best regards,
> Kirill
>
>
>>
>> 3.) See troubleshooting steps above.
>>
>> Dan
>>
>>>
>>>
>>> Thanks a lot in advance,
>>> Kirill
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Webapp reload corrupts Tomcat
Posted by Kirill Ilyukhin <ki...@gmail.com>.
On Feb 22, 2013, at 22:32 , Daniel Mikusa <dm...@vmware.com> wrote:
> On Feb 22, 2013, at 12:09 AM, Kirill Ilyukhin wrote:
>
>> Hi!
>>
>> I have 5 servers with the following configuration:
>> Windows Server 2008 R2 64-bit,
>> Oracle JDK 1.7.0_11-b21 64-bit,
>> Tomcat 7.0.35 running as a service.
>> A few webapps are running perfectly well under the Tomcat until one of
>> them is reloaded.
>>
>> Reload of a webapp (either by pressing [reload] on manager page or by
>> changing web.xml file) makes Tomcat extremely unstable.
>> The first (pre-production) server sporadically fails with absolutely
>> no traces in log files or Windows event logs:
>
> These three symptoms can often mean different things. For example, if it's not accepting connections then you can't really get a 404 error. Can you be more specific about when you see them and the status of your Tomcat server when you see them.
>
> Some troubleshooting thoughts...
>
>> does not accept connections,
>
> 1.) Is the process still running? Did the JVM crash? Is there a crash file? hs_err_podXXXX.log?
> 2.) Is the socket still listening? Have you changed any firewall rules?
> 3.) Try "telnet <port>". What happens?
> 4.) Try connecting to Tomcat with jconsole or jvisualvm? Can you? If so look at the connector mbean. What is its state?
> 5.) Take some thread dumps.
>
>> responds with HTTP 404 or
>
> Assuming the URL that you entered is valid, this sounds like your application may have failed to reload. Can you turn up the log level in your application to see what it is doing?
>
>> does not respond at all.
>
> Assuming that you mean it accepts your connection and request, but never processes and returns a response. In this case, take some thread dumps after you have sent the response. That should show you what is going on inside the JVM.
Here is more details.
All these was perfectly working on JDK 1.6 and Tomcat 7 a few builds earlier.
A webapp is serving real-time data with updates. Open the page in a browser, it downloads an HTML (a static file), then a bunch of CSS and Javascripts (also static files), and starts fetching the data and updates (from a servlet) - this makes an indefinite sequence of HTTP requests.
When I open the page after the webapp is reloaded, some of this requests (including the static files) fail with either 404 code or no response or not accepted connection. Requests are being failed quite randomly. If I reload the page, also a lot of fails, but slightly different - e.g. I might get the file which was 404 earlier. The data requests also fail randomly, some of them are successful, which means that the webapp is loaded.
So, answering to the questions above: the process is definitely running, socket is definitely listening, telneting Tomcat gives random results, JConsole shows nothing uncommon.
>
>> Other 4 (production) servers also become unstable but in a different
>> manner - a lot of HTTP 500 responses (logs are also clean).
>
> It would be very odd to see nothing in the logs after a 500 error. Have you looked at all of the log files, in particular "localhost-YYYY-MM-DD.log"? That file should contain a stack trace after a 500 error. Also, what do the see in your browser when you get the 500 error? The default error page should print the stack trace for the 500 error.
That is what I am talking about - quite odd behaviour. Nothing in log files, no body with HTTP 500 response in a browser.
>
>> Plus on
>> some of these servers I have a lot (less than HTTP 500 responses
>> though) of NPEs:
>> -------------
>> SEVERE: Exception Processing /path/here
>> java.lang.NullPointerException
>> at org.apache.catalina.core.StandardWrapper.servletSecurityAnnotationScan(StandardWrapper.java:1216)
>> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:461)
>> at org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:305)
>> at org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:83)
>> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>> at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
>> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
>> at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
>> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
>> at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:1822)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> at java.lang.Thread.run(Thread.java:722)
>> -------------
>> These exception occurs for any webapp, not only the one being reloaded.
>
> What's the specific version of Tomcat where you saw this error occur? Is it 7.0.37? or was this on an older version?
This was on 7.0.35
>
>>
>> All the above happens irregularly, quite often on the pre-production
>> server, only once on production servers. Tomcat service restart fixes
>> the problem. Updating Tomcat to 7.0.37 did not help.
>> I cannot reproduce this on any of developers hosts with the same
>> versions of JDK and Tomcat.
>>
>> Any ideas?
>
> 1.) An easy temporary fix would be to not reload the application in production. If you need to redeploy or reload the application, stop the server, undeploy, redeploy and start your server.
>
> 2.) Include your Tomcat configuration, minus comments.
Please see below:
<Server port="8005" shutdown="SHUTDOWN">
<Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
<Listener className="org.apache.catalina.core.JasperListener" />
<Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
<Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
<Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />
<GlobalNamingResources>
<Resource name="UserDatabase" auth="Container"
type="org.apache.catalina.UserDatabase"
description="User database that can be updated and saved"
factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
pathname="conf/tomcat-users.xml" />
<!-- A couple of custom resources here -->
</GlobalNamingResources>
<Service name="Catalina">
<Connector port="80" protocol="HTTP/1.1"
asyncTimeout="20000"
URIEncoding="utf-8"
acceptorThreadCount="2"
compressableMimeType="text/html,text/xml,text/plain,text/x-json,application/javascript,application/json,text/css"
compression="on"
connectionTimeout="20000"
maxThreads="1024"
processorCache="512"
redirectPort="8443" />
<Engine name="Catalina" defaultHost="localhost">
<Realm className="org.apache.catalina.realm.LockOutRealm">
<Realm className="org.apache.catalina.realm.UserDatabaseRealm"
resourceName="UserDatabase"/>
</Realm>
<Host name="localhost" appBase="webapps"
unpackWARs="true" autoDeploy="true">
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
prefix="localhost_access_log." suffix=".txt"
pattern="%h %l %u %t "%r" %s %b %Dms" resolveHosts="false"/>
</Host>
</Engine>
</Service>
</Server>
Best regards,
Kirill
>
> 3.) See troubleshooting steps above.
>
> Dan
>
>>
>>
>> Thanks a lot in advance,
>> Kirill
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Webapp reload corrupts Tomcat
Posted by Daniel Mikusa <dm...@vmware.com>.
On Feb 22, 2013, at 12:09 AM, Kirill Ilyukhin wrote:
> Hi!
>
> I have 5 servers with the following configuration:
> Windows Server 2008 R2 64-bit,
> Oracle JDK 1.7.0_11-b21 64-bit,
> Tomcat 7.0.35 running as a service.
> A few webapps are running perfectly well under the Tomcat until one of
> them is reloaded.
>
> Reload of a webapp (either by pressing [reload] on manager page or by
> changing web.xml file) makes Tomcat extremely unstable.
> The first (pre-production) server sporadically fails with absolutely
> no traces in log files or Windows event logs:
These three symptoms can often mean different things. For example, if it's not accepting connections then you can't really get a 404 error. Can you be more specific about when you see them and the status of your Tomcat server when you see them.
Some troubleshooting thoughts...
> does not accept connections,
1.) Is the process still running? Did the JVM crash? Is there a crash file? hs_err_podXXXX.log?
2.) Is the socket still listening? Have you changed any firewall rules?
3.) Try "telnet <port>". What happens?
4.) Try connecting to Tomcat with jconsole or jvisualvm? Can you? If so look at the connector mbean. What is its state?
5.) Take some thread dumps.
> responds with HTTP 404 or
Assuming the URL that you entered is valid, this sounds like your application may have failed to reload. Can you turn up the log level in your application to see what it is doing?
> does not respond at all.
Assuming that you mean it accepts your connection and request, but never processes and returns a response. In this case, take some thread dumps after you have sent the response. That should show you what is going on inside the JVM.
> Other 4 (production) servers also become unstable but in a different
> manner - a lot of HTTP 500 responses (logs are also clean).
It would be very odd to see nothing in the logs after a 500 error. Have you looked at all of the log files, in particular "localhost-YYYY-MM-DD.log"? That file should contain a stack trace after a 500 error. Also, what do the see in your browser when you get the 500 error? The default error page should print the stack trace for the 500 error.
> Plus on
> some of these servers I have a lot (less than HTTP 500 responses
> though) of NPEs:
> -------------
> SEVERE: Exception Processing /path/here
> java.lang.NullPointerException
> at org.apache.catalina.core.StandardWrapper.servletSecurityAnnotationScan(StandardWrapper.java:1216)
> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:461)
> at org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:305)
> at org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:83)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
> at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
> at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
> at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:1822)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> -------------
> These exception occurs for any webapp, not only the one being reloaded.
What's the specific version of Tomcat where you saw this error occur? Is it 7.0.37? or was this on an older version?
>
> All the above happens irregularly, quite often on the pre-production
> server, only once on production servers. Tomcat service restart fixes
> the problem. Updating Tomcat to 7.0.37 did not help.
> I cannot reproduce this on any of developers hosts with the same
> versions of JDK and Tomcat.
>
> Any ideas?
1.) An easy temporary fix would be to not reload the application in production. If you need to redeploy or reload the application, stop the server, undeploy, redeploy and start your server.
2.) Include your Tomcat configuration, minus comments.
3.) See troubleshooting steps above.
Dan
>
>
> Thanks a lot in advance,
> Kirill
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org