You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Rainer Jung <ra...@kippdata.de> on 2008/08/01 10:24:31 UTC
Re: Apache/mod_jk serves random files from tomcat
dave.smith wrote:
> As I mentioned upgrading to mod_jk 1.2.26 was very easy. Unfortunately,
> Tomcat is now crashing with "An unexpected error has been detected by
> HotSpot Virtual Machine."
>
> # SIGSEGV (0xb) at pc=0xb7aeaf7b, pid=19887, tid=2991246224
> #
> # Java VM: Java HotSpot(TM) Client VM (1.5.0_12-b04 mixed mode, sharing)
> # Problematic frame:
> # V [libjvm.so+0x2ccf7b]
> #
>
> There's a lot more. Also, here are the jvm args:
> jvm_args: -Djava.library.path=/usr/local/apr/lib -Xms512m -Xmx512m
> -XX:PermSize=256m -XX:MaxPermSize=256m
> -Djava.util.logging.manager=org.apache.juli.ClassLo
> aderLogManager
> -Djava.util.logging.config.file=/usr/share/tomcat5/conf/logging.properties
> -Djava.endorsed.dirs=/usr/share/tomcat5/common/endorsed -Dcatalina.
> base=/usr/share/tomcat5 -Dcatalina.home=/usr/share/tomcat5
> -Djava.io.tmpdir=/usr/share/tomcat5/temp
> java_command: org.apache.catalina.startup.Bootstrap start
>
> Do you believe these are related to the mod_jk upgrade? It's happened twice
> (once on each server) since the upgrade that was made 2 weeks ago.
>
> Please, let me know if you need anymore information.
>
> Thanks,
> Dave
I don't know of any similar case. The full hot spot error file would be
useful though. It e.g. contains a stack of the thread were the crash
happened.
Regards,
Rainer
> Rainer Jung-3 wrote:
>> dave.smith schrieb:
>>> Yesterday, I upgraded our dev environment to mod_jk 1.2.26, which
>>> couldn't
>>> have been easier. It will probably take me a couple of days before I can
>>> get this done in production, though.
>>>
>>> I terminate all HTTPS requests before they get to the web server, so from
>>> what you have described, it is probably safe to disable the APR
>>> connector.
>>> How do I disable it, though? I will look into disabling this after I
>>> have
>>> updated mod_jk in production.
>> Locate the tcnative shared object file (tcnative.so or tcnative-1.so)
>> and renme it, so that the linker loader does not find it (e.g. add an
>> underscore at the end of the file name).
>>
>> Then during the next startup, Tomcat should emit an info level log
>> message telling you, that it couldn't find the lib.
>>
>>> Here's the full stack trace for that exception, displayed in my Tomcat
>>> logs:
>>>
>>> Jul 10, 2008 10:06:50 PM org.apache.catalina.connector.Request
>>> parseParameters
>>> WARNING: Exception thrown whilst processing POSTed parameters
>>> java.io.IOException: Socket read failed
>>> at
>>> org.apache.coyote.ajp.AjpAprProcessor.read(AjpAprProcessor.java:1037)
>>> at
>>> org.apache.coyote.ajp.AjpAprProcessor.readMessage(AjpAprProcessor.java:1158)
>>> at
>>> org.apache.coyote.ajp.AjpAprProcessor.receive(AjpAprProcessor.java:1090)
>>> at
>>> org.apache.coyote.ajp.AjpAprProcessor$SocketInputBuffer.doRead(AjpAprProcessor.java:1228)
>>> at org.apache.coyote.Request.doRead(Request.java:419)
>>> at
>>> org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:265)
>>> at
>>> org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:403)
>>> at
>>> org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:280)
>>> at
>>> org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:193)
>>> at
>>> org.apache.catalina.connector.Request.readPostBody(Request.java:2400)
>>> at
>>> org.apache.catalina.connector.Request.parseParameters(Request.java:2379)
>>> at
>>> org.apache.catalina.connector.Request.getParameterNames(Request.java:1047)
>>> at
>>> org.apache.catalina.connector.RequestFacade.getParameterNames(RequestFacade.java:369)
>>> at
>>> org.apache.struts.util.RequestUtils.populate(RequestUtils.java:1225)
>>> at
>>> org.apache.struts.action.RequestProcessor.processPopulate(RequestProcessor.java:821)
>>> at
>>> org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:254)
>>> at
>>> org.apache.struts.action.ActionServlet.process(ActionServlet.java:1482)
>>> at
>>> org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:525)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
>>> at
>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
>>> at
>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)
>>> at
>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>>> at
>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
>>> at
>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
>>> at
>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
>>> at
>>> org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:444)
>>> at
>>> org.apache.coyote.ajp.AjpAprProtocol$AjpConnectionHandler.process(AjpAprProtocol.java:472)
>>> at
>>> org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1286)
>>> at java.lang.Thread.run(Thread.java:595)
>> OK, thanks. You are sure it is not 5.5.26?
>>
>> Regards,
>>
>> Rainer
>>
>>> Rainer Jung-3 wrote:
>>>> Hi David,
>>>>
>>>> dave.smith schrieb:
>>>>> Hi Rainer,
>>>>>
>>>>> Thanks a lot for the reply.
>>>>>
>>>>> I am using Tomcat 5.5.25 (rpm from jpackage.org). CentOS Linux 2.6.18.
>>>>>
>>>>> httpd was compiled in prefork mode. The prefork settings are:
>>>>>
>>>>> StartServers 8
>>>>> MinSpareServers 5
>>>>> MaxSpareServers 20
>>>>> ServerLimit 256
>>>>> MaxClients 256
>>>>> MaxRequestsPerChild 4000
>>>>>
>>>>> I have setup JMeter to run against a test environment, but was unable
>>>>> to
>>>>> reproduce. These random responses occur in production about once every
>>>>> week
>>>>> or so/more. The problem will often (temporarily) correct itself, but
>>>>> sometimes I will need to restart httpd if the problem persists --
>>>>> restarting
>>>>> tomcat also works to temporarily correct the problem.
>>>>>
>>>>> The only thing strange that I see in my logs are in the
>>>>> test_client.log:
>>>>>
>>>>> WARNING: Exception thrown whilst processing POSTed parameters
>>>>> java.io.IOException: Socket read failed
>>>>> at
>>>>> org.apache.coyote.ajp.AjpAprProcessor.read(AjpAprProcessor.java:1037)
>>>>> ...
>>>> Thanks for the information. What is "test_client.log"? It looks like a
>>>> Tomcat log file? Could you also post a larger part of the stack, or do
>>>> you only get one line?
>>>>
>>>> Would you be able to do the following two things, maybe not both at the
>>>> same time:
>>>>
>>>> - disable the apr connector (tcnative.so)
>>>> - upgrade jk to 1.2.26
>>>>
>>>> Concerning the apr connector: If you are using OpenSSL with apr and
>>>> Tomcat or you have some similar reason you really need it, then don't
>>>> switch. But if you use it without a very specific reason, disabling it
>>>> for a week or two would help us isolate the problem.
>>>>
>>>> Concerning mod_jk upgrade: That should be very easy, apart from the
>>>> following: if your httpd uses VirtualHost in the configuration, you have
>>>> to include your JkMount inside the VirtualHost, not in the global part,
>>>> or you add JkMountCopy On to the VirtualHost.
>>>>
>>>> Regards,
>>>>
>>>> Rainer
>>>>
>>>>> Rainer Jung-3 wrote:
>>>>>> dave.smith schrieb:
>>>>>>>> Wow. That's weird. Is Tomcat serving the file, or is httpd serving
>>>>>>>> it?
>>>>>>> Not too weird. I am experiencing the same thing with Tomcat 5.5 and
>>>>>>> mod_jk
>>>>>>> 1.2.23. I have Tomcat serving everything.
>>>>>>>
>>>>>>> I am also using a load balancer that sends an OPTION every 2 seconds
>>>>>>> to
>>>>>>> each
>>>>>>> web server to make sure that the server is alive.
>>>>>>>
>>>>>>> This intermittent random response issue is really killing me.
>>>>>> Could you please also add some info:
>>>>>>
>>>>>> Tomcat version?
>>>>>>
>>>>>> And from my previous mail:
>>>>>>
>>>>>> What's you platform and which httpd MPM (prefork orworker or something
>>>>>> else) do you use? For some platforms (e.g. AIX) the detection of
>>>>>> multi-threading in httpd during mpod_jk build-time was broken.
>>>>>> Starting
>>>>>> with 1.2.24 we build always including multi-thread support unless
>>>>>> explicitely stated via a configure option. If you 1.2.23 build is not
>>>>>> thread safe, but your httpd uses threads (like with worker mpm), then
>>>>>> such trouble is possible, although more likely you would see crashes
>>>>>> etc. For most platforms like Linux and Solaris the threading detection
>>>>>> was OK already before 1.2.24.
>>>>>>
>>>>>> Another possible (but not very likely) cause could be bug 44494 of
>>>>>> Tomcat 6.0.16/5.5.26 which under certain circumstances could leave
>>>>>> data
>>>>>> in the request object after request handling completed. You could try
>>>>>> either downgrading to 6.0.15/5.5.25 or upgrading to the soon to be
>>>>>> expected 6.0.17/5.5.27.
>>>>>>
>>>>>> I would also add the access log on the Tomcat side. If you find the
>>>>>> same
>>>>>> phenomenon there, then it's unlikely, that httpd/mod_jk are
>>>>>> responsible
>>>>>> and the reason should be inside Tomcat or the webapp.
>>>>>>
>>>>>> Can you reproduce the problem on a test system?
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Rainer
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org