You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by John Smith <to...@gmail.com> on 2014/08/22 17:29:02 UTC
Daily catalina.out has thousands of NPEs
TC 7.0.54, RHEL 6, JDK 1.7.0_60. I have two RH servers with one instance of
TC on each, set up for clustering. There's a HWLB routing 80 to 8080 with 5
min sticky sessions.
My daily catalina logs are set up to just show SEVERE errors, every day
they average about *30mb* per server with the same set of errors, over and
over:
Aug 21, 2014 12:00:04 AM org.apache.catalina.core.StandardWrapperValve
invoke
SEVERE: Servlet.service() for servlet [GetLevelServlet] in context with
path [] threw exception
java.lang.NullPointerException
Aug 21, 2014 12:00:04 AM org.apache.coyote.http11.AbstractHttp11Processor
process
SEVERE: Error processing request
java.lang.NullPointerException
Aug 21, 2014 12:00:04 AM org.apache.coyote.http11.AbstractHttp11Processor
endRequest
SEVERE: Error finishing response
java.lang.NullPointerException
Aug 21, 2014 12:00:13 AM org.apache.catalina.core.StandardWrapperValve
invoke
SEVERE: Servlet.service() for servlet [GetLevelServlet] in context with
path [] threw exception
java.lang.NullPointerException
Aug 21, 2014 12:00:13 AM org.apache.coyote.http11.AbstractHttp11Processor
process
SEVERE: Error processing request
java.lang.NullPointerException
Aug 21, 2014 12:00:13 AM org.apache.coyote.http11.AbstractHttp11Processor
endRequest
SEVERE: Error finishing response
java.lang.NullPointerException
GetLevelServlet is the workhorse of the site, providing XML to a SWF
through a simple method:
protected void writeXML(HttpServletResponse res, String xml) throws
IOException {
if (xml == null || xml.isEmpty()) return;
res.setContentType("text/xml");
PrintWriter out = res.getWriter();
out.write(xml);
out.close();
}
About 99% of the site usage is through that servlet. I'm catching all 500
errors and logging them with Logback to a different log file, and I'm not
seeing the NPE's generated on the application level (i.e., com.mysite.*)
I'm using the NIO connector.
<!-- Using the non blocking NIO connector -->
<Connector port="8080"
protocol="org.apache.coyote.http11.Http11NioProtocol"
connectionTimeout="20000"
redirectPort="443"
maxConnections="7500"
maxThreads="400"
/>
There's a sizable download (8mg) for the SWF (it's a game site). So
useSendFile being on (the default) works incredibly well. The errors in
catalina aren't presenting on the site as much as they are in the logs,
that is, the site seems to be behaving reasonably well despite the enormous
amount of errors in the logs.
Anyone know what might be causing it?
Best,
John
Re: Daily catalina.out has thousands of NPEs
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
John,
On 8/27/14, 9:29 AM, John Smith wrote:
>>
>>
>>
>> You haven't really provided any information, here. Please post
>> full stack traces.
>>
>> You might want to do something like catch NPE around the area(s)
>> where they are being thrown, and log information about the
>> requests that are causing them. It may be something like a web
>> search crawler requesting the XML with parameters you don't
>> expect, etc.
>>
>>
>>
> Chris,
>
> Sorry for the delayed reply. The thing is, there are no stack
> traces in the catalina logs. Just the report of the NPE's as cut
> and pasted above, over and over again. So I can't "get to" where
> the NPE is even being thrown. Again, the errors are not generated
> at the application level code, or at least they're not 500 errors
> or they would be written to the logback logs, instead of
> catalina.out.
>
> I have an a really high amount of traffic (500,000 visits a day)
> so detailed catalina logs become so massive they become untenable.
> I thinned out logging.properties to simply:
>
> ___________________ handlers =
> 1catalina.org.apache.juli.FileHandler
>
> 1catalina.org.apache.juli.FileHandler.level = SEVERE
> 1catalina.org.apache.juli.FileHandler.directory =
> ${catalina.base}/logs 1catalina.org.apache.juli.FileHandler.prefix
> = catalina. ___________________
>
> Do you know if this configuration removing the full trace?
That configuration does not affect catalina.out, but
catalina.[date].log instead. But if you've set your level to SEVERE
you might not get any of this logged.
Since you can see the component which throws the exception
(StandardWrapperValve), you can change the log level for just that class:
org.apache.catalina.core.StandardWrapperValve.level=INFO
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
iQIcBAEBCAAGBQJT/zTRAAoJEBzwKT+lPKRYr/gQAIZ9w7bxsIc5/IEOm0ZkyK6G
9vbsjuD9zJPZ9FzdOwUvCJ+DFY6d7NfKbGddUB4lyCgmFM/CTadjRtfEVcz17g37
YfV+EPYCZ7OCQM7TI4y61MPkDuo5qiVTc2IdO//dyYlCSOV8xpQH1BSoqzdPvJQt
iRqEVXuvU0utf/ZwiGCid7hAc8gpcyPJX1OhvYXHJgvNiyM654BcY/VjDosL4v+e
N09kDcCLOqViphQaUoNCU8JDBQT8E7UXzMNWDCHUmvAjHNbxkiDKiS2zbHUjxqxI
lwfw5JjJoVMXc5yKMNIoK/1jDmqJoOA5tciSuAw2MZHuu19hXkOjO0W3xd43SW1C
anf6QWus9qEwxs/TokadCefKpdQ/9ohrwH/ecKBMxnKQhXZTiOYALMFLeZy7RiiY
X5+a3SgXSKYjQxHFtUk81lwH5TTv0syhwu6JEqRIHl9csb0AuBA1ORXrSdnjqtd6
alYdst53aN5UE21UMCd7RQ5vzSEFkLCRyO37l0ZlTdZFNWTMjnaQBg15Lm5IBp+8
NqKRFzEcWU95U9YuRSYCMIvWgBKOAQYxwbQ57C9Oq1lclDXcvZWQf+ZZ4JXb5axR
Oxo4Qz2U0xkkDEd2t6Y0lEtCvRH6SlJuR0MGCDRfpAwnldVCKOjjgWMdqxkSWfMo
FEGllQEnJb9g0RPiWQqY
=i7p1
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Daily catalina.out has thousands of NPEs
Posted by Konstantin Kolinko <kn...@gmail.com>.
2014-08-27 17:29 GMT+04:00 John Smith <to...@gmail.com>:
>>
>>
>>
>> You haven't really provided any information, here. Please post full
>> stack traces.
>>
>> You might want to do something like catch NPE around the area(s) where
>> they are being thrown, and log information about the requests that are
>> causing them. It may be something like a web search crawler requesting
>> the XML with parameters you don't expect, etc.
>>
>>
>>
> Chris,
>
> Sorry for the delayed reply. The thing is, there are no stack traces in the
> catalina logs. Just the report of the NPE's as cut and pasted above, over
> and over again. So I can't "get to" where the NPE is even being thrown.
> Again, the errors are not generated at the application level code, or at
> least they're not 500 errors or they would be written to the logback logs,
> instead of catalina.out.
>
> I have an a really high amount of traffic (500,000 visits a day) so
> detailed catalina logs become so massive they become untenable. I thinned
> out logging.properties to simply:
>
> ___________________
> handlers = 1catalina.org.apache.juli.FileHandler
>
> 1catalina.org.apache.juli.FileHandler.level = SEVERE
> 1catalina.org.apache.juli.FileHandler.directory = ${catalina.base}/logs
> 1catalina.org.apache.juli.FileHandler.prefix = catalina.
> ___________________
>
> Do you know if this configuration removing the full trace?
I think I saw several times that the JVM itself may omit generating
stacktraces if the same exception is being raised too often. I think
it should have printed the full stacktrace for several first
occurrences of this error.
There is a way to match log messages to access log entries by thread
name. The key is to
- Configure a FileHandler to use Tomcat's OneLineFormatter
1catalina.org.apache.juli.FileHandler.formatter =
org.apache.juli.OneLineFormatter
- Configure AccessLogValve to log thread name with "%I"
> 1catalina.org.apache.juli.FileHandler.level = SEVERE
Note that the above setting filters what log messages are being written out.
If you want to filter what log messages are being generated (thus
saving some time on processing them), you can set level for a logger,
e.g.
".level = SEVERE" for the default log level.
Best regards,
Konstantin Kolinko
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Daily catalina.out has thousands of NPEs
Posted by John Smith <to...@gmail.com>.
>
>
>
> You haven't really provided any information, here. Please post full
> stack traces.
>
> You might want to do something like catch NPE around the area(s) where
> they are being thrown, and log information about the requests that are
> causing them. It may be something like a web search crawler requesting
> the XML with parameters you don't expect, etc.
>
>
>
Chris,
Sorry for the delayed reply. The thing is, there are no stack traces in the
catalina logs. Just the report of the NPE's as cut and pasted above, over
and over again. So I can't "get to" where the NPE is even being thrown.
Again, the errors are not generated at the application level code, or at
least they're not 500 errors or they would be written to the logback logs,
instead of catalina.out.
I have an a really high amount of traffic (500,000 visits a day) so
detailed catalina logs become so massive they become untenable. I thinned
out logging.properties to simply:
___________________
handlers = 1catalina.org.apache.juli.FileHandler
1catalina.org.apache.juli.FileHandler.level = SEVERE
1catalina.org.apache.juli.FileHandler.directory = ${catalina.base}/logs
1catalina.org.apache.juli.FileHandler.prefix = catalina.
___________________
Do you know if this configuration removing the full trace?
Best,
John
Re: Daily catalina.out has thousands of NPEs
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
John,
On 8/22/14, 11:29 AM, John Smith wrote:
> TC 7.0.54, RHEL 6, JDK 1.7.0_60. I have two RH servers with one
> instance of TC on each, set up for clustering. There's a HWLB
> routing 80 to 8080 with 5 min sticky sessions.
>
> My daily catalina logs are set up to just show SEVERE errors, every
> day they average about *30mb* per server with the same set of
> errors, over and over:
>
>
> Aug 21, 2014 12:00:04 AM
> org.apache.catalina.core.StandardWrapperValve invoke SEVERE:
> Servlet.service() for servlet [GetLevelServlet] in context with
> path [] threw exception java.lang.NullPointerException
Stack trace(s)?
> Aug 21, 2014 12:00:04 AM
> org.apache.coyote.http11.AbstractHttp11Processor process SEVERE:
> Error processing request java.lang.NullPointerException
>
> Aug 21, 2014 12:00:04 AM
> org.apache.coyote.http11.AbstractHttp11Processor endRequest SEVERE:
> Error finishing response java.lang.NullPointerException
>
> Aug 21, 2014 12:00:13 AM
> org.apache.catalina.core.StandardWrapperValve invoke SEVERE:
> Servlet.service() for servlet [GetLevelServlet] in context with
> path [] threw exception java.lang.NullPointerException
>
> Aug 21, 2014 12:00:13 AM
> org.apache.coyote.http11.AbstractHttp11Processor process SEVERE:
> Error processing request java.lang.NullPointerException
>
> Aug 21, 2014 12:00:13 AM
> org.apache.coyote.http11.AbstractHttp11Processor endRequest SEVERE:
> Error finishing response java.lang.NullPointerException
Stack trace(s)?
These look like at least 2 different NPEs. Stack traces would
certainly help.
> GetLevelServlet is the workhorse of the site, providing XML to a
> SWF through a simple method:
>
>
> protected void writeXML(HttpServletResponse res, String xml)
> throws IOException {
>
> if (xml == null || xml.isEmpty()) return;
>
> res.setContentType("text/xml"); PrintWriter out = res.getWriter();
> out.write(xml); out.close(); }
>
>
>
> About 99% of the site usage is through that servlet. I'm catching
> all 500 errors and logging them with Logback to a different log
> file, and I'm not seeing the NPE's generated on the application
> level (i.e., com.mysite.*)
>
> I'm using the NIO connector.
>
>
> <!-- Using the non blocking NIO connector --> <Connector
> port="8080" protocol="org.apache.coyote.http11.Http11NioProtocol"
> connectionTimeout="20000" redirectPort="443" maxConnections="7500"
> maxThreads="400" />
>
>
> There's a sizable download (8mg) for the SWF (it's a game site).
> So useSendFile being on (the default) works incredibly well. The
> errors in catalina aren't presenting on the site as much as they
> are in the logs, that is, the site seems to be behaving reasonably
> well despite the enormous amount of errors in the logs.
>
> Anyone know what might be causing it?
You haven't really provided any information, here. Please post full
stack traces.
You might want to do something like catch NPE around the area(s) where
they are being thrown, and log information about the requests that are
causing them. It may be something like a web search crawler requesting
the XML with parameters you don't expect, etc.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
iQIcBAEBCAAGBQJT965sAAoJEBzwKT+lPKRYaiQP/1ZmHnP2Zn1DJ2T6Ha9axKbD
24hyv2dtBr5rxeVgN2/JF4nH7q7FL4AgfC3EtgofLMxrbWG64HXTcPWhZDeS38zI
usHU7LvZOldgEAzMTBTFL+aSjrSxuXf68pU2nWivhjJ9cksRaeeEMcaFobsw7kFY
nQMDtajUg4zuqz6io8oc66E1CKi/1RHjixLpOmrCCBTolGZdoGLTgDI8UhqsyAoa
xSlOvswcUizrrvyj91oQqv8XL6OKr3KDS73NbMLttu0RHyvY1NP7mR9TKu4dDyFb
JcWBjV7vaV2/Rv9WSN2e5sXrcgI5+8K6cw7xwovG6Sd7w7HOgeq/r42wfAQQ6lsm
i61Z60IziMaT1jZkWp2zCPs6Di5DFQDXq4x03WopgpV8Lv7J863T5KrjofctDjuj
ZCvvRsELcnKyc27rtODdA5MdmzrjiRtpFXQYBy5oDVgUonz5VnriOYKqA5XAiuoN
U/lqWAHsPfLw0iknZ8mTk88wYsRifDkEYAgFqS5J4UhphLMU72hUMeGbms3LYi37
I+xshgL0CXRju2c+C1EY2ApMYNQOs1cof5nfEDxxwOLaLIshyV9ONUFRuT2RXJ/2
J+aVHRWwH70OvnwUfONax+D/4gvrLLEXx5ZfXpNRIlVmgjIpb1xj5TL9y0Z1O7ea
NJySyyaOV3PNkWUVcMXv
=95HE
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org