CXF app running on Tomee gets numerous "GC overhead limit exceeded" errors

["KARR, DAVID" <dk...@att.com>]

I have a CXF app running in TomEE on a Win7 service.  It was running fine for several days.  I don't know if my new issues have to do with CXF, TomEE, Windows, or multiple of those elements.  I'm writing a similar note on the TomEE users list.

Today we saw some issues with the box, so we restarted the box, but we weren't getting a response from the service, so I checked the Catalina log, and we saw numerous errors like this:
---------------------
org.apache.cxf.interceptor.Fault: GC overhead limit exceeded
	at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.handleWriteException(JAXRSOutInterceptor.java:391)
	at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.serializeMessage(JAXRSOutInterceptor.java:266)
	at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.processResponse(JAXRSOutInterceptor.java:120)
	at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.handleMessage(JAXRSOutInterceptor.java:83)
	at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
	at org.apache.cxf.interceptor.OutgoingChainInterceptor.handleMessage(OutgoingChainInterceptor.java:83)
	at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
	at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
	at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:254)
	at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234)
	at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208)
	at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160)
	at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:180)
	at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:299)
	at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:223)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
	at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:274)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
	at org.apache.logging.log4j.web.Log4jServletFilter.doFilter(Log4jServletFilter.java:71)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
--------------------------

I'm not sure what to do about this one.

I tried just stopping the service, but somehow it "failed to stop".  It's now in a "Stopping" state that I don't appear to be able to change.  I tried opening up SysInternals ProcessExplorer to see if I could manually stop the TomEE/Tomcat process, but I can't even find it.  There must be some other wrapper that is somehow failing to communicate.

RE: CXF app running on Tomee gets numerous "GC overhead limit exceeded" errors

["KARR, DAVID" <dk...@att.com>]

> -----Original Message-----
> From: KARR, DAVID
> Sent: Friday, September 09, 2016 11:03 AM
> To: cxf-users@apache.org
> Subject: CXF app running on Tomee gets numerous "GC overhead limit
> exceeded" errors
> 
> I have a CXF app running in TomEE on a Win7 service.  It was running
> fine for several days.  I don't know if my new issues have to do with
> CXF, TomEE, Windows, or multiple of those elements.  I'm writing a
> similar note on the TomEE users list.
> 
> Today we saw some issues with the box, so we restarted the box, but we
> weren't getting a response from the service, so I checked the Catalina
> log, and we saw numerous errors like this:
> ---------------------
> org.apache.cxf.interceptor.Fault: GC overhead limit exceeded

This could have been caused by the fact that when we were running it today, the data it was building was more than 20 times larger than when I was testing it before.  That was simply because of the nature of the data.  I've since restarted the service with 2.5 times the maxmem value, so perhaps it will stand up better now.  Figuring out how to make it scale more gracefully is a harder problem.