Ilegal state excepton and parallel deployment

[Усманов Азат Анварович <us...@ieml.ru>]

Ĺ
Hello everyone!I have only recently discovered a wonderful posibility of parallel deployment in tomcat 7 and it works great for jsp and most of the servlets . however there is a small problem .Amongs regular jsp and servlets we have around 300 different reports developed with BIRT designer and run under birt0runtime in Tomcat (some are more heavier in terms of memory consumtions than others)Problem is that after I parallel deploy a webapp some of yhese reports refuse to run throwing the following stack trace in the log




дек 13, 2019 8:44:26 PM org.apache.catalina.loader.WebappClassLoaderBase findResourceInternal
INFO: Illegal access: this web application instance has been stopped already. Could not load META-INF/services/org.apache.xerces.xni.parser.XMLParserConfiguration. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
Birt engine error. Path: /opt/tomcat/webapps/Education##12846/birt/TeacherPlan.rptdesign, format: html, ErrorMessage:Error happened while running the report.
org.eclipse.birt.report.engine.api.EngineException: Error happened while running the report.
at org.eclipse.birt.report.engine.api.impl.EngineTask.handleFatalExceptions(EngineTask.java:2248)
at org.eclipse.birt.report.engine.api.impl.RunAndRenderTask.doRun(RunAndRenderTask.java:190)
at org.eclipse.birt.report.engine.api.impl.RunAndRenderTask.run(RunAndRenderTask.java:77)
at com.ieml.basedata.BirtService.runReport(BirtService.java:146)
at com.ieml.basedata.BIRTReportServlet.doPost(BIRTReportServlet.java:85)
at com.ieml.basedata.BIRTReportServlet.doGet(BIRTReportServlet.java:19)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:624)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.filters.ExpiresFilter.doFilter(ExpiresFilter.java:1203)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.planetj.servlet.filter.compression.CompressingFilter.doFilter(CompressingFilter.java:222)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.ieml.filters.LogFilter.doFilter(LogFilter.java:33)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.ieml.filters.ProtectFilter.doFilter(ProtectFilter.java:92)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.ieml.filters.saml.SAMLSPFilter.doFilter(SAMLSPFilter.java:169)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.ieml.filters.CharsetFilter.doFilter(CharsetFilter.java:51)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:126)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.ieml.filters.HeadersFilter.doFilter(HeadersFilter.java:19)
at

Азат Усманов, сегодня в 0:29
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:603)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1137)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(AprEndpoint.java:2574)
at  org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:2563)
and this dissapears only after tomcat restart which in terms kinda defeats the whole benefit of parallel deployment(which I used to minimize number of TC restarts )


I do have undeployOldversions set to true in server.xml on the Host
<Host undeployOldVersions="true" startStopThreads="0" name="localhost" appBase="webapps"
unpackWARs="true" autoDeploy="true" deployXML="false"> )

here is what the webapps folder looks like at the moment of the error

ll -h /opt/tomcat/webapps/
drwxrwxr-x. 14 idis idis 4,0K Дек 13 14:53 Education##12843
-rw-r--r--. 1 root root 17M Дек 13 14:53 Education##12843.war
drwxrwxr-x. 14 idis idis 4,0K Дек 13 19:11 Education##12846
-rw-r--r--. 1 root root 17M Дек 13 19:11 Education##12846.war
drwxrwxr-x. 13 idis idis 4,0K Дек 13 19:11 Professional##12846
-rw-r--r--. 1 root root 16M Дек 13 19:11 Professional##12846.war
drwxr-xr-x. 3 idis idis 4,0K Фев 19 2019 ROOT
drwxr-xr-x. 6 idis idis 4,0K Ноя 19 2018 yui


googling the issue I thought it might be permgen related so I increased MaxPem size to 1536m from original 768m .total server ram is 8GB


I also have specified org.apache.catalina.connector.RECYCLE_FACADES=true inside of catalina.properties file to no avail
I am also a bit confused why the error happens when accessing the newest version of the app (##12846) and not the oldest one
Any suggestions one how to avoid the error without tomcat restart ?

Tomcat version 7.0.92
OS RHEL 6.12

JVM startup options(catalina.sh)

export JRE_HOME=/usr/java/jdk1.7.0_67/jre
export CATALINA_OPTS="-Dorg.apache.juli.logging.UserDataHelper.CONFIG=DEBUG_ALL -Djava.awt.headless=true -Xmx2048m -XX:MaxPermSize=1536m -XX:+HeapDumpOnOutOfMemoryError"
jVM vendor oracle

java version "1.7.0_67"
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)

С уважением,
Азат Усманов