You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by James Boggs <jb...@rightdirectiontech.com.INVALID> on 2023/07/11 14:21:29 UTC
Tomcat 9.0.76 Memory leak with Java 17
Hi,
We had a stable SSL enabled website with Apache Tomcat 9.0.73 on Windows Server 2012 o/s, Java 8, Oracle ORDS 21.4 and SSL.
We simultaneously upgraded to Tomcat 9.0.75, upgraded to Java 17 and to ORDS 22.1, then used Java 17 to create a new Java Keystore and a new SSL csr file, and imported a new SSL certificate from the CA into the new keystore.
The website works but after logging in there are memory leak warnings and the Tomcat service crashes within just a couple of minutes.
We even upgraded to 9.0.76 and the issue persists.
Below is an excerpt from the stderr log file.
I have been unable to find any recent threads on this, any help is appreciated. Is any other information needed?
------------------------------------------ start of logfile
2023-07-10 16:51:33 Apache Commons Daemon procrun stderr initialized.
10-Jul-2023 16:51:35.434 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name: Apache Tomcat/9.0.76
10-Jul-2023 16:51:35.449 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built: Jun 5 2023 07:17:04 UTC
10-Jul-2023 16:51:35.451 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version number: 9.0.76.0
10-Jul-2023 16:51:35.451 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name: Windows Server 2012 R2
10-Jul-2023 16:51:35.451 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version: 6.3
10-Jul-2023 16:51:35.451 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture: amd64
10-Jul-2023 16:51:35.452 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home: D:\JAVA\JDK
10-Jul-2023 16:51:35.452 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version: 17.0.7+8-LTS-224
10-Jul-2023 16:51:35.452 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor: Oracle Corporation
10-Jul-2023 16:51:35.452 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE: D:\tomcat9
10-Jul-2023 16:51:35.454 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME: D:\tomcat9
10-Jul-2023 16:51:35.481 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dconfig.url=D:\ords222
10-Jul-2023 16:51:35.481 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xms1024M
10-Jul-2023 16:51:35.481 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xmx1024M
10-Jul-2023 16:51:35.481 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=D:\tomcat9
10-Jul-2023 16:51:35.481 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=D:\tomcat9
10-Jul-2023 16:51:35.481 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dignore.endorsed.dirs=D:\tomcat9\endorsed
10-Jul-2023 16:51:35.481 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=D:\tomcat9\temp
10-Jul-2023 16:51:35.486 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
10-Jul-2023 16:51:35.486 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=D:\tomcat9\conf\logging.properties
10-Jul-2023 16:51:35.486 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.lang=ALL-UNNAMED
10-Jul-2023 16:51:35.486 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.io=ALL-UNNAMED
10-Jul-2023 16:51:35.486 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.util=ALL-UNNAMED
10-Jul-2023 16:51:35.486 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.util.concurrent=ALL-UNNAMED
10-Jul-2023 16:51:35.486 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
10-Jul-2023 16:51:35.486 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: exit
10-Jul-2023 16:51:35.486 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: abort
10-Jul-2023 16:51:35.486 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xms128m
10-Jul-2023 16:51:35.486 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xmx256m
10-Jul-2023 16:51:35.496 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded Apache Tomcat Native library [1.2.37] using APR version [1.7.4].
10-Jul-2023 16:51:35.496 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true], UDS [true].
10-Jul-2023 16:51:35.496 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true]
10-Jul-2023 16:51:35.581 INFO [main] org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL successfully initialized [OpenSSL 1.1.1u 30 May 2023]
10-Jul-2023 16:51:36.476 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["https-openssl-nio-10.2.251.132-443"]
10-Jul-2023 16:51:37.386 INFO [main] org.apache.tomcat.util.net.AbstractEndpoint.logCertificate Connector [https-openssl-nio-10.2.251.132-443], TLS virtual host [_default_], certificate type [UNDEFINED] configured from [D:\JAVA\JRE\bin\rplansstage.keystore] using alias [tomcat] and with trust store [null]
10-Jul-2023 16:51:37.406 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [2624] milliseconds
10-Jul-2023 16:51:37.506 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
10-Jul-2023 16:51:37.511 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/9.0.76]
10-Jul-2023 16:51:37.586 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [D:\tomcat9\webapps\rplans-vpd.war]
10-Jul-2023 16:51:40.432 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
2023-07-10T20:51:57.839Z INFO Configuration properties for: |default|lo|
db.servicename=prplans.c11.ds.altess.army.mil
java.specification.version=17
conf.use.wallet=true
sun.cpu.isalist=amd64
sun.jnu.encoding=Cp1252
java.class.path=D:\tomcat9\bin\bootstrap.jar;D:\tomcat9\bin\tomcat-juli.jar
java.vm.vendor=Oracle Corporation
sun.arch.data.model=64
user.variant=
catalina.useNaming=true
java.vendor.url=https://java.oracle.com/
resource.templates.enabled=false
user.timezone=America/New_York
db.port=1521
debug.printDebugToScreen=true
org.apache.el.GET_CLASSLOADER_USE_PRIVILEGED=false
java.vm.specification.version=17
os.name=Windows Server 2012 R2
jdbc.MinLimit=15
user.country=US
sun.boot.library.path=D:\JAVA\JDK\bin
jdk.debug=release
sun.cpu.endian=little
user.home=C:\Windows\ServiceProfiles\LocalService
user.language=en
java.specification.vendor=Oracle Corporation
java.naming.factory.url.pkgs=org.apache.naming
misc.defaultPage=apex
java.version.date=2023-04-18
database.api.enabled=false
version=22.2.1.r2021302
java.home=D:\JAVA\JDK
db.username=ORDS_PUBLIC_USER
ignore.endorsed.dirs=D:\tomcat9\endorsed
file.separator=\
java.vm.compressedOopsMode=32-bit
line.separator=
restEnabledSql.active=false
java.specification.name=Java Platform API Specification
java.vm.specification.vendor=Oracle Corporation
jdbc.MaxStatementsLimit=10
feature.sdw=false
package.access=sun.,org.apache.catalina.,org.apache.coyote.,org.apache.jasper.,org.apache.tomcat.
config.url=D:\ords222
package.definition=sun.,java.,org.apache.catalina.,org.apache.coyote.,org.apache.jasper.,org.apache.naming.,org.apache.tomcat.
db.password=******
db.hostname=HQDAW089AAA4S05.c11.ds.altess.army.mil
server.loader=
user.script=
java.util.logging.config.file=D:\tomcat9\conf\logging.properties
sun.management.compiler=HotSpot 64-Bit Tiered Compilers
java.naming.factory.initial=org.apache.naming.java.javaURLContextFactory
java.runtime.version=17.0.7+8-LTS-224
user.name=LOCAL SERVICE
path.separator=;
common.loader="${catalina.base}/lib","${catalina.base}/lib/*.jar","${catalina.home}/lib","${catalina.home}/lib/*.jar"
os.version=6.3
java.runtime.name=Java(TM) SE Runtime Environment
file.encoding=Cp1252
plsql.gateway.mode=proxied
java.vm.name=Java HotSpot(TM) 64-Bit Server VM
jdbc.DriverType=thin
jdbc.statementTimeout=900
java.vendor.url.bug=https://bugreport.java.com/bugreport/
java.io.tmpdir=D:\tomcat9\temp
tomcat.util.scan.StandardJarScanFilter.jarsToScan=log4j-taglib*.jar,log4j-web*.jar,log4javascript*.jar,slf4j-taglib*.jar
catalina.home=D:\tomcat9
java.version=17.0.7
tomcat.util.scan.StandardJarScanFilter.jarsToSkip=annotations-api.jar,ant-junit*.jar,ant-launcher*.jar,ant*.jar,asm-*.jar,aspectj*.jar,bcel*.jar,biz.aQute.bnd*.jar,bootstrap.jar,catalina-ant.jar,catalina-ha.jar,catalina-ssi.jar,catalina-storeconfig.jar,catalina-tribes.jar,catalina.jar,cglib-*.jar,cobertura-*.jar,commons-beanutils*.jar,commons-codec*.jar,commons-collections*.jar,commons-compress*.jar,commons-daemon.jar,commons-dbcp*.jar,commons-digester*.jar,commons-fileupload*.jar,commons-httpclient*.jar,commons-io*.jar,commons-lang*.jar,commons-logging*.jar,commons-math*.jar,commons-pool*.jar,derby-*.jar,dom4j-*.jar,easymock-*.jar,ecj-*.jar,el-api.jar,geronimo-spec-jaxrpc*.jar,h2*.jar,ha-api-*.jar,hamcrest-*.jar,hibernate*.jar,httpclient*.jar,icu4j-*.jar,jasper-el.jar,jasper.jar,jaspic-api.jar,jaxb-*.jar,jaxen-*.jar,jaxws-rt-*.jar,jdom-*.jar,jetty-*.jar,jmx-tools.jar,jmx.jar,jsp-api.jar,jstl.jar,jta*.jar,junit-*.jar,junit.jar,log4j*.jar,mail*.jar,objenesis-*.jar,oraclepki.jar,org.hamcrest.core_*.jar,org.junit_*.jar,oro-*.jar,servlet-api-*.jar,servlet-api.jar,slf4j*.jar,taglibs-standard-spec-*.jar,tagsoup-*.jar,tomcat-api.jar,tomcat-coyote.jar,tomcat-dbcp.jar,tomcat-i18n-*.jar,tomcat-jdbc.jar,tomcat-jni.jar,tomcat-juli-adapters.jar,tomcat-juli.jar,tomcat-util-scan.jar,tomcat-util.jar,tomcat-websocket.jar,tools.jar,unboundid-ldapsdk-*.jar,websocket-api.jar,wsdl4j*.jar,xercesImpl.jar,xml-apis.jar,xmlParserAPIs-*.jar,xmlParserAPIs.jar,xom-*.jar
user.dir=D:\tomcat9
os.arch=amd64
java.vm.specification.name=Java Virtual Machine Specification
jdbc.InactivityTimeout=1800
jdbc.MaxLimit=100
sun.os.patch.level=
catalina.base=D:\tomcat9
shared.loader=
native.encoding=Cp1252
java.util.logging.manager=org.apache.juli.ClassLoaderLogManager
jdbc.MaxConnectionReuseCount=1000
java.library.path=D:\tomcat9\bin;C:\windows\Sun\Java\bin;C:\windows\system32;C:\windows;D:\JAVA\JDK\bin;C:\Program Files\Common Files\Oracle\Java\javapath;D:\JAVA\JDK\bin;C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\HID Global\ActivClient\;C:\Program Files\HID Global\ActivClient\;C:\Program Files\McAfee\Solidcore\Tools\GatherInfo;C:\Program Files\McAfee\Solidcore\Tools\Scanalyzer;C:\Program Files\McAfee\Solidcore\;C:\Program Files\McAfee\Solidcore\Tools\ScGetCerts;C:\Program Files\Tumbleweed\Desktop Validator\;C:\Program Files\Tumbleweed\Desktop Validator\x86;.
java.vendor=Oracle Corporation
java.vm.info=mixed mode, sharing
java.vm.version=17.0.7+8-LTS-224
sun.io.unicode.encoding=UnicodeLittle
tomcat.util.buf.StringCache.byte.enabled=true
jdbc.InitialLimit=15
db.connectionType=basic
java.class.version=61.0
db.sid=prplans
2023-07-10T20:52:04.298Z INFO Oracle REST Data Services initialized
Oracle REST Data Services version : 22.2.1.r2021302
Oracle REST Data Services server info: Apache Tomcat/9.0.76
Oracle REST Data Services java info: Java HotSpot(TM) 64-Bit Server VM 17.0.7+8-LTS-224
2023-07-10T20:52:04.463Z INFO Deployment of web application archive [D:\tomcat9\webapps\rplans-vpd.war] has finished in [26,876] ms
2023-07-10T20:52:04.466Z INFO Deploying web application directory [D:\tomcat9\webapps\docs]
2023-07-10T20:52:04.584Z INFO Deployment of web application directory [D:\tomcat9\webapps\docs] has finished in [117] ms
2023-07-10T20:52:04.585Z INFO Deploying web application directory [D:\tomcat9\webapps\examples]
2023-07-10T20:52:05.942Z INFO Deployment of web application directory [D:\tomcat9\webapps\examples] has finished in [1,356] ms
2023-07-10T20:52:05.942Z INFO Deploying web application directory [D:\tomcat9\webapps\host-manager]
2023-07-10T20:52:06.056Z INFO Deployment of web application directory [D:\tomcat9\webapps\host-manager] has finished in [113] ms
2023-07-10T20:52:06.057Z INFO Deploying web application directory [D:\tomcat9\webapps\i]
2023-07-10T20:52:06.133Z INFO Deployment of web application directory [D:\tomcat9\webapps\i] has finished in [76] ms
2023-07-10T20:52:06.133Z INFO Deploying web application directory [D:\tomcat9\webapps\manager]
2023-07-10T20:52:06.246Z INFO Deployment of web application directory [D:\tomcat9\webapps\manager] has finished in [112] ms
2023-07-10T20:52:06.247Z INFO Deploying web application directory [D:\tomcat9\webapps\ROOT]
2023-07-10T20:52:06.288Z INFO Deployment of web application directory [D:\tomcat9\webapps\ROOT] has finished in [40] ms
2023-07-10T20:52:06.292Z INFO Starting ProtocolHandler ["https-openssl-nio-10.2.251.132-443"]
2023-07-10T20:52:06.346Z INFO Server startup in [28980] milliseconds
2023-07-10T21:35:40.487Z INFO Pausing ProtocolHandler ["https-openssl-nio-10.2.251.132-443"]
2023-07-10T21:35:40.495Z INFO Stopping service [Catalina]
2023-07-10T21:35:40.910Z INFO Destroyed pool : |default|lo|-2023-07-10T20-51-53.099285500Z
2023-07-10T21:35:40.939Z WARNING The web application [rplans-vpd] registered the JDBC driver [oracle.jdbc.OracleDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
2023-07-10T21:35:40.944Z WARNING The web application [rplans-vpd] appears to have started a thread named [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
2023-07-10T21:35:40.944Z WARNING The web application [rplans-vpd] appears to have started a thread named [UCP Clock] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@17.0.7/java.lang.Thread.sleep(Native<mailto:java.base@17.0.7/java.lang.Thread.sleep(Native> Method)
oracle.ucp.common.Clock.lambda$static$0(Clock.java:91)
oracle.ucp.common.Clock$$Lambda$314/0x0000000801128ae8.run(Unknown Source)
java.base@17.0.7/java.lang.Thread.run(Thread.java:833)<mailto:java.base@17.0.7/java.lang.Thread.run(Thread.java:833)>
2023-07-10T21:35:40.944Z WARNING The web application [rplans-vpd] appears to have started a thread named [oracle.ucp.actors.InterruptableActor-control] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@17.0.7/jdk.internal.misc.Unsafe.park(Native<mailto:java.base@17.0.7/jdk.internal.misc.Unsafe.park(Native> Method)
java.base@17.0.7/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)<mailto:java.base@17.0.7/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)>
java.base@17.0.7/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1757)<mailto:java.base@17.0.7/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1757)>
oracle.ucp.actors.InterruptableActor.lambda$buildControlThread$1(InterruptableActor.java:66)
oracle.ucp.actors.InterruptableActor$$Lambda$323/0x000000080113d1a8.run(Unknown Source)
java.base@17.0.7/java.lang.Thread.run(Thread.java:833)<mailto:java.base@17.0.7/java.lang.Thread.run(Thread.java:833)>
2023-07-10T21:35:40.944Z SEVERE The web application [rplans-vpd] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@427241b4]) and a value of type [oracle.ucp.common.Core.Match] (value [UNKNOWN]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
2023-07-10T21:35:40.949Z SEVERE The web application [rplans-vpd] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@56fb380d]) and a value of type [java.util.LinkedList] (value [[oracle.ucp.common.waitfreepool.LinkedListPool$Element@e8ba2bf]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
2023-07-10T21:35:40.949Z SEVERE The web application [rplans-vpd] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@427241b4]) and a value of type [oracle.ucp.common.Core.Match] (value [UNKNOWN]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
2023-07-10T21:35:40.949Z SEVERE The web application [rplans-vpd] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@56fb380d]) and a value of type [java.util.LinkedList] (value [[oracle.ucp.common.waitfreepool.LinkedListPool$Element@e8ba2bf]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
2023-07-10T21:35:40.949Z SEVERE The web application [rplans-vpd] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@427241b4]) and a value of type [oracle.ucp.common.Core.Match] (value [UNKNOWN]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
2023-07-10T21:35:40.949Z SEVERE The web application [rplans-vpd] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@56fb380d]) and a value of type [java.util.LinkedList] (value [[oracle.ucp.common.waitfreepool.LinkedListPool$Element@e8ba2bf]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
2023-07-10T21:35:40.949Z SEVERE The web application [rplans-vpd] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@427241b4]) and a value of type [oracle.ucp.common.Core.Match] (value [UNKNOWN]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
2023-07-10T21:35:40.949Z SEVERE The web application [rplans-vpd] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@56fb380d]) and a value of type [java.util.LinkedList] (value [[oracle.ucp.common.waitfreepool.LinkedListPool$Element@e8ba2bf]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
2023-07-10T21:35:40.949Z SEVERE The web application [rplans-vpd] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@427241b4]) and a value of type [oracle.ucp.common.Core.Match] (value [UNKNOWN]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
2023-07-10T21:35:40.949Z SEVERE The web application [rplans-vpd] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@56fb380d]) and a value of type [java.util.LinkedList] (value [[oracle.ucp.common.waitfreepool.LinkedListPool$Element@e8ba2bf, oracle.ucp.common.waitfreepool.LinkedListPool$Element@2edff08b<ma...@2edff08b>]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
2023-07-10T21:35:40.949Z SEVERE The web application [rplans-vpd] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@427241b4]) and a value of type [oracle.ucp.common.Core.Match] (value [UNKNOWN]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
2023-07-10T21:35:40.949Z SEVERE The web application [rplans-vpd] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@56fb380d]) and a value of type [java.util.LinkedList] (value [[oracle.ucp.common.waitfreepool.LinkedListPool$Element@e8ba2bf]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
2023-07-10T21:35:40.949Z SEVERE The web application [rplans-vpd] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@427241b4]) and a value of type [oracle.ucp.common.Core.Match] (value [UNKNOWN]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
2023-07-10T21:35:40.949Z SEVERE The web application [rplans-vpd] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@56fb380d]) and a value of type [java.util.LinkedList] (value [[oracle.ucp.common.waitfreepool.LinkedListPool$Element@e8ba2bf]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
2023-07-10T21:35:40.949Z SEVERE The web application [rplans-vpd] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@427241b4]) and a value of type [oracle.ucp.common.Core.Match] (value [UNKNOWN]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
2023-07-10T21:35:40.949Z SEVERE The web application [rplans-vpd] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@56fb380d]) and a value of type [java.util.LinkedList] (value [[oracle.ucp.common.waitfreepool.LinkedListPool$Element@e8ba2bf]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
2023-07-10T21:35:40.949Z SEVERE The web application [rplans-vpd] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@427241b4]) and a value of type [oracle.ucp.common.Core.Match] (value [UNKNOWN]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
2023-07-10T21:35:40.949Z SEVERE The web application [rplans-vpd] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@56fb380d]) and a value of type [java.util.LinkedList] (value [[oracle.ucp.common.waitfreepool.LinkedListPool$Element@e8ba2bf]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
2023-07-10T21:35:40.954Z SEVERE The web application [rplans-vpd] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@427241b4]) and a value of type [oracle.ucp.common.Core.Match] (value [UNKNOWN]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
2023-07-10T21:35:40.954Z SEVERE The web application [rplans-vpd] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@56fb380d]) and a value of type [java.util.LinkedList] (value [[oracle.ucp.common.waitfreepool.LinkedListPool$Element@e8ba2bf]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
2023-07-10T21:35:40.954Z SEVERE The web application [rplans-vpd] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@427241b4]) and a value of type [oracle.ucp.common.Core.Match] (value [UNKNOWN]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
2023-07-10T21:35:40.954Z SEVERE The web application [rplans-vpd] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@56fb380d]) and a value of type [java.util.LinkedList] (value [[oracle.ucp.common.waitfreepool.LinkedListPool$Element@e8ba2bf]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
2023-07-10T21:35:40.959Z SEVERE The web application [rplans-vpd] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@427241b4]) and a value of type [oracle.ucp.common.Core.Match] (value [UNKNOWN]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
2023-07-10T21:35:40.959Z SEVERE The web application [rplans-vpd] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@56fb380d]) and a value of type [java.util.LinkedList] (value [[oracle.ucp.common.waitfreepool.LinkedListPool$Element@e8ba2bf]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
2023-07-10T21:35:40.959Z SEVERE The web application [rplans-vpd] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@427241b4]) and a value of type [oracle.ucp.common.Core.Match] (value [UNKNOWN]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
2023-07-10T21:35:40.964Z SEVERE The web application [rplans-vpd] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@56fb380d]) and a value of type [java.util.LinkedList] (value [[oracle.ucp.common.waitfreepool.LinkedListPool$Element@e8ba2bf]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
2023-07-10T21:35:40.964Z SEVERE The web application [rplans-vpd] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@427241b4]) and a value of type [oracle.ucp.common.Core.Match] (value [UNKNOWN]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
2023-07-10T21:35:40.964Z SEVERE The web application [rplans-vpd] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@56fb380d]) and a value of type [java.util.LinkedList] (value [[oracle.ucp.common.waitfreepool.LinkedListPool$Element@e8ba2bf]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
2023-07-10T21:35:40.964Z SEVERE The web application [rplans-vpd] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@427241b4]) and a value of type [oracle.ucp.common.Core.Match] (value [UNKNOWN]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
2023-07-10T21:35:40.964Z SEVERE The web application [rplans-vpd] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@56fb380d]) and a value of type [java.util.LinkedList] (value [[oracle.ucp.common.waitfreepool.LinkedListPool$Element@e8ba2bf]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
2023-07-10T21:35:40.964Z SEVERE The web application [rplans-vpd] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@427241b4]) and a value of type [oracle.ucp.common.Core.Match] (value [UNKNOWN]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
2023-07-10T21:35:40.989Z INFO Stopping ProtocolHandler ["https-openssl-nio-10.2.251.132-443"]
2023-07-10T21:35:41.009Z INFO Destroying ProtocolHandler ["https-openssl-nio-10.2.251.132-443"]
------------------------------------------ end of logfile
Regards,
James Boggs | Senior DBA/SA | Mobile: 571-337-0535
"Trust, Integrity, Loyalty to Our Customers, Employees and Partner"
VA Verified (SDVOSB) | SBA Certified 8(a) | SB | SDB | MBE/DBE (MD) | SWaM (VA)
ISO 9001:2015|ISO/IEC 20000-1:2018|ISO/IEC 27001:2013|
CMMI-DEV Level 3 Appraised |
GSA Schedule Holder: IT-70#:GS35F237AA
GSA 8(a) STARS III#: 47QTCB21D0030
CIO-SP3 Contract#: HHSN316201800033W(SDVOSB)
CIO-SP3 Contract#: HHSN316201800054W(HUBZone)
Seaport-NXG Contract#: N00178-19-D-8420
eFAST Contract#: DTFAWA-13-A-00074
[cid:image001.png@01D9B3DF.B2B781F0]
[cid:image002.png@01D9B3DF.B2B781F0]
Fax: 410-814-7539 |jboggs@rightdirectiontech.com<ma...@rightdirectiontech.com>
RightDirection Technology Solutions, LLC | 300 E. Lombard St Suite 840 | Baltimore, MD 21202 |
www.rightdirectiontech.com<http://www.rightdirectiontech.com/>
Please Go Green! Please do not print this e-mail unless necessary.
Notice of Confidentiality: This e-mail and any attachments thereto, are intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail (or the person responsible for delivering this document to the intended recipient), you are hereby notified that any dissemination, distribution, printing or copying of this e-mail, and any attachment thereto, is strictly prohibited. If you have received this e-mail in error, please respond to the individual sending the message, and permanently delete the original and any copy of any e-mail and printout thereof.
Re: AW: Tomcat 9.0.76 Memory leak with Java 17
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Thomas,
On 7/13/23 02:19, Thomas Hoffmann (Speed4Trade GmbH) wrote:
> Hello,
>
>> -----Ursprüngliche Nachricht-----
>> Von: Christopher Schultz <ch...@christopherschultz.net>
>> Gesendet: Mittwoch, 12. Juli 2023 21:34
>> An: users@tomcat.apache.org
>> Betreff: Re: Tomcat 9.0.76 Memory leak with Java 17
>>
>> Michael,
>>
>> On 7/12/23 07:33, Michael Osipov wrote:
>>> On 2023/07/11 18:16:24 Christopher Schultz wrote:
>>>> You should report all of the previous issues to Oracle against their
>>>> ORDS version 22.1 and ask them to fix them. It's why you write those
>>>> big, fat checks in the first place ;)
>>>
>>> This doesn't really matter. I have reported a memory leak in OJDBC
>>> many years ago where a background thread pins the WebappClassLoader.
>>> Answer from Oracle: This happens only once, does not repeat. We won't
>>> address that...
>> Fair enough, but that doesn't mean it's not worth reporting.
>>
>> BTW the fine folks at Sun/Oracle and Connector/J are similarly confused
>> when it comes to ClassLoader pinning in /their/ JDBC driver. I think they had a
>> few different patches that all waved their hands _still_ failing to actually
>> understand and fix the problem.
>>
>> -chris
>>
>
> It's not just Oracle JDBC driver.
> Same can happen with SQL-Server JDBC driver when you use Kerberos Auth.
> Ticket was also closed without a fix.
Yeah, and I think it just comes down to failure to understand the problem.
The problem with Connector/J has always been that they launch a
background thread to clean things up. All they have to do is provide a
method to "shut down background threads".
I kept getting pushback like "they are daemon threads, they won't keep
your JVM running when your application stops" and I was like "hey
sometimes the application stops and restarts without shutting down the
JVM #webapps" and they were like "what does that even mean CLOSED WONTFIX".
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
AW: Tomcat 9.0.76 Memory leak with Java 17
Posted by "Thomas Hoffmann (Speed4Trade GmbH)" <Th...@speed4trade.com.INVALID>.
Hello,
> -----Ursprüngliche Nachricht-----
> Von: Christopher Schultz <ch...@christopherschultz.net>
> Gesendet: Mittwoch, 12. Juli 2023 21:34
> An: users@tomcat.apache.org
> Betreff: Re: Tomcat 9.0.76 Memory leak with Java 17
>
> Michael,
>
> On 7/12/23 07:33, Michael Osipov wrote:
> > On 2023/07/11 18:16:24 Christopher Schultz wrote:
> >> You should report all of the previous issues to Oracle against their
> >> ORDS version 22.1 and ask them to fix them. It's why you write those
> >> big, fat checks in the first place ;)
> >
> > This doesn't really matter. I have reported a memory leak in OJDBC
> > many years ago where a background thread pins the WebappClassLoader.
> > Answer from Oracle: This happens only once, does not repeat. We won't
> > address that...
> Fair enough, but that doesn't mean it's not worth reporting.
>
> BTW the fine folks at Sun/Oracle and Connector/J are similarly confused
> when it comes to ClassLoader pinning in /their/ JDBC driver. I think they had a
> few different patches that all waved their hands _still_ failing to actually
> understand and fix the problem.
>
> -chris
>
It's not just Oracle JDBC driver.
Same can happen with SQL-Server JDBC driver when you use Kerberos Auth.
Ticket was also closed without a fix.
Re: Tomcat 9.0.76 Memory leak with Java 17
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Michael,
On 7/12/23 07:33, Michael Osipov wrote:
> On 2023/07/11 18:16:24 Christopher Schultz wrote:
>> You should report all of the previous issues to Oracle against their
>> ORDS version 22.1 and ask them to fix them. It's why you write those
>> big, fat checks in the first place ;)
>
> This doesn't really matter. I have reported a memory leak in OJDBC
> many years ago where a background thread pins the WebappClassLoader.
> Answer from Oracle: This happens only once, does not repeat. We won't
> address that...
Fair enough, but that doesn't mean it's not worth reporting.
BTW the fine folks at Sun/Oracle and Connector/J are similarly confused
when it comes to ClassLoader pinning in /their/ JDBC driver. I think
they had a few different patches that all waved their hands _still_
failing to actually understand and fix the problem.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 9.0.76 Memory leak with Java 17
Posted by Michael Osipov <mi...@apache.org>.
On 2023/07/11 18:16:24 Christopher Schultz wrote:
> You should report all of the previous issues to Oracle against their
> ORDS version 22.1 and ask them to fix them. It's why you write those
> big, fat checks in the first place ;)
This doesn't really matter. I have reported a memory leak in OJDBC many years ago where a background thread pins the WebappClassLoader. Answer from Oracle: This happens only once, does not repeat. We won't address that...
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat 9.0.76 Memory leak with Java 17
Posted by James Boggs <jb...@rightdirectiontech.com.INVALID>.
Chris,
Yes it is unintentional. Actually once we start it with the Windows service, and run through a few reports on the website, it stops in just ba few minutes.
We will look at the java heap size settings.
Regards,
James Boggs
-----Original Message-----
From: Christopher Schultz <ch...@christopherschultz.net>
Sent: Wednesday, July 12, 2023 3:55 PM
To: users@tomcat.apache.org
Subject: Re: Tomcat 9.0.76 Memory leak with Java 17
James,
On 7/12/23 15:41, James Boggs wrote:
> Thanks for the input. I will forward the email to our developers to
> look at the heap size settings being different.
>
> We have a Windows service that is used to start/stop Tomcat. When this
> happens we find that the Windows service is no longer running.
Depending upon your settings, you may have more than one log file to look at. Look at everything in CATALINA_BASE/logs the next time the service goes down to see if there is a clue in any of them.
You could also look for hs_pid[process-id] files (or the Windows equivalent, if it's different) that are usually dumped when a JVM crash occurs.
Each of your logs seems to show that Tomcat is being shut-down in an orderly way, though. Are you sure it's crashing and not being intentionally stopped?
-chris
> -----Original Message-----
> From: Christopher Schultz <ch...@christopherschultz.net>
> Sent: Wednesday, July 12, 2023 3:32 PM
> To: users@tomcat.apache.org
> Subject: Re: Tomcat 9.0.76 Memory leak with Java 17
>
> Suvendu,
>
> On 7/12/23 07:11, Suvendu Sekhar Mondal wrote:
>> On Tue, Jul 11, 2023 at 11:48 PM Christopher Schultz
>> <ch...@christopherschultz.net> wrote:
>>>
>>> James,
>>>
>>> On 7/11/23 10:21, James Boggs wrote:
>>>> We had a stable SSL enabled website with Apache Tomcat 9.0.73 on
>>>> Windows Server 2012 o/s, Java 8, Oracle ORDS 21.4 and SSL.
>>>>
>>>> We simultaneously upgraded to Tomcat 9.0.75, upgraded to Java 17
>>>> and to ORDS 22.1, then used Java 17 to create a new Java Keystore
>>>> and a new SSL csr file, and imported a new SSL certificate from the
>>>> CA into the new keystore.
>>>>
>>>> The website works but after logging in there are memory leak
>>>> warnings and the Tomcat service crashes within just a couple of minutes.
>>>>
>>>> We even upgraded to 9.0.76 and the issue persists.
>>>>
>>>> Below is an excerpt from the stderr log file.
>>>>
>>>> I have been unable to find any recent threads on this, any help is
>>>> appreciated. Is any other information needed?
>>>
>>> I think you have included all necessary information. I'm
>>> chopping-out the irrrelevant bits:
>>>
>>>> 2023-07-10T21:35:40.939Z WARNING The web application [rplans-vpd]
>>>> registered the JDBC driver [oracle.jdbc.OracleDriver] but failed to
>>>> unregister it when the web application was stopped. To prevent a
>>>> memory leak, the JDBC Driver has been forcibly unregistered.
>>>
>>> This is actually "okay" in that Tomcat has detected a global JDBC
>>> driver registration performed by the application, and is fixing the
>>> problem for you. The application, however, is making a mistake by
>>> not de-registering that JDBC driver. Your options are to move the
>>> driver library from your application into Tomcat's lib/ directory,
>>> fix the application to de-register the driver when it shuts down, or just ignore the warning.
>>> But you should fix the application.
>>>
>>>> 2023-07-10T21:35:40.944Z WARNING The web application [rplans-vpd]
>>>> appears to have started a thread named
>>>> [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
>>>
>>> There are multiple instances of this same message and THIS is your
>>> problem. The problem is what the error message says: your
>>> application has started a Thread and never stopped it. The "memory
>>> leak" comes from the fact that the Thread has inherited the web
>>> application's ClassLoader and the web application is being
>>> re-loaded. When that happens, Tomcat discards the ClassLoader which
>>> usually means the GC will clean up after it at some point in the
>>> future. But that Thread is still running and will keep the ClassLoader in memory, likely forever.
>>>
>>> You have a few options:
>>>
>>> 1. Fix the application. The application needs to shut-down any
>>> threads is starts during its operation, preferrably in a
>>> ServletContextListener's destroy method or similar.
>>>
>>> 2. Don't hot-reload your application. Instead, shut-down the JVM and
>>> re-start it. Ovbviously, this may have availability implications,
>>> but then again so does running out of memory and having to bounce
>>> the JVM, anyway.
>>>
>>
>> I was checking the stacks and saw this: They might not be doing
>> "hot-deploy" of the app. AFAIK those messages come once someone stops
>> Tomcat.
>>
>>> 2023-07-10T20:52:06.292Z INFO Starting ProtocolHandler ["https-openssl-nio-10.2.251.132-443"]
>>> 2023-07-10T20:52:06.346Z INFO Server startup in [28980] milliseconds
>>> 2023-07-10T21:35:40.487Z INFO Pausing ProtocolHandler ["https-openssl-nio-10.2.251.132-443"]
>>> 2023-07-10T21:35:40.495Z INFO Stopping service [Catalina]
>>> 2023-07-10T21:35:40.910Z INFO Destroyed pool : |default|lo|-2023-07-10T20-51-53.099285500Z
>
> Good point: Tomcat will complain about this stuff on shutdown even if the warnings are irrelevant because the application-stop leak-checks are run whether Tomcat is going to stop or not.
>
> So operationally, these warnings may be irrelevant.
>
> (IMHO you should still fix the application!)
>
>> Another thing I noticed was the difference in allocated Java heap. It
>> appears that first it sets min=max=1GB and after that it's setting
>> max heap to 256MB. That could be the problem. JVM might be crashing
>> because of heap shortage.
>>
>>> 10-Jul-2023 16:51:35.481 INFO [main]
>>> org.apache.catalina.startup.VersionLoggerListener.log Command line
>>> argument: -Dconfig.url=D:\ords222
>>> 10-Jul-2023 16:51:35.481 INFO [main]
>>> org.apache.catalina.startup.VersionLoggerListener.log Command line
>>> argument: -Xms1024M
>>> 10-Jul-2023 16:51:35.481 INFO [main]
>>> org.apache.catalina.startup.VersionLoggerListener.log Command line
>>> argument: -Xmx1024M
>>
>>> 10-Jul-2023 16:51:35.486 INFO [main]
>>> org.apache.catalina.startup.VersionLoggerListener.log Command line
>>> argument: exit
>>> 10-Jul-2023 16:51:35.486 INFO [main]
>>> org.apache.catalina.startup.VersionLoggerListener.log Command line
>>> argument: abort
>>> 10-Jul-2023 16:51:35.486 INFO [main]
>>> org.apache.catalina.startup.VersionLoggerListener.log Command line
>>> argument: -Xms128m
>>> 10-Jul-2023 16:51:35.486 INFO [main]
>>> org.apache.catalina.startup.VersionLoggerListener.log Command line
>>> argument: -Xmx256m
>
> Oh, I hadn't noticed that, either.
>
> Dropping your heap to 25% of original values could certainly cause
> problems :)
>
> I don't see a "crash" in the log, so it's hard to tell what exactly is being reported, here.
>
> -chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 9.0.76 Memory leak with Java 17
Posted by Christopher Schultz <ch...@christopherschultz.net>.
James,
On 7/12/23 15:41, James Boggs wrote:
> Thanks for the input. I will forward the email to our developers to
> look at the heap size settings being different.
>
> We have a Windows service that is used to start/stop Tomcat. When
> this happens we find that the Windows service is no longer running.
Depending upon your settings, you may have more than one log file to
look at. Look at everything in CATALINA_BASE/logs the next time the
service goes down to see if there is a clue in any of them.
You could also look for hs_pid[process-id] files (or the Windows
equivalent, if it's different) that are usually dumped when a JVM crash
occurs.
Each of your logs seems to show that Tomcat is being shut-down in an
orderly way, though. Are you sure it's crashing and not being
intentionally stopped?
-chris
> -----Original Message-----
> From: Christopher Schultz <ch...@christopherschultz.net>
> Sent: Wednesday, July 12, 2023 3:32 PM
> To: users@tomcat.apache.org
> Subject: Re: Tomcat 9.0.76 Memory leak with Java 17
>
> Suvendu,
>
> On 7/12/23 07:11, Suvendu Sekhar Mondal wrote:
>> On Tue, Jul 11, 2023 at 11:48 PM Christopher Schultz
>> <ch...@christopherschultz.net> wrote:
>>>
>>> James,
>>>
>>> On 7/11/23 10:21, James Boggs wrote:
>>>> We had a stable SSL enabled website with Apache Tomcat 9.0.73 on
>>>> Windows Server 2012 o/s, Java 8, Oracle ORDS 21.4 and SSL.
>>>>
>>>> We simultaneously upgraded to Tomcat 9.0.75, upgraded to Java 17 and
>>>> to ORDS 22.1, then used Java 17 to create a new Java Keystore and a
>>>> new SSL csr file, and imported a new SSL certificate from the CA
>>>> into the new keystore.
>>>>
>>>> The website works but after logging in there are memory leak
>>>> warnings and the Tomcat service crashes within just a couple of minutes.
>>>>
>>>> We even upgraded to 9.0.76 and the issue persists.
>>>>
>>>> Below is an excerpt from the stderr log file.
>>>>
>>>> I have been unable to find any recent threads on this, any help is
>>>> appreciated. Is any other information needed?
>>>
>>> I think you have included all necessary information. I'm chopping-out
>>> the irrrelevant bits:
>>>
>>>> 2023-07-10T21:35:40.939Z WARNING The web application [rplans-vpd]
>>>> registered the JDBC driver [oracle.jdbc.OracleDriver] but failed to
>>>> unregister it when the web application was stopped. To prevent a
>>>> memory leak, the JDBC Driver has been forcibly unregistered.
>>>
>>> This is actually "okay" in that Tomcat has detected a global JDBC
>>> driver registration performed by the application, and is fixing the
>>> problem for you. The application, however, is making a mistake by not
>>> de-registering that JDBC driver. Your options are to move the driver
>>> library from your application into Tomcat's lib/ directory, fix the
>>> application to de-register the driver when it shuts down, or just ignore the warning.
>>> But you should fix the application.
>>>
>>>> 2023-07-10T21:35:40.944Z WARNING The web application [rplans-vpd]
>>>> appears to have started a thread named
>>>> [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
>>>
>>> There are multiple instances of this same message and THIS is your
>>> problem. The problem is what the error message says: your application
>>> has started a Thread and never stopped it. The "memory leak" comes
>>> from the fact that the Thread has inherited the web application's
>>> ClassLoader and the web application is being re-loaded. When that
>>> happens, Tomcat discards the ClassLoader which usually means the GC
>>> will clean up after it at some point in the future. But that Thread
>>> is still running and will keep the ClassLoader in memory, likely forever.
>>>
>>> You have a few options:
>>>
>>> 1. Fix the application. The application needs to shut-down any
>>> threads is starts during its operation, preferrably in a
>>> ServletContextListener's destroy method or similar.
>>>
>>> 2. Don't hot-reload your application. Instead, shut-down the JVM and
>>> re-start it. Ovbviously, this may have availability implications, but
>>> then again so does running out of memory and having to bounce the
>>> JVM, anyway.
>>>
>>
>> I was checking the stacks and saw this: They might not be doing
>> "hot-deploy" of the app. AFAIK those messages come once someone stops
>> Tomcat.
>>
>>> 2023-07-10T20:52:06.292Z INFO Starting ProtocolHandler ["https-openssl-nio-10.2.251.132-443"]
>>> 2023-07-10T20:52:06.346Z INFO Server startup in [28980] milliseconds
>>> 2023-07-10T21:35:40.487Z INFO Pausing ProtocolHandler ["https-openssl-nio-10.2.251.132-443"]
>>> 2023-07-10T21:35:40.495Z INFO Stopping service [Catalina]
>>> 2023-07-10T21:35:40.910Z INFO Destroyed pool : |default|lo|-2023-07-10T20-51-53.099285500Z
>
> Good point: Tomcat will complain about this stuff on shutdown even if the warnings are irrelevant because the application-stop leak-checks are run whether Tomcat is going to stop or not.
>
> So operationally, these warnings may be irrelevant.
>
> (IMHO you should still fix the application!)
>
>> Another thing I noticed was the difference in allocated Java heap. It
>> appears that first it sets min=max=1GB and after that it's setting max
>> heap to 256MB. That could be the problem. JVM might be crashing
>> because of heap shortage.
>>
>>> 10-Jul-2023 16:51:35.481 INFO [main]
>>> org.apache.catalina.startup.VersionLoggerListener.log Command line
>>> argument: -Dconfig.url=D:\ords222
>>> 10-Jul-2023 16:51:35.481 INFO [main]
>>> org.apache.catalina.startup.VersionLoggerListener.log Command line
>>> argument: -Xms1024M
>>> 10-Jul-2023 16:51:35.481 INFO [main]
>>> org.apache.catalina.startup.VersionLoggerListener.log Command line
>>> argument: -Xmx1024M
>>
>>> 10-Jul-2023 16:51:35.486 INFO [main]
>>> org.apache.catalina.startup.VersionLoggerListener.log Command line
>>> argument: exit
>>> 10-Jul-2023 16:51:35.486 INFO [main]
>>> org.apache.catalina.startup.VersionLoggerListener.log Command line
>>> argument: abort
>>> 10-Jul-2023 16:51:35.486 INFO [main]
>>> org.apache.catalina.startup.VersionLoggerListener.log Command line
>>> argument: -Xms128m
>>> 10-Jul-2023 16:51:35.486 INFO [main]
>>> org.apache.catalina.startup.VersionLoggerListener.log Command line
>>> argument: -Xmx256m
>
> Oh, I hadn't noticed that, either.
>
> Dropping your heap to 25% of original values could certainly cause problems :)
>
> I don't see a "crash" in the log, so it's hard to tell what exactly is being reported, here.
>
> -chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat 9.0.76 Memory leak with Java 17
Posted by James Boggs <jb...@rightdirectiontech.com.INVALID>.
Thanks for the input. I will forward the email to our developers to look at the heap size settings being different.
We have a Windows service that is used to start/stop Tomcat. When this happens we find that the Windows service is no longer running.
Thanks,
James Boggs
-----Original Message-----
From: Christopher Schultz <ch...@christopherschultz.net>
Sent: Wednesday, July 12, 2023 3:32 PM
To: users@tomcat.apache.org
Subject: Re: Tomcat 9.0.76 Memory leak with Java 17
Suvendu,
On 7/12/23 07:11, Suvendu Sekhar Mondal wrote:
> On Tue, Jul 11, 2023 at 11:48 PM Christopher Schultz
> <ch...@christopherschultz.net> wrote:
>>
>> James,
>>
>> On 7/11/23 10:21, James Boggs wrote:
>>> We had a stable SSL enabled website with Apache Tomcat 9.0.73 on
>>> Windows Server 2012 o/s, Java 8, Oracle ORDS 21.4 and SSL.
>>>
>>> We simultaneously upgraded to Tomcat 9.0.75, upgraded to Java 17 and
>>> to ORDS 22.1, then used Java 17 to create a new Java Keystore and a
>>> new SSL csr file, and imported a new SSL certificate from the CA
>>> into the new keystore.
>>>
>>> The website works but after logging in there are memory leak
>>> warnings and the Tomcat service crashes within just a couple of minutes.
>>>
>>> We even upgraded to 9.0.76 and the issue persists.
>>>
>>> Below is an excerpt from the stderr log file.
>>>
>>> I have been unable to find any recent threads on this, any help is
>>> appreciated. Is any other information needed?
>>
>> I think you have included all necessary information. I'm chopping-out
>> the irrrelevant bits:
>>
>>> 2023-07-10T21:35:40.939Z WARNING The web application [rplans-vpd]
>>> registered the JDBC driver [oracle.jdbc.OracleDriver] but failed to
>>> unregister it when the web application was stopped. To prevent a
>>> memory leak, the JDBC Driver has been forcibly unregistered.
>>
>> This is actually "okay" in that Tomcat has detected a global JDBC
>> driver registration performed by the application, and is fixing the
>> problem for you. The application, however, is making a mistake by not
>> de-registering that JDBC driver. Your options are to move the driver
>> library from your application into Tomcat's lib/ directory, fix the
>> application to de-register the driver when it shuts down, or just ignore the warning.
>> But you should fix the application.
>>
>>> 2023-07-10T21:35:40.944Z WARNING The web application [rplans-vpd]
>>> appears to have started a thread named
>>> [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
>>
>> There are multiple instances of this same message and THIS is your
>> problem. The problem is what the error message says: your application
>> has started a Thread and never stopped it. The "memory leak" comes
>> from the fact that the Thread has inherited the web application's
>> ClassLoader and the web application is being re-loaded. When that
>> happens, Tomcat discards the ClassLoader which usually means the GC
>> will clean up after it at some point in the future. But that Thread
>> is still running and will keep the ClassLoader in memory, likely forever.
>>
>> You have a few options:
>>
>> 1. Fix the application. The application needs to shut-down any
>> threads is starts during its operation, preferrably in a
>> ServletContextListener's destroy method or similar.
>>
>> 2. Don't hot-reload your application. Instead, shut-down the JVM and
>> re-start it. Ovbviously, this may have availability implications, but
>> then again so does running out of memory and having to bounce the
>> JVM, anyway.
>>
>
> I was checking the stacks and saw this: They might not be doing
> "hot-deploy" of the app. AFAIK those messages come once someone stops
> Tomcat.
>
>> 2023-07-10T20:52:06.292Z INFO Starting ProtocolHandler ["https-openssl-nio-10.2.251.132-443"]
>> 2023-07-10T20:52:06.346Z INFO Server startup in [28980] milliseconds
>> 2023-07-10T21:35:40.487Z INFO Pausing ProtocolHandler ["https-openssl-nio-10.2.251.132-443"]
>> 2023-07-10T21:35:40.495Z INFO Stopping service [Catalina]
>> 2023-07-10T21:35:40.910Z INFO Destroyed pool : |default|lo|-2023-07-10T20-51-53.099285500Z
Good point: Tomcat will complain about this stuff on shutdown even if the warnings are irrelevant because the application-stop leak-checks are run whether Tomcat is going to stop or not.
So operationally, these warnings may be irrelevant.
(IMHO you should still fix the application!)
> Another thing I noticed was the difference in allocated Java heap. It
> appears that first it sets min=max=1GB and after that it's setting max
> heap to 256MB. That could be the problem. JVM might be crashing
> because of heap shortage.
>
>> 10-Jul-2023 16:51:35.481 INFO [main]
>> org.apache.catalina.startup.VersionLoggerListener.log Command line
>> argument: -Dconfig.url=D:\ords222
>> 10-Jul-2023 16:51:35.481 INFO [main]
>> org.apache.catalina.startup.VersionLoggerListener.log Command line
>> argument: -Xms1024M
>> 10-Jul-2023 16:51:35.481 INFO [main]
>> org.apache.catalina.startup.VersionLoggerListener.log Command line
>> argument: -Xmx1024M
>
>> 10-Jul-2023 16:51:35.486 INFO [main]
>> org.apache.catalina.startup.VersionLoggerListener.log Command line
>> argument: exit
>> 10-Jul-2023 16:51:35.486 INFO [main]
>> org.apache.catalina.startup.VersionLoggerListener.log Command line
>> argument: abort
>> 10-Jul-2023 16:51:35.486 INFO [main]
>> org.apache.catalina.startup.VersionLoggerListener.log Command line
>> argument: -Xms128m
>> 10-Jul-2023 16:51:35.486 INFO [main]
>> org.apache.catalina.startup.VersionLoggerListener.log Command line
>> argument: -Xmx256m
Oh, I hadn't noticed that, either.
Dropping your heap to 25% of original values could certainly cause problems :)
I don't see a "crash" in the log, so it's hard to tell what exactly is being reported, here.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 9.0.76 Memory leak with Java 17
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Suvendu,
On 7/12/23 07:11, Suvendu Sekhar Mondal wrote:
> On Tue, Jul 11, 2023 at 11:48 PM Christopher Schultz
> <ch...@christopherschultz.net> wrote:
>>
>> James,
>>
>> On 7/11/23 10:21, James Boggs wrote:
>>> We had a stable SSL enabled website with Apache Tomcat 9.0.73 on Windows
>>> Server 2012 o/s, Java 8, Oracle ORDS 21.4 and SSL.
>>>
>>> We simultaneously upgraded to Tomcat 9.0.75, upgraded to Java 17 and to
>>> ORDS 22.1, then used Java 17 to create a new Java Keystore and a new SSL
>>> csr file, and imported a new SSL certificate from the CA into the new
>>> keystore.
>>>
>>> The website works but after logging in there are memory leak warnings
>>> and the Tomcat service crashes within just a couple of minutes.
>>>
>>> We even upgraded to 9.0.76 and the issue persists.
>>>
>>> Below is an excerpt from the stderr log file.
>>>
>>> I have been unable to find any recent threads on this, any help is
>>> appreciated. Is any other information needed?
>>
>> I think you have included all necessary information. I'm chopping-out
>> the irrrelevant bits:
>>
>>> 2023-07-10T21:35:40.939Z WARNING The web application [rplans-vpd]
>>> registered the JDBC driver [oracle.jdbc.OracleDriver] but failed to
>>> unregister it when the web application was stopped. To prevent a memory
>>> leak, the JDBC Driver has been forcibly unregistered.
>>
>> This is actually "okay" in that Tomcat has detected a global JDBC driver
>> registration performed by the application, and is fixing the problem for
>> you. The application, however, is making a mistake by not de-registering
>> that JDBC driver. Your options are to move the driver library from your
>> application into Tomcat's lib/ directory, fix the application to
>> de-register the driver when it shuts down, or just ignore the warning.
>> But you should fix the application.
>>
>>> 2023-07-10T21:35:40.944Z WARNING The web application [rplans-vpd]
>>> appears to have started a thread named
>>> [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
>>
>> There are multiple instances of this same message and THIS is your
>> problem. The problem is what the error message says: your application
>> has started a Thread and never stopped it. The "memory leak" comes from
>> the fact that the Thread has inherited the web application's ClassLoader
>> and the web application is being re-loaded. When that happens, Tomcat
>> discards the ClassLoader which usually means the GC will clean up after
>> it at some point in the future. But that Thread is still running and
>> will keep the ClassLoader in memory, likely forever.
>>
>> You have a few options:
>>
>> 1. Fix the application. The application needs to shut-down any threads
>> is starts during its operation, preferrably in a
>> ServletContextListener's destroy method or similar.
>>
>> 2. Don't hot-reload your application. Instead, shut-down the JVM and
>> re-start it. Ovbviously, this may have availability implications, but
>> then again so does running out of memory and having to bounce the JVM,
>> anyway.
>>
>
> I was checking the stacks and saw this: They might not be doing
> "hot-deploy" of the app. AFAIK those messages come once someone stops
> Tomcat.
>
>> 2023-07-10T20:52:06.292Z INFO Starting ProtocolHandler ["https-openssl-nio-10.2.251.132-443"]
>> 2023-07-10T20:52:06.346Z INFO Server startup in [28980] milliseconds
>> 2023-07-10T21:35:40.487Z INFO Pausing ProtocolHandler ["https-openssl-nio-10.2.251.132-443"]
>> 2023-07-10T21:35:40.495Z INFO Stopping service [Catalina]
>> 2023-07-10T21:35:40.910Z INFO Destroyed pool : |default|lo|-2023-07-10T20-51-53.099285500Z
Good point: Tomcat will complain about this stuff on shutdown even if
the warnings are irrelevant because the application-stop leak-checks are
run whether Tomcat is going to stop or not.
So operationally, these warnings may be irrelevant.
(IMHO you should still fix the application!)
> Another thing I noticed was the difference in allocated Java heap. It
> appears that first it sets min=max=1GB and after that it's setting max
> heap to 256MB. That could be the problem. JVM might be crashing
> because of heap shortage.
>
>> 10-Jul-2023 16:51:35.481 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dconfig.url=D:\ords222
>> 10-Jul-2023 16:51:35.481 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xms1024M
>> 10-Jul-2023 16:51:35.481 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xmx1024M
>
>> 10-Jul-2023 16:51:35.486 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: exit
>> 10-Jul-2023 16:51:35.486 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: abort
>> 10-Jul-2023 16:51:35.486 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xms128m
>> 10-Jul-2023 16:51:35.486 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xmx256m
Oh, I hadn't noticed that, either.
Dropping your heap to 25% of original values could certainly cause
problems :)
I don't see a "crash" in the log, so it's hard to tell what exactly is
being reported, here.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 9.0.76 Memory leak with Java 17
Posted by Suvendu Sekhar Mondal <su...@gmail.com>.
Hi Chris,
On Tue, Jul 11, 2023 at 11:48 PM Christopher Schultz
<ch...@christopherschultz.net> wrote:
>
> James,
>
> On 7/11/23 10:21, James Boggs wrote:
> > We had a stable SSL enabled website with Apache Tomcat 9.0.73 on Windows
> > Server 2012 o/s, Java 8, Oracle ORDS 21.4 and SSL.
> >
> > We simultaneously upgraded to Tomcat 9.0.75, upgraded to Java 17 and to
> > ORDS 22.1, then used Java 17 to create a new Java Keystore and a new SSL
> > csr file, and imported a new SSL certificate from the CA into the new
> > keystore.
> >
> > The website works but after logging in there are memory leak warnings
> > and the Tomcat service crashes within just a couple of minutes.
> >
> > We even upgraded to 9.0.76 and the issue persists.
> >
> > Below is an excerpt from the stderr log file.
> >
> > I have been unable to find any recent threads on this, any help is
> > appreciated. Is any other information needed?
>
> I think you have included all necessary information. I'm chopping-out
> the irrrelevant bits:
>
> > 2023-07-10T21:35:40.939Z WARNING The web application [rplans-vpd]
> > registered the JDBC driver [oracle.jdbc.OracleDriver] but failed to
> > unregister it when the web application was stopped. To prevent a memory
> > leak, the JDBC Driver has been forcibly unregistered.
>
> This is actually "okay" in that Tomcat has detected a global JDBC driver
> registration performed by the application, and is fixing the problem for
> you. The application, however, is making a mistake by not de-registering
> that JDBC driver. Your options are to move the driver library from your
> application into Tomcat's lib/ directory, fix the application to
> de-register the driver when it shuts down, or just ignore the warning.
> But you should fix the application.
>
> > 2023-07-10T21:35:40.944Z WARNING The web application [rplans-vpd]
> > appears to have started a thread named
> > [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
>
> There are multiple instances of this same message and THIS is your
> problem. The problem is what the error message says: your application
> has started a Thread and never stopped it. The "memory leak" comes from
> the fact that the Thread has inherited the web application's ClassLoader
> and the web application is being re-loaded. When that happens, Tomcat
> discards the ClassLoader which usually means the GC will clean up after
> it at some point in the future. But that Thread is still running and
> will keep the ClassLoader in memory, likely forever.
>
> You have a few options:
>
> 1. Fix the application. The application needs to shut-down any threads
> is starts during its operation, preferrably in a
> ServletContextListener's destroy method or similar.
>
> 2. Don't hot-reload your application. Instead, shut-down the JVM and
> re-start it. Ovbviously, this may have availability implications, but
> then again so does running out of memory and having to bounce the JVM,
> anyway.
>
I was checking the stacks and saw this: They might not be doing
"hot-deploy" of the app. AFAIK those messages come once someone stops
Tomcat.
> 2023-07-10T20:52:06.292Z INFO Starting ProtocolHandler ["https-openssl-nio-10.2.251.132-443"]
> 2023-07-10T20:52:06.346Z INFO Server startup in [28980] milliseconds
> 2023-07-10T21:35:40.487Z INFO Pausing ProtocolHandler ["https-openssl-nio-10.2.251.132-443"]
> 2023-07-10T21:35:40.495Z INFO Stopping service [Catalina]
> 2023-07-10T21:35:40.910Z INFO Destroyed pool : |default|lo|-2023-07-10T20-51-53.099285500Z
Another thing I noticed was the difference in allocated Java heap. It
appears that first it sets min=max=1GB and after that it's setting max
heap to 256MB. That could be the problem. JVM might be crashing
because of heap shortage.
> 10-Jul-2023 16:51:35.481 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dconfig.url=D:\ords222
> 10-Jul-2023 16:51:35.481 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xms1024M
> 10-Jul-2023 16:51:35.481 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xmx1024M
> 10-Jul-2023 16:51:35.486 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: exit
> 10-Jul-2023 16:51:35.486 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: abort
> 10-Jul-2023 16:51:35.486 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xms128m
> 10-Jul-2023 16:51:35.486 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xmx256m
> > 2023-07-10T21:35:40.944Z SEVERE The web application [rplans-vpd]
> > created a ThreadLocal with key of type
> > [java.lang.ThreadLocal.SuppliedThreadLocal] (value
> > [java.lang.ThreadLocal$SuppliedThreadLocal@427241b4]) and a value of
> > type [oracle.ucp.common.Core.Match] (value [UNKNOWN]) but failed to
> > remove it when the web application was stopped. Threads are going to be
> > renewed over time to try and avoid a probable memory leak.
>
> This is actually "okay" in that Tomcat has detected your application's
> ThreadLocal variables (objects bound to one or more Threads which are
> owned by the application) which are not being cleaned-up by the
> application, and it's fixing the problem for you by, over time, killing
> each of those Threads and replacing them in the Thread pool for you.
> Your options are to fix the application or to ignore the warning. But
> you should fix the application.
>
> It appears that your upgrade of ORDS has introduced a lot of stuff that
> doesn't play well with hot-reloading of the application. I'm assuming
> that you aren't responsible for maintaining ORDS... Oracle is.
>
> You should report all of the previous issues to Oracle against their
> ORDS version 22.1 and ask them to fix them. It's why you write those
> big, fat checks in the first place ;)
>
> -chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 9.0.76 Memory leak with Java 17
Posted by Christopher Schultz <ch...@christopherschultz.net>.
James,
On 7/11/23 10:21, James Boggs wrote:
> We had a stable SSL enabled website with Apache Tomcat 9.0.73 on Windows
> Server 2012 o/s, Java 8, Oracle ORDS 21.4 and SSL.
>
> We simultaneously upgraded to Tomcat 9.0.75, upgraded to Java 17 and to
> ORDS 22.1, then used Java 17 to create a new Java Keystore and a new SSL
> csr file, and imported a new SSL certificate from the CA into the new
> keystore.
>
> The website works but after logging in there are memory leak warnings
> and the Tomcat service crashes within just a couple of minutes.
>
> We even upgraded to 9.0.76 and the issue persists.
>
> Below is an excerpt from the stderr log file.
>
> I have been unable to find any recent threads on this, any help is
> appreciated. Is any other information needed?
I think you have included all necessary information. I'm chopping-out
the irrrelevant bits:
> 2023-07-10T21:35:40.939Z WARNING The web application [rplans-vpd]
> registered the JDBC driver [oracle.jdbc.OracleDriver] but failed to
> unregister it when the web application was stopped. To prevent a memory
> leak, the JDBC Driver has been forcibly unregistered.
This is actually "okay" in that Tomcat has detected a global JDBC driver
registration performed by the application, and is fixing the problem for
you. The application, however, is making a mistake by not de-registering
that JDBC driver. Your options are to move the driver library from your
application into Tomcat's lib/ directory, fix the application to
de-register the driver when it shuts down, or just ignore the warning.
But you should fix the application.
> 2023-07-10T21:35:40.944Z WARNING The web application [rplans-vpd]
> appears to have started a thread named
> [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
There are multiple instances of this same message and THIS is your
problem. The problem is what the error message says: your application
has started a Thread and never stopped it. The "memory leak" comes from
the fact that the Thread has inherited the web application's ClassLoader
and the web application is being re-loaded. When that happens, Tomcat
discards the ClassLoader which usually means the GC will clean up after
it at some point in the future. But that Thread is still running and
will keep the ClassLoader in memory, likely forever.
You have a few options:
1. Fix the application. The application needs to shut-down any threads
is starts during its operation, preferrably in a
ServletContextListener's destroy method or similar.
2. Don't hot-reload your application. Instead, shut-down the JVM and
re-start it. Ovbviously, this may have availability implications, but
then again so does running out of memory and having to bounce the JVM,
anyway.
> 2023-07-10T21:35:40.944Z SEVERE The web application [rplans-vpd]
> created a ThreadLocal with key of type
> [java.lang.ThreadLocal.SuppliedThreadLocal] (value
> [java.lang.ThreadLocal$SuppliedThreadLocal@427241b4]) and a value of
> type [oracle.ucp.common.Core.Match] (value [UNKNOWN]) but failed to
> remove it when the web application was stopped. Threads are going to be
> renewed over time to try and avoid a probable memory leak.
This is actually "okay" in that Tomcat has detected your application's
ThreadLocal variables (objects bound to one or more Threads which are
owned by the application) which are not being cleaned-up by the
application, and it's fixing the problem for you by, over time, killing
each of those Threads and replacing them in the Thread pool for you.
Your options are to fix the application or to ignore the warning. But
you should fix the application.
It appears that your upgrade of ORDS has introduced a lot of stuff that
doesn't play well with hot-reloading of the application. I'm assuming
that you aren't responsible for maintaining ORDS... Oracle is.
You should report all of the previous issues to Oracle against their
ORDS version 22.1 and ask them to fix them. It's why you write those
big, fat checks in the first place ;)
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org