You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tomcat.apache.org by GitBox <gi...@apache.org> on 2020/02/18 14:04:15 UTC
[GitHub] [tomcat] mads1980 opened a new pull request #245: OpenSSLEngine
improvements to guard against multiple shutdown() calls triggered by
construction exception and finalize() later
mads1980 opened a new pull request #245: OpenSSLEngine improvements to guard against multiple shutdown() calls triggered by construction exception and finalize() later
URL: https://github.com/apache/tomcat/pull/245
We started getting random SIGSEGV JVM crashes on our servers after upgrading to OpenJDK 13.0.2
The crashes are always in tcnative code, on the JVM Finalizer daemon thread, at OpenSSLEngine.shutdown(). We're on Tomcat 9.0.31, tcnative 1.2.23, OpenSSL 1.1.1d and APR 1.7.0 using NIO connectors. This combination works fine with Oracle JDK 8u202 as well as OpenJDK 8u242, but crashes began immediately after upgrading to OpenJDK 13.0.2
Since the failure is related to tcnative code and finalization, we guess the new JVM is probably invoking the finalizers sooner, so this is exposing a bug in OpenSSLEngine which was already present, but was not being triggered on previous JVMs because of different timing.
The crash occurs when SSL.freeSSL() is invoked, so this probably means either the pointer is zero, or a double SSL.freeSSL() is invoked. This can happen if there are uncaught exceptions within the OpenSSLEngine constructor, as finalization can execute concurrently with object construction.
We adapted the solution found in OpenSSLContext.destroy() which evidently was causing similar issues. The current volatile "destroyed" checks found in OpenSSLEngine seem to be insufficient, so we replaced it with an AtomicBoolean. Also, we check whether "networkBIO" and
"ssl" are zero in shutdown() to avoid attempting to free a null pointer.
We found a discussing regarding these issues in the following thread:
http://mail-archives.apache.org/mod_mbox/tomcat-dev/201901.mbox/%3C44ce40f1-36a1-df9b-b648-8635d9d84eb1@kippdata.de%3E
It seems the improvements made it to OpenSSLContext but not OpenSSLEngine. This pull request should fix remaining issues.
See sample crash dump extract below:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007fc98d024f58, pid=32426, tid=32435
#
# JRE version: OpenJDK Runtime Environment (13.0.2+8) (build 13.0.2+8)
# Java VM: OpenJDK 64-Bit Server VM (13.0.2+8, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# C [libcrypto.so.1.1+0x16ff58] CRYPTO_get_ex_data+0x8
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t e" (or dumping to //core.32426)
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
--------------- S U M M A R Y ------------
Command Line: -Dcatalina.home=/home/tomcat -Dcatalina.base=/home/tomcat -Djava.library.path=/home/tomcat/lib -Djava.security.properties=/home/tomcat/conf/java.security.properties -Djava.io.tmpdir=/var/tmp -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/home/tomcat/conf/logging.properties -XX:ErrorFile=/home/tomcat/logs/hs_err_pid%p.log -XX:HeapDumpPath=/home/tomcat/logs/ --illegal-access=permit -Xshare:off -XX:+PrintVMOptions -XX:+UnlockExperimentalVMOptions -XX:+UnlockDiagnosticVMOptions -XX:+UsePerfData -XX:+DoEscapeAnalysis -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -XX:-OmitStackTraceInFastThrow -XX:+PrintStringTableStatistics -XX:StringTableSize=1000003 -XX:SymbolTableSize=1000003 -XX:+PrintCompilation -XX:+PrintInlining -XX:+UseBiasedLocking -XX:BiasedLockingStartupDelay=0 -XX:MaxInlineLevel=15 -Xms5G -Xmx5G -Xss512K -XX:MetaspaceSize=128M -XX:MaxMetaspaceSize=256M -XX:+UseLargePages -XX:InitialCodeCacheSize=128M -XX:ReservedCodeCacheSize=128M -XX:+UseCodeCacheFlushing -XX:+UseCompressedOops -verbose:gc -XX:+DisableExplicitGC -XX:+ParallelRefProcEnabled -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:GCTimeRatio=4 -XX:G1NewSizePercent=30 -XX:G1MaxNewSizePercent=60 -XX:InitiatingHeapOccupancyPercent=0 -XX:G1ReservePercent=10 -XX:G1HeapRegionSize=32M -XX:+UseStringDeduplication -Xlog:gc*,gc+ergo*=trace,gc+heap=debug,classhisto*=trace,safepoint:file=/home/tomcat/logs/gclog.2020-02-17:level,time,uptime,tags -Dcom.renxo.cms.util.io.network.NetworkUtils.preferredLocalHostSubnet=192.168.0.0/16 -Dcom.renxo.cms.util.io.network.NetworkUtils.developmentEnvironmentSubnet=192.168.3.0/24 -javaagent:/home/tomcat/apminsight/apminsight-javaagent.jar -Dapminsight.application.name=homer.renxo.net applications -Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8 -Djava.security.egd=file:/dev/urandom -Djava.rmi.server.hostname=homer.renxo.net -Djava.io.tmpdir=/home/tomcat/temp/ -Dcommons.daemon.process.id=32426 -Dcommons.daemon.process.parent=28765 -Dcommons.daemon.version=1.2.2 abort org.apache.catalina.startup.Bootstrap
Host: Intel(R) Xeon(R) CPU 3050 @ 2.13GHz, 2 cores, 7G, Red Hat Enterprise Linux Server release 6.5 (Santiago)
Time: Tue Feb 18 00:13:06 2020 GMT+3 elapsed time: 18291 seconds (0d 5h 4m 51s)
--------------- T H R E A D ---------------
Current thread (0x0000000000d75800): JavaThread "Finalizer" daemon [_thread_in_native, id=32435, stack(0x00007fc994890000,0x00007fc994911000)]
Stack: [0x00007fc994890000,0x00007fc994911000], sp=0x00007fc99490f770, free space=509k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [libcrypto.so.1.1+0x16ff58] CRYPTO_get_ex_data+0x8
j org.apache.tomcat.util.net.openssl.OpenSSLEngine.shutdown()V+23
j org.apache.tomcat.util.net.openssl.OpenSSLEngine.finalize()V+5
J 45265 c1 java.lang.ref.Finalizer.runFinalizer(Ljdk/internal/access/JavaLangAccess;)V java.base@13.0.2 (117 bytes) @ 0x00007fc9a2afd804 [0x00007fc9a2afd1a0+0x0000000000000664]
j java.lang.ref.Finalizer$FinalizerThread.run()V+46 java.base@13.0.2
v ~StubRoutines::call_stub
V [libjvm.so+0x821d79] JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, Thread*)+0x319
V [libjvm.so+0x820070] JavaCalls::call_virtual(JavaValue*, Klass*, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x160
V [libjvm.so+0x820141] JavaCalls::call_virtual(JavaValue*, Handle, Klass*, Symbol*, Symbol*, Thread*)+0x81
V [libjvm.so+0x8bf30c] thread_entry(JavaThread*, Thread*)+0x6c
V [libjvm.so+0xd51712] JavaThread::thread_main_inner()+0xe2
V [libjvm.so+0xd5637d] Thread::call_run()+0x10d
V [libjvm.so+0xba8b37] thread_native_entry(Thread*)+0xe7
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j org.apache.tomcat.jni.SSL.freeSSL(J)V+0
j org.apache.tomcat.util.net.openssl.OpenSSLEngine.shutdown()V+23
j org.apache.tomcat.util.net.openssl.OpenSSLEngine.finalize()V+5
J 45265 c1 java.lang.ref.Finalizer.runFinalizer(Ljdk/internal/access/JavaLangAccess;)V java.base@13.0.2 (117 bytes) @ 0x00007fc9a2afd804 [0x00007fc9a2afd1a0+0x0000000000000664]
j java.lang.ref.Finalizer$FinalizerThread.run()V+46 java.base@13.0.2
v ~StubRoutines::call_stub
siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x00000000000005b8
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[GitHub] [tomcat] mads1980 closed pull request #245: OpenSSLEngine
improvements to guard against multiple shutdown() calls triggered by
construction exception and finalize() later
Posted by GitBox <gi...@apache.org>.
mads1980 closed pull request #245: OpenSSLEngine improvements to guard against multiple shutdown() calls triggered by construction exception and finalize() later
URL: https://github.com/apache/tomcat/pull/245
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org