You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tomcat.apache.org by Rainer Jung <ra...@kippdata.de> on 2022/07/18 10:09:23 UTC
tcnative crashes during shutdown (TC 10.x unit tests)
Hi there,
this is just an info, at this point probably not a showstopper. The
topic is crashes in tcnative 1.2 and 2.0 for TC 10.0 during shutdown
after TLS unit tests.
Details:
I ran the TC unit tests for latest 9.x, 10.x and 10.1.x with tcnative
1.2.35 OpenSSL 1.1.1q, 1.2.35 OpenSSL 3.0.2 and 2.0.1 OpenSSL 3.0.2.
I ran the test for a variety of OpenJDK builds (Adoptium, Zulu, Oracle,
RedHat) and versions (latest 1.8.0 except for 10.1, 11, 17 and current 19).
The platforms where SLES 11, 12 and 15 and RHEL 6, 7 and 8. For RHEL 7
and 8 there were 48 runs, for the other platforms 39 (no RedHat JDK).
I only ran about 150 test classes (for NIO and also for NIO2), because I
also ran the full unit tests (about 450 classes) for JSSE and didn't
want to rerun all tests for time and efficiency reasons.
For TC 10 I observed crashes in TLS tests during shutdown: Out of the
roughly 250 test runs, 5 produced such a crash. For TC 9 I did not
observe a single one. Tests for TC 10.1 are ongoing, until now no crash,
but it is a bit early for a final result. I think the crashes are not
new. All hapened in the TLS tests in org.apache.tomcat.util.net.
The list of crashes I saw for TC 10.0.23:
RHEL7 jdk1.8.0 tcnative 1.2.35 OpenSSL 3.0.2
org.apache.tomcat.util.net.TestSsl FAILED (crashed)
openjdk version "1.8.0_332-ea"
OpenJDK Runtime Environment (build 1.8.0_332-ea-b06)
OpenJDK 64-Bit Server VM (build 25.332-b06, mixed mode)
double free or corruption (!prev): 0x00007f473c19df50
======= Backtrace: =========
/lib64/libc.so.6(+0x7d56d)[0x7f4742aa456d]
/.../tcnative-deps/libapr-1.so.0(apr_allocator_destroy+0x1d)[0x7f472871923d]
/.../tcnative-deps/libapr-1.so.0(apr_pool_terminate+0x30)[0x7f4728719c10]
[0x7f472d018427]
RHEL7 jdk17 tcnative 1.2.35 OpenSSL 1.1.1q
org.apache.tomcat.util.net.TestCustomSslTrustManager FAILED (crashed)
openjdk version "17.0.2" 2022-01-18
OpenJDK Runtime Environment (build 17.0.2+8-86)
OpenJDK 64-Bit Server VM (build 17.0.2+8-86, mixed mode, sharing)
corrupted double-linked list: 0x00007f6bb8001d10
======= Backtrace: =========
/lib64/libc.so.6(+0x7bfc7)[0x7f6bf481dfc7]
/lib64/libc.so.6(+0x7d774)[0x7f6bf481f774]
/.../tcnative-deps/libapr-1.so.0(apr_allocator_destroy+0x1d)[0x7f6bc543223d]
/.../tcnative-deps/libapr-1.so.0(apr_pool_terminate+0x30)[0x7f6bc5432c10]
[0x7f6bd572249a]
SLES11 oracle_jdk1.8.0 tcnative 2.0.1 OpenSSL 3.0.2
org.apache.tomcat.util.net.TestSsl FAILED (crashed)
java version "1.8.0_331"
Java(TM) SE Runtime Environment (build 1.8.0_331-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.331-b09, mixed mode)
double free or corruption (!prev): 0x00007fbf88c1de10
======= Backtrace: =========
/lib64/libc.so.6(+0x75018)[0x7fbf87b35018]
/lib64/libc.so.6(cfree+0x6c)[0x7fbf87b39f6c]
/.../tcnative-deps/libapr-1.so.0(apr_allocator_destroy+0x1d)[0x7fbf718a5aad]
/.../tcnative-deps/libapr-1.so.0(apr_pool_terminate+0x34)[0x7fbf718a66f4]
[0x7fbf770264a7]
SLES11 jdk11 tcnative 1.2.35 OpenSSL 1.1.1q
org.apache.tomcat.util.net.TestSsl FAILED (crashed)
openjdk version "11.0.15" 2022-04-19
OpenJDK Runtime Environment 18.9 (build 11.0.15+10)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.15+10, mixed mode)
double free or corruption (!prev): 0x00007f4f6bb93040
======= Backtrace: =========
/lib64/libc.so.6(+0x75018)[0x7f4f6a171018]
/lib64/libc.so.6(cfree+0x6c)[0x7f4f6a175f6c]
/.../tcnative-deps/libapr-1.so.0(apr_allocator_destroy+0x1d)[0x7f4f49403aad]
/.../tcnative-deps/libapr-1.so.0(apr_pool_terminate+0x34)[0x7f4f494046f4]
[0x7f4f508b88b0]
RHEL 8 Adoptium jdk11 tcnative 1.2.35 OpenSSL 1.1.1q
Test org.apache.tomcat.util.net.TestClientCert FAILED (crashed)
Since they are rare and happen in various tests and version
combinations, it seems the general shutdown behavior w.r.t. the library
is not yet perfect.
Once the tests for 10.1 complete, I will see, whether I can force the
crashes more often by focusing on the TLS tests in
org.apache.tomcat.util.net.
Best regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: tcnative crashes during shutdown (TC 10.x unit tests)
Posted by Rainer Jung <ra...@kippdata.de>.
Hi Jean-Frederic,
Am 20.07.2022 um 08:47 schrieb jean-frederic clere:
> On 20/07/2022 00:16, Rainer Jung wrote:
>> Roughly the same pattern I saw for TC 10.0 now also seen for TC 10.1.
>
> May be something wrong in apr? Which apr version are you using?
- the same APR library for tcnative 1.2.35 and 2.0.1
- the same tcnative plus APR for TC 9.x and 10.x. Observer crashes for
TC 10, but not for TC 9.
- APR version is 1.7.0 plus r1878355, r1877195, r1872035, r1891198:
*) Restore fix for out-of-bounds array dereference in apr_time_exp*()
functions.
(This issue was addressed as CVE-2017-12613 in APR 1.6.3 and
later 1.6.x releases, but was missing in 1.7.0.) [Stefan Sperling]
*) Don't silently set APR_FOPEN_NOCLEANUP for apr_file_mktemp()
created file to avoid a fd and inode leak when/if later passed to
apr_file_setaside().
[Yann Ylavic]
*) Address some warnings raised by MSVC-32/64.
*) Avoid an overflow on 32 bit platforms. [René Hjortskov Nielsen
<r... hjortskov.dk>]
I think for the current topic it is just as good as 1.7.0 without the
four patches.
Regards,
Rainer
>> Am 18.07.2022 um 12:09 schrieb Rainer Jung:
>>> Hi there,
>>>
>>> this is just an info, at this point probably not a showstopper. The
>>> topic is crashes in tcnative 1.2 and 2.0 for TC 10.0 during shutdown
>>> after TLS unit tests.
>>>
>>> Details:
>>>
>>> I ran the TC unit tests for latest 9.x, 10.x and 10.1.x with tcnative
>>> 1.2.35 OpenSSL 1.1.1q, 1.2.35 OpenSSL 3.0.2 and 2.0.1 OpenSSL 3.0.2.
>>>
>>> I ran the test for a variety of OpenJDK builds (Adoptium, Zulu,
>>> Oracle, RedHat) and versions (latest 1.8.0 except for 10.1, 11, 17
>>> and current 19).
>>>
>>> The platforms where SLES 11, 12 and 15 and RHEL 6, 7 and 8. For RHEL
>>> 7 and 8 there were 48 runs, for the other platforms 39 (no RedHat JDK).
>>>
>>> I only ran about 150 test classes (for NIO and also for NIO2),
>>> because I also ran the full unit tests (about 450 classes) for JSSE
>>> and didn't want to rerun all tests for time and efficiency reasons.
>>>
>>> For TC 10 I observed crashes in TLS tests during shutdown: Out of the
>>> roughly 250 test runs, 5 produced such a crash. For TC 9 I did not
>>> observe a single one. Tests for TC 10.1 are ongoing, until now no
>>> crash, but it is a bit early for a final result. I think the crashes
>>> are not new. All hapened in the TLS tests in org.apache.tomcat.util.net.
>>>
>>> The list of crashes I saw for TC 10.0.23:
>>>
>>> RHEL7 jdk1.8.0 tcnative 1.2.35 OpenSSL 3.0.2
>>> org.apache.tomcat.util.net.TestSsl FAILED (crashed)
>>> openjdk version "1.8.0_332-ea"
>>> OpenJDK Runtime Environment (build 1.8.0_332-ea-b06)
>>> OpenJDK 64-Bit Server VM (build 25.332-b06, mixed mode)
>>> double free or corruption (!prev): 0x00007f473c19df50
>>> ======= Backtrace: =========
>>> /lib64/libc.so.6(+0x7d56d)[0x7f4742aa456d]
>>> /.../tcnative-deps/libapr-1.so.0(apr_allocator_destroy+0x1d)[0x7f472871923d]
>>>
>>> /.../tcnative-deps/libapr-1.so.0(apr_pool_terminate+0x30)[0x7f4728719c10]
>>>
>>> [0x7f472d018427]
>>>
>>> RHEL7 jdk17 tcnative 1.2.35 OpenSSL 1.1.1q
>>> org.apache.tomcat.util.net.TestCustomSslTrustManager FAILED (crashed)
>>> openjdk version "17.0.2" 2022-01-18
>>> OpenJDK Runtime Environment (build 17.0.2+8-86)
>>> OpenJDK 64-Bit Server VM (build 17.0.2+8-86, mixed mode, sharing)
>>> corrupted double-linked list: 0x00007f6bb8001d10
>>> ======= Backtrace: =========
>>> /lib64/libc.so.6(+0x7bfc7)[0x7f6bf481dfc7]
>>> /lib64/libc.so.6(+0x7d774)[0x7f6bf481f774]
>>> /.../tcnative-deps/libapr-1.so.0(apr_allocator_destroy+0x1d)[0x7f6bc543223d]
>>>
>>> /.../tcnative-deps/libapr-1.so.0(apr_pool_terminate+0x30)[0x7f6bc5432c10]
>>>
>>> [0x7f6bd572249a]
>>>
>>> SLES11 oracle_jdk1.8.0 tcnative 2.0.1 OpenSSL 3.0.2
>>> org.apache.tomcat.util.net.TestSsl FAILED (crashed)
>>> java version "1.8.0_331"
>>> Java(TM) SE Runtime Environment (build 1.8.0_331-b09)
>>> Java HotSpot(TM) 64-Bit Server VM (build 25.331-b09, mixed mode)
>>> double free or corruption (!prev): 0x00007fbf88c1de10
>>> ======= Backtrace: =========
>>> /lib64/libc.so.6(+0x75018)[0x7fbf87b35018]
>>> /lib64/libc.so.6(cfree+0x6c)[0x7fbf87b39f6c]
>>> /.../tcnative-deps/libapr-1.so.0(apr_allocator_destroy+0x1d)[0x7fbf718a5aad]
>>>
>>> /.../tcnative-deps/libapr-1.so.0(apr_pool_terminate+0x34)[0x7fbf718a66f4]
>>>
>>> [0x7fbf770264a7]
>>>
>>> SLES11 jdk11 tcnative 1.2.35 OpenSSL 1.1.1q
>>> org.apache.tomcat.util.net.TestSsl FAILED (crashed)
>>> openjdk version "11.0.15" 2022-04-19
>>> OpenJDK Runtime Environment 18.9 (build 11.0.15+10)
>>> OpenJDK 64-Bit Server VM 18.9 (build 11.0.15+10, mixed mode)
>>> double free or corruption (!prev): 0x00007f4f6bb93040
>>> ======= Backtrace: =========
>>> /lib64/libc.so.6(+0x75018)[0x7f4f6a171018]
>>> /lib64/libc.so.6(cfree+0x6c)[0x7f4f6a175f6c]
>>> /.../tcnative-deps/libapr-1.so.0(apr_allocator_destroy+0x1d)[0x7f4f49403aad]
>>>
>>> /.../tcnative-deps/libapr-1.so.0(apr_pool_terminate+0x34)[0x7f4f494046f4]
>>>
>>> [0x7f4f508b88b0]
>>>
>>> RHEL 8 Adoptium jdk11 tcnative 1.2.35 OpenSSL 1.1.1q
>>> Test org.apache.tomcat.util.net.TestClientCert FAILED (crashed)
>>>
>>> Since they are rare and happen in various tests and version
>>> combinations, it seems the general shutdown behavior w.r.t. the
>>> library is not yet perfect.
>>>
>>> Once the tests for 10.1 complete, I will see, whether I can force the
>>> crashes more often by focusing on the TLS tests in
>>> org.apache.tomcat.util.net.
>>>
>>> Best regards,
>>>
>>> Rainer
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: tcnative crashes during shutdown (TC 10.x unit tests)
Posted by jean-frederic clere <jf...@gmail.com>.
On 20/07/2022 00:16, Rainer Jung wrote:
> Roughly the same pattern I saw for TC 10.0 now also seen for TC 10.1.
May be something wrong in apr? Which apr version are you using?
>
> Am 18.07.2022 um 12:09 schrieb Rainer Jung:
>> Hi there,
>>
>> this is just an info, at this point probably not a showstopper. The
>> topic is crashes in tcnative 1.2 and 2.0 for TC 10.0 during shutdown
>> after TLS unit tests.
>>
>> Details:
>>
>> I ran the TC unit tests for latest 9.x, 10.x and 10.1.x with tcnative
>> 1.2.35 OpenSSL 1.1.1q, 1.2.35 OpenSSL 3.0.2 and 2.0.1 OpenSSL 3.0.2.
>>
>> I ran the test for a variety of OpenJDK builds (Adoptium, Zulu,
>> Oracle, RedHat) and versions (latest 1.8.0 except for 10.1, 11, 17 and
>> current 19).
>>
>> The platforms where SLES 11, 12 and 15 and RHEL 6, 7 and 8. For RHEL 7
>> and 8 there were 48 runs, for the other platforms 39 (no RedHat JDK).
>>
>> I only ran about 150 test classes (for NIO and also for NIO2), because
>> I also ran the full unit tests (about 450 classes) for JSSE and didn't
>> want to rerun all tests for time and efficiency reasons.
>>
>> For TC 10 I observed crashes in TLS tests during shutdown: Out of the
>> roughly 250 test runs, 5 produced such a crash. For TC 9 I did not
>> observe a single one. Tests for TC 10.1 are ongoing, until now no
>> crash, but it is a bit early for a final result. I think the crashes
>> are not new. All hapened in the TLS tests in org.apache.tomcat.util.net.
>>
>> The list of crashes I saw for TC 10.0.23:
>>
>> RHEL7 jdk1.8.0 tcnative 1.2.35 OpenSSL 3.0.2
>> org.apache.tomcat.util.net.TestSsl FAILED (crashed)
>> openjdk version "1.8.0_332-ea"
>> OpenJDK Runtime Environment (build 1.8.0_332-ea-b06)
>> OpenJDK 64-Bit Server VM (build 25.332-b06, mixed mode)
>> double free or corruption (!prev): 0x00007f473c19df50
>> ======= Backtrace: =========
>> /lib64/libc.so.6(+0x7d56d)[0x7f4742aa456d]
>> /.../tcnative-deps/libapr-1.so.0(apr_allocator_destroy+0x1d)[0x7f472871923d]
>>
>> /.../tcnative-deps/libapr-1.so.0(apr_pool_terminate+0x30)[0x7f4728719c10]
>> [0x7f472d018427]
>>
>> RHEL7 jdk17 tcnative 1.2.35 OpenSSL 1.1.1q
>> org.apache.tomcat.util.net.TestCustomSslTrustManager FAILED (crashed)
>> openjdk version "17.0.2" 2022-01-18
>> OpenJDK Runtime Environment (build 17.0.2+8-86)
>> OpenJDK 64-Bit Server VM (build 17.0.2+8-86, mixed mode, sharing)
>> corrupted double-linked list: 0x00007f6bb8001d10
>> ======= Backtrace: =========
>> /lib64/libc.so.6(+0x7bfc7)[0x7f6bf481dfc7]
>> /lib64/libc.so.6(+0x7d774)[0x7f6bf481f774]
>> /.../tcnative-deps/libapr-1.so.0(apr_allocator_destroy+0x1d)[0x7f6bc543223d]
>>
>> /.../tcnative-deps/libapr-1.so.0(apr_pool_terminate+0x30)[0x7f6bc5432c10]
>> [0x7f6bd572249a]
>>
>> SLES11 oracle_jdk1.8.0 tcnative 2.0.1 OpenSSL 3.0.2
>> org.apache.tomcat.util.net.TestSsl FAILED (crashed)
>> java version "1.8.0_331"
>> Java(TM) SE Runtime Environment (build 1.8.0_331-b09)
>> Java HotSpot(TM) 64-Bit Server VM (build 25.331-b09, mixed mode)
>> double free or corruption (!prev): 0x00007fbf88c1de10
>> ======= Backtrace: =========
>> /lib64/libc.so.6(+0x75018)[0x7fbf87b35018]
>> /lib64/libc.so.6(cfree+0x6c)[0x7fbf87b39f6c]
>> /.../tcnative-deps/libapr-1.so.0(apr_allocator_destroy+0x1d)[0x7fbf718a5aad]
>>
>> /.../tcnative-deps/libapr-1.so.0(apr_pool_terminate+0x34)[0x7fbf718a66f4]
>> [0x7fbf770264a7]
>>
>> SLES11 jdk11 tcnative 1.2.35 OpenSSL 1.1.1q
>> org.apache.tomcat.util.net.TestSsl FAILED (crashed)
>> openjdk version "11.0.15" 2022-04-19
>> OpenJDK Runtime Environment 18.9 (build 11.0.15+10)
>> OpenJDK 64-Bit Server VM 18.9 (build 11.0.15+10, mixed mode)
>> double free or corruption (!prev): 0x00007f4f6bb93040
>> ======= Backtrace: =========
>> /lib64/libc.so.6(+0x75018)[0x7f4f6a171018]
>> /lib64/libc.so.6(cfree+0x6c)[0x7f4f6a175f6c]
>> /.../tcnative-deps/libapr-1.so.0(apr_allocator_destroy+0x1d)[0x7f4f49403aad]
>>
>> /.../tcnative-deps/libapr-1.so.0(apr_pool_terminate+0x34)[0x7f4f494046f4]
>> [0x7f4f508b88b0]
>>
>> RHEL 8 Adoptium jdk11 tcnative 1.2.35 OpenSSL 1.1.1q
>> Test org.apache.tomcat.util.net.TestClientCert FAILED (crashed)
>>
>> Since they are rare and happen in various tests and version
>> combinations, it seems the general shutdown behavior w.r.t. the
>> library is not yet perfect.
>>
>> Once the tests for 10.1 complete, I will see, whether I can force the
>> crashes more often by focusing on the TLS tests in
>> org.apache.tomcat.util.net.
>>
>> Best regards,
>>
>> Rainer
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: dev-help@tomcat.apache.org
>
--
Cheers
Jean-Frederic
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: tcnative crashes during shutdown (TC 10.x unit tests)
Posted by Mark Thomas <ma...@apache.org>.
On 19/07/2022 23:16, Rainer Jung wrote:
> Roughly the same pattern I saw for TC 10.0 now also seen for TC 10.1.
A failure rate of 1 in 50 is going to make any testing a relatively slow
process. We are going to need at least 250 test runs to get a reasonable
degree of certainty in the results.
I agree that it seems likely that the Connector shutdown code isn't 100%
robust. I took a quick look and cleanup seems to be triggered by GC
which should avoid these sorts of issues. This one may take a while to
track down.
Did you have any success in narrowing down the scope of the tests while
still reproducing the issue?
Mark
>
> Am 18.07.2022 um 12:09 schrieb Rainer Jung:
>> Hi there,
>>
>> this is just an info, at this point probably not a showstopper. The
>> topic is crashes in tcnative 1.2 and 2.0 for TC 10.0 during shutdown
>> after TLS unit tests.
>>
>> Details:
>>
>> I ran the TC unit tests for latest 9.x, 10.x and 10.1.x with tcnative
>> 1.2.35 OpenSSL 1.1.1q, 1.2.35 OpenSSL 3.0.2 and 2.0.1 OpenSSL 3.0.2.
>>
>> I ran the test for a variety of OpenJDK builds (Adoptium, Zulu,
>> Oracle, RedHat) and versions (latest 1.8.0 except for 10.1, 11, 17 and
>> current 19).
>>
>> The platforms where SLES 11, 12 and 15 and RHEL 6, 7 and 8. For RHEL 7
>> and 8 there were 48 runs, for the other platforms 39 (no RedHat JDK).
>>
>> I only ran about 150 test classes (for NIO and also for NIO2), because
>> I also ran the full unit tests (about 450 classes) for JSSE and didn't
>> want to rerun all tests for time and efficiency reasons.
>>
>> For TC 10 I observed crashes in TLS tests during shutdown: Out of the
>> roughly 250 test runs, 5 produced such a crash. For TC 9 I did not
>> observe a single one. Tests for TC 10.1 are ongoing, until now no
>> crash, but it is a bit early for a final result. I think the crashes
>> are not new. All hapened in the TLS tests in org.apache.tomcat.util.net.
>>
>> The list of crashes I saw for TC 10.0.23:
>>
>> RHEL7 jdk1.8.0 tcnative 1.2.35 OpenSSL 3.0.2
>> org.apache.tomcat.util.net.TestSsl FAILED (crashed)
>> openjdk version "1.8.0_332-ea"
>> OpenJDK Runtime Environment (build 1.8.0_332-ea-b06)
>> OpenJDK 64-Bit Server VM (build 25.332-b06, mixed mode)
>> double free or corruption (!prev): 0x00007f473c19df50
>> ======= Backtrace: =========
>> /lib64/libc.so.6(+0x7d56d)[0x7f4742aa456d]
>> /.../tcnative-deps/libapr-1.so.0(apr_allocator_destroy+0x1d)[0x7f472871923d]
>> /.../tcnative-deps/libapr-1.so.0(apr_pool_terminate+0x30)[0x7f4728719c10]
>> [0x7f472d018427]
>>
>> RHEL7 jdk17 tcnative 1.2.35 OpenSSL 1.1.1q
>> org.apache.tomcat.util.net.TestCustomSslTrustManager FAILED (crashed)
>> openjdk version "17.0.2" 2022-01-18
>> OpenJDK Runtime Environment (build 17.0.2+8-86)
>> OpenJDK 64-Bit Server VM (build 17.0.2+8-86, mixed mode, sharing)
>> corrupted double-linked list: 0x00007f6bb8001d10
>> ======= Backtrace: =========
>> /lib64/libc.so.6(+0x7bfc7)[0x7f6bf481dfc7]
>> /lib64/libc.so.6(+0x7d774)[0x7f6bf481f774]
>> /.../tcnative-deps/libapr-1.so.0(apr_allocator_destroy+0x1d)[0x7f6bc543223d]
>> /.../tcnative-deps/libapr-1.so.0(apr_pool_terminate+0x30)[0x7f6bc5432c10]
>> [0x7f6bd572249a]
>>
>> SLES11 oracle_jdk1.8.0 tcnative 2.0.1 OpenSSL 3.0.2
>> org.apache.tomcat.util.net.TestSsl FAILED (crashed)
>> java version "1.8.0_331"
>> Java(TM) SE Runtime Environment (build 1.8.0_331-b09)
>> Java HotSpot(TM) 64-Bit Server VM (build 25.331-b09, mixed mode)
>> double free or corruption (!prev): 0x00007fbf88c1de10
>> ======= Backtrace: =========
>> /lib64/libc.so.6(+0x75018)[0x7fbf87b35018]
>> /lib64/libc.so.6(cfree+0x6c)[0x7fbf87b39f6c]
>> /.../tcnative-deps/libapr-1.so.0(apr_allocator_destroy+0x1d)[0x7fbf718a5aad]
>> /.../tcnative-deps/libapr-1.so.0(apr_pool_terminate+0x34)[0x7fbf718a66f4]
>> [0x7fbf770264a7]
>>
>> SLES11 jdk11 tcnative 1.2.35 OpenSSL 1.1.1q
>> org.apache.tomcat.util.net.TestSsl FAILED (crashed)
>> openjdk version "11.0.15" 2022-04-19
>> OpenJDK Runtime Environment 18.9 (build 11.0.15+10)
>> OpenJDK 64-Bit Server VM 18.9 (build 11.0.15+10, mixed mode)
>> double free or corruption (!prev): 0x00007f4f6bb93040
>> ======= Backtrace: =========
>> /lib64/libc.so.6(+0x75018)[0x7f4f6a171018]
>> /lib64/libc.so.6(cfree+0x6c)[0x7f4f6a175f6c]
>> /.../tcnative-deps/libapr-1.so.0(apr_allocator_destroy+0x1d)[0x7f4f49403aad]
>> /.../tcnative-deps/libapr-1.so.0(apr_pool_terminate+0x34)[0x7f4f494046f4]
>> [0x7f4f508b88b0]
>>
>> RHEL 8 Adoptium jdk11 tcnative 1.2.35 OpenSSL 1.1.1q
>> Test org.apache.tomcat.util.net.TestClientCert FAILED (crashed)
>>
>> Since they are rare and happen in various tests and version
>> combinations, it seems the general shutdown behavior w.r.t. the
>> library is not yet perfect.
>>
>> Once the tests for 10.1 complete, I will see, whether I can force the
>> crashes more often by focusing on the TLS tests in
>> org.apache.tomcat.util.net.
>>
>> Best regards,
>>
>> Rainer
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: dev-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: tcnative crashes during shutdown (TC 10.x unit tests)
Posted by Rainer Jung <ra...@kippdata.de>.
Roughly the same pattern I saw for TC 10.0 now also seen for TC 10.1.
Am 18.07.2022 um 12:09 schrieb Rainer Jung:
> Hi there,
>
> this is just an info, at this point probably not a showstopper. The
> topic is crashes in tcnative 1.2 and 2.0 for TC 10.0 during shutdown
> after TLS unit tests.
>
> Details:
>
> I ran the TC unit tests for latest 9.x, 10.x and 10.1.x with tcnative
> 1.2.35 OpenSSL 1.1.1q, 1.2.35 OpenSSL 3.0.2 and 2.0.1 OpenSSL 3.0.2.
>
> I ran the test for a variety of OpenJDK builds (Adoptium, Zulu, Oracle,
> RedHat) and versions (latest 1.8.0 except for 10.1, 11, 17 and current 19).
>
> The platforms where SLES 11, 12 and 15 and RHEL 6, 7 and 8. For RHEL 7
> and 8 there were 48 runs, for the other platforms 39 (no RedHat JDK).
>
> I only ran about 150 test classes (for NIO and also for NIO2), because I
> also ran the full unit tests (about 450 classes) for JSSE and didn't
> want to rerun all tests for time and efficiency reasons.
>
> For TC 10 I observed crashes in TLS tests during shutdown: Out of the
> roughly 250 test runs, 5 produced such a crash. For TC 9 I did not
> observe a single one. Tests for TC 10.1 are ongoing, until now no crash,
> but it is a bit early for a final result. I think the crashes are not
> new. All hapened in the TLS tests in org.apache.tomcat.util.net.
>
> The list of crashes I saw for TC 10.0.23:
>
> RHEL7 jdk1.8.0 tcnative 1.2.35 OpenSSL 3.0.2
> org.apache.tomcat.util.net.TestSsl FAILED (crashed)
> openjdk version "1.8.0_332-ea"
> OpenJDK Runtime Environment (build 1.8.0_332-ea-b06)
> OpenJDK 64-Bit Server VM (build 25.332-b06, mixed mode)
> double free or corruption (!prev): 0x00007f473c19df50
> ======= Backtrace: =========
> /lib64/libc.so.6(+0x7d56d)[0x7f4742aa456d]
> /.../tcnative-deps/libapr-1.so.0(apr_allocator_destroy+0x1d)[0x7f472871923d]
>
> /.../tcnative-deps/libapr-1.so.0(apr_pool_terminate+0x30)[0x7f4728719c10]
> [0x7f472d018427]
>
> RHEL7 jdk17 tcnative 1.2.35 OpenSSL 1.1.1q
> org.apache.tomcat.util.net.TestCustomSslTrustManager FAILED (crashed)
> openjdk version "17.0.2" 2022-01-18
> OpenJDK Runtime Environment (build 17.0.2+8-86)
> OpenJDK 64-Bit Server VM (build 17.0.2+8-86, mixed mode, sharing)
> corrupted double-linked list: 0x00007f6bb8001d10
> ======= Backtrace: =========
> /lib64/libc.so.6(+0x7bfc7)[0x7f6bf481dfc7]
> /lib64/libc.so.6(+0x7d774)[0x7f6bf481f774]
> /.../tcnative-deps/libapr-1.so.0(apr_allocator_destroy+0x1d)[0x7f6bc543223d]
>
> /.../tcnative-deps/libapr-1.so.0(apr_pool_terminate+0x30)[0x7f6bc5432c10]
> [0x7f6bd572249a]
>
> SLES11 oracle_jdk1.8.0 tcnative 2.0.1 OpenSSL 3.0.2
> org.apache.tomcat.util.net.TestSsl FAILED (crashed)
> java version "1.8.0_331"
> Java(TM) SE Runtime Environment (build 1.8.0_331-b09)
> Java HotSpot(TM) 64-Bit Server VM (build 25.331-b09, mixed mode)
> double free or corruption (!prev): 0x00007fbf88c1de10
> ======= Backtrace: =========
> /lib64/libc.so.6(+0x75018)[0x7fbf87b35018]
> /lib64/libc.so.6(cfree+0x6c)[0x7fbf87b39f6c]
> /.../tcnative-deps/libapr-1.so.0(apr_allocator_destroy+0x1d)[0x7fbf718a5aad]
>
> /.../tcnative-deps/libapr-1.so.0(apr_pool_terminate+0x34)[0x7fbf718a66f4]
> [0x7fbf770264a7]
>
> SLES11 jdk11 tcnative 1.2.35 OpenSSL 1.1.1q
> org.apache.tomcat.util.net.TestSsl FAILED (crashed)
> openjdk version "11.0.15" 2022-04-19
> OpenJDK Runtime Environment 18.9 (build 11.0.15+10)
> OpenJDK 64-Bit Server VM 18.9 (build 11.0.15+10, mixed mode)
> double free or corruption (!prev): 0x00007f4f6bb93040
> ======= Backtrace: =========
> /lib64/libc.so.6(+0x75018)[0x7f4f6a171018]
> /lib64/libc.so.6(cfree+0x6c)[0x7f4f6a175f6c]
> /.../tcnative-deps/libapr-1.so.0(apr_allocator_destroy+0x1d)[0x7f4f49403aad]
>
> /.../tcnative-deps/libapr-1.so.0(apr_pool_terminate+0x34)[0x7f4f494046f4]
> [0x7f4f508b88b0]
>
> RHEL 8 Adoptium jdk11 tcnative 1.2.35 OpenSSL 1.1.1q
> Test org.apache.tomcat.util.net.TestClientCert FAILED (crashed)
>
> Since they are rare and happen in various tests and version
> combinations, it seems the general shutdown behavior w.r.t. the library
> is not yet perfect.
>
> Once the tests for 10.1 complete, I will see, whether I can force the
> crashes more often by focusing on the TLS tests in
> org.apache.tomcat.util.net.
>
> Best regards,
>
> Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org