You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Eduardo Quintanilla <eq...@bnext.mx> on 2019/08/21 21:36:44 UTC
Tomcat 8.5 - Native Library - Crash
We have been getting some crashes in Tomcat 8.5.43 lately.
The environment is:
* JDK 1.8.0_202
* Windows Server 2012 R2
The logs shows a tcnative exception.
Crash Log:
Current thread JavaThread "https-openssl-apr-8081-exec-298" daemon _thread_in_native
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 15034 org.apache.tomcat.jni.Socket.sendb(JLjava/nio/ByteBuffer;II)I (0 bytes) @ 0x0000000002962ddf [0x0000000002962d80+0x5f]
J 29164 C2 org.apache.tomcat.util.net.AprEndpoint$AprSocketWrapper.doWrite(ZLjava/nio/ByteBuffer;)V (242 bytes) @ 0x0000000006d6933c [0x0000000006d68b20+0x81c]
J 27479 C2 org.apache.tomcat.util.net.SocketWrapperBase.flush(Z)Z (20 bytes) @ 0x00000000053f6144 [0x00000000053f60a0+0xa4]
J 24666 C2 org.apache.coyote.http2.Http2OutputBuffer.doWrite(Ljava/nio/ByteBuffer;)I (28 bytes) @ 0x0000000006b51b74 [0x0000000006b51660+0x514]
J 24689 C2 org.apache.catalina.connector.OutputBuffer.writeBytes([BII)V (38 bytes) @ 0x00000000031c09f8 [0x00000000031c04c0+0x538]
J 27348 C2 org.apache.catalina.connector.CoyoteOutputStream.write([BII)V (26 bytes) @ 0x00000000071d51a8 [0x00000000071d5120+0x88]
Tomcat logs:
org.apache.catalina.connector.ClientAbortException: org.apache.coyote.CloseNowException: Connection [214], Stream [279], This stream is not writable
I was able to cause a crash in my development environment but I do not think that is the same error, I got the same error with version 9.0.24
Sample app[1]
Crash logs:
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j org.apache.tomcat.jni.Address.get(IJ)J+0
j org.apache.tomcat.util.net.AprEndpoint$AprSocketWrapper.populateRemoteAddr()V+21
J 4645 C1 org.apache.coyote.AbstractProcessor.action(Lorg/apache/coyote/ActionCode;Ljava/lang/Object;)V (1019 bytes) @ 0x00000000024a4774 [0x000000000249f700+0x5074]
j org.apache.coyote.Request.action(Lorg/apache/coyote/ActionCode;Ljava/lang/Object;)V+31
j org.apache.catalina.connector.Request.getRemoteAddr()Ljava/lang/String;+18
j org.apache.catalina.connector.RequestFacade.getRemoteAddr()Ljava/lang/String;+27
j org.springframework.web.servlet.FrameworkServlet.publishRequestHandledEvent(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;JLjava/lang/Throwable;)V+37
Dev environment:
* Tomcat 8.5.43
* Windows Server 2012 R2
* Windows 10.0.18362.295
The keystore was created with:
keytool -genkey -keyalg RSA -alias tomcat -keystore tomcat.jks -storepass tomcat -validity 360 -keysize 2048
The HTTPS connector is configured with:
<Connector port="8443" protocol="org.apache.coyote.http11.Http11AprProtocol"
maxThreads="150" SSLEnabled="true" >
<UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" />
<SSLHostConfig>
<Certificate certificateKeystoreFile="conf/tomcat.jks" certificateKeystorePassword="tomcat" certificateKeyPassword="tomcat"
type="RSA" />
</SSLHostConfig>
</Connector>
I am running the tests with Gatling 3.2.0[3] configured with requestTimeout = 60 so it causes client disconnections.
Sample test[2]
The issue could be fixed in the app code but I think that is also a Tomcat bug.
Any tipos or suggestions?
[1][ https://github.com/lalo-mx/app-tomcat-sample/blob/master/src/main/java/sample/AppController.java]
[2][ https://gist.github.com/lalo-mx/0a690290f6aac5017f15a65fc100eaa2]
[3][ https://gatling.io/open-source]
Eduardo Quintanilla
Software Developer
The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer.
La información transmitida está destinada únicamente a la persona o entidad a quien que va dirigida y puede contener información confidencial y/o material privilegiado. Cualquier revisión, retransmisión, difusión u otros usos, o cualquier acción tomada por personas o entidades distintas al destinatario basándose en esta información está prohibida. Si usted recibe este mensaje por error, por favor contacte al remitente y elimine el material de cualquier computadora.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat 8.5 - Native Library - Crash
Posted by Eduardo Quintanilla <eq...@bnext.mx>.
Tomcat Native versión 1.2.23.0, the included with Tomcat.
Eduardo Quintanilla
Software Developer
-----Original Message-----
From: Mark Thomas <ma...@apache.org>
Sent: jueves, 22 de agosto de 2019 1:11 a. m.
To: Tomcat Users List <us...@tomcat.apache.org>
Subject: Re: Tomcat 8.5 - Native Library - Crash
Tomcat Native version?
Mark
On August 21, 2019 9:36:44 PM UTC, Eduardo Quintanilla <eq...@bnext.mx> wrote:
>We have been getting some crashes in Tomcat 8.5.43 lately.
>
>The environment is:
>* JDK 1.8.0_202
>* Windows Server 2012 R2
>
>The logs shows a tcnative exception.
>
>Crash Log:
>Current thread JavaThread "https-openssl-apr-8081-exec-298" daemon
>_thread_in_native Java frames: (J=compiled Java code, j=interpreted,
>Vv=VM code) J 15034
>org.apache.tomcat.jni.Socket.sendb(JLjava/nio/ByteBuffer;II)I
>(0 bytes) @ 0x0000000002962ddf [0x0000000002962d80+0x5f] J 29164 C2
>org.apache.tomcat.util.net.AprEndpoint$AprSocketWrapper.doWrite(ZLjava/
>nio/ByteBuffer;)V
>(242 bytes) @ 0x0000000006d6933c [0x0000000006d68b20+0x81c] J 27479 C2
>org.apache.tomcat.util.net.SocketWrapperBase.flush(Z)Z (20
>bytes) @ 0x00000000053f6144 [0x00000000053f60a0+0xa4] J 24666 C2
>org.apache.coyote.http2.Http2OutputBuffer.doWrite(Ljava/nio/ByteBuffer;
>)I
>(28 bytes) @ 0x0000000006b51b74 [0x0000000006b51660+0x514] J 24689 C2
>org.apache.catalina.connector.OutputBuffer.writeBytes([BII)V
>(38 bytes) @ 0x00000000031c09f8 [0x00000000031c04c0+0x538] J 27348 C2
>org.apache.catalina.connector.CoyoteOutputStream.write([BII)V (26
>bytes) @ 0x00000000071d51a8 [0x00000000071d5120+0x88]
>
>Tomcat logs:
>org.apache.catalina.connector.ClientAbortException:
>org.apache.coyote.CloseNowException: Connection [214], Stream [279],
>This stream is not writable
>
>
>I was able to cause a crash in my development environment but I do not
>think that is the same error, I got the same error with version 9.0.24
>Sample app[1] Crash logs:
>Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) j
>org.apache.tomcat.jni.Address.get(IJ)J+0
>j
>org.apache.tomcat.util.net.AprEndpoint$AprSocketWrapper.populateRemoteA
>ddr()V+21
>J 4645 C1
>org.apache.coyote.AbstractProcessor.action(Lorg/apache/coyote/ActionCod
>e;Ljava/lang/Object;)V
>(1019 bytes) @ 0x00000000024a4774 [0x000000000249f700+0x5074] j
>org.apache.coyote.Request.action(Lorg/apache/coyote/ActionCode;Ljava/la
>ng/Object;)V+31
>j
>org.apache.catalina.connector.Request.getRemoteAddr()Ljava/lang/String;
>+18
>j
>org.apache.catalina.connector.RequestFacade.getRemoteAddr()Ljava/lang/S
>tring;+27
>j
>org.springframework.web.servlet.FrameworkServlet.publishRequestHandledE
>vent(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpSer
>vletResponse;JLjava/lang/Throwable;)V+37
>
>
>Dev environment:
>* Tomcat 8.5.43
>* Windows Server 2012 R2
>* Windows 10.0.18362.295
>
>The keystore was created with:
>
>keytool -genkey -keyalg RSA -alias tomcat -keystore tomcat.jks
>-storepass tomcat -validity 360 -keysize 2048
>
>
>The HTTPS connector is configured with:
>
><Connector port="8443"
>protocol="org.apache.coyote.http11.Http11AprProtocol"
> maxThreads="150" SSLEnabled="true" >
> <UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" />
> <SSLHostConfig>
><Certificate certificateKeystoreFile="conf/tomcat.jks"
>certificateKeystorePassword="tomcat" certificateKeyPassword="tomcat"
> type="RSA" />
> </SSLHostConfig>
> </Connector>
>
>I am running the tests with Gatling 3.2.0[3] configured with
>requestTimeout = 60 so it causes client disconnections.
>Sample test[2]
>
>The issue could be fixed in the app code but I think that is also a
>Tomcat bug.
>Any tipos or suggestions?
>
>[1][
>https://github.com/lalo-mx/app-tomcat-sample/blob/master/src/main/java/
>sample/AppController.java] [2][
>https://gist.github.com/lalo-mx/0a690290f6aac5017f15a65fc100eaa2]
>[3][ https://gatling.io/open-source]
>
>Eduardo Quintanilla
>Software Developer
>
>
>The information transmitted is intended only for the person or entity
>to which it is addressed and may contain confidential and/or privileged
>material. Any review, retransmission, dissemination or other use of, or
>taking of any action in reliance upon, this information by persons or
>entities other than the intended recipient is prohibited. If you
>received this in error, please contact the sender and delete the
>material from any computer.
>La información transmitida está destinada únicamente a la persona o
>entidad a quien que va dirigida y puede contener información
>confidencial y/o material privilegiado. Cualquier revisión,
>retransmisión, difusión u otros usos, o cualquier acción tomada por
>personas o entidades distintas al destinatario basándose en esta
>información está prohibida. Si usted recibe este mensaje por error, por
>favor contacte al remitente y elimine el material de cualquier
>computadora.
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>For additional commands, e-mail: users-help@tomcat.apache.org
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer.
La información transmitida está destinada únicamente a la persona o entidad a quien que va dirigida y puede contener información confidencial y/o material privilegiado. Cualquier revisión, retransmisión, difusión u otros usos, o cualquier acción tomada por personas o entidades distintas al destinatario basándose en esta información está prohibida. Si usted recibe este mensaje por error, por favor contacte al remitente y elimine el material de cualquier computadora.
Re: Tomcat 8.5 - Native Library - Crash
Posted by Mark Thomas <ma...@apache.org>.
Tomcat Native version?
Mark
On August 21, 2019 9:36:44 PM UTC, Eduardo Quintanilla <eq...@bnext.mx> wrote:
>We have been getting some crashes in Tomcat 8.5.43 lately.
>
>The environment is:
>* JDK 1.8.0_202
>* Windows Server 2012 R2
>
>The logs shows a tcnative exception.
>
>Crash Log:
>Current thread JavaThread "https-openssl-apr-8081-exec-298" daemon
>_thread_in_native
>Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
>J 15034 org.apache.tomcat.jni.Socket.sendb(JLjava/nio/ByteBuffer;II)I
>(0 bytes) @ 0x0000000002962ddf [0x0000000002962d80+0x5f]
>J 29164 C2
>org.apache.tomcat.util.net.AprEndpoint$AprSocketWrapper.doWrite(ZLjava/nio/ByteBuffer;)V
>(242 bytes) @ 0x0000000006d6933c [0x0000000006d68b20+0x81c]
>J 27479 C2 org.apache.tomcat.util.net.SocketWrapperBase.flush(Z)Z (20
>bytes) @ 0x00000000053f6144 [0x00000000053f60a0+0xa4]
>J 24666 C2
>org.apache.coyote.http2.Http2OutputBuffer.doWrite(Ljava/nio/ByteBuffer;)I
>(28 bytes) @ 0x0000000006b51b74 [0x0000000006b51660+0x514]
>J 24689 C2 org.apache.catalina.connector.OutputBuffer.writeBytes([BII)V
>(38 bytes) @ 0x00000000031c09f8 [0x00000000031c04c0+0x538]
>J 27348 C2
>org.apache.catalina.connector.CoyoteOutputStream.write([BII)V (26
>bytes) @ 0x00000000071d51a8 [0x00000000071d5120+0x88]
>
>Tomcat logs:
>org.apache.catalina.connector.ClientAbortException:
>org.apache.coyote.CloseNowException: Connection [214], Stream [279],
>This stream is not writable
>
>
>I was able to cause a crash in my development environment but I do not
>think that is the same error, I got the same error with version 9.0.24
>Sample app[1]
>Crash logs:
>Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
>j org.apache.tomcat.jni.Address.get(IJ)J+0
>j
>org.apache.tomcat.util.net.AprEndpoint$AprSocketWrapper.populateRemoteAddr()V+21
>J 4645 C1
>org.apache.coyote.AbstractProcessor.action(Lorg/apache/coyote/ActionCode;Ljava/lang/Object;)V
>(1019 bytes) @ 0x00000000024a4774 [0x000000000249f700+0x5074]
>j
>org.apache.coyote.Request.action(Lorg/apache/coyote/ActionCode;Ljava/lang/Object;)V+31
>j
>org.apache.catalina.connector.Request.getRemoteAddr()Ljava/lang/String;+18
>j
>org.apache.catalina.connector.RequestFacade.getRemoteAddr()Ljava/lang/String;+27
>j
>org.springframework.web.servlet.FrameworkServlet.publishRequestHandledEvent(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;JLjava/lang/Throwable;)V+37
>
>
>Dev environment:
>* Tomcat 8.5.43
>* Windows Server 2012 R2
>* Windows 10.0.18362.295
>
>The keystore was created with:
>
>keytool -genkey -keyalg RSA -alias tomcat -keystore tomcat.jks
>-storepass tomcat -validity 360 -keysize 2048
>
>
>The HTTPS connector is configured with:
>
><Connector port="8443"
>protocol="org.apache.coyote.http11.Http11AprProtocol"
> maxThreads="150" SSLEnabled="true" >
> <UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" />
> <SSLHostConfig>
><Certificate certificateKeystoreFile="conf/tomcat.jks"
>certificateKeystorePassword="tomcat" certificateKeyPassword="tomcat"
> type="RSA" />
> </SSLHostConfig>
> </Connector>
>
>I am running the tests with Gatling 3.2.0[3] configured with
>requestTimeout = 60 so it causes client disconnections.
>Sample test[2]
>
>The issue could be fixed in the app code but I think that is also a
>Tomcat bug.
>Any tipos or suggestions?
>
>[1][
>https://github.com/lalo-mx/app-tomcat-sample/blob/master/src/main/java/sample/AppController.java]
>[2][ https://gist.github.com/lalo-mx/0a690290f6aac5017f15a65fc100eaa2]
>[3][ https://gatling.io/open-source]
>
>Eduardo Quintanilla
>Software Developer
>
>
>The information transmitted is intended only for the person or entity
>to which it is addressed and may contain confidential and/or privileged
>material. Any review, retransmission, dissemination or other use of, or
>taking of any action in reliance upon, this information by persons or
>entities other than the intended recipient is prohibited. If you
>received this in error, please contact the sender and delete the
>material from any computer.
>La información transmitida está destinada únicamente a la persona o
>entidad a quien que va dirigida y puede contener información
>confidencial y/o material privilegiado. Cualquier revisión,
>retransmisión, difusión u otros usos, o cualquier acción tomada por
>personas o entidades distintas al destinatario basándose en esta
>información está prohibida. Si usted recibe este mensaje por error, por
>favor contacte al remitente y elimine el material de cualquier
>computadora.
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>For additional commands, e-mail: users-help@tomcat.apache.org
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 8.5 - Native Library - Crash
Posted by Mark Thomas <ma...@apache.org>.
On 26/08/2019 16:14, Eduardo Quintanilla wrote:
> Any tips for building Tomcat Native on Windows 10?
https://cwiki.apache.org/confluence/display/TOMCAT/Building+the+Tomcat+Native+Connector+binaries+for+Windows
You can build it with later versions of Visual Studio but that will end
up creating additional dependencies on various Windows DLLs.
Mark
>
> Best regards,
> Eduardo Quintanilla
> Software Developer
> Block Networks<http://www.blocknetworks.com.mx/>
>
> From: Christopher Schultz <ch...@christopherschultz.net>
> Sent: jueves, 22 de agosto de 2019 11:57 a. m.
> To: users@tomcat.apache.org
> Subject: Re: Tomcat 8.5 - Native Library - Crash
>
>
> Eduardo,
>
> On 8/22/19 10:18, Eduardo Quintanilla wrote:
>> Chris,
>>
>> The crash dump and logs[1] from my dev environment.
>>
>> [1]https://1drv.ms/u/s!AjSSUmqOgKmfi1g7z4wUvfYPksCk?e=CLH6VF
>
> Okay, as I suspected, it's a SIGSEGV, or what Java programmers would
> call a NullPointerException.
>
> The C code is computing a pointer with a 44 byte offset to a NULL
> pointer and trying to dereference it, which causes the problem.
>
> I'm not sure which field has a 44-byte offset into the structure it's
> supposed to be contained in. That highly depends upon the
> architecture, compiler, etc. and those are the things I don't have
> access to.
>
> If someone in the community is able to build tcnative for Windows in
> the same way[1] that the Tomcat team builds it, can you identify the
> following:
>
> 1. What line of code corresponds to tcnative-1.dll+0x14d32
> 2. Which structure is being used
> 3. Which field is 44 bytes into that structure
>
> Knowing all those things should make this a very easy fix.
>
> Of course, the "fix" will involve throwing an exception back over into
> the Java space, to the application will still throw an error. But at
> least it won't bring-down the whole JVM.
>
>>> Do you always get the "stream is not writable" error before the
>>> JVM performs the crash-dump?
>>
>> Yes but only in the Tomcat of the production environment.
>
> That's ... interesting. If Tomcat knows it can't write to the
> stream... why is it still writing to the stream?
>
> Note that the crash dump you posted is for an error in
> jni.Address.get() and not the one for Socket.sendb(). If you can
> generate a crash dump in Socket.sendb() that would be helpful, too.
>
> Also, please see:
>
> https://bz.apache.org/bugzilla/show_bug.cgi?id=62626
> https://bz.apache.org/bugzilla/show_bug.cgi?id=63405
>
> -chris
>
>>
>> Best regards, Eduardo Quintanilla Software Developer
>>
>> From: Christopher Schultz <ch...@christopherschultz.net>> Sent:
>> jueves, 22 de agosto de 2019 8:25 a. m. To:
>> users@tomcat.apache.org<ma...@tomcat.apache.org> Subject: Re: Tomcat 8.5 - Native Library -
>> Crash
>>
>>
>> Eduardo,
>>
>> On 8/21/19 17:36, Eduardo Quintanilla wrote:
>>> We have been getting some crashes in Tomcat 8.5.43 lately.
>>>
>>> The environment is: * JDK 1.8.0_202 * Windows Server 2012 R2
>>>
>>> The logs shows a tcnative exception.
>>>
>>> Crash Log: Current thread JavaThread
>>> "https-openssl-apr-8081-exec-298" daemon _thread_in_native Java
>>> frames: (J=compiled Java code, j=interpreted, Vv=VM code) J
>>> 15034
>>> org.apache.tomcat.jni.Socket.sendb(JLjava/nio/ByteBuffer;II)I (0
>>> bytes) @ 0x0000000002962ddf [0x0000000002962d80+0x5f] J 29164 C2
>>> org.apache.tomcat.util.net.AprEndpoint$AprSocketWrapper.doWrite(ZLjav
> a
>>
>>>
> /nio/ByteBuffer;)V
>>> (242 bytes) @ 0x0000000006d6933c [0x0000000006d68b20+0x81c] J
>>> 27479 C2 org.apache.tomcat.util.net.SocketWrapperBase.flush(Z)Z
>>> (20 bytes) @ 0x00000000053f6144 [0x00000000053f60a0+0xa4] J 24666
>>> C2
>>> org.apache.coyote.http2.Http2OutputBuffer.doWrite(Ljava/nio/ByteBuffe
> r
>>
>>>
> ;)I
>>> (28 bytes) @ 0x0000000006b51b74 [0x0000000006b51660+0x514] J
>>> 24689 C2
>>> org.apache.catalina.connector.OutputBuffer.writeBytes([BII)V (38
>>> bytes) @ 0x00000000031c09f8 [0x00000000031c04c0+0x538] J 27348
>>> C2 org.apache.catalina.connector.CoyoteOutputStream.write([BII)V
>>> (26 bytes) @ 0x00000000071d51a8 [0x00000000071d5120+0x88]
>>
>> What's the actual error? Can you post the entire crash dump?
>>
>>> Tomcat logs: org.apache.catalina.connector.ClientAbortException:
>>> org.apache.coyote.CloseNowException: Connection [214], Stream
>>> [279], This stream is not writable
>>
>> Do you always get the "stream is not writable" error before the
>> JVM performs the crash-dump?
>>
>> I believe there is at least one lingering SEGV hiding in
>> Socket.sendb(), possibly only on Windows. I don't have a Windows
>> development environment to be able to disassemble the crash dump
>> and locate the line of code being executed when the SEGV occurs, so
>> I have added NULL-checks to everything I can find in there but it
>> looks like it's not enough. Something is missing, somewhere.
>>
>> -chris
>>
>>
>>
>> The information transmitted is intended only for the person or
>> entity to which it is addressed and may contain confidential and/or
>> privileged material. Any review, retransmission, dissemination or
>> other use of, or taking of any action in reliance upon, this
>> information by persons or entities other than the intended
>> recipient is prohibited. If you received this in error, please
>> contact the sender and delete the material from any computer. La
>> información transmitida está destinada únicamente a la persona o
>> entidad a quien que va dirigida y puede contener información
>> confidencial y/o material privilegiado. Cualquier revisión,
>> retransmisión, difusión u otros usos, o cualquier acción tomada por
>> personas o entidades distintas al destinatario basándose en esta
>> información está prohibida. Si usted recibe este mensaje por error,
>> por favor contacte al remitente y elimine el material de cualquier
>> computadora.
>>
>
>
>
> The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer.
> La información transmitida está destinada únicamente a la persona o entidad a quien que va dirigida y puede contener información confidencial y/o material privilegiado. Cualquier revisión, retransmisión, difusión u otros usos, o cualquier acción tomada por personas o entidades distintas al destinatario basándose en esta información está prohibida. Si usted recibe este mensaje por error, por favor contacte al remitente y elimine el material de cualquier computadora.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat 8.5 - Native Library - Crash
Posted by Eduardo Quintanilla <eq...@bnext.mx>.
Any tips for building Tomcat Native on Windows 10?
Best regards,
Eduardo Quintanilla
Software Developer
Block Networks<http://www.blocknetworks.com.mx/>
From: Christopher Schultz <ch...@christopherschultz.net>
Sent: jueves, 22 de agosto de 2019 11:57 a. m.
To: users@tomcat.apache.org
Subject: Re: Tomcat 8.5 - Native Library - Crash
Eduardo,
On 8/22/19 10:18, Eduardo Quintanilla wrote:
> Chris,
>
> The crash dump and logs[1] from my dev environment.
>
> [1]https://1drv.ms/u/s!AjSSUmqOgKmfi1g7z4wUvfYPksCk?e=CLH6VF
Okay, as I suspected, it's a SIGSEGV, or what Java programmers would
call a NullPointerException.
The C code is computing a pointer with a 44 byte offset to a NULL
pointer and trying to dereference it, which causes the problem.
I'm not sure which field has a 44-byte offset into the structure it's
supposed to be contained in. That highly depends upon the
architecture, compiler, etc. and those are the things I don't have
access to.
If someone in the community is able to build tcnative for Windows in
the same way[1] that the Tomcat team builds it, can you identify the
following:
1. What line of code corresponds to tcnative-1.dll+0x14d32
2. Which structure is being used
3. Which field is 44 bytes into that structure
Knowing all those things should make this a very easy fix.
Of course, the "fix" will involve throwing an exception back over into
the Java space, to the application will still throw an error. But at
least it won't bring-down the whole JVM.
>> Do you always get the "stream is not writable" error before the
>> JVM performs the crash-dump?
>
> Yes but only in the Tomcat of the production environment.
That's ... interesting. If Tomcat knows it can't write to the
stream... why is it still writing to the stream?
Note that the crash dump you posted is for an error in
jni.Address.get() and not the one for Socket.sendb(). If you can
generate a crash dump in Socket.sendb() that would be helpful, too.
Also, please see:
https://bz.apache.org/bugzilla/show_bug.cgi?id=62626
https://bz.apache.org/bugzilla/show_bug.cgi?id=63405
-chris
>
> Best regards, Eduardo Quintanilla Software Developer
>
> From: Christopher Schultz <ch...@christopherschultz.net>> Sent:
> jueves, 22 de agosto de 2019 8:25 a. m. To:
> users@tomcat.apache.org<ma...@tomcat.apache.org> Subject: Re: Tomcat 8.5 - Native Library -
> Crash
>
>
> Eduardo,
>
> On 8/21/19 17:36, Eduardo Quintanilla wrote:
>> We have been getting some crashes in Tomcat 8.5.43 lately.
>>
>> The environment is: * JDK 1.8.0_202 * Windows Server 2012 R2
>>
>> The logs shows a tcnative exception.
>>
>> Crash Log: Current thread JavaThread
>> "https-openssl-apr-8081-exec-298" daemon _thread_in_native Java
>> frames: (J=compiled Java code, j=interpreted, Vv=VM code) J
>> 15034
>> org.apache.tomcat.jni.Socket.sendb(JLjava/nio/ByteBuffer;II)I (0
>> bytes) @ 0x0000000002962ddf [0x0000000002962d80+0x5f] J 29164 C2
>> org.apache.tomcat.util.net.AprEndpoint$AprSocketWrapper.doWrite(ZLjav
a
>
>>
/nio/ByteBuffer;)V
>> (242 bytes) @ 0x0000000006d6933c [0x0000000006d68b20+0x81c] J
>> 27479 C2 org.apache.tomcat.util.net.SocketWrapperBase.flush(Z)Z
>> (20 bytes) @ 0x00000000053f6144 [0x00000000053f60a0+0xa4] J 24666
>> C2
>> org.apache.coyote.http2.Http2OutputBuffer.doWrite(Ljava/nio/ByteBuffe
r
>
>>
;)I
>> (28 bytes) @ 0x0000000006b51b74 [0x0000000006b51660+0x514] J
>> 24689 C2
>> org.apache.catalina.connector.OutputBuffer.writeBytes([BII)V (38
>> bytes) @ 0x00000000031c09f8 [0x00000000031c04c0+0x538] J 27348
>> C2 org.apache.catalina.connector.CoyoteOutputStream.write([BII)V
>> (26 bytes) @ 0x00000000071d51a8 [0x00000000071d5120+0x88]
>
> What's the actual error? Can you post the entire crash dump?
>
>> Tomcat logs: org.apache.catalina.connector.ClientAbortException:
>> org.apache.coyote.CloseNowException: Connection [214], Stream
>> [279], This stream is not writable
>
> Do you always get the "stream is not writable" error before the
> JVM performs the crash-dump?
>
> I believe there is at least one lingering SEGV hiding in
> Socket.sendb(), possibly only on Windows. I don't have a Windows
> development environment to be able to disassemble the crash dump
> and locate the line of code being executed when the SEGV occurs, so
> I have added NULL-checks to everything I can find in there but it
> looks like it's not enough. Something is missing, somewhere.
>
> -chris
>
>
>
> The information transmitted is intended only for the person or
> entity to which it is addressed and may contain confidential and/or
> privileged material. Any review, retransmission, dissemination or
> other use of, or taking of any action in reliance upon, this
> information by persons or entities other than the intended
> recipient is prohibited. If you received this in error, please
> contact the sender and delete the material from any computer. La
> información transmitida está destinada únicamente a la persona o
> entidad a quien que va dirigida y puede contener información
> confidencial y/o material privilegiado. Cualquier revisión,
> retransmisión, difusión u otros usos, o cualquier acción tomada por
> personas o entidades distintas al destinatario basándose en esta
> información está prohibida. Si usted recibe este mensaje por error,
> por favor contacte al remitente y elimine el material de cualquier
> computadora.
>
The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer.
La información transmitida está destinada únicamente a la persona o entidad a quien que va dirigida y puede contener información confidencial y/o material privilegiado. Cualquier revisión, retransmisión, difusión u otros usos, o cualquier acción tomada por personas o entidades distintas al destinatario basándose en esta información está prohibida. Si usted recibe este mensaje por error, por favor contacte al remitente y elimine el material de cualquier computadora.
Re: Tomcat 8.5 - Native Library - Crash
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Eduardo,
On 8/22/19 10:18, Eduardo Quintanilla wrote:
> Chris,
>
> The crash dump and logs[1] from my dev environment.
>
> [1]https://1drv.ms/u/s!AjSSUmqOgKmfi1g7z4wUvfYPksCk?e=CLH6VF
Okay, as I suspected, it's a SIGSEGV, or what Java programmers would
call a NullPointerException.
The C code is computing a pointer with a 44 byte offset to a NULL
pointer and trying to dereference it, which causes the problem.
I'm not sure which field has a 44-byte offset into the structure it's
supposed to be contained in. That highly depends upon the
architecture, compiler, etc. and those are the things I don't have
access to.
If someone in the community is able to build tcnative for Windows in
the same way[1] that the Tomcat team builds it, can you identify the
following:
1. What line of code corresponds to tcnative-1.dll+0x14d32
2. Which structure is being used
3. Which field is 44 bytes into that structure
Knowing all those things should make this a very easy fix.
Of course, the "fix" will involve throwing an exception back over into
the Java space, to the application will still throw an error. But at
least it won't bring-down the whole JVM.
>> Do you always get the "stream is not writable" error before the
>> JVM performs the crash-dump?
>
> Yes but only in the Tomcat of the production environment.
That's ... interesting. If Tomcat knows it can't write to the
stream... why is it still writing to the stream?
Note that the crash dump you posted is for an error in
jni.Address.get() and not the one for Socket.sendb(). If you can
generate a crash dump in Socket.sendb() that would be helpful, too.
Also, please see:
https://bz.apache.org/bugzilla/show_bug.cgi?id=62626
https://bz.apache.org/bugzilla/show_bug.cgi?id=63405
- -chris
>
> Best regards, Eduardo Quintanilla Software Developer
>
> From: Christopher Schultz <ch...@christopherschultz.net> Sent:
> jueves, 22 de agosto de 2019 8:25 a. m. To:
> users@tomcat.apache.org Subject: Re: Tomcat 8.5 - Native Library -
> Crash
>
>
> Eduardo,
>
> On 8/21/19 17:36, Eduardo Quintanilla wrote:
>> We have been getting some crashes in Tomcat 8.5.43 lately.
>>
>> The environment is: * JDK 1.8.0_202 * Windows Server 2012 R2
>>
>> The logs shows a tcnative exception.
>>
>> Crash Log: Current thread JavaThread
>> "https-openssl-apr-8081-exec-298" daemon _thread_in_native Java
>> frames: (J=compiled Java code, j=interpreted, Vv=VM code) J
>> 15034
>> org.apache.tomcat.jni.Socket.sendb(JLjava/nio/ByteBuffer;II)I (0
>> bytes) @ 0x0000000002962ddf [0x0000000002962d80+0x5f] J 29164 C2
>> org.apache.tomcat.util.net.AprEndpoint$AprSocketWrapper.doWrite(ZLjav
a
>
>>
/nio/ByteBuffer;)V
>> (242 bytes) @ 0x0000000006d6933c [0x0000000006d68b20+0x81c] J
>> 27479 C2 org.apache.tomcat.util.net.SocketWrapperBase.flush(Z)Z
>> (20 bytes) @ 0x00000000053f6144 [0x00000000053f60a0+0xa4] J 24666
>> C2
>> org.apache.coyote.http2.Http2OutputBuffer.doWrite(Ljava/nio/ByteBuffe
r
>
>>
;)I
>> (28 bytes) @ 0x0000000006b51b74 [0x0000000006b51660+0x514] J
>> 24689 C2
>> org.apache.catalina.connector.OutputBuffer.writeBytes([BII)V (38
>> bytes) @ 0x00000000031c09f8 [0x00000000031c04c0+0x538] J 27348
>> C2 org.apache.catalina.connector.CoyoteOutputStream.write([BII)V
>> (26 bytes) @ 0x00000000071d51a8 [0x00000000071d5120+0x88]
>
> What's the actual error? Can you post the entire crash dump?
>
>> Tomcat logs: org.apache.catalina.connector.ClientAbortException:
>> org.apache.coyote.CloseNowException: Connection [214], Stream
>> [279], This stream is not writable
>
> Do you always get the "stream is not writable" error before the
> JVM performs the crash-dump?
>
> I believe there is at least one lingering SEGV hiding in
> Socket.sendb(), possibly only on Windows. I don't have a Windows
> development environment to be able to disassemble the crash dump
> and locate the line of code being executed when the SEGV occurs, so
> I have added NULL-checks to everything I can find in there but it
> looks like it's not enough. Something is missing, somewhere.
>
> -chris
>
>
>
> The information transmitted is intended only for the person or
> entity to which it is addressed and may contain confidential and/or
> privileged material. Any review, retransmission, dissemination or
> other use of, or taking of any action in reliance upon, this
> information by persons or entities other than the intended
> recipient is prohibited. If you received this in error, please
> contact the sender and delete the material from any computer. La
> información transmitida está destinada únicamente a la persona o
> entidad a quien que va dirigida y puede contener información
> confidencial y/o material privilegiado. Cualquier revisión,
> retransmisión, difusión u otros usos, o cualquier acción tomada por
> personas o entidades distintas al destinatario basándose en esta
> información está prohibida. Si usted recibe este mensaje por error,
> por favor contacte al remitente y elimine el material de cualquier
> computadora.
>
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl1eyVAACgkQHPApP6U8
pFhZGg//SjOsRFjkeXF/pCqjGI8O0w/Ap98RyR9IV1/XfY8CrhNo4dRHnqhwmkx9
lNUEOK9VMhQWH4B8MNQ/AqdyYoN5+zAjkUIH0Q+bpldXmqQFl8RKI/9mvO5jcBGJ
1IO6TtsHc3SsW2qWxyloBt30iZ83Vc97yLkUVzYt9FCJZMN2RD1wE633WgiRr5HO
Vlh2UVKInMak62V00RdvyoxWrzxhcH0qLoun7AqKpq8aGuLy8gi44sSx7n+SgcFH
l3o2EdJRWDSSS+2sg2Y0AZLnFE8RkrZn7O0F9wPuSZ/ksuRPZJKouO+mYLda1yBv
fc98yCg88etcqYVLneYpSbyPTYReDBYMprAyNfBFcxM28m6JyOBEvavifDbkzR1z
IuIr716NK/CW30GJDIbYp5Z9zWD45A1p2rhcNhlLofpojwfKpv6Acf+oOJlCZ4it
Ra561uXnwXth0MOujISUmdBpg6oy6UvRDwquc1T/WqpmhtqT7vVFhRHLZDgQs5Tx
6WFZn4dAJnDp42aPJkiQNE4uNTpi1lWHFzBkipH75Kx5YzzBwh2orNYjXYLFXyfs
6RV81kpP2cbVpKMiYb75+CxsVEwK3X+LXYIM85ZIuoE0SMUSBfE+IU3tRW6+4Tfn
w7sV9lyiIolKE0T3tpIMjJHIB3FO0DNik9m47jd5qr5dHlvUnIg=
=gR2W
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat 8.5 - Native Library - Crash
Posted by Eduardo Quintanilla <eq...@bnext.mx>.
Chris,
The crash dump and logs[1] from my dev environment.
[1]https://1drv.ms/u/s!AjSSUmqOgKmfi1g7z4wUvfYPksCk?e=CLH6VF
> Do you always get the "stream is not writable" error before the JVM
> performs the crash-dump?
Yes but only in the Tomcat of the production environment.
Best regards,
Eduardo Quintanilla
Software Developer
From: Christopher Schultz <ch...@christopherschultz.net>
Sent: jueves, 22 de agosto de 2019 8:25 a. m.
To: users@tomcat.apache.org
Subject: Re: Tomcat 8.5 - Native Library - Crash
Eduardo,
On 8/21/19 17:36, Eduardo Quintanilla wrote:
> We have been getting some crashes in Tomcat 8.5.43 lately.
>
> The environment is: * JDK 1.8.0_202 * Windows Server 2012 R2
>
> The logs shows a tcnative exception.
>
> Crash Log: Current thread JavaThread
> "https-openssl-apr-8081-exec-298" daemon _thread_in_native Java
> frames: (J=compiled Java code, j=interpreted, Vv=VM code) J 15034
> org.apache.tomcat.jni.Socket.sendb(JLjava/nio/ByteBuffer;II)I (0
> bytes) @ 0x0000000002962ddf [0x0000000002962d80+0x5f] J 29164 C2
> org.apache.tomcat.util.net.AprEndpoint$AprSocketWrapper.doWrite(ZLjava
/nio/ByteBuffer;)V
> (242 bytes) @ 0x0000000006d6933c [0x0000000006d68b20+0x81c] J 27479
> C2 org.apache.tomcat.util.net.SocketWrapperBase.flush(Z)Z (20
> bytes) @ 0x00000000053f6144 [0x00000000053f60a0+0xa4] J 24666 C2
> org.apache.coyote.http2.Http2OutputBuffer.doWrite(Ljava/nio/ByteBuffer
;)I
> (28 bytes) @ 0x0000000006b51b74 [0x0000000006b51660+0x514] J 24689
> C2 org.apache.catalina.connector.OutputBuffer.writeBytes([BII)V (38
> bytes) @ 0x00000000031c09f8 [0x00000000031c04c0+0x538] J 27348 C2
> org.apache.catalina.connector.CoyoteOutputStream.write([BII)V (26
> bytes) @ 0x00000000071d51a8 [0x00000000071d5120+0x88]
What's the actual error? Can you post the entire crash dump?
> Tomcat logs: org.apache.catalina.connector.ClientAbortException:
> org.apache.coyote.CloseNowException: Connection [214], Stream
> [279], This stream is not writable
Do you always get the "stream is not writable" error before the JVM
performs the crash-dump?
I believe there is at least one lingering SEGV hiding in
Socket.sendb(), possibly only on Windows. I don't have a Windows
development environment to be able to disassemble the crash dump and
locate the line of code being executed when the SEGV occurs, so I have
added NULL-checks to everything I can find in there but it looks like
it's not enough. Something is missing, somewhere.
-chris
The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer.
La información transmitida está destinada únicamente a la persona o entidad a quien que va dirigida y puede contener información confidencial y/o material privilegiado. Cualquier revisión, retransmisión, difusión u otros usos, o cualquier acción tomada por personas o entidades distintas al destinatario basándose en esta información está prohibida. Si usted recibe este mensaje por error, por favor contacte al remitente y elimine el material de cualquier computadora.
Re: Tomcat 8.5 - Native Library - Crash
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Eduardo,
On 8/21/19 17:36, Eduardo Quintanilla wrote:
> We have been getting some crashes in Tomcat 8.5.43 lately.
>
> The environment is: * JDK 1.8.0_202 * Windows Server 2012 R2
>
> The logs shows a tcnative exception.
>
> Crash Log: Current thread JavaThread
> "https-openssl-apr-8081-exec-298" daemon _thread_in_native Java
> frames: (J=compiled Java code, j=interpreted, Vv=VM code) J 15034
> org.apache.tomcat.jni.Socket.sendb(JLjava/nio/ByteBuffer;II)I (0
> bytes) @ 0x0000000002962ddf [0x0000000002962d80+0x5f] J 29164 C2
> org.apache.tomcat.util.net.AprEndpoint$AprSocketWrapper.doWrite(ZLjava
/nio/ByteBuffer;)V
> (242 bytes) @ 0x0000000006d6933c [0x0000000006d68b20+0x81c] J 27479
> C2 org.apache.tomcat.util.net.SocketWrapperBase.flush(Z)Z (20
> bytes) @ 0x00000000053f6144 [0x00000000053f60a0+0xa4] J 24666 C2
> org.apache.coyote.http2.Http2OutputBuffer.doWrite(Ljava/nio/ByteBuffer
;)I
> (28 bytes) @ 0x0000000006b51b74 [0x0000000006b51660+0x514] J 24689
> C2 org.apache.catalina.connector.OutputBuffer.writeBytes([BII)V (38
> bytes) @ 0x00000000031c09f8 [0x00000000031c04c0+0x538] J 27348 C2
> org.apache.catalina.connector.CoyoteOutputStream.write([BII)V (26
> bytes) @ 0x00000000071d51a8 [0x00000000071d5120+0x88]
What's the actual error? Can you post the entire crash dump?
> Tomcat logs: org.apache.catalina.connector.ClientAbortException:
> org.apache.coyote.CloseNowException: Connection [214], Stream
> [279], This stream is not writable
Do you always get the "stream is not writable" error before the JVM
performs the crash-dump?
I believe there is at least one lingering SEGV hiding in
Socket.sendb(), possibly only on Windows. I don't have a Windows
development environment to be able to disassemble the crash dump and
locate the line of code being executed when the SEGV occurs, so I have
added NULL-checks to everything I can find in there but it looks like
it's not enough. Something is missing, somewhere.
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl1el8QACgkQHPApP6U8
pFiXsg//fUdmDBlrzNhbz+cbtk7B71WY+BYo8o2KiKUUOwrOcibs8nBoINeAWzpA
Q9ICA1vMfZST1N2uqoSI+vVtYOZe5fYF2iNdgfZSi8feiedzYU0ZiKf6qg1tCO8B
BfUkTGR4UuJe4uFV9QkkxVUjqcfNNRJriKd+uGi9rHL6pKjGt1K855MOXNgeIY9G
GD034hwEmY0hxr8sV48TnTOPdxEL0jEWoAqhKDLkRcnmjqCPTJUOnXE0tntVNO2R
s4D/N1rEP7KtBn610h7lqzmvr5PjpDL3tgt4UueGHbPWLiy0UJgHiKbmDM6GUjz/
7jySHUyyZM4TVynjYKhJHM/J7oEVpTEY9mvrZJNjnOxPQRN9AZmr4lDyS9Okdo2q
3wXgVXvqy3e9UI6CJdRISg5ItxBRZhQSSGlaDgVMq9o8S4xAHmk0Q+ee9Y3T+vto
L+PV5rz0vY16tcM3A/x+QnGJMOsX+a4mx9EQQblU19l2e+1JEgGcYsworltFufld
VrRZxilVkZdPyDnXMN8w/C345PeQquNp0AWcSYhtkyr2fHlMsxpUdFwDLUYHp8Iz
NU4uNsYgsGKn4Qoy2CdHzuNigTPcXco+VpUBTBV0p46nWxkY8yPFjvtZOADMcaNv
4LInt+ouJhje2avzV1IS3M2RxEvGnKwAVBREvg6l+rDahvKhc3M=
=3Tgo
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org