You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@trafficserver.apache.org by Matthieu Bienvenüe <ma...@exultet.net> on 2014/10/01 09:50:58 UTC
Re: SSL results in segmentation fault
Le 30/09/2014 17:47, Leif Hedstrom a écrit :
>
> On Sep 30, 2014, at 9:00 AM, Matthieu Bienvenüe <matthieu@exultet.net
> <ma...@exultet.net>> wrote:
>
>> Is that possible to do it on config instead of recompiling ATS ?
>
>
> What version are you using? I’m not 100% certain, but I’d expect
> Geffon’s additions to not have dedicated SSL threads would avoid the
> need for that patch as well? Brian? If I recall, with a recent version
> of ATS, you’d simply set proxy.config.ssl.number.threads to -1.
When I set this settings SSL don't work and I've the following stack
trace :
NOTE: Traffic Server received Sig 11: Segmentation fault
/usr/bin/traffic_server - STACK TRACE:
[0x4001e500]
/usr/lib/trafficserver/libtsutil.so.5(ink_hash_table_lookup_entry+0x12)[0x4003c0f2]
/usr/lib/trafficserver/libtsutil.so.5(ink_hash_table_lookup+0x24)[0x4003c3b4]
/usr/bin/traffic_server[0x8308185]
/usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0(+0x12844)[0x40067844]
/usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0(SSL_accept+0x2a)[0x4008c73a]
/usr/bin/traffic_server(_ZN17SSLNetVConnection23sslServerHandShakeEventERi+0x19)[0x8303d89]
/usr/bin/traffic_server(_ZN17SSLNetVConnection17sslStartHandShakeEiRi+0x2b)[0x830446b]
/usr/bin/traffic_server(_ZN17SSLNetVConnection11net_read_ioEP10NetHandlerP7EThread+0xb30)[0x8305270]
/usr/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x27f)[0x830dd7f]
/usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x98)[0x8339cf8]
/usr/bin/traffic_server(_ZN7EThread7executeEv+0x419)[0x833a449]
/usr/bin/traffic_server(main+0xf40)[0x80d4e30]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xe6)[0x405c9e36]
/usr/bin/traffic_server[0x80da229]
[TrafficServer] using root directory '/usr'
>
> In either case, why is that patch not committed? Is there a Jira for it?
>
> — Leif
>
>>
>> Regards,
>>
>> Matt
>> Le 30/09/2014 16:49, 英才 a écrit :
>>>
>>> disable AIO or patch https://github.com/phonehold/with-aio-ssl-init
>>> may solve your problem
>>>
>>> 在 2014年9月30日，下午10:41，Matthieu Bienvenüe
>>> <matthieu@exultet.net <ma...@exultet.net>> 写道：
>>>
>>>> Hello,
>>>>
>>>>
>>>> SSL works fine with my certs, but it crashes only after a certain
>>>> amount of time/requests.
>>>>
>>>> Here is the stack trace from traffic.out:
>>>>
>>>> NOTE: Traffic Server received Sig 11: Segmentation fault
>>>> /usr/bin/traffic_server - STACK TRACE:
>>>> [0x4001e500]
>>>> /usr/bin/traffic_server(_Z12ink_aio_readP11AIOCallbacki+0x2a)[0x830056a]
>>>> /usr/bin/traffic_server(_ZN7CacheVC10handleReadEiP5Event+0x282)[0x82c4402]
>>>> /usr/bin/traffic_server(_ZN5Cache9open_readEP12ContinuationP7INK_MD5P7HTTPHdrP21CacheLookupHttpConfig13CacheFragTypePci+0x5be)[0x82df68e]
>>>> /usr/bin/traffic_server(_ZN14CacheProcessor9open_readEP12ContinuationP3URLbP7HTTPHdrP21CacheLookupHttpConfigl13CacheFragType+0xdc)[0x82c2b4c]
>>>> /usr/bin/traffic_server(_ZN11HttpCacheSM18do_cache_open_readEv+0x63)[0x81ab6f3]
>>>> /usr/bin/traffic_server(_ZN11HttpCacheSM9open_readEP3URLP7HTTPHdrP21CacheLookupHttpConfigl+0x4c)[0x81aba0c]
>>>> /usr/bin/traffic_server(_ZN6HttpSM24do_cache_lookup_and_readEv+0x115)[0x81bd105]
>>>> /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x6af)[0x81ce7bf]
>>>> /usr/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x108)[0x81cc8d8]
>>>> /usr/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x300)[0x81c9940]
>>>> /usr/bin/traffic_server(_ZN6HttpSM23do_api_callout_internalEv+0x54)[0x81c9da4]
>>>> /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x250)[0x81ce360]
>>>> /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x7eb)[0x81ce8fb]
>>>> /usr/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x108)[0x81cc8d8]
>>>> /usr/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x300)[0x81c9940]
>>>> /usr/bin/traffic_server(_ZN6HttpSM23do_api_callout_internalEv+0x54)[0x81c9da4]
>>>> /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x250)[0x81ce360]
>>>> /usr/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x108)[0x81cc8d8]
>>>> /usr/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x300)[0x81c9940]
>>>> /usr/bin/traffic_server(_ZN6HttpSM18state_api_callbackEiPv+0x78)[0x81cc398]
>>>> /usr/bin/traffic_server(TSHttpTxnReenable+0x1f0)[0x810ef50]
>>>> /usr/lib/trafficserver/modules/stats_over_http.so(+0x102e)[0x4095f02e]
>>>> /usr/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0xd8)[0x81c9718]
>>>> /usr/bin/traffic_server(_ZN6HttpSM23do_api_callout_internalEv+0x54)[0x81c9da4]
>>>> /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x250)[0x81ce360]
>>>> /usr/bin/traffic_server(_ZN6HttpSM32state_read_client_request_headerEiPv+0x1e8)[0x81c5738]
>>>> /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x7e)[0x81ca93e]
>>>> /usr/bin/traffic_server(_ZN18UnixNetVConnection19readSignalAndUpdateEi+0x45)[0x83166a5]
>>>> /usr/bin/traffic_server(_ZN17SSLNetVConnection11net_read_ioEP10NetHandlerP7EThread+0x10b0)[0x83057f0]
>>>> /usr/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x27f)[0x830dd7f]
>>>> /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x98)[0x8339cf8]
>>>> /usr/bin/traffic_server(_ZN7EThread7executeEv+0x419)[0x833a449]
>>>> /usr/bin/traffic_server[0x8338ebb]
>>>> /lib/i386-linux-gnu/libpthread.so.0(+0x5954)[0x4046b954]
>>>> /lib/i386-linux-gnu/libc.so.6(clone+0x5e)[0x40688cbe]
>>>> [E. Mgmt] log ==> [TrafficManager] using root directory '/usr'
>>>> [TrafficServer] using root directory '/usr'
>>>>
>>>> Here is my record.config for SSL parameters:
>>>>
>>>> CONFIG proxy.config.http.server_ports STRING 8080 4443:ssl
>>>>
>>>> CONFIG proxy.config.ssl.enabled INT 1
>>>> CONFIG proxy.config.ssl.server.cert.path STRING /etc/trafficserver/ssl/
>>>> CONFIG proxy.config.ssl.server.private_key.path STRING
>>>> /etc/trafficserver/ssl/
>>>>
>>>> And for ssl_multicert.config:
>>>>
>>>> ssl_cert_name=new2014/100.pem ssl_key_name=new2014/100.key
>>>>
>>>>
>>>>
>>>>
>>>> Le 30/09/2014 15:54, Susan Hinrichs a écrit :
>>>>> Matt,
>>>>>
>>>>> Is there a basic stack trace in traffic.out? What is your SSL
>>>>> configuration? Do you have certs set up in ssl_multicert.config?
>>>>> Or are you doing a blind tunnel on the SSL traffic?
>>>>>
>>>>> Susan
>>>>>
>>>>> On 9/30/2014 2:14 AM, Matthieu Bienvenüe wrote:
>>>>>> Hello !
>>>>>>
>>>>>> I'm configuring ATS as a reverse proxy and I need SSL support.
>>>>>>
>>>>>> ATS runs on OpenVZ on Debian. It's the version 5.0.1 installed
>>>>>> from backport packages.
>>>>>>
>>>>>> ATS works fine, SSL too. But after a while SSL makes ATS crash.
>>>>>>
>>>>>> In manager.log I found that there is a segmentation fault:
>>>>>>
>>>>>> [Sep 29 16:08:33.020] Manager {0xb6fb76d0} ERROR:
>>>>>> [LocalManager::pollMgmtProcessServer] Server Process terminated
>>>>>> due to Sig 11: Segmentation fault
>>>>>> [Sep 29 16:08:33.021] Manager {0xb6fb76d0} ERROR:
>>>>>> [Alarms::signalAlarm] Server Process was reset
>>>>>> [Sep 29 16:08:34.041] Manager {0xb6fb76d0} NOTE:
>>>>>> [LocalManager::startProxy] Launching ts process
>>>>>> [Sep 29 16:08:34.049] Manager {0xb6fb76d0} NOTE:
>>>>>> [LocalManager::pollMgmtProcessServer] New process connecting fd '16'
>>>>>> [Sep 29 16:08:34.049] Manager {0xb6fb76d0} NOTE:
>>>>>> [Alarms::signalAlarm] Server Process born
>>>>>>
>>>>>> Here is a dump of the syslog when crashing:
>>>>>>
>>>>>> Sep 30 07:05:09 ats traffic_manager[5471]: {0xb704d6d0} FATAL:
>>>>>> [LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
>>>>>> Sep 30 07:05:09 ats traffic_manager[5471]: {0xb704d6d0} ERROR:
>>>>>> [LocalManager::sendMgmtMsgToProcesses] Error writing message
>>>>>> Sep 30 07:05:09 ats traffic_manager[5471]: {0xb704d6d0} ERROR:
>>>>>> (last system error 32: Broken pipe)
>>>>>> Sep 30 07:05:09 ats traffic_cop[23694]: cop received child status
>>>>>> signal [5471 256]
>>>>>> Sep 30 07:05:09 ats traffic_cop[23694]: traffic_manager not
>>>>>> running, making sure traffic_server is dead
>>>>>> Sep 30 07:05:09 ats traffic_cop[23694]: spawning traffic_manager
>>>>>> Sep 30 07:05:09 ats traffic_manager[6938]: NOTE: --- Manager
>>>>>> Starting ---
>>>>>> Sep 30 07:05:09 ats traffic_manager[6938]: NOTE: Manager Version:
>>>>>> Apache Traffic Server - traffic_manager - 5.0.1 - (build # 7259
>>>>>> on Aug 25 2014 at 09:26:11)
>>>>>> Sep 30 07:05:09 ats traffic_manager[6938]: NOTE: Unable to set
>>>>>> RLIMIT_NOFILE(7):cur(1475961),max(1475961)
>>>>>> Sep 30 07:05:09 ats traffic_manager[6938]: NOTE:
>>>>>> RLIMIT_NOFILE(7):cur(30000),max(30000)
>>>>>> Sep 30 07:05:09 ats traffic_manager[6938]: ERROR ==> [runAsUser]
>>>>>> Error: Failed to restore capabilities after switch to user
>>>>>> trafficserver.
>>>>>> Sep 30 07:05:11 ats traffic_server[6946]: NOTE: ---
>>>>>> traffic_server Starting ---
>>>>>> Sep 30 07:05:11 ats traffic_server[6946]: NOTE: traffic_server
>>>>>> Version: Apache Traffic Server - traffic_server - 5.0.1 - (build
>>>>>> # 7259 on Aug 25 2014 at 09:27:18)
>>>>>> Sep 30 07:05:11 ats traffic_server[6946]: NOTE: Unable to set
>>>>>> RLIMIT_NOFILE(7):cur(-611778560),max(-611778560)
>>>>>> Sep 30 07:05:13 ats traffic_manager[6938]: {0xb708b6d0} ERROR:
>>>>>> [LocalManager::pollMgmtProcessServer] Server Process terminated
>>>>>> due to Sig 11: Segmentation fault
>>>>>> Sep 30 07:05:13 ats traffic_manager[6938]: {0xb708b6d0} ERROR:
>>>>>> [Alarms::signalAlarm] Server Process was reset
>>>>>>
>>>>>>
>>>>>> Any idea where to look for to solve this problem ?
>>>>>>
>>>>>> Thanks a lot !
>>>>>>
>>>>>> Matt
>>>>>
>>>>
>>>>
>>>
>>
>
Re: SSL results in segmentation fault
Posted by Matthieu Bienvenüe <ma...@exultet.net>.
Hello,
thanks for the advice ! I've tried with an amd64 template (because as I
said earlier ATS runs on a OpenVZ container) but I've the same problem.
The server starts and handle HTTP traffic correctly. But it crashes when
there is an HTTPS request that reaches it.
Here is the new dump :
[Oct 7 10:56:26.404] Server {0x2b93aaaed320} DEBUG: (ssl)
[SSLNextProtocolAccept:mainEvent] event 202 netvc 0x2b93aea40000
[Oct 7 10:56:26.405] Server {0x2b93aaaed320} DEBUG: (ssl)
ssl_callback_info ssl: 0x2e4ba60 where: 16 ret: 1
[Oct 7 10:56:26.405] Server {0x2b93aaaed320} DEBUG: (ssl)
ssl_callback_info ssl: 0x2e4ba60 where: 8193 ret: 1
[Oct 7 10:56:26.405] Server {0x2b93aaaed320} DEBUG: (ssl)
ssl_servername_callback ssl=0x2e4ba60 ad=112 lookup=0x2de9680
server=XXXXX handshake_complete=0
[Oct 7 10:56:26.405] Server {0x2b93aaaed320} DEBUG: (ssl)
ssl_servername_callback found SSL context 0x2dea9f0 for requested name
'XXXXX'
[Oct 7 10:56:26.405] Server {0x2b93aaaed320} DEBUG: (ssl)
ssl_callback_info ssl: 0x2e4ba60 where: 8193 ret: 1
[Oct 7 10:56:26.405] Server {0x2b93aaaed320} DEBUG: (ssl) advertising
protocol http/1.1
[Oct 7 10:56:26.405] Server {0x2b93aaaed320} DEBUG: (ssl) advertising
protocol http/1.0
[Oct 7 10:56:26.405] Server {0x2b93aaaed320} DEBUG: (ssl)
ssl_callback_info ssl: 0x2e4ba60 where: 8193 ret: 1
[Oct 7 10:56:26.405] Server {0x2b93aaaed320} DEBUG: (ssl)
ssl_callback_info ssl: 0x2e4ba60 where: 8193 ret: 1
[Oct 7 10:56:26.413] Server {0x2b93aaaed320} DEBUG: (ssl)
ssl_callback_info ssl: 0x2e4ba60 where: 8193 ret: 1
[Oct 7 10:56:26.413] Server {0x2b93aaaed320} DEBUG: (ssl)
ssl_callback_info ssl: 0x2e4ba60 where: 8193 ret: 1
[Oct 7 10:56:26.413] Server {0x2b93aaaed320} DEBUG: (ssl)
ssl_callback_info ssl: 0x2e4ba60 where: 8193 ret: 1
[Oct 7 10:56:26.413] Server {0x2b93aaaed320} DEBUG: (ssl)
ssl_callback_info ssl: 0x2e4ba60 where: 8194 ret: -1
[Oct 7 10:56:26.413] Server {0x2b93aaaed320} DEBUG: (ssl)
ssl_callback_info ssl: 0x2e4ba60 where: 8194 ret: -1
[Oct 7 10:56:26.414] Server {0x2b93aaaed320} DEBUG:
<SSLNetVConnection.cc:558 (sslServerHandShakeEvent)> (ssl) SSL handshake
error: SSL_ERROR_WANT_READ (2), errno=11
[Oct 7 10:56:26.577] Server {0x2b93aaaed320} DEBUG: (ssl)
ssl_callback_info ssl: 0x2e4ba60 where: 8193 ret: 1
[Oct 7 10:56:26.578] Server {0x2b93aaaed320} DEBUG: (ssl)
ssl_callback_info ssl: 0x2e4ba60 where: 8193 ret: 1
[Oct 7 10:56:26.578] Server {0x2b93aaaed320} DEBUG: (ssl)
ssl_callback_info ssl: 0x2e4ba60 where: 8193 ret: 1
[Oct 7 10:56:26.578] Server {0x2b93aaaed320} DEBUG: (ssl)
ssl_callback_info ssl: 0x2e4ba60 where: 8193 ret: 1
[Oct 7 10:56:26.578] Server {0x2b93aaaed320} DEBUG: (ssl)
ssl_callback_info ssl: 0x2e4ba60 where: 8193 ret: 1
[Oct 7 10:56:26.578] Server {0x2b93aaaed320} DEBUG: (ssl)
ssl_callback_info ssl: 0x2e4ba60 where: 8193 ret: 1
[Oct 7 10:56:26.578] Server {0x2b93aaaed320} DEBUG: (ssl)
ssl_callback_info ssl: 0x2e4ba60 where: 8193 ret: 1
[Oct 7 10:56:26.578] Server {0x2b93aaaed320} DEBUG: (ssl)
ssl_callback_info ssl: 0x2e4ba60 where: 32 ret: 1
NOTE: Traffic Server received Sig 11: Segmentation fault
/usr/bin/traffic_server - STACK TRACE:
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf030)[0x2b93a99a7030]
/usr/lib/trafficserver/libtsutil.so.5(ink_hash_table_lookup_entry+0x0)[0x2b93a7d2c350]
/usr/lib/trafficserver/libtsutil.so.5(ink_hash_table_lookup+0x19)[0x2b93a7d2c549]
/usr/bin/traffic_server[0x6b0519]
/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0(+0x1a35f)[0x2b93a7f5d35f]
/usr/bin/traffic_server(_ZN17SSLNetVConnection23sslServerHandShakeEventERi+0x2a)[0x6ac72a]
/usr/bin/traffic_server(_ZN17SSLNetVConnection17sslStartHandShakeEiRi+0x32)[0x6acd72]
/usr/bin/traffic_server(_ZN17SSLNetVConnection11net_read_ioEP10NetHandlerP7EThread+0x70c)[0x6ad74c]
/usr/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x26c)[0x6b57dc]
/usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x98)[0x6de338]
/usr/bin/traffic_server(_ZN7EThread7executeEv+0x448)[0x6dea68]
/usr/bin/traffic_server(main+0xc55)[0x49fa45]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd)[0x2b93aa577eed]
/usr/bin/traffic_server[0x4a40ed]
[TrafficServer] using root directory '/usr'
[Oct 7 10:56:28.662] Server {0x2ab0a4989320} DEBUG: (ssl) setting SNI
callbacks with for ctx 0x2229570
[Oct 7 10:56:28.663] Server {0x2ab0a4989320} DEBUG: (ssl) importing SNI
names from /etc/trafficserver/ssl/new2014/100.pem
[Oct 7 10:56:28.663] Server {0x2ab0a4989320} DEBUG: (ssl) mapping
'XXXXX' to certificate /etc/trafficserver/ssl/new2014/100.pem
[Oct 7 10:56:28.663] Server {0x2ab0a4989320} DEBUG: (ssl) indexed
'XXXXX' with SSL_CTX 0x2229570
[Oct 7 10:56:28.663] Server {0x2ab0a4989320} DEBUG: (ssl) mapping
'XXXXX' to certificate /etc/trafficserver/ssl/new2014/100.pem
[Oct 7 10:56:28.663] Server {0x2ab0a4989320} DEBUG: (ssl) indexed
'XXXXX' with SSL_CTX 0x2229570
[Oct 7 10:56:28.663] Server {0x2ab0a4989320} DEBUG: (ssl) mapping
'YYYYY' to certificate /etc/trafficserver/ssl/new2014/100.pem
[Oct 7 10:56:28.663] Server {0x2ab0a4989320} DEBUG: (ssl) indexed
'YYYYY' with SSL_CTX 0x2229570
[Oct 7 10:56:28.664] Server {0x2ab0a4989320} DEBUG: (ssl) setting SNI
callbacks with for ctx 0x222d4c0
[Oct 7 10:56:28.664] Server {0x2ab0a4989320} DEBUG: (ssl) indexed '*'
with SSL_CTX 0x222d4c0
[Oct 7 10:56:28.664] Server {0x2ab0a4989320} DEBUG: (ssl) importing SNI
names from /etc/trafficserver/ssl
Le 07/10/2014 16:01, Leif Hedstrom a écrit :
> On Oct 7, 2014, at 1:26 AM, Matthieu Bienvenüe <ma...@exultet.net> wrote:
>
>> OK so here is what I get in traffic.out when setting :
>>
>> CONFIG proxy.config.ssl.number.threads INT -1
>> CONFIG proxy.config.diags.debug.enabled INT 1
>> CONFIG proxy.config.diags.debug.tags STRING ssl
>
> Could it be a 32-bit issue? I don’t have any 32-bit boxes any more, and we don’t have any CI for it as well. I believe the decision was that as of 5.0, we would only officially support 64-bit.
>
> Can you test on a 64-bit box as well ? Also, did you send any request to get this to trigger, or does it segfault right out the gate on startup?
>
> — leif
>
>>
>>
>> [Oct 7 09:24:57.497] Server {0x40709730} DEBUG: (ssl) [SSLNextProtocolAccept:mainEvent] event 202 netvc 0x438d0000
>> [Oct 7 09:24:57.497] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl: 0xa5e0ee0 where: 16 ret: 1
>> [Oct 7 09:24:57.497] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl: 0xa5e0ee0 where: 8193 ret: 1
>> [Oct 7 09:24:57.497] Server {0x40709730} DEBUG: (ssl) ssl_servername_callback ssl=0xa5e0ee0 ad=112 lookup=0xa54f9c0 server=XXXX handshake_complete=0
>> [Oct 7 09:24:57.497] Server {0x40709730} DEBUG: (ssl) ssl_servername_callback found SSL context 0xa552960 for requested name 'XXXX'
>> [Oct 7 09:24:57.498] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl: 0xa5e0ee0 where: 8193 ret: 1
>> [Oct 7 09:24:57.498] Server {0x40709730} DEBUG: (ssl) advertising protocol http/1.1
>> [Oct 7 09:24:57.498] Server {0x40709730} DEBUG: (ssl) advertising protocol http/1.0
>> [Oct 7 09:24:57.498] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl: 0xa5e0ee0 where: 8193 ret: 1
>> [Oct 7 09:24:57.498] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl: 0xa5e0ee0 where: 8193 ret: 1
>> [Oct 7 09:24:57.527] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl: 0xa5e0ee0 where: 8193 ret: 1
>> [Oct 7 09:24:57.527] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl: 0xa5e0ee0 where: 8193 ret: 1
>> [Oct 7 09:24:57.527] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl: 0xa5e0ee0 where: 8193 ret: 1
>> [Oct 7 09:24:57.528] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl: 0xa5e0ee0 where: 8194 ret: -1
>> [Oct 7 09:24:57.528] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl: 0xa5e0ee0 where: 8194 ret: -1
>> [Oct 7 09:24:57.528] Server {0x40709730} DEBUG: <SSLNetVConnection.cc:558 (sslServerHandShakeEvent)> (ssl) SSL handshake error: SSL_ERROR_WANT_READ (2), errno=11
>> [Oct 7 09:24:57.589] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl: 0xa5e0ee0 where: 8193 ret: 1
>> [Oct 7 09:24:57.589] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl: 0xa5e0ee0 where: 8193 ret: 1
>> [Oct 7 09:24:57.589] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl: 0xa5e0ee0 where: 8193 ret: 1
>> [Oct 7 09:24:57.590] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl: 0xa5e0ee0 where: 8193 ret: 1
>> [Oct 7 09:24:57.590] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl: 0xa5e0ee0 where: 8193 ret: 1
>> [Oct 7 09:24:57.590] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl: 0xa5e0ee0 where: 8193 ret: 1
>> [Oct 7 09:24:57.590] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl: 0xa5e0ee0 where: 8193 ret: 1
>> [Oct 7 09:24:57.590] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl: 0xa5e0ee0 where: 32 ret: 1
>> NOTE: Traffic Server received Sig 11: Segmentation fault
>> /usr/bin/traffic_server - STACK TRACE:
>> [0x4001e500]
>> /usr/lib/trafficserver/libtsutil.so.5(ink_hash_table_lookup_entry+0x12)[0x4003c0f2]
>> /usr/lib/trafficserver/libtsutil.so.5(ink_hash_table_lookup+0x24)[0x4003c3b4]
>> /usr/bin/traffic_server[0x8308185]
>> /usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0(+0x12844)[0x40067844]
>> /usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0(SSL_accept+0x2a)[0x4008c73a]
>> /usr/bin/traffic_server(_ZN17SSLNetVConnection23sslServerHandShakeEventERi+0x19)[0x8303d89]
>> /usr/bin/traffic_server(_ZN17SSLNetVConnection17sslStartHandShakeEiRi+0x2b)[0x830446b]
>> /usr/bin/traffic_server(_ZN17SSLNetVConnection11net_read_ioEP10NetHandlerP7EThread+0xb30)[0x8305270]
>> /usr/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x27f)[0x830dd7f]
>> /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x98)[0x8339cf8]
>> /usr/bin/traffic_server(_ZN7EThread7executeEv+0x419)[0x833a449]
>> /usr/bin/traffic_server(main+0xf40)[0x80d4e30]
>> /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xe6)[0x405c9e36]
>> /usr/bin/traffic_server[0x80da229]
>> [TrafficServer] using root directory '/usr'
>> [Oct 7 09:24:59.682] Server {0x40709730} DEBUG: (ssl) setting SNI callbacks with for ctx 0x8a6b8b8
>> [Oct 7 09:24:59.682] Server {0x40709730} DEBUG: (ssl) importing SNI names from /etc/trafficserver/ssl/new2014/100.pem
>> [Oct 7 09:24:59.683] Server {0x40709730} DEBUG: (ssl) mapping 'XXXX' to certificate /etc/trafficserver/ssl/new2014/100.pem
>> [Oct 7 09:24:59.683] Server {0x40709730} DEBUG: (ssl) indexed 'XXXX' with SSL_CTX 0x8a6b8b8
>> [Oct 7 09:24:59.683] Server {0x40709730} DEBUG: (ssl) mapping 'XXXX' to certificate /etc/trafficserver/ssl/new2014/100.pem
>> [Oct 7 09:24:59.683] Server {0x40709730} DEBUG: (ssl) indexed 'XXXX' with SSL_CTX 0x8a6b8b8
>> [Oct 7 09:24:59.683] Server {0x40709730} DEBUG: (ssl) mapping 'YYYYYY' to certificate /etc/trafficserver/ssl/new2014/100.pem
>> [Oct 7 09:24:59.683] Server {0x40709730} DEBUG: (ssl) indexed 'YYYYYY' with SSL_CTX 0x8a6b8b8
>> [Oct 7 09:24:59.683] Server {0x40709730} DEBUG: (ssl) setting SNI callbacks with for ctx 0x8a76b30
>> [Oct 7 09:24:59.684] Server {0x40709730} DEBUG: (ssl) indexed '*' with SSL_CTX 0x8a76b30
>> [Oct 7 09:24:59.684] Server {0x40709730} DEBUG: (ssl) importing SNI names from /etc/trafficserver/ssl
>>
>>
>>
>>
>>
>> Le 03/10/2014 18:43, James Peach a écrit :
>>> On Oct 3, 2014, at 3:32 AM, Matthieu Bienvenüe <ma...@exultet.net> wrote:
>>>
>>>> Any idea to solve this isssu ?
>>> I did a quick test of setting proxy.config.ssl.number.threads to -1, and it didn't crash for me. Can you enable ssl diagnostics and try again?
>>>
>>> CONFIG proxy.config.diags.debug.enabled INT 1
>>> CONFIG proxy.config.diags.debug.tags STRING ssl
>>>
>>>> Matthieu
>>>>
>>>>
>>>> Le 01/10/2014 09:50, Matthieu Bienvenüe a écrit :
>>>>> Le 30/09/2014 17:47, Leif Hedstrom a écrit :
>>>>>> On Sep 30, 2014, at 9:00 AM, Matthieu Bienvenüe <ma...@exultet.net> wrote:
>>>>>>
>>>>>>> Is that possible to do it on config instead of recompiling ATS ?
>>>>>> What version are you using? I’m not 100% certain, but I’d expect Geffon’s additions to not have dedicated SSL threads would avoid the need for that patch as well? Brian? If I recall, with a recent version of ATS, you’d simply set proxy.config.ssl.number.threads to -1.
>>>>> When I set this settings SSL don't work and I've the following stack trace :
>>>>>
>>>>> NOTE: Traffic Server received Sig 11: Segmentation fault
>>>>> /usr/bin/traffic_server - STACK TRACE:
>>>>> [0x4001e500]
>>>>> /usr/lib/trafficserver/libtsutil.so.5(ink_hash_table_lookup_entry+0x12)[0x4003c0f2]
>>>>> /usr/lib/trafficserver/libtsutil.so.5(ink_hash_table_lookup+0x24)[0x4003c3b4]
>>>>> /usr/bin/traffic_server[0x8308185]
>>>>> /usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0(+0x12844)[0x40067844]
>>>>> /usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0(SSL_accept+0x2a)[0x4008c73a]
>>>>> /usr/bin/traffic_server(_ZN17SSLNetVConnection23sslServerHandShakeEventERi+0x19)[0x8303d89]
>>>>> /usr/bin/traffic_server(_ZN17SSLNetVConnection17sslStartHandShakeEiRi+0x2b)[0x830446b]
>>>>> /usr/bin/traffic_server(_ZN17SSLNetVConnection11net_read_ioEP10NetHandlerP7EThread+0xb30)[0x8305270]
>>>>> /usr/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x27f)[0x830dd7f]
>>>>> /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x98)[0x8339cf8]
>>>>> /usr/bin/traffic_server(_ZN7EThread7executeEv+0x419)[0x833a449]
>>>>> /usr/bin/traffic_server(main+0xf40)[0x80d4e30]
>>>>> /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xe6)[0x405c9e36]
>>>>> /usr/bin/traffic_server[0x80da229]
>>>>> [TrafficServer] using root directory '/usr'
>>>>>
>>>>>> In either case, why is that patch not committed? Is there a Jira for it?
>>>>>>
>>>>>> — Leif
>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Matt
>>>>>>> Le 30/09/2014 16:49, 英才 a écrit :
>>>>>>>> disable AIO or patch https://github.com/phonehold/with-aio-ssl-init may solve your problem
>>>>>>>>
>>>>>>>> 在 2014年9月30日，下午10:41，Matthieu Bienvenüe <ma...@exultet.net> 写道：
>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> SSL works fine with my certs, but it crashes only after a certain amount of time/requests.
>>>>>>>>>
>>>>>>>>> Here is the stack trace from traffic.out:
>>>>>>>>>
>>>>>>>>> NOTE: Traffic Server received Sig 11: Segmentation fault
>>>>>>>>> /usr/bin/traffic_server - STACK TRACE:
>>>>>>>>> [0x4001e500]
>>>>>>>>> /usr/bin/traffic_server(_Z12ink_aio_readP11AIOCallbacki+0x2a)[0x830056a]
>>>>>>>>> /usr/bin/traffic_server(_ZN7CacheVC10handleReadEiP5Event+0x282)[0x82c4402]
>>>>>>>>> /usr/bin/traffic_server(_ZN5Cache9open_readEP12ContinuationP7INK_MD5P7HTTPHdrP21CacheLookupHttpConfig13CacheFragTypePci+0x5be)[0x82df68e]
>>>>>>>>> /usr/bin/traffic_server(_ZN14CacheProcessor9open_readEP12ContinuationP3URLbP7HTTPHdrP21CacheLookupHttpConfigl13CacheFragType+0xdc)[0x82c2b4c]
>>>>>>>>> /usr/bin/traffic_server(_ZN11HttpCacheSM18do_cache_open_readEv+0x63)[0x81ab6f3]
>>>>>>>>> /usr/bin/traffic_server(_ZN11HttpCacheSM9open_readEP3URLP7HTTPHdrP21CacheLookupHttpConfigl+0x4c)[0x81aba0c]
>>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM24do_cache_lookup_and_readEv+0x115)[0x81bd105]
>>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x6af)[0x81ce7bf]
>>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x108)[0x81cc8d8]
>>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x300)[0x81c9940]
>>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM23do_api_callout_internalEv+0x54)[0x81c9da4]
>>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x250)[0x81ce360]
>>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x7eb)[0x81ce8fb]
>>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x108)[0x81cc8d8]
>>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x300)[0x81c9940]
>>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM23do_api_callout_internalEv+0x54)[0x81c9da4]
>>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x250)[0x81ce360]
>>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x108)[0x81cc8d8]
>>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x300)[0x81c9940]
>>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM18state_api_callbackEiPv+0x78)[0x81cc398]
>>>>>>>>> /usr/bin/traffic_server(TSHttpTxnReenable+0x1f0)[0x810ef50]
>>>>>>>>> /usr/lib/trafficserver/modules/stats_over_http.so(+0x102e)[0x4095f02e]
>>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0xd8)[0x81c9718]
>>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM23do_api_callout_internalEv+0x54)[0x81c9da4]
>>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x250)[0x81ce360]
>>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM32state_read_client_request_headerEiPv+0x1e8)[0x81c5738]
>>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x7e)[0x81ca93e]
>>>>>>>>> /usr/bin/traffic_server(_ZN18UnixNetVConnection19readSignalAndUpdateEi+0x45)[0x83166a5]
>>>>>>>>> /usr/bin/traffic_server(_ZN17SSLNetVConnection11net_read_ioEP10NetHandlerP7EThread+0x10b0)[0x83057f0]
>>>>>>>>> /usr/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x27f)[0x830dd7f]
>>>>>>>>> /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x98)[0x8339cf8]
>>>>>>>>> /usr/bin/traffic_server(_ZN7EThread7executeEv+0x419)[0x833a449]
>>>>>>>>> /usr/bin/traffic_server[0x8338ebb]
>>>>>>>>> /lib/i386-linux-gnu/libpthread.so.0(+0x5954)[0x4046b954]
>>>>>>>>> /lib/i386-linux-gnu/libc.so.6(clone+0x5e)[0x40688cbe]
>>>>>>>>> [E. Mgmt] log ==> [TrafficManager] using root directory '/usr'
>>>>>>>>> [TrafficServer] using root directory '/usr'
>>>>>>>>>
>>>>>>>>> Here is my record.config for SSL parameters:
>>>>>>>>>
>>>>>>>>> CONFIG proxy.config.http.server_ports STRING 8080 4443:ssl
>>>>>>>>>
>>>>>>>>> CONFIG proxy.config.ssl.enabled INT 1
>>>>>>>>> CONFIG proxy.config.ssl.server.cert.path STRING /etc/trafficserver/ssl/
>>>>>>>>> CONFIG proxy.config.ssl.server.private_key.path STRING /etc/trafficserver/ssl/
>>>>>>>>>
>>>>>>>>> And for ssl_multicert.config:
>>>>>>>>>
>>>>>>>>> ssl_cert_name=new2014/100.pem ssl_key_name=new2014/100.key
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Le 30/09/2014 15:54, Susan Hinrichs a écrit :
>>>>>>>>>> Matt,
>>>>>>>>>>
>>>>>>>>>> Is there a basic stack trace in traffic.out? What is your SSL configuration? Do you have certs set up in ssl_multicert.config? Or are you doing a blind tunnel on the SSL traffic?
>>>>>>>>>>
>>>>>>>>>> Susan
>>>>>>>>>>
>>>>>>>>>> On 9/30/2014 2:14 AM, Matthieu Bienvenüe wrote:
>>>>>>>>>>> Hello !
>>>>>>>>>>>
>>>>>>>>>>> I'm configuring ATS as a reverse proxy and I need SSL support.
>>>>>>>>>>>
>>>>>>>>>>> ATS runs on OpenVZ on Debian. It's the version 5.0.1 installed from backport packages.
>>>>>>>>>>>
>>>>>>>>>>> ATS works fine, SSL too. But after a while SSL makes ATS crash.
>>>>>>>>>>>
>>>>>>>>>>> In manager.log I found that there is a segmentation fault:
>>>>>>>>>>>
>>>>>>>>>>> [Sep 29 16:08:33.020] Manager {0xb6fb76d0} ERROR: [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 11: Segmentation fault
>>>>>>>>>>> [Sep 29 16:08:33.021] Manager {0xb6fb76d0} ERROR: [Alarms::signalAlarm] Server Process was reset
>>>>>>>>>>> [Sep 29 16:08:34.041] Manager {0xb6fb76d0} NOTE: [LocalManager::startProxy] Launching ts process
>>>>>>>>>>> [Sep 29 16:08:34.049] Manager {0xb6fb76d0} NOTE: [LocalManager::pollMgmtProcessServer] New process connecting fd '16'
>>>>>>>>>>> [Sep 29 16:08:34.049] Manager {0xb6fb76d0} NOTE: [Alarms::signalAlarm] Server Process born
>>>>>>>>>>>
>>>>>>>>>>> Here is a dump of the syslog when crashing:
>>>>>>>>>>>
>>>>>>>>>>> Sep 30 07:05:09 ats traffic_manager[5471]: {0xb704d6d0} FATAL: [LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
>>>>>>>>>>> Sep 30 07:05:09 ats traffic_manager[5471]: {0xb704d6d0} ERROR: [LocalManager::sendMgmtMsgToProcesses] Error writing message
>>>>>>>>>>> Sep 30 07:05:09 ats traffic_manager[5471]: {0xb704d6d0} ERROR: (last system error 32: Broken pipe)
>>>>>>>>>>> Sep 30 07:05:09 ats traffic_cop[23694]: cop received child status signal [5471 256]
>>>>>>>>>>> Sep 30 07:05:09 ats traffic_cop[23694]: traffic_manager not running, making sure traffic_server is dead
>>>>>>>>>>> Sep 30 07:05:09 ats traffic_cop[23694]: spawning traffic_manager
>>>>>>>>>>> Sep 30 07:05:09 ats traffic_manager[6938]: NOTE: --- Manager Starting ---
>>>>>>>>>>> Sep 30 07:05:09 ats traffic_manager[6938]: NOTE: Manager Version: Apache Traffic Server - traffic_manager - 5.0.1 - (build # 7259 on Aug 25 2014 at 09:26:11)
>>>>>>>>>>> Sep 30 07:05:09 ats traffic_manager[6938]: NOTE: Unable to set RLIMIT_NOFILE(7):cur(1475961),max(1475961)
>>>>>>>>>>> Sep 30 07:05:09 ats traffic_manager[6938]: NOTE: RLIMIT_NOFILE(7):cur(30000),max(30000)
>>>>>>>>>>> Sep 30 07:05:09 ats traffic_manager[6938]: ERROR ==> [runAsUser] Error: Failed to restore capabilities after switch to user trafficserver.
>>>>>>>>>>> Sep 30 07:05:11 ats traffic_server[6946]: NOTE: --- traffic_server Starting ---
>>>>>>>>>>> Sep 30 07:05:11 ats traffic_server[6946]: NOTE: traffic_server Version: Apache Traffic Server - traffic_server - 5.0.1 - (build # 7259 on Aug 25 2014 at 09:27:18)
>>>>>>>>>>> Sep 30 07:05:11 ats traffic_server[6946]: NOTE: Unable to set RLIMIT_NOFILE(7):cur(-611778560),max(-611778560)
>>>>>>>>>>> Sep 30 07:05:13 ats traffic_manager[6938]: {0xb708b6d0} ERROR: [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 11: Segmentation fault
>>>>>>>>>>> Sep 30 07:05:13 ats traffic_manager[6938]: {0xb708b6d0} ERROR: [Alarms::signalAlarm] Server Process was reset
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Any idea where to look for to solve this problem ?
>>>>>>>>>>>
>>>>>>>>>>> Thanks a lot !
>>>>>>>>>>>
>>>>>>>>>>> Matt
Re: SSL results in segmentation fault
Posted by Leif Hedstrom <zw...@apache.org>.
On Oct 7, 2014, at 1:26 AM, Matthieu Bienvenüe <ma...@exultet.net> wrote:
> OK so here is what I get in traffic.out when setting :
>
> CONFIG proxy.config.ssl.number.threads INT -1
> CONFIG proxy.config.diags.debug.enabled INT 1
> CONFIG proxy.config.diags.debug.tags STRING ssl
Could it be a 32-bit issue? I don’t have any 32-bit boxes any more, and we don’t have any CI for it as well. I believe the decision was that as of 5.0, we would only officially support 64-bit.
Can you test on a 64-bit box as well ? Also, did you send any request to get this to trigger, or does it segfault right out the gate on startup?
— leif
>
>
>
> [Oct 7 09:24:57.497] Server {0x40709730} DEBUG: (ssl) [SSLNextProtocolAccept:mainEvent] event 202 netvc 0x438d0000
> [Oct 7 09:24:57.497] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl: 0xa5e0ee0 where: 16 ret: 1
> [Oct 7 09:24:57.497] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl: 0xa5e0ee0 where: 8193 ret: 1
> [Oct 7 09:24:57.497] Server {0x40709730} DEBUG: (ssl) ssl_servername_callback ssl=0xa5e0ee0 ad=112 lookup=0xa54f9c0 server=XXXX handshake_complete=0
> [Oct 7 09:24:57.497] Server {0x40709730} DEBUG: (ssl) ssl_servername_callback found SSL context 0xa552960 for requested name 'XXXX'
> [Oct 7 09:24:57.498] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl: 0xa5e0ee0 where: 8193 ret: 1
> [Oct 7 09:24:57.498] Server {0x40709730} DEBUG: (ssl) advertising protocol http/1.1
> [Oct 7 09:24:57.498] Server {0x40709730} DEBUG: (ssl) advertising protocol http/1.0
> [Oct 7 09:24:57.498] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl: 0xa5e0ee0 where: 8193 ret: 1
> [Oct 7 09:24:57.498] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl: 0xa5e0ee0 where: 8193 ret: 1
> [Oct 7 09:24:57.527] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl: 0xa5e0ee0 where: 8193 ret: 1
> [Oct 7 09:24:57.527] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl: 0xa5e0ee0 where: 8193 ret: 1
> [Oct 7 09:24:57.527] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl: 0xa5e0ee0 where: 8193 ret: 1
> [Oct 7 09:24:57.528] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl: 0xa5e0ee0 where: 8194 ret: -1
> [Oct 7 09:24:57.528] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl: 0xa5e0ee0 where: 8194 ret: -1
> [Oct 7 09:24:57.528] Server {0x40709730} DEBUG: <SSLNetVConnection.cc:558 (sslServerHandShakeEvent)> (ssl) SSL handshake error: SSL_ERROR_WANT_READ (2), errno=11
> [Oct 7 09:24:57.589] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl: 0xa5e0ee0 where: 8193 ret: 1
> [Oct 7 09:24:57.589] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl: 0xa5e0ee0 where: 8193 ret: 1
> [Oct 7 09:24:57.589] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl: 0xa5e0ee0 where: 8193 ret: 1
> [Oct 7 09:24:57.590] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl: 0xa5e0ee0 where: 8193 ret: 1
> [Oct 7 09:24:57.590] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl: 0xa5e0ee0 where: 8193 ret: 1
> [Oct 7 09:24:57.590] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl: 0xa5e0ee0 where: 8193 ret: 1
> [Oct 7 09:24:57.590] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl: 0xa5e0ee0 where: 8193 ret: 1
> [Oct 7 09:24:57.590] Server {0x40709730} DEBUG: (ssl) ssl_callback_info ssl: 0xa5e0ee0 where: 32 ret: 1
> NOTE: Traffic Server received Sig 11: Segmentation fault
> /usr/bin/traffic_server - STACK TRACE:
> [0x4001e500]
> /usr/lib/trafficserver/libtsutil.so.5(ink_hash_table_lookup_entry+0x12)[0x4003c0f2]
> /usr/lib/trafficserver/libtsutil.so.5(ink_hash_table_lookup+0x24)[0x4003c3b4]
> /usr/bin/traffic_server[0x8308185]
> /usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0(+0x12844)[0x40067844]
> /usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0(SSL_accept+0x2a)[0x4008c73a]
> /usr/bin/traffic_server(_ZN17SSLNetVConnection23sslServerHandShakeEventERi+0x19)[0x8303d89]
> /usr/bin/traffic_server(_ZN17SSLNetVConnection17sslStartHandShakeEiRi+0x2b)[0x830446b]
> /usr/bin/traffic_server(_ZN17SSLNetVConnection11net_read_ioEP10NetHandlerP7EThread+0xb30)[0x8305270]
> /usr/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x27f)[0x830dd7f]
> /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x98)[0x8339cf8]
> /usr/bin/traffic_server(_ZN7EThread7executeEv+0x419)[0x833a449]
> /usr/bin/traffic_server(main+0xf40)[0x80d4e30]
> /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xe6)[0x405c9e36]
> /usr/bin/traffic_server[0x80da229]
> [TrafficServer] using root directory '/usr'
> [Oct 7 09:24:59.682] Server {0x40709730} DEBUG: (ssl) setting SNI callbacks with for ctx 0x8a6b8b8
> [Oct 7 09:24:59.682] Server {0x40709730} DEBUG: (ssl) importing SNI names from /etc/trafficserver/ssl/new2014/100.pem
> [Oct 7 09:24:59.683] Server {0x40709730} DEBUG: (ssl) mapping 'XXXX' to certificate /etc/trafficserver/ssl/new2014/100.pem
> [Oct 7 09:24:59.683] Server {0x40709730} DEBUG: (ssl) indexed 'XXXX' with SSL_CTX 0x8a6b8b8
> [Oct 7 09:24:59.683] Server {0x40709730} DEBUG: (ssl) mapping 'XXXX' to certificate /etc/trafficserver/ssl/new2014/100.pem
> [Oct 7 09:24:59.683] Server {0x40709730} DEBUG: (ssl) indexed 'XXXX' with SSL_CTX 0x8a6b8b8
> [Oct 7 09:24:59.683] Server {0x40709730} DEBUG: (ssl) mapping 'YYYYYY' to certificate /etc/trafficserver/ssl/new2014/100.pem
> [Oct 7 09:24:59.683] Server {0x40709730} DEBUG: (ssl) indexed 'YYYYYY' with SSL_CTX 0x8a6b8b8
> [Oct 7 09:24:59.683] Server {0x40709730} DEBUG: (ssl) setting SNI callbacks with for ctx 0x8a76b30
> [Oct 7 09:24:59.684] Server {0x40709730} DEBUG: (ssl) indexed '*' with SSL_CTX 0x8a76b30
> [Oct 7 09:24:59.684] Server {0x40709730} DEBUG: (ssl) importing SNI names from /etc/trafficserver/ssl
>
>
>
>
>
> Le 03/10/2014 18:43, James Peach a écrit :
>> On Oct 3, 2014, at 3:32 AM, Matthieu Bienvenüe <ma...@exultet.net> wrote:
>>
>>> Any idea to solve this isssu ?
>> I did a quick test of setting proxy.config.ssl.number.threads to -1, and it didn't crash for me. Can you enable ssl diagnostics and try again?
>>
>> CONFIG proxy.config.diags.debug.enabled INT 1
>> CONFIG proxy.config.diags.debug.tags STRING ssl
>>
>>> Matthieu
>>>
>>>
>>> Le 01/10/2014 09:50, Matthieu Bienvenüe a écrit :
>>>> Le 30/09/2014 17:47, Leif Hedstrom a écrit :
>>>>> On Sep 30, 2014, at 9:00 AM, Matthieu Bienvenüe <ma...@exultet.net> wrote:
>>>>>
>>>>>> Is that possible to do it on config instead of recompiling ATS ?
>>>>>
>>>>> What version are you using? I’m not 100% certain, but I’d expect Geffon’s additions to not have dedicated SSL threads would avoid the need for that patch as well? Brian? If I recall, with a recent version of ATS, you’d simply set proxy.config.ssl.number.threads to -1.
>>>> When I set this settings SSL don't work and I've the following stack trace :
>>>>
>>>> NOTE: Traffic Server received Sig 11: Segmentation fault
>>>> /usr/bin/traffic_server - STACK TRACE:
>>>> [0x4001e500]
>>>> /usr/lib/trafficserver/libtsutil.so.5(ink_hash_table_lookup_entry+0x12)[0x4003c0f2]
>>>> /usr/lib/trafficserver/libtsutil.so.5(ink_hash_table_lookup+0x24)[0x4003c3b4]
>>>> /usr/bin/traffic_server[0x8308185]
>>>> /usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0(+0x12844)[0x40067844]
>>>> /usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0(SSL_accept+0x2a)[0x4008c73a]
>>>> /usr/bin/traffic_server(_ZN17SSLNetVConnection23sslServerHandShakeEventERi+0x19)[0x8303d89]
>>>> /usr/bin/traffic_server(_ZN17SSLNetVConnection17sslStartHandShakeEiRi+0x2b)[0x830446b]
>>>> /usr/bin/traffic_server(_ZN17SSLNetVConnection11net_read_ioEP10NetHandlerP7EThread+0xb30)[0x8305270]
>>>> /usr/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x27f)[0x830dd7f]
>>>> /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x98)[0x8339cf8]
>>>> /usr/bin/traffic_server(_ZN7EThread7executeEv+0x419)[0x833a449]
>>>> /usr/bin/traffic_server(main+0xf40)[0x80d4e30]
>>>> /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xe6)[0x405c9e36]
>>>> /usr/bin/traffic_server[0x80da229]
>>>> [TrafficServer] using root directory '/usr'
>>>>
>>>>> In either case, why is that patch not committed? Is there a Jira for it?
>>>>>
>>>>> — Leif
>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Matt
>>>>>> Le 30/09/2014 16:49, 英才 a écrit :
>>>>>>> disable AIO or patch https://github.com/phonehold/with-aio-ssl-init may solve your problem
>>>>>>>
>>>>>>> 在 2014年9月30日，下午10:41，Matthieu Bienvenüe <ma...@exultet.net> 写道：
>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>>
>>>>>>>> SSL works fine with my certs, but it crashes only after a certain amount of time/requests.
>>>>>>>>
>>>>>>>> Here is the stack trace from traffic.out:
>>>>>>>>
>>>>>>>> NOTE: Traffic Server received Sig 11: Segmentation fault
>>>>>>>> /usr/bin/traffic_server - STACK TRACE:
>>>>>>>> [0x4001e500]
>>>>>>>> /usr/bin/traffic_server(_Z12ink_aio_readP11AIOCallbacki+0x2a)[0x830056a]
>>>>>>>> /usr/bin/traffic_server(_ZN7CacheVC10handleReadEiP5Event+0x282)[0x82c4402]
>>>>>>>> /usr/bin/traffic_server(_ZN5Cache9open_readEP12ContinuationP7INK_MD5P7HTTPHdrP21CacheLookupHttpConfig13CacheFragTypePci+0x5be)[0x82df68e]
>>>>>>>> /usr/bin/traffic_server(_ZN14CacheProcessor9open_readEP12ContinuationP3URLbP7HTTPHdrP21CacheLookupHttpConfigl13CacheFragType+0xdc)[0x82c2b4c]
>>>>>>>> /usr/bin/traffic_server(_ZN11HttpCacheSM18do_cache_open_readEv+0x63)[0x81ab6f3]
>>>>>>>> /usr/bin/traffic_server(_ZN11HttpCacheSM9open_readEP3URLP7HTTPHdrP21CacheLookupHttpConfigl+0x4c)[0x81aba0c]
>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM24do_cache_lookup_and_readEv+0x115)[0x81bd105]
>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x6af)[0x81ce7bf]
>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x108)[0x81cc8d8]
>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x300)[0x81c9940]
>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM23do_api_callout_internalEv+0x54)[0x81c9da4]
>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x250)[0x81ce360]
>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x7eb)[0x81ce8fb]
>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x108)[0x81cc8d8]
>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x300)[0x81c9940]
>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM23do_api_callout_internalEv+0x54)[0x81c9da4]
>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x250)[0x81ce360]
>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x108)[0x81cc8d8]
>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x300)[0x81c9940]
>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM18state_api_callbackEiPv+0x78)[0x81cc398]
>>>>>>>> /usr/bin/traffic_server(TSHttpTxnReenable+0x1f0)[0x810ef50]
>>>>>>>> /usr/lib/trafficserver/modules/stats_over_http.so(+0x102e)[0x4095f02e]
>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0xd8)[0x81c9718]
>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM23do_api_callout_internalEv+0x54)[0x81c9da4]
>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x250)[0x81ce360]
>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM32state_read_client_request_headerEiPv+0x1e8)[0x81c5738]
>>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x7e)[0x81ca93e]
>>>>>>>> /usr/bin/traffic_server(_ZN18UnixNetVConnection19readSignalAndUpdateEi+0x45)[0x83166a5]
>>>>>>>> /usr/bin/traffic_server(_ZN17SSLNetVConnection11net_read_ioEP10NetHandlerP7EThread+0x10b0)[0x83057f0]
>>>>>>>> /usr/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x27f)[0x830dd7f]
>>>>>>>> /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x98)[0x8339cf8]
>>>>>>>> /usr/bin/traffic_server(_ZN7EThread7executeEv+0x419)[0x833a449]
>>>>>>>> /usr/bin/traffic_server[0x8338ebb]
>>>>>>>> /lib/i386-linux-gnu/libpthread.so.0(+0x5954)[0x4046b954]
>>>>>>>> /lib/i386-linux-gnu/libc.so.6(clone+0x5e)[0x40688cbe]
>>>>>>>> [E. Mgmt] log ==> [TrafficManager] using root directory '/usr'
>>>>>>>> [TrafficServer] using root directory '/usr'
>>>>>>>>
>>>>>>>> Here is my record.config for SSL parameters:
>>>>>>>>
>>>>>>>> CONFIG proxy.config.http.server_ports STRING 8080 4443:ssl
>>>>>>>>
>>>>>>>> CONFIG proxy.config.ssl.enabled INT 1
>>>>>>>> CONFIG proxy.config.ssl.server.cert.path STRING /etc/trafficserver/ssl/
>>>>>>>> CONFIG proxy.config.ssl.server.private_key.path STRING /etc/trafficserver/ssl/
>>>>>>>>
>>>>>>>> And for ssl_multicert.config:
>>>>>>>>
>>>>>>>> ssl_cert_name=new2014/100.pem ssl_key_name=new2014/100.key
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Le 30/09/2014 15:54, Susan Hinrichs a écrit :
>>>>>>>>> Matt,
>>>>>>>>>
>>>>>>>>> Is there a basic stack trace in traffic.out? What is your SSL configuration? Do you have certs set up in ssl_multicert.config? Or are you doing a blind tunnel on the SSL traffic?
>>>>>>>>>
>>>>>>>>> Susan
>>>>>>>>>
>>>>>>>>> On 9/30/2014 2:14 AM, Matthieu Bienvenüe wrote:
>>>>>>>>>> Hello !
>>>>>>>>>>
>>>>>>>>>> I'm configuring ATS as a reverse proxy and I need SSL support.
>>>>>>>>>>
>>>>>>>>>> ATS runs on OpenVZ on Debian. It's the version 5.0.1 installed from backport packages.
>>>>>>>>>>
>>>>>>>>>> ATS works fine, SSL too. But after a while SSL makes ATS crash.
>>>>>>>>>>
>>>>>>>>>> In manager.log I found that there is a segmentation fault:
>>>>>>>>>>
>>>>>>>>>> [Sep 29 16:08:33.020] Manager {0xb6fb76d0} ERROR: [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 11: Segmentation fault
>>>>>>>>>> [Sep 29 16:08:33.021] Manager {0xb6fb76d0} ERROR: [Alarms::signalAlarm] Server Process was reset
>>>>>>>>>> [Sep 29 16:08:34.041] Manager {0xb6fb76d0} NOTE: [LocalManager::startProxy] Launching ts process
>>>>>>>>>> [Sep 29 16:08:34.049] Manager {0xb6fb76d0} NOTE: [LocalManager::pollMgmtProcessServer] New process connecting fd '16'
>>>>>>>>>> [Sep 29 16:08:34.049] Manager {0xb6fb76d0} NOTE: [Alarms::signalAlarm] Server Process born
>>>>>>>>>>
>>>>>>>>>> Here is a dump of the syslog when crashing:
>>>>>>>>>>
>>>>>>>>>> Sep 30 07:05:09 ats traffic_manager[5471]: {0xb704d6d0} FATAL: [LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
>>>>>>>>>> Sep 30 07:05:09 ats traffic_manager[5471]: {0xb704d6d0} ERROR: [LocalManager::sendMgmtMsgToProcesses] Error writing message
>>>>>>>>>> Sep 30 07:05:09 ats traffic_manager[5471]: {0xb704d6d0} ERROR: (last system error 32: Broken pipe)
>>>>>>>>>> Sep 30 07:05:09 ats traffic_cop[23694]: cop received child status signal [5471 256]
>>>>>>>>>> Sep 30 07:05:09 ats traffic_cop[23694]: traffic_manager not running, making sure traffic_server is dead
>>>>>>>>>> Sep 30 07:05:09 ats traffic_cop[23694]: spawning traffic_manager
>>>>>>>>>> Sep 30 07:05:09 ats traffic_manager[6938]: NOTE: --- Manager Starting ---
>>>>>>>>>> Sep 30 07:05:09 ats traffic_manager[6938]: NOTE: Manager Version: Apache Traffic Server - traffic_manager - 5.0.1 - (build # 7259 on Aug 25 2014 at 09:26:11)
>>>>>>>>>> Sep 30 07:05:09 ats traffic_manager[6938]: NOTE: Unable to set RLIMIT_NOFILE(7):cur(1475961),max(1475961)
>>>>>>>>>> Sep 30 07:05:09 ats traffic_manager[6938]: NOTE: RLIMIT_NOFILE(7):cur(30000),max(30000)
>>>>>>>>>> Sep 30 07:05:09 ats traffic_manager[6938]: ERROR ==> [runAsUser] Error: Failed to restore capabilities after switch to user trafficserver.
>>>>>>>>>> Sep 30 07:05:11 ats traffic_server[6946]: NOTE: --- traffic_server Starting ---
>>>>>>>>>> Sep 30 07:05:11 ats traffic_server[6946]: NOTE: traffic_server Version: Apache Traffic Server - traffic_server - 5.0.1 - (build # 7259 on Aug 25 2014 at 09:27:18)
>>>>>>>>>> Sep 30 07:05:11 ats traffic_server[6946]: NOTE: Unable to set RLIMIT_NOFILE(7):cur(-611778560),max(-611778560)
>>>>>>>>>> Sep 30 07:05:13 ats traffic_manager[6938]: {0xb708b6d0} ERROR: [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 11: Segmentation fault
>>>>>>>>>> Sep 30 07:05:13 ats traffic_manager[6938]: {0xb708b6d0} ERROR: [Alarms::signalAlarm] Server Process was reset
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Any idea where to look for to solve this problem ?
>>>>>>>>>>
>>>>>>>>>> Thanks a lot !
>>>>>>>>>>
>>>>>>>>>> Matt
>>>>>>>>
>
Re: SSL results in segmentation fault
Posted by Matthieu Bienvenüe <ma...@exultet.net>.
OK so here is what I get in traffic.out when setting :
CONFIG proxy.config.ssl.number.threads INT -1
CONFIG proxy.config.diags.debug.enabled INT 1
CONFIG proxy.config.diags.debug.tags STRING ssl
[Oct 7 09:24:57.497] Server {0x40709730} DEBUG: (ssl)
[SSLNextProtocolAccept:mainEvent] event 202 netvc 0x438d0000
[Oct 7 09:24:57.497] Server {0x40709730} DEBUG: (ssl) ssl_callback_info
ssl: 0xa5e0ee0 where: 16 ret: 1
[Oct 7 09:24:57.497] Server {0x40709730} DEBUG: (ssl) ssl_callback_info
ssl: 0xa5e0ee0 where: 8193 ret: 1
[Oct 7 09:24:57.497] Server {0x40709730} DEBUG: (ssl)
ssl_servername_callback ssl=0xa5e0ee0 ad=112 lookup=0xa54f9c0
server=XXXX handshake_complete=0
[Oct 7 09:24:57.497] Server {0x40709730} DEBUG: (ssl)
ssl_servername_callback found SSL context 0xa552960 for requested name
'XXXX'
[Oct 7 09:24:57.498] Server {0x40709730} DEBUG: (ssl) ssl_callback_info
ssl: 0xa5e0ee0 where: 8193 ret: 1
[Oct 7 09:24:57.498] Server {0x40709730} DEBUG: (ssl) advertising
protocol http/1.1
[Oct 7 09:24:57.498] Server {0x40709730} DEBUG: (ssl) advertising
protocol http/1.0
[Oct 7 09:24:57.498] Server {0x40709730} DEBUG: (ssl) ssl_callback_info
ssl: 0xa5e0ee0 where: 8193 ret: 1
[Oct 7 09:24:57.498] Server {0x40709730} DEBUG: (ssl) ssl_callback_info
ssl: 0xa5e0ee0 where: 8193 ret: 1
[Oct 7 09:24:57.527] Server {0x40709730} DEBUG: (ssl) ssl_callback_info
ssl: 0xa5e0ee0 where: 8193 ret: 1
[Oct 7 09:24:57.527] Server {0x40709730} DEBUG: (ssl) ssl_callback_info
ssl: 0xa5e0ee0 where: 8193 ret: 1
[Oct 7 09:24:57.527] Server {0x40709730} DEBUG: (ssl) ssl_callback_info
ssl: 0xa5e0ee0 where: 8193 ret: 1
[Oct 7 09:24:57.528] Server {0x40709730} DEBUG: (ssl) ssl_callback_info
ssl: 0xa5e0ee0 where: 8194 ret: -1
[Oct 7 09:24:57.528] Server {0x40709730} DEBUG: (ssl) ssl_callback_info
ssl: 0xa5e0ee0 where: 8194 ret: -1
[Oct 7 09:24:57.528] Server {0x40709730} DEBUG:
<SSLNetVConnection.cc:558 (sslServerHandShakeEvent)> (ssl) SSL handshake
error: SSL_ERROR_WANT_READ (2), errno=11
[Oct 7 09:24:57.589] Server {0x40709730} DEBUG: (ssl) ssl_callback_info
ssl: 0xa5e0ee0 where: 8193 ret: 1
[Oct 7 09:24:57.589] Server {0x40709730} DEBUG: (ssl) ssl_callback_info
ssl: 0xa5e0ee0 where: 8193 ret: 1
[Oct 7 09:24:57.589] Server {0x40709730} DEBUG: (ssl) ssl_callback_info
ssl: 0xa5e0ee0 where: 8193 ret: 1
[Oct 7 09:24:57.590] Server {0x40709730} DEBUG: (ssl) ssl_callback_info
ssl: 0xa5e0ee0 where: 8193 ret: 1
[Oct 7 09:24:57.590] Server {0x40709730} DEBUG: (ssl) ssl_callback_info
ssl: 0xa5e0ee0 where: 8193 ret: 1
[Oct 7 09:24:57.590] Server {0x40709730} DEBUG: (ssl) ssl_callback_info
ssl: 0xa5e0ee0 where: 8193 ret: 1
[Oct 7 09:24:57.590] Server {0x40709730} DEBUG: (ssl) ssl_callback_info
ssl: 0xa5e0ee0 where: 8193 ret: 1
[Oct 7 09:24:57.590] Server {0x40709730} DEBUG: (ssl) ssl_callback_info
ssl: 0xa5e0ee0 where: 32 ret: 1
NOTE: Traffic Server received Sig 11: Segmentation fault
/usr/bin/traffic_server - STACK TRACE:
[0x4001e500]
/usr/lib/trafficserver/libtsutil.so.5(ink_hash_table_lookup_entry+0x12)[0x4003c0f2]
/usr/lib/trafficserver/libtsutil.so.5(ink_hash_table_lookup+0x24)[0x4003c3b4]
/usr/bin/traffic_server[0x8308185]
/usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0(+0x12844)[0x40067844]
/usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0(SSL_accept+0x2a)[0x4008c73a]
/usr/bin/traffic_server(_ZN17SSLNetVConnection23sslServerHandShakeEventERi+0x19)[0x8303d89]
/usr/bin/traffic_server(_ZN17SSLNetVConnection17sslStartHandShakeEiRi+0x2b)[0x830446b]
/usr/bin/traffic_server(_ZN17SSLNetVConnection11net_read_ioEP10NetHandlerP7EThread+0xb30)[0x8305270]
/usr/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x27f)[0x830dd7f]
/usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x98)[0x8339cf8]
/usr/bin/traffic_server(_ZN7EThread7executeEv+0x419)[0x833a449]
/usr/bin/traffic_server(main+0xf40)[0x80d4e30]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xe6)[0x405c9e36]
/usr/bin/traffic_server[0x80da229]
[TrafficServer] using root directory '/usr'
[Oct 7 09:24:59.682] Server {0x40709730} DEBUG: (ssl) setting SNI
callbacks with for ctx 0x8a6b8b8
[Oct 7 09:24:59.682] Server {0x40709730} DEBUG: (ssl) importing SNI
names from /etc/trafficserver/ssl/new2014/100.pem
[Oct 7 09:24:59.683] Server {0x40709730} DEBUG: (ssl) mapping 'XXXX' to
certificate /etc/trafficserver/ssl/new2014/100.pem
[Oct 7 09:24:59.683] Server {0x40709730} DEBUG: (ssl) indexed 'XXXX'
with SSL_CTX 0x8a6b8b8
[Oct 7 09:24:59.683] Server {0x40709730} DEBUG: (ssl) mapping 'XXXX' to
certificate /etc/trafficserver/ssl/new2014/100.pem
[Oct 7 09:24:59.683] Server {0x40709730} DEBUG: (ssl) indexed 'XXXX'
with SSL_CTX 0x8a6b8b8
[Oct 7 09:24:59.683] Server {0x40709730} DEBUG: (ssl) mapping 'YYYYYY'
to certificate /etc/trafficserver/ssl/new2014/100.pem
[Oct 7 09:24:59.683] Server {0x40709730} DEBUG: (ssl) indexed 'YYYYYY'
with SSL_CTX 0x8a6b8b8
[Oct 7 09:24:59.683] Server {0x40709730} DEBUG: (ssl) setting SNI
callbacks with for ctx 0x8a76b30
[Oct 7 09:24:59.684] Server {0x40709730} DEBUG: (ssl) indexed '*' with
SSL_CTX 0x8a76b30
[Oct 7 09:24:59.684] Server {0x40709730} DEBUG: (ssl) importing SNI
names from /etc/trafficserver/ssl
Le 03/10/2014 18:43, James Peach a écrit :
> On Oct 3, 2014, at 3:32 AM, Matthieu Bienvenüe <ma...@exultet.net> wrote:
>
>> Any idea to solve this isssu ?
> I did a quick test of setting proxy.config.ssl.number.threads to -1, and it didn't crash for me. Can you enable ssl diagnostics and try again?
>
> CONFIG proxy.config.diags.debug.enabled INT 1
> CONFIG proxy.config.diags.debug.tags STRING ssl
>
>> Matthieu
>>
>>
>> Le 01/10/2014 09:50, Matthieu Bienvenüe a écrit :
>>> Le 30/09/2014 17:47, Leif Hedstrom a écrit :
>>>> On Sep 30, 2014, at 9:00 AM, Matthieu Bienvenüe <ma...@exultet.net> wrote:
>>>>
>>>>> Is that possible to do it on config instead of recompiling ATS ?
>>>>
>>>> What version are you using? I’m not 100% certain, but I’d expect Geffon’s additions to not have dedicated SSL threads would avoid the need for that patch as well? Brian? If I recall, with a recent version of ATS, you’d simply set proxy.config.ssl.number.threads to -1.
>>> When I set this settings SSL don't work and I've the following stack trace :
>>>
>>> NOTE: Traffic Server received Sig 11: Segmentation fault
>>> /usr/bin/traffic_server - STACK TRACE:
>>> [0x4001e500]
>>> /usr/lib/trafficserver/libtsutil.so.5(ink_hash_table_lookup_entry+0x12)[0x4003c0f2]
>>> /usr/lib/trafficserver/libtsutil.so.5(ink_hash_table_lookup+0x24)[0x4003c3b4]
>>> /usr/bin/traffic_server[0x8308185]
>>> /usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0(+0x12844)[0x40067844]
>>> /usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0(SSL_accept+0x2a)[0x4008c73a]
>>> /usr/bin/traffic_server(_ZN17SSLNetVConnection23sslServerHandShakeEventERi+0x19)[0x8303d89]
>>> /usr/bin/traffic_server(_ZN17SSLNetVConnection17sslStartHandShakeEiRi+0x2b)[0x830446b]
>>> /usr/bin/traffic_server(_ZN17SSLNetVConnection11net_read_ioEP10NetHandlerP7EThread+0xb30)[0x8305270]
>>> /usr/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x27f)[0x830dd7f]
>>> /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x98)[0x8339cf8]
>>> /usr/bin/traffic_server(_ZN7EThread7executeEv+0x419)[0x833a449]
>>> /usr/bin/traffic_server(main+0xf40)[0x80d4e30]
>>> /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xe6)[0x405c9e36]
>>> /usr/bin/traffic_server[0x80da229]
>>> [TrafficServer] using root directory '/usr'
>>>
>>>> In either case, why is that patch not committed? Is there a Jira for it?
>>>>
>>>> — Leif
>>>>
>>>>> Regards,
>>>>>
>>>>> Matt
>>>>> Le 30/09/2014 16:49, 英才 a écrit :
>>>>>> disable AIO or patch https://github.com/phonehold/with-aio-ssl-init may solve your problem
>>>>>>
>>>>>> 在 2014年9月30日，下午10:41，Matthieu Bienvenüe <ma...@exultet.net> 写道：
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>>
>>>>>>> SSL works fine with my certs, but it crashes only after a certain amount of time/requests.
>>>>>>>
>>>>>>> Here is the stack trace from traffic.out:
>>>>>>>
>>>>>>> NOTE: Traffic Server received Sig 11: Segmentation fault
>>>>>>> /usr/bin/traffic_server - STACK TRACE:
>>>>>>> [0x4001e500]
>>>>>>> /usr/bin/traffic_server(_Z12ink_aio_readP11AIOCallbacki+0x2a)[0x830056a]
>>>>>>> /usr/bin/traffic_server(_ZN7CacheVC10handleReadEiP5Event+0x282)[0x82c4402]
>>>>>>> /usr/bin/traffic_server(_ZN5Cache9open_readEP12ContinuationP7INK_MD5P7HTTPHdrP21CacheLookupHttpConfig13CacheFragTypePci+0x5be)[0x82df68e]
>>>>>>> /usr/bin/traffic_server(_ZN14CacheProcessor9open_readEP12ContinuationP3URLbP7HTTPHdrP21CacheLookupHttpConfigl13CacheFragType+0xdc)[0x82c2b4c]
>>>>>>> /usr/bin/traffic_server(_ZN11HttpCacheSM18do_cache_open_readEv+0x63)[0x81ab6f3]
>>>>>>> /usr/bin/traffic_server(_ZN11HttpCacheSM9open_readEP3URLP7HTTPHdrP21CacheLookupHttpConfigl+0x4c)[0x81aba0c]
>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM24do_cache_lookup_and_readEv+0x115)[0x81bd105]
>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x6af)[0x81ce7bf]
>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x108)[0x81cc8d8]
>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x300)[0x81c9940]
>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM23do_api_callout_internalEv+0x54)[0x81c9da4]
>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x250)[0x81ce360]
>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x7eb)[0x81ce8fb]
>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x108)[0x81cc8d8]
>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x300)[0x81c9940]
>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM23do_api_callout_internalEv+0x54)[0x81c9da4]
>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x250)[0x81ce360]
>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x108)[0x81cc8d8]
>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x300)[0x81c9940]
>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM18state_api_callbackEiPv+0x78)[0x81cc398]
>>>>>>> /usr/bin/traffic_server(TSHttpTxnReenable+0x1f0)[0x810ef50]
>>>>>>> /usr/lib/trafficserver/modules/stats_over_http.so(+0x102e)[0x4095f02e]
>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0xd8)[0x81c9718]
>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM23do_api_callout_internalEv+0x54)[0x81c9da4]
>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x250)[0x81ce360]
>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM32state_read_client_request_headerEiPv+0x1e8)[0x81c5738]
>>>>>>> /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x7e)[0x81ca93e]
>>>>>>> /usr/bin/traffic_server(_ZN18UnixNetVConnection19readSignalAndUpdateEi+0x45)[0x83166a5]
>>>>>>> /usr/bin/traffic_server(_ZN17SSLNetVConnection11net_read_ioEP10NetHandlerP7EThread+0x10b0)[0x83057f0]
>>>>>>> /usr/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x27f)[0x830dd7f]
>>>>>>> /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x98)[0x8339cf8]
>>>>>>> /usr/bin/traffic_server(_ZN7EThread7executeEv+0x419)[0x833a449]
>>>>>>> /usr/bin/traffic_server[0x8338ebb]
>>>>>>> /lib/i386-linux-gnu/libpthread.so.0(+0x5954)[0x4046b954]
>>>>>>> /lib/i386-linux-gnu/libc.so.6(clone+0x5e)[0x40688cbe]
>>>>>>> [E. Mgmt] log ==> [TrafficManager] using root directory '/usr'
>>>>>>> [TrafficServer] using root directory '/usr'
>>>>>>>
>>>>>>> Here is my record.config for SSL parameters:
>>>>>>>
>>>>>>> CONFIG proxy.config.http.server_ports STRING 8080 4443:ssl
>>>>>>>
>>>>>>> CONFIG proxy.config.ssl.enabled INT 1
>>>>>>> CONFIG proxy.config.ssl.server.cert.path STRING /etc/trafficserver/ssl/
>>>>>>> CONFIG proxy.config.ssl.server.private_key.path STRING /etc/trafficserver/ssl/
>>>>>>>
>>>>>>> And for ssl_multicert.config:
>>>>>>>
>>>>>>> ssl_cert_name=new2014/100.pem ssl_key_name=new2014/100.key
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Le 30/09/2014 15:54, Susan Hinrichs a écrit :
>>>>>>>> Matt,
>>>>>>>>
>>>>>>>> Is there a basic stack trace in traffic.out? What is your SSL configuration? Do you have certs set up in ssl_multicert.config? Or are you doing a blind tunnel on the SSL traffic?
>>>>>>>>
>>>>>>>> Susan
>>>>>>>>
>>>>>>>> On 9/30/2014 2:14 AM, Matthieu Bienvenüe wrote:
>>>>>>>>> Hello !
>>>>>>>>>
>>>>>>>>> I'm configuring ATS as a reverse proxy and I need SSL support.
>>>>>>>>>
>>>>>>>>> ATS runs on OpenVZ on Debian. It's the version 5.0.1 installed from backport packages.
>>>>>>>>>
>>>>>>>>> ATS works fine, SSL too. But after a while SSL makes ATS crash.
>>>>>>>>>
>>>>>>>>> In manager.log I found that there is a segmentation fault:
>>>>>>>>>
>>>>>>>>> [Sep 29 16:08:33.020] Manager {0xb6fb76d0} ERROR: [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 11: Segmentation fault
>>>>>>>>> [Sep 29 16:08:33.021] Manager {0xb6fb76d0} ERROR: [Alarms::signalAlarm] Server Process was reset
>>>>>>>>> [Sep 29 16:08:34.041] Manager {0xb6fb76d0} NOTE: [LocalManager::startProxy] Launching ts process
>>>>>>>>> [Sep 29 16:08:34.049] Manager {0xb6fb76d0} NOTE: [LocalManager::pollMgmtProcessServer] New process connecting fd '16'
>>>>>>>>> [Sep 29 16:08:34.049] Manager {0xb6fb76d0} NOTE: [Alarms::signalAlarm] Server Process born
>>>>>>>>>
>>>>>>>>> Here is a dump of the syslog when crashing:
>>>>>>>>>
>>>>>>>>> Sep 30 07:05:09 ats traffic_manager[5471]: {0xb704d6d0} FATAL: [LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
>>>>>>>>> Sep 30 07:05:09 ats traffic_manager[5471]: {0xb704d6d0} ERROR: [LocalManager::sendMgmtMsgToProcesses] Error writing message
>>>>>>>>> Sep 30 07:05:09 ats traffic_manager[5471]: {0xb704d6d0} ERROR: (last system error 32: Broken pipe)
>>>>>>>>> Sep 30 07:05:09 ats traffic_cop[23694]: cop received child status signal [5471 256]
>>>>>>>>> Sep 30 07:05:09 ats traffic_cop[23694]: traffic_manager not running, making sure traffic_server is dead
>>>>>>>>> Sep 30 07:05:09 ats traffic_cop[23694]: spawning traffic_manager
>>>>>>>>> Sep 30 07:05:09 ats traffic_manager[6938]: NOTE: --- Manager Starting ---
>>>>>>>>> Sep 30 07:05:09 ats traffic_manager[6938]: NOTE: Manager Version: Apache Traffic Server - traffic_manager - 5.0.1 - (build # 7259 on Aug 25 2014 at 09:26:11)
>>>>>>>>> Sep 30 07:05:09 ats traffic_manager[6938]: NOTE: Unable to set RLIMIT_NOFILE(7):cur(1475961),max(1475961)
>>>>>>>>> Sep 30 07:05:09 ats traffic_manager[6938]: NOTE: RLIMIT_NOFILE(7):cur(30000),max(30000)
>>>>>>>>> Sep 30 07:05:09 ats traffic_manager[6938]: ERROR ==> [runAsUser] Error: Failed to restore capabilities after switch to user trafficserver.
>>>>>>>>> Sep 30 07:05:11 ats traffic_server[6946]: NOTE: --- traffic_server Starting ---
>>>>>>>>> Sep 30 07:05:11 ats traffic_server[6946]: NOTE: traffic_server Version: Apache Traffic Server - traffic_server - 5.0.1 - (build # 7259 on Aug 25 2014 at 09:27:18)
>>>>>>>>> Sep 30 07:05:11 ats traffic_server[6946]: NOTE: Unable to set RLIMIT_NOFILE(7):cur(-611778560),max(-611778560)
>>>>>>>>> Sep 30 07:05:13 ats traffic_manager[6938]: {0xb708b6d0} ERROR: [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 11: Segmentation fault
>>>>>>>>> Sep 30 07:05:13 ats traffic_manager[6938]: {0xb708b6d0} ERROR: [Alarms::signalAlarm] Server Process was reset
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Any idea where to look for to solve this problem ?
>>>>>>>>>
>>>>>>>>> Thanks a lot !
>>>>>>>>>
>>>>>>>>> Matt
>>>>>>>
Re: SSL results in segmentation fault
Posted by James Peach <jp...@apache.org>.
On Oct 3, 2014, at 3:32 AM, Matthieu Bienvenüe <ma...@exultet.net> wrote:
> Any idea to solve this isssu ?
I did a quick test of setting proxy.config.ssl.number.threads to -1, and it didn't crash for me. Can you enable ssl diagnostics and try again?
CONFIG proxy.config.diags.debug.enabled INT 1
CONFIG proxy.config.diags.debug.tags STRING ssl
>
> Matthieu
>
>
> Le 01/10/2014 09:50, Matthieu Bienvenüe a écrit :
>>
>> Le 30/09/2014 17:47, Leif Hedstrom a écrit :
>>>
>>> On Sep 30, 2014, at 9:00 AM, Matthieu Bienvenüe <ma...@exultet.net> wrote:
>>>
>>>> Is that possible to do it on config instead of recompiling ATS ?
>>>
>>>
>>> What version are you using? I’m not 100% certain, but I’d expect Geffon’s additions to not have dedicated SSL threads would avoid the need for that patch as well? Brian? If I recall, with a recent version of ATS, you’d simply set proxy.config.ssl.number.threads to -1.
>> When I set this settings SSL don't work and I've the following stack trace :
>>
>> NOTE: Traffic Server received Sig 11: Segmentation fault
>> /usr/bin/traffic_server - STACK TRACE:
>> [0x4001e500]
>> /usr/lib/trafficserver/libtsutil.so.5(ink_hash_table_lookup_entry+0x12)[0x4003c0f2]
>> /usr/lib/trafficserver/libtsutil.so.5(ink_hash_table_lookup+0x24)[0x4003c3b4]
>> /usr/bin/traffic_server[0x8308185]
>> /usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0(+0x12844)[0x40067844]
>> /usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0(SSL_accept+0x2a)[0x4008c73a]
>> /usr/bin/traffic_server(_ZN17SSLNetVConnection23sslServerHandShakeEventERi+0x19)[0x8303d89]
>> /usr/bin/traffic_server(_ZN17SSLNetVConnection17sslStartHandShakeEiRi+0x2b)[0x830446b]
>> /usr/bin/traffic_server(_ZN17SSLNetVConnection11net_read_ioEP10NetHandlerP7EThread+0xb30)[0x8305270]
>> /usr/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x27f)[0x830dd7f]
>> /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x98)[0x8339cf8]
>> /usr/bin/traffic_server(_ZN7EThread7executeEv+0x419)[0x833a449]
>> /usr/bin/traffic_server(main+0xf40)[0x80d4e30]
>> /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xe6)[0x405c9e36]
>> /usr/bin/traffic_server[0x80da229]
>> [TrafficServer] using root directory '/usr'
>>
>>>
>>> In either case, why is that patch not committed? Is there a Jira for it?
>>>
>>> — Leif
>>>
>>>>
>>>> Regards,
>>>>
>>>> Matt
>>>> Le 30/09/2014 16:49, 英才 a écrit :
>>>>>
>>>>> disable AIO or patch https://github.com/phonehold/with-aio-ssl-init may solve your problem
>>>>>
>>>>> 在 2014年9月30日，下午10:41，Matthieu Bienvenüe <ma...@exultet.net> 写道：
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>>
>>>>>> SSL works fine with my certs, but it crashes only after a certain amount of time/requests.
>>>>>>
>>>>>> Here is the stack trace from traffic.out:
>>>>>>
>>>>>> NOTE: Traffic Server received Sig 11: Segmentation fault
>>>>>> /usr/bin/traffic_server - STACK TRACE:
>>>>>> [0x4001e500]
>>>>>> /usr/bin/traffic_server(_Z12ink_aio_readP11AIOCallbacki+0x2a)[0x830056a]
>>>>>> /usr/bin/traffic_server(_ZN7CacheVC10handleReadEiP5Event+0x282)[0x82c4402]
>>>>>> /usr/bin/traffic_server(_ZN5Cache9open_readEP12ContinuationP7INK_MD5P7HTTPHdrP21CacheLookupHttpConfig13CacheFragTypePci+0x5be)[0x82df68e]
>>>>>> /usr/bin/traffic_server(_ZN14CacheProcessor9open_readEP12ContinuationP3URLbP7HTTPHdrP21CacheLookupHttpConfigl13CacheFragType+0xdc)[0x82c2b4c]
>>>>>> /usr/bin/traffic_server(_ZN11HttpCacheSM18do_cache_open_readEv+0x63)[0x81ab6f3]
>>>>>> /usr/bin/traffic_server(_ZN11HttpCacheSM9open_readEP3URLP7HTTPHdrP21CacheLookupHttpConfigl+0x4c)[0x81aba0c]
>>>>>> /usr/bin/traffic_server(_ZN6HttpSM24do_cache_lookup_and_readEv+0x115)[0x81bd105]
>>>>>> /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x6af)[0x81ce7bf]
>>>>>> /usr/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x108)[0x81cc8d8]
>>>>>> /usr/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x300)[0x81c9940]
>>>>>> /usr/bin/traffic_server(_ZN6HttpSM23do_api_callout_internalEv+0x54)[0x81c9da4]
>>>>>> /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x250)[0x81ce360]
>>>>>> /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x7eb)[0x81ce8fb]
>>>>>> /usr/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x108)[0x81cc8d8]
>>>>>> /usr/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x300)[0x81c9940]
>>>>>> /usr/bin/traffic_server(_ZN6HttpSM23do_api_callout_internalEv+0x54)[0x81c9da4]
>>>>>> /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x250)[0x81ce360]
>>>>>> /usr/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x108)[0x81cc8d8]
>>>>>> /usr/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x300)[0x81c9940]
>>>>>> /usr/bin/traffic_server(_ZN6HttpSM18state_api_callbackEiPv+0x78)[0x81cc398]
>>>>>> /usr/bin/traffic_server(TSHttpTxnReenable+0x1f0)[0x810ef50]
>>>>>> /usr/lib/trafficserver/modules/stats_over_http.so(+0x102e)[0x4095f02e]
>>>>>> /usr/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0xd8)[0x81c9718]
>>>>>> /usr/bin/traffic_server(_ZN6HttpSM23do_api_callout_internalEv+0x54)[0x81c9da4]
>>>>>> /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x250)[0x81ce360]
>>>>>> /usr/bin/traffic_server(_ZN6HttpSM32state_read_client_request_headerEiPv+0x1e8)[0x81c5738]
>>>>>> /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x7e)[0x81ca93e]
>>>>>> /usr/bin/traffic_server(_ZN18UnixNetVConnection19readSignalAndUpdateEi+0x45)[0x83166a5]
>>>>>> /usr/bin/traffic_server(_ZN17SSLNetVConnection11net_read_ioEP10NetHandlerP7EThread+0x10b0)[0x83057f0]
>>>>>> /usr/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x27f)[0x830dd7f]
>>>>>> /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x98)[0x8339cf8]
>>>>>> /usr/bin/traffic_server(_ZN7EThread7executeEv+0x419)[0x833a449]
>>>>>> /usr/bin/traffic_server[0x8338ebb]
>>>>>> /lib/i386-linux-gnu/libpthread.so.0(+0x5954)[0x4046b954]
>>>>>> /lib/i386-linux-gnu/libc.so.6(clone+0x5e)[0x40688cbe]
>>>>>> [E. Mgmt] log ==> [TrafficManager] using root directory '/usr'
>>>>>> [TrafficServer] using root directory '/usr'
>>>>>>
>>>>>> Here is my record.config for SSL parameters:
>>>>>>
>>>>>> CONFIG proxy.config.http.server_ports STRING 8080 4443:ssl
>>>>>>
>>>>>> CONFIG proxy.config.ssl.enabled INT 1
>>>>>> CONFIG proxy.config.ssl.server.cert.path STRING /etc/trafficserver/ssl/
>>>>>> CONFIG proxy.config.ssl.server.private_key.path STRING /etc/trafficserver/ssl/
>>>>>>
>>>>>> And for ssl_multicert.config:
>>>>>>
>>>>>> ssl_cert_name=new2014/100.pem ssl_key_name=new2014/100.key
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Le 30/09/2014 15:54, Susan Hinrichs a écrit :
>>>>>>> Matt,
>>>>>>>
>>>>>>> Is there a basic stack trace in traffic.out? What is your SSL configuration? Do you have certs set up in ssl_multicert.config? Or are you doing a blind tunnel on the SSL traffic?
>>>>>>>
>>>>>>> Susan
>>>>>>>
>>>>>>> On 9/30/2014 2:14 AM, Matthieu Bienvenüe wrote:
>>>>>>>> Hello !
>>>>>>>>
>>>>>>>> I'm configuring ATS as a reverse proxy and I need SSL support.
>>>>>>>>
>>>>>>>> ATS runs on OpenVZ on Debian. It's the version 5.0.1 installed from backport packages.
>>>>>>>>
>>>>>>>> ATS works fine, SSL too. But after a while SSL makes ATS crash.
>>>>>>>>
>>>>>>>> In manager.log I found that there is a segmentation fault:
>>>>>>>>
>>>>>>>> [Sep 29 16:08:33.020] Manager {0xb6fb76d0} ERROR: [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 11: Segmentation fault
>>>>>>>> [Sep 29 16:08:33.021] Manager {0xb6fb76d0} ERROR: [Alarms::signalAlarm] Server Process was reset
>>>>>>>> [Sep 29 16:08:34.041] Manager {0xb6fb76d0} NOTE: [LocalManager::startProxy] Launching ts process
>>>>>>>> [Sep 29 16:08:34.049] Manager {0xb6fb76d0} NOTE: [LocalManager::pollMgmtProcessServer] New process connecting fd '16'
>>>>>>>> [Sep 29 16:08:34.049] Manager {0xb6fb76d0} NOTE: [Alarms::signalAlarm] Server Process born
>>>>>>>>
>>>>>>>> Here is a dump of the syslog when crashing:
>>>>>>>>
>>>>>>>> Sep 30 07:05:09 ats traffic_manager[5471]: {0xb704d6d0} FATAL: [LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
>>>>>>>> Sep 30 07:05:09 ats traffic_manager[5471]: {0xb704d6d0} ERROR: [LocalManager::sendMgmtMsgToProcesses] Error writing message
>>>>>>>> Sep 30 07:05:09 ats traffic_manager[5471]: {0xb704d6d0} ERROR: (last system error 32: Broken pipe)
>>>>>>>> Sep 30 07:05:09 ats traffic_cop[23694]: cop received child status signal [5471 256]
>>>>>>>> Sep 30 07:05:09 ats traffic_cop[23694]: traffic_manager not running, making sure traffic_server is dead
>>>>>>>> Sep 30 07:05:09 ats traffic_cop[23694]: spawning traffic_manager
>>>>>>>> Sep 30 07:05:09 ats traffic_manager[6938]: NOTE: --- Manager Starting ---
>>>>>>>> Sep 30 07:05:09 ats traffic_manager[6938]: NOTE: Manager Version: Apache Traffic Server - traffic_manager - 5.0.1 - (build # 7259 on Aug 25 2014 at 09:26:11)
>>>>>>>> Sep 30 07:05:09 ats traffic_manager[6938]: NOTE: Unable to set RLIMIT_NOFILE(7):cur(1475961),max(1475961)
>>>>>>>> Sep 30 07:05:09 ats traffic_manager[6938]: NOTE: RLIMIT_NOFILE(7):cur(30000),max(30000)
>>>>>>>> Sep 30 07:05:09 ats traffic_manager[6938]: ERROR ==> [runAsUser] Error: Failed to restore capabilities after switch to user trafficserver.
>>>>>>>> Sep 30 07:05:11 ats traffic_server[6946]: NOTE: --- traffic_server Starting ---
>>>>>>>> Sep 30 07:05:11 ats traffic_server[6946]: NOTE: traffic_server Version: Apache Traffic Server - traffic_server - 5.0.1 - (build # 7259 on Aug 25 2014 at 09:27:18)
>>>>>>>> Sep 30 07:05:11 ats traffic_server[6946]: NOTE: Unable to set RLIMIT_NOFILE(7):cur(-611778560),max(-611778560)
>>>>>>>> Sep 30 07:05:13 ats traffic_manager[6938]: {0xb708b6d0} ERROR: [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 11: Segmentation fault
>>>>>>>> Sep 30 07:05:13 ats traffic_manager[6938]: {0xb708b6d0} ERROR: [Alarms::signalAlarm] Server Process was reset
>>>>>>>>
>>>>>>>>
>>>>>>>> Any idea where to look for to solve this problem ?
>>>>>>>>
>>>>>>>> Thanks a lot !
>>>>>>>>
>>>>>>>> Matt
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: SSL results in segmentation fault
Posted by Matthieu Bienvenüe <ma...@exultet.net>.
Any idea to solve this isssu ?
Matthieu
Le 01/10/2014 09:50, Matthieu Bienvenüe a écrit :
>
> Le 30/09/2014 17:47, Leif Hedstrom a écrit :
>>
>> On Sep 30, 2014, at 9:00 AM, Matthieu Bienvenüe <matthieu@exultet.net
>> <ma...@exultet.net>> wrote:
>>
>>> Is that possible to do it on config instead of recompiling ATS ?
>>
>>
>> What version are you using? I’m not 100% certain, but I’d expect
>> Geffon’s additions to not have dedicated SSL threads would avoid the
>> need for that patch as well? Brian? If I recall, with a recent
>> version of ATS, you’d simply set proxy.config.ssl.number.threads to -1.
> When I set this settings SSL don't work and I've the following stack
> trace :
>
> NOTE: Traffic Server received Sig 11: Segmentation fault
> /usr/bin/traffic_server - STACK TRACE:
> [0x4001e500]
> /usr/lib/trafficserver/libtsutil.so.5(ink_hash_table_lookup_entry+0x12)[0x4003c0f2]
> /usr/lib/trafficserver/libtsutil.so.5(ink_hash_table_lookup+0x24)[0x4003c3b4]
> /usr/bin/traffic_server[0x8308185]
> /usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0(+0x12844)[0x40067844]
> /usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0(SSL_accept+0x2a)[0x4008c73a]
> /usr/bin/traffic_server(_ZN17SSLNetVConnection23sslServerHandShakeEventERi+0x19)[0x8303d89]
> /usr/bin/traffic_server(_ZN17SSLNetVConnection17sslStartHandShakeEiRi+0x2b)[0x830446b]
> /usr/bin/traffic_server(_ZN17SSLNetVConnection11net_read_ioEP10NetHandlerP7EThread+0xb30)[0x8305270]
> /usr/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x27f)[0x830dd7f]
> /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x98)[0x8339cf8]
> /usr/bin/traffic_server(_ZN7EThread7executeEv+0x419)[0x833a449]
> /usr/bin/traffic_server(main+0xf40)[0x80d4e30]
> /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xe6)[0x405c9e36]
> /usr/bin/traffic_server[0x80da229]
> [TrafficServer] using root directory '/usr'
>
>>
>> In either case, why is that patch not committed? Is there a Jira for it?
>>
>> — Leif
>>
>>>
>>> Regards,
>>>
>>> Matt
>>> Le 30/09/2014 16:49, 英才 a écrit :
>>>>
>>>> disable AIO or patch https://github.com/phonehold/with-aio-ssl-init
>>>> may solve your problem
>>>>
>>>> 在 2014年9月30日，下午10:41，Matthieu Bienvenüe
>>>> <matthieu@exultet.net <ma...@exultet.net>> 写道：
>>>>
>>>>> Hello,
>>>>>
>>>>>
>>>>> SSL works fine with my certs, but it crashes only after a certain
>>>>> amount of time/requests.
>>>>>
>>>>> Here is the stack trace from traffic.out:
>>>>>
>>>>> NOTE: Traffic Server received Sig 11: Segmentation fault
>>>>> /usr/bin/traffic_server - STACK TRACE:
>>>>> [0x4001e500]
>>>>> /usr/bin/traffic_server(_Z12ink_aio_readP11AIOCallbacki+0x2a)[0x830056a]
>>>>> /usr/bin/traffic_server(_ZN7CacheVC10handleReadEiP5Event+0x282)[0x82c4402]
>>>>> /usr/bin/traffic_server(_ZN5Cache9open_readEP12ContinuationP7INK_MD5P7HTTPHdrP21CacheLookupHttpConfig13CacheFragTypePci+0x5be)[0x82df68e]
>>>>> /usr/bin/traffic_server(_ZN14CacheProcessor9open_readEP12ContinuationP3URLbP7HTTPHdrP21CacheLookupHttpConfigl13CacheFragType+0xdc)[0x82c2b4c]
>>>>> /usr/bin/traffic_server(_ZN11HttpCacheSM18do_cache_open_readEv+0x63)[0x81ab6f3]
>>>>> /usr/bin/traffic_server(_ZN11HttpCacheSM9open_readEP3URLP7HTTPHdrP21CacheLookupHttpConfigl+0x4c)[0x81aba0c]
>>>>> /usr/bin/traffic_server(_ZN6HttpSM24do_cache_lookup_and_readEv+0x115)[0x81bd105]
>>>>> /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x6af)[0x81ce7bf]
>>>>> /usr/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x108)[0x81cc8d8]
>>>>> /usr/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x300)[0x81c9940]
>>>>> /usr/bin/traffic_server(_ZN6HttpSM23do_api_callout_internalEv+0x54)[0x81c9da4]
>>>>> /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x250)[0x81ce360]
>>>>> /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x7eb)[0x81ce8fb]
>>>>> /usr/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x108)[0x81cc8d8]
>>>>> /usr/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x300)[0x81c9940]
>>>>> /usr/bin/traffic_server(_ZN6HttpSM23do_api_callout_internalEv+0x54)[0x81c9da4]
>>>>> /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x250)[0x81ce360]
>>>>> /usr/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x108)[0x81cc8d8]
>>>>> /usr/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x300)[0x81c9940]
>>>>> /usr/bin/traffic_server(_ZN6HttpSM18state_api_callbackEiPv+0x78)[0x81cc398]
>>>>> /usr/bin/traffic_server(TSHttpTxnReenable+0x1f0)[0x810ef50]
>>>>> /usr/lib/trafficserver/modules/stats_over_http.so(+0x102e)[0x4095f02e]
>>>>> /usr/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0xd8)[0x81c9718]
>>>>> /usr/bin/traffic_server(_ZN6HttpSM23do_api_callout_internalEv+0x54)[0x81c9da4]
>>>>> /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x250)[0x81ce360]
>>>>> /usr/bin/traffic_server(_ZN6HttpSM32state_read_client_request_headerEiPv+0x1e8)[0x81c5738]
>>>>> /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x7e)[0x81ca93e]
>>>>> /usr/bin/traffic_server(_ZN18UnixNetVConnection19readSignalAndUpdateEi+0x45)[0x83166a5]
>>>>> /usr/bin/traffic_server(_ZN17SSLNetVConnection11net_read_ioEP10NetHandlerP7EThread+0x10b0)[0x83057f0]
>>>>> /usr/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x27f)[0x830dd7f]
>>>>> /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x98)[0x8339cf8]
>>>>> /usr/bin/traffic_server(_ZN7EThread7executeEv+0x419)[0x833a449]
>>>>> /usr/bin/traffic_server[0x8338ebb]
>>>>> /lib/i386-linux-gnu/libpthread.so.0(+0x5954)[0x4046b954]
>>>>> /lib/i386-linux-gnu/libc.so.6(clone+0x5e)[0x40688cbe]
>>>>> [E. Mgmt] log ==> [TrafficManager] using root directory '/usr'
>>>>> [TrafficServer] using root directory '/usr'
>>>>>
>>>>> Here is my record.config for SSL parameters:
>>>>>
>>>>> CONFIG proxy.config.http.server_ports STRING 8080 4443:ssl
>>>>>
>>>>> CONFIG proxy.config.ssl.enabled INT 1
>>>>> CONFIG proxy.config.ssl.server.cert.path STRING
>>>>> /etc/trafficserver/ssl/
>>>>> CONFIG proxy.config.ssl.server.private_key.path STRING
>>>>> /etc/trafficserver/ssl/
>>>>>
>>>>> And for ssl_multicert.config:
>>>>>
>>>>> ssl_cert_name=new2014/100.pem ssl_key_name=new2014/100.key
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Le 30/09/2014 15:54, Susan Hinrichs a écrit :
>>>>>> Matt,
>>>>>>
>>>>>> Is there a basic stack trace in traffic.out? What is your SSL
>>>>>> configuration? Do you have certs set up in ssl_multicert.config?
>>>>>> Or are you doing a blind tunnel on the SSL traffic?
>>>>>>
>>>>>> Susan
>>>>>>
>>>>>> On 9/30/2014 2:14 AM, Matthieu Bienvenüe wrote:
>>>>>>> Hello !
>>>>>>>
>>>>>>> I'm configuring ATS as a reverse proxy and I need SSL support.
>>>>>>>
>>>>>>> ATS runs on OpenVZ on Debian. It's the version 5.0.1 installed
>>>>>>> from backport packages.
>>>>>>>
>>>>>>> ATS works fine, SSL too. But after a while SSL makes ATS crash.
>>>>>>>
>>>>>>> In manager.log I found that there is a segmentation fault:
>>>>>>>
>>>>>>> [Sep 29 16:08:33.020] Manager {0xb6fb76d0} ERROR:
>>>>>>> [LocalManager::pollMgmtProcessServer] Server Process terminated
>>>>>>> due to Sig 11: Segmentation fault
>>>>>>> [Sep 29 16:08:33.021] Manager {0xb6fb76d0} ERROR:
>>>>>>> [Alarms::signalAlarm] Server Process was reset
>>>>>>> [Sep 29 16:08:34.041] Manager {0xb6fb76d0} NOTE:
>>>>>>> [LocalManager::startProxy] Launching ts process
>>>>>>> [Sep 29 16:08:34.049] Manager {0xb6fb76d0} NOTE:
>>>>>>> [LocalManager::pollMgmtProcessServer] New process connecting fd '16'
>>>>>>> [Sep 29 16:08:34.049] Manager {0xb6fb76d0} NOTE:
>>>>>>> [Alarms::signalAlarm] Server Process born
>>>>>>>
>>>>>>> Here is a dump of the syslog when crashing:
>>>>>>>
>>>>>>> Sep 30 07:05:09 ats traffic_manager[5471]: {0xb704d6d0} FATAL:
>>>>>>> [LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
>>>>>>> Sep 30 07:05:09 ats traffic_manager[5471]: {0xb704d6d0} ERROR:
>>>>>>> [LocalManager::sendMgmtMsgToProcesses] Error writing message
>>>>>>> Sep 30 07:05:09 ats traffic_manager[5471]: {0xb704d6d0} ERROR:
>>>>>>> (last system error 32: Broken pipe)
>>>>>>> Sep 30 07:05:09 ats traffic_cop[23694]: cop received child
>>>>>>> status signal [5471 256]
>>>>>>> Sep 30 07:05:09 ats traffic_cop[23694]: traffic_manager not
>>>>>>> running, making sure traffic_server is dead
>>>>>>> Sep 30 07:05:09 ats traffic_cop[23694]: spawning traffic_manager
>>>>>>> Sep 30 07:05:09 ats traffic_manager[6938]: NOTE: --- Manager
>>>>>>> Starting ---
>>>>>>> Sep 30 07:05:09 ats traffic_manager[6938]: NOTE: Manager
>>>>>>> Version: Apache Traffic Server - traffic_manager - 5.0.1 -
>>>>>>> (build # 7259 on Aug 25 2014 at 09:26:11)
>>>>>>> Sep 30 07:05:09 ats traffic_manager[6938]: NOTE: Unable to set
>>>>>>> RLIMIT_NOFILE(7):cur(1475961),max(1475961)
>>>>>>> Sep 30 07:05:09 ats traffic_manager[6938]: NOTE:
>>>>>>> RLIMIT_NOFILE(7):cur(30000),max(30000)
>>>>>>> Sep 30 07:05:09 ats traffic_manager[6938]: ERROR ==> [runAsUser]
>>>>>>> Error: Failed to restore capabilities after switch to user
>>>>>>> trafficserver.
>>>>>>> Sep 30 07:05:11 ats traffic_server[6946]: NOTE: ---
>>>>>>> traffic_server Starting ---
>>>>>>> Sep 30 07:05:11 ats traffic_server[6946]: NOTE: traffic_server
>>>>>>> Version: Apache Traffic Server - traffic_server - 5.0.1 - (build
>>>>>>> # 7259 on Aug 25 2014 at 09:27:18)
>>>>>>> Sep 30 07:05:11 ats traffic_server[6946]: NOTE: Unable to set
>>>>>>> RLIMIT_NOFILE(7):cur(-611778560),max(-611778560)
>>>>>>> Sep 30 07:05:13 ats traffic_manager[6938]: {0xb708b6d0} ERROR:
>>>>>>> [LocalManager::pollMgmtProcessServer] Server Process terminated
>>>>>>> due to Sig 11: Segmentation fault
>>>>>>> Sep 30 07:05:13 ats traffic_manager[6938]: {0xb708b6d0} ERROR:
>>>>>>> [Alarms::signalAlarm] Server Process was reset
>>>>>>>
>>>>>>>
>>>>>>> Any idea where to look for to solve this problem ?
>>>>>>>
>>>>>>> Thanks a lot !
>>>>>>>
>>>>>>> Matt
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>