You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Stefan Priebe - Profihost AG <s....@profihost.ag> on 2017/03/03 20:02:55 UTC
Re: release v1.9.2
Hi Stefan,
live. Sorry for the late reply.
Stefan
Am 28.02.2017 um 11:49 schrieb Stefan Eissing:
> Meh, my mistake. Sorry about that. Did not cleanup properly. Dare you with an improved version:
>
>
>
>
>
>> Am 28.02.2017 um 08:41 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>
>> That one breaks everyting. Multiple crashes per second.
>>
>> [Thread debugging using libthread_db enabled]
>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>> Program terminated with signal SIGABRT, Aborted.
>> #0 0x00007f02724ea067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>> #0 0x00007f02724ea067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>> #1 0x00007f02724eb448 in abort () from /lib/x86_64-linux-gnu/libc.so.6
>> #2 0x00007f02724e3266 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>> #3 0x00007f02724e3312 in __assert_fail () from
>> /lib/x86_64-linux-gnu/libc.so.6
>> #4 0x00007f027287062f in __pthread_tpp_change_priority ()
>> from /lib/x86_64-linux-gnu/libpthread.so.0
>> #5 0x00007f0272865e9f in __pthread_mutex_lock_full ()
>> from /lib/x86_64-linux-gnu/libpthread.so.0
>> #6 0x00007f0272a98db8 in allocator_alloc (in_size=in_size@entry=8032,
>> allocator=0x7f025c03f3c0) at memory/unix/apr_pools.c:244
>> #7 apr_allocator_alloc (allocator=0x7f025c03f3c0, size=size@entry=8032)
>> at memory/unix/apr_pools.c:438
>> #8 0x00007f0272cbcac4 in apr_bucket_alloc (size=8032, size@entry=8000,
>> list=0x7f022c0008e8) at buckets/apr_buckets_alloc.c:157
>> #9 0x00007f0272cbdca2 in socket_bucket_read (a=0x7f022c000b08,
>> str=0x7f02627a87b8, len=0x7f02627a87c0, block=<optimized out>)
>> at buckets/apr_buckets_socket.c:34
>> #10 0x0000565098cf1fb1 in ap_core_input_filter (f=0x7f022c0056b0,
>> b=0x7f022c005630, mode=<optimized out>, block=APR_NONBLOCK_READ,
>> readbytes=5) at core_filters.c:235
>> #11 0x0000565098d3aaca in logio_in_filter (f=<optimized out>,
>> bb=0x7f022c005630, mode=<optimized out>, block=<optimized out>,
>> readbytes=<optimized out>) at mod_logio.c:165
>> #12 0x0000565098d45b9a in bio_filter_in_read (bio=0x7f024800b460,
>> in=0x7f02480492a3 "", inlen=5) at ssl_engine_io.c:483
>> #13 0x00007f02736b014c in BIO_read ()
>> from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
>> #14 0x00007f0273a13c92 in ?? () from
>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>> #15 0x00007f0273a1548d in ?? () from
>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>> #16 0x00007f0273a12024 in ?? () from
>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>> #17 0x0000565098d469f3 in ssl_io_input_read (inctx=0x7f022c0035b8,
>> buf=0x7f022c003600 "GET
>> /media/image/07/3c/c2/4612_13721_160192280322.jpeg HTTP/1.1\r\nHost:
>> www.onlinestore.it\r\nUser-Agent: curl/7.15+ (x64-criteo) libcurl/7.15+
>> OpenSSL zlib libidn\r\nAccept: */*\r\n\r\n", len=0x7f02627a8b38)
>> at ssl_engine_io.c:614
>> #18 0x0000565098d493ec in ssl_io_filter_input (f=0x7f022c005608,
>> bb=0x7f022c006dd0, mode=<optimized out>, block=<optimized out>,
>> readbytes=8000) at ssl_engine_io.c:1474
>> #19 0x0000565098d5cd85 in h2_filter_core_input (f=0x7f022c005980,
>> brigade=0x7f022c006dd0, mode=738225624, block=APR_NONBLOCK_READ,
>> readbytes=8000) at h2_filter.c:149
>> #20 0x0000565098d6809b in h2_session_read (session=0x7f022c006920,
>> block=<optimized out>) at h2_session.c:1440
>> #21 0x0000565098d6b97f in h2_session_process (session=0x7f022c006920,
>> async=3964) at h2_session.c:2074
>> #22 0x0000565098d5b84e in h2_conn_run (ctx=<optimized out>,
>> c=0x7f025c03f7d8)
>> at h2_conn.c:218
>> #23 0x0000565098d5e551 in h2_h2_process_conn (c=0x7f025c03f7d8) at
>> h2_h2.c:658
>> #24 0x0000565098d00fd0 in ap_run_process_connection (c=0x7f025c03f7d8)
>> at connection.c:42
>> #25 0x0000565098d9b6d0 in process_socket (my_thread_num=<optimized out>,
>> my_child_num=<optimized out>, cs=0x7f025c03f748, sock=<optimized out>,
>> p=<optimized out>, thd=<optimized out>) at event.c:1134
>> #26 worker_thread (thd=0xf5e, dummy=0xf7c) at event.c:2137
>> #27 0x00007f02728680a4 in start_thread ()
>> from /lib/x86_64-linux-gnu/libpthread.so.0
>> #28 0x00007f027259d62d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> (gdb) (gdb) quit
>> Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
>> /usr/lib/debug//usr/local/apache2/bin/httpd...done.
>> done.
>> [Thread debugging using libthread_db enabled]
>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>> Program terminated with signal SIGABRT, Aborted.
>> #0 0x00007f02724ea067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>> #0 0x00007f02724ea067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>> #1 0x00007f02724eb448 in abort () from /lib/x86_64-linux-gnu/libc.so.6
>> #2 0x00007f02724e3266 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>> #3 0x00007f02724e3312 in __assert_fail () from
>> /lib/x86_64-linux-gnu/libc.so.6
>> #4 0x00007f027287062f in __pthread_tpp_change_priority ()
>> from /lib/x86_64-linux-gnu/libpthread.so.0
>> #5 0x00007f0272865e9f in __pthread_mutex_lock_full ()
>> from /lib/x86_64-linux-gnu/libpthread.so.0
>> #6 0x00007f0272a98db8 in allocator_alloc (in_size=in_size@entry=8032,
>> allocator=0x7f025c03d320) at memory/unix/apr_pools.c:244
>> #7 apr_allocator_alloc (allocator=0x7f025c03d320, size=size@entry=8032)
>> at memory/unix/apr_pools.c:438
>> #8 0x00007f0272cbcac4 in apr_bucket_alloc (size=8032, size@entry=8000,
>> list=0x7f02340008e8) at buckets/apr_buckets_alloc.c:157
>> #9 0x00007f0272cbdca2 in socket_bucket_read (a=0x7f0234000b08,
>> str=0x7f0260fa57b8, len=0x7f0260fa57c0, block=<optimized out>)
>> at buckets/apr_buckets_socket.c:34
>> #10 0x0000565098cf1fb1 in ap_core_input_filter (f=0x7f02340056b0,
>> b=0x7f0234005630, mode=<optimized out>, block=APR_NONBLOCK_READ,
>> readbytes=5) at core_filters.c:235
>> #11 0x0000565098d3aaca in logio_in_filter (f=<optimized out>,
>> bb=0x7f0234005630, mode=<optimized out>, block=<optimized out>,
>> readbytes=<optimized out>) at mod_logio.c:165
>> #12 0x0000565098d45b9a in bio_filter_in_read (bio=0x7f022801ff50,
>> in=0x7f02280345d3 "(\002\177", inlen=5) at ssl_engine_io.c:483
>> #13 0x00007f02736b014c in BIO_read ()
>> from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
>> #14 0x00007f0273a13c92 in ?? () from
>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>> #15 0x00007f0273a1548d in ?? () from
>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>> #16 0x00007f0273a12024 in ?? () from
>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>> #17 0x0000565098d469f3 in ssl_io_input_read (inctx=0x7f02340035b8,
>> buf=0x7f0234003600 "", len=0x7f0260fa5b38) at ssl_engine_io.c:614
>> #18 0x0000565098d493ec in ssl_io_filter_input (f=0x7f0234005608,
>> bb=0x7f023400cd00, mode=<optimized out>, block=<optimized out>,
>> readbytes=8000) at ssl_engine_io.c:1474
>> #19 0x0000565098d5cd85 in h2_filter_core_input (f=0x7f0234005980,
>> brigade=0x7f023400cd00, mode=872467720, block=APR_NONBLOCK_READ,
>> readbytes=8000) at h2_filter.c:149
>> #20 0x0000565098d6809b in h2_session_read (session=0x7f023400c850,
>> block=<optimized out>) at h2_session.c:1440
>> #21 0x0000565098d6b97f in h2_session_process (session=0x7f023400c850,
>> async=4029) at h2_session.c:2074
>> #22 0x0000565098d5b84e in h2_conn_run (ctx=<optimized out>,
>> c=0x7f025c03d738)
>> at h2_conn.c:218
>> #23 0x0000565098d5e551 in h2_h2_process_conn (c=0x7f025c03d738) at
>> h2_h2.c:658
>> #24 0x0000565098d00fd0 in ap_run_process_connection (c=0x7f025c03d738)
>> at connection.c:42
>> #25 0x0000565098d9b6d0 in process_socket (my_thread_num=<optimized out>,
>> my_child_num=<optimized out>, cs=0x7f025c03d6a8, sock=<optimized out>,
>> p=<optimized out>, thd=<optimized out>) at event.c:1134
>> #26 worker_thread (thd=0xf9c, dummy=0xfbd) at event.c:2137
>> #27 0x00007f02728680a4 in start_thread ()
>> from /lib/x86_64-linux-gnu/libpthread.so.0
>> #28 0x00007f027259d62d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>
>> Stefan
>>
>> Am 27.02.2017 um 19:44 schrieb Stefan Eissing:
>>> Damnit. Can you try the attached patch on top of mod_http2 v1.9.2? This one sets a mutex on the main connection allocator if missing and is pretty close to the version we ran successfully with on your site for days.
>>>
>>> Thanks again!
>>>
>>> -Stefan
>>>
>>>
>>>
>>>
>>>> Am 27.02.2017 um 16:22 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>>
>>>> Hi Stefan,
>>>>
>>>> two new crashes here.
>>>>
>>>> Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
>>>> /usr/lib/debug//usr/local/apache2/bin/httpd...done.
>>>> done.
>>>> (gdb) (gdb) quit
>>>> Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
>>>> /usr/lib/debug//usr/local/apache2/bin/httpd...done.
>>>> done.
>>>> [Thread debugging using libthread_db enabled]
>>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>> #0 allocator_free (node=0x0, allocator=0x7f458005a320)
>>>> at memory/unix/apr_pools.c:381
>>>> #0 allocator_free (node=0x0, allocator=0x7f458005a320)
>>>> at memory/unix/apr_pools.c:381
>>>> #1 apr_pool_clear (pool=0x7f458004bec8) at memory/unix/apr_pools.c:793
>>>> #2 0x0000560ce83e5718 in ap_push_pool (queue_info=0x0,
>>>> pool_to_recycle=0x7f458005a328) at fdqueue.c:234
>>>> #3 0x0000560ce83e0a5a in process_lingering_close (cs=0x7f458004c158,
>>>> pfd=0x560ce8b8dda8) at event.c:1513
>>>> #4 0x0000560ce83e4590 in listener_thread (thd=0x0, dummy=0x5497f5242585c)
>>>> at event.c:1837
>>>> #5 0x00007f45967610a4 in start_thread ()
>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>> #6 0x00007f459649662d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>> (gdb) (gdb) quit
>>>>
>>>>
>>>> Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
>>>> /usr/lib/debug//usr/local/apache2/bin/httpd...done.
>>>> done.
>>>> [Thread debugging using libthread_db enabled]
>>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>> #0 allocator_free (node=0x0, allocator=0x7f4580062840)
>>>> at memory/unix/apr_pools.c:381
>>>> #0 allocator_free (node=0x0, allocator=0x7f4580062840)
>>>> at memory/unix/apr_pools.c:381
>>>> #1 apr_pool_clear (pool=0x7f4580069188) at memory/unix/apr_pools.c:793
>>>> #2 0x0000560ce83e5718 in ap_push_pool (queue_info=0x0,
>>>> pool_to_recycle=0x7f4580062848) at fdqueue.c:234
>>>> #3 0x0000560ce83e0a5a in process_lingering_close (cs=0x7f4580069418,
>>>> pfd=0x560ce8b8dda8) at event.c:1513
>>>> #4 0x0000560ce83e4590 in listener_thread (thd=0x0, dummy=0x549845279ce62)
>>>> at event.c:1837
>>>> #5 0x00007f45967610a4 in start_thread ()
>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>> #6 0x00007f459649662d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>> (gdb) (gdb) quit
>>>>
>>>> Stefan
>>>> Am 26.02.2017 um 19:46 schrieb Stefan Priebe - Profihost AG:
>>>>> Hi Stefan,
>>>>>
>>>>> currently everything fine. No segfaults.
>>>>>
>>>>> Stefan
>>>>>
>>>>> Am 25.02.2017 um 20:40 schrieb Stefan Priebe - Profihost AG:
>>>>>> Hi Stefan,
>>>>>> Am 25.02.2017 um 13:51 schrieb Stefan Eissing:
>>>>>>> Stefan,
>>>>>>>
>>>>>>> whenever you have time, please deploy https://github.com/icing/mod_h2/releases/tag/v1.9.2
>>>>>>>
>>>>>>> I added an own allocator with mutex to the http/2 main session. That is something of a middle-ground between placing that on the main conn (as we had in the crash free version) and 1.9.1 behaviour. Thanks!
>>>>>>
>>>>>> done. But please keep in mind that this crash might be very rare and
>>>>>> might even have happened with v1.9.0 if we've waited more time.
>>>>>>
>>>>>> Greets,
>>>>>> Stefan
>>>>>>
>>>>>>> -Stefan
>>>>>>>
>>>>>>>> Am 24.02.2017 um 19:31 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>>>>>>
>>>>>>>> Hi Yann,
>>>>>>>>
>>>>>>>> here we go:
>>>>>>>>
>>>>>>>> (gdb) p *ipsub
>>>>>>>> $1 = {family = 2, sub = {1367753145, 0, 0, 0}, mask = {4294967295,
>>>>>>>> 4294967295, 4294967295, 4294967295}}
>>>>>>>>
>>>>>>>> (gdb) p *sa
>>>>>>>> $2 = {pool = 0x10b500c4ff0b0a, hostname = 0x503040203030102 <error:
>>>>>>>> Cannot access memory at address 0x503040203030102>,
>>>>>>>> servname = 0x17d010000040405 <error: Cannot access memory at address
>>>>>>>> 0x17d010000040405>, port = 770, family = 554829073,
>>>>>>>> salen = 319177009, ipaddr_len = 570909009, addr_str_len = -2127424399,
>>>>>>>> ipaddr_ptr = 0x24f0d15215c1b142,
>>>>>>>> next = 0x17160a0982726233, sa = {sin = {sin_family = 6424, sin_port =
>>>>>>>> 9498, sin_addr = {s_addr = 690497318},
>>>>>>>> sin_zero = "*456789:"}, sin6 = {sin6_family = 6424, sin6_port =
>>>>>>>> 9498, sin6_flowinfo = 690497318, sin6_addr = {__in6_u = {
>>>>>>>> __u6_addr8 = "*456789:CDEFGHIJ", __u6_addr16 = {13354, 13877,
>>>>>>>> 14391, 14905, 17475, 17989, 18503, 19017}, __u6_addr32 = {
>>>>>>>> 909456426, 976828471, 1178944579, 1246316615}}},
>>>>>>>> sin6_scope_id = 1448432723}, sas = {ss_family = 6424,
>>>>>>>> __ss_align = 4195446337656140842,
>>>>>>>> __ss_padding =
>>>>>>>> "CDEFGHIJSTUVWXYZcdefghijstuvwxyz\203\204\205\206\207\210\211\212\222\223\224\225\226\227\230\231\232\242\243\244\245\246\247\250\251\252\262\263\264\265\266\267\270\271\272\302\303\304\305\306\307\310\311\312\322\323\324\325\326\327\330\331\332\341\342\343\344\345\346\347\350\351\352\361\362\363\364\365\366\367\370\371\372\377\304\000\037\001\000\003"}}}
>>>>>>>>
>>>>>>>> (gdb) p *(struct in6_addr *)sa
>>>>>>>> $3 = {__in6_u = {__u6_addr8 =
>>>>>>>> "\n\v\377\304\000\265\020\000\002\001\003\003\002\004\003\005",
>>>>>>>> __u6_addr16 = {2826, 50431, 46336,
>>>>>>>> 16, 258, 771, 1026, 1283}, __u6_addr32 = {3305048842, 1094912,
>>>>>>>> 50528514, 84083714}}}
>>>>>>>>
>>>>>>>>
>>>>>>>> Stefan
>>>>>>>>
>>>>>>>> Am 24.02.2017 um 14:18 schrieb Yann Ylavic:
>>>>>>>>> Hi Stefan (Priebe),
>>>>>>>>>
>>>>>>>>> Is IPv6 (really) involved in your network?
>>>>>>>>>
>>>>>>>>> Could you please show up the gdb output of the below ?
>>>>>>>>>
>>>>>>>>> On Fri, Feb 24, 2017 at 2:07 PM, Yann Ylavic <yl...@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>> 1078 APR_DECLARE(int) apr_ipsubnet_test(apr_ipsubnet_t *ipsub,
>>>>>>>>>> apr_sockaddr_t *sa)
>>>>>>>>>> 1079 {
>>>>>>>>>> 1080 #if APR_HAVE_IPV6
>>>>>>>>>> 1081 /* XXX This line will segv on Win32 build with APR_HAVE_IPV6,
>>>>>>>>>> 1082 * but without the IPV6 drivers installed.
>>>>>>>>>> 1083 */
>>>>>>>>>> 1084 if (sa->family == AF_INET) {
>>>>>>>>>> 1085 if (ipsub->family == AF_INET &&
>>>>>>>>>> 1086 ((sa->sa.sin.sin_addr.s_addr & ipsub->mask[0]) ==
>>>>>>>>>> ipsub->sub[0])) {
>>>>>>>>>> 1087 return 1;
>>>>>>>>>> 1088 }
>>>>>>>>>> 1089 }
>>>>>>>>>> 1090 else if (IN6_IS_ADDR_V4MAPPED((struct in6_addr *)sa->ipaddr_ptr)) {
>>>>>>>>>> 1091 if (ipsub->family == AF_INET &&
>>>>>>>>>> 1092 (((apr_uint32_t *)sa->ipaddr_ptr)[3] &
>>>>>>>>>> ipsub->mask[0]) == ipsub->sub[0]) {
>>>>>>>>>> 1093 return 1;
>>>>>>>>>> 1094 }
>>>>>>>>>> 1095 }
>>>>>>>>>
>>>>>>>>> (gdb) p *ipsub
>>>>>>>>> (gdb) p *sa
>>>>>>>>> (gdb) p *(struct in6_addr *)sa
>>>>>>>>>
>>>>>>>>> and possibly more to come...
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Yann.
>>>>>>>>>
>>>>>>>
>>>>>>> Stefan Eissing
>>>>>>>
>>>>>>> <green/>bytes GmbH
>>>>>>> Hafenstrasse 16
>>>>>>> 48155 M�nster
>>>>>>> www.greenbytes.de
>>>>>>>
>>>
>>> Stefan Eissing
>>>
>>> <green/>bytes GmbH
>>> Hafenstrasse 16
>>> 48155 M�nster
>>> www.greenbytes.de
>>>
>
> Stefan Eissing
>
> <green/>bytes GmbH
> Hafenstrasse 16
> 48155 M�nster
> www.greenbytes.de
>
Re: release v1.9.2
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Am 09.03.2017 um 18:09 schrieb Stefan Eissing:
> Thanks for running our patches with many changes! Not a mistake to have it running fine for four days. ;-)
>
> Hmm, so we hunt a rare beast. After much effort from all sides we made it rare. So, nothing to be ashamed of. Need to keep looking...
May be Yann has an idea? May be it's not related to http2 at all?
Stefan
>> Am 09.03.2017 um 17:32 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>
>> Hi Stefan,
>>
>> while doing longer testing i can say that also the version which was
>> working for 4 days segfaults. So it was never solved ;-( Sorry about
>> that. Testing was just too short.
>>
>> Greets,
>> Stefan
>>
>> Am 05.03.2017 um 14:49 schrieb Stefan Priebe - Profihost AG:
>>> Hi Stefan,
>>>
>>> yes this seems to correct BUT i'm not sure i the test was long enough
>>> where i hadn't a segfault. I'll rerun a test with that version. To
>>> verify this was no just a "fault".
>>>
>>> Greets,
>>> Stefan
>>> Am 05.03.2017 um 14:34 schrieb Stefan Eissing:
>>>> Thanks. I try to summarize:
>>>>
>>>> - we did see this rarely with unpatched v1.9.2 and Yann's mpm patch v??
>>>> - we still see this rarely with the patch adding mutex to the main conn allocator (so this seems not to change anything)
>>>> - we did not see this with v1.9.2 and Yann's patch on ptrans version ??
>>>>
>>>> Is this correct?
>>>>
>>>>> Am 04.03.2017 um 22:34 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>>>
>>>>> Hi Stefan,
>>>>>
>>>>> current trace - not sure whether this is http2 related at all:
>>>>>
>>>>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>> #0 allocator_free (node=0x0, allocator=0x7fbf4c05a410)
>>>>> at memory/unix/apr_pools.c:381
>>>>> #0 allocator_free (node=0x0, allocator=0x7fbf4c05a410)
>>>>> at memory/unix/apr_pools.c:381
>>>>> #1 apr_pool_clear (pool=0x7fbf4c04f888) at memory/unix/apr_pools.c:793
>>>>> #2 0x00005642878c2818 in ap_push_pool (queue_info=0x0,
>>>>> pool_to_recycle=0x7fbf4c05a418) at fdqueue.c:234
>>>>> #3 0x00005642878bdb5a in process_lingering_close (cs=0x7fbf4c04fb18,
>>>>> pfd=0x5642886c7078) at event.c:1513
>>>>> #4 0x00005642878c1690 in listener_thread (thd=0x0, dummy=0x549eb52311a6f)
>>>>> at event.c:1837
>>>>> #5 0x00007fbf5f2940a4 in start_thread ()
>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>> #6 0x00007fbf5efc962d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>
>>>>> Stefan
>>>>>
>>>>>> Am 03.03.2017 um 21:02 schrieb Stefan Priebe - Profihost AG:
>>>>>> Hi Stefan,
>>>>>>
>>>>>> live. Sorry for the late reply.
>>>>>>
>>>>>> Stefan
>>>>>>
>>>>>>> Am 28.02.2017 um 11:49 schrieb Stefan Eissing:
>>>>>>> Meh, my mistake. Sorry about that. Did not cleanup properly. Dare you with an improved version:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Am 28.02.2017 um 08:41 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>>>>>>
>>>>>>>> That one breaks everyting. Multiple crashes per second.
>>>>>>>>
>>>>>>>> [Thread debugging using libthread_db enabled]
>>>>>>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>>>>>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>>>>>>>> Program terminated with signal SIGABRT, Aborted.
>>>>>>>> #0 0x00007f02724ea067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>>>> #0 0x00007f02724ea067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>>>> #1 0x00007f02724eb448 in abort () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>>>> #2 0x00007f02724e3266 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>>>> #3 0x00007f02724e3312 in __assert_fail () from
>>>>>>>> /lib/x86_64-linux-gnu/libc.so.6
>>>>>>>> #4 0x00007f027287062f in __pthread_tpp_change_priority ()
>>>>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>>>> #5 0x00007f0272865e9f in __pthread_mutex_lock_full ()
>>>>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>>>> #6 0x00007f0272a98db8 in allocator_alloc (in_size=in_size@entry=8032,
>>>>>>>> allocator=0x7f025c03f3c0) at memory/unix/apr_pools.c:244
>>>>>>>> #7 apr_allocator_alloc (allocator=0x7f025c03f3c0, size=size@entry=8032)
>>>>>>>> at memory/unix/apr_pools.c:438
>>>>>>>> #8 0x00007f0272cbcac4 in apr_bucket_alloc (size=8032, size@entry=8000,
>>>>>>>> list=0x7f022c0008e8) at buckets/apr_buckets_alloc.c:157
>>>>>>>> #9 0x00007f0272cbdca2 in socket_bucket_read (a=0x7f022c000b08,
>>>>>>>> str=0x7f02627a87b8, len=0x7f02627a87c0, block=<optimized out>)
>>>>>>>> at buckets/apr_buckets_socket.c:34
>>>>>>>> #10 0x0000565098cf1fb1 in ap_core_input_filter (f=0x7f022c0056b0,
>>>>>>>> b=0x7f022c005630, mode=<optimized out>, block=APR_NONBLOCK_READ,
>>>>>>>> readbytes=5) at core_filters.c:235
>>>>>>>> #11 0x0000565098d3aaca in logio_in_filter (f=<optimized out>,
>>>>>>>> bb=0x7f022c005630, mode=<optimized out>, block=<optimized out>,
>>>>>>>> readbytes=<optimized out>) at mod_logio.c:165
>>>>>>>> #12 0x0000565098d45b9a in bio_filter_in_read (bio=0x7f024800b460,
>>>>>>>> in=0x7f02480492a3 "", inlen=5) at ssl_engine_io.c:483
>>>>>>>> #13 0x00007f02736b014c in BIO_read ()
>>>>>>>> from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
>>>>>>>> #14 0x00007f0273a13c92 in ?? () from
>>>>>>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>>>>>>>> #15 0x00007f0273a1548d in ?? () from
>>>>>>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>>>>>>>> #16 0x00007f0273a12024 in ?? () from
>>>>>>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>>>>>>>> #17 0x0000565098d469f3 in ssl_io_input_read (inctx=0x7f022c0035b8,
>>>>>>>> buf=0x7f022c003600 "GET
>>>>>>>> /media/image/07/3c/c2/4612_13721_160192280322.jpeg HTTP/1.1\r\nHost:
>>>>>>>> www.onlinestore.it\r\nUser-Agent: curl/7.15+ (x64-criteo) libcurl/7.15+
>>>>>>>> OpenSSL zlib libidn\r\nAccept: */*\r\n\r\n", len=0x7f02627a8b38)
>>>>>>>> at ssl_engine_io.c:614
>>>>>>>> #18 0x0000565098d493ec in ssl_io_filter_input (f=0x7f022c005608,
>>>>>>>> bb=0x7f022c006dd0, mode=<optimized out>, block=<optimized out>,
>>>>>>>> readbytes=8000) at ssl_engine_io.c:1474
>>>>>>>> #19 0x0000565098d5cd85 in h2_filter_core_input (f=0x7f022c005980,
>>>>>>>> brigade=0x7f022c006dd0, mode=738225624, block=APR_NONBLOCK_READ,
>>>>>>>> readbytes=8000) at h2_filter.c:149
>>>>>>>> #20 0x0000565098d6809b in h2_session_read (session=0x7f022c006920,
>>>>>>>> block=<optimized out>) at h2_session.c:1440
>>>>>>>> #21 0x0000565098d6b97f in h2_session_process (session=0x7f022c006920,
>>>>>>>> async=3964) at h2_session.c:2074
>>>>>>>> #22 0x0000565098d5b84e in h2_conn_run (ctx=<optimized out>,
>>>>>>>> c=0x7f025c03f7d8)
>>>>>>>> at h2_conn.c:218
>>>>>>>> #23 0x0000565098d5e551 in h2_h2_process_conn (c=0x7f025c03f7d8) at
>>>>>>>> h2_h2.c:658
>>>>>>>> #24 0x0000565098d00fd0 in ap_run_process_connection (c=0x7f025c03f7d8)
>>>>>>>> at connection.c:42
>>>>>>>> #25 0x0000565098d9b6d0 in process_socket (my_thread_num=<optimized out>,
>>>>>>>> my_child_num=<optimized out>, cs=0x7f025c03f748, sock=<optimized out>,
>>>>>>>> p=<optimized out>, thd=<optimized out>) at event.c:1134
>>>>>>>> #26 worker_thread (thd=0xf5e, dummy=0xf7c) at event.c:2137
>>>>>>>> #27 0x00007f02728680a4 in start_thread ()
>>>>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>>>> #28 0x00007f027259d62d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>>>> (gdb) (gdb) quit
>>>>>>>> Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
>>>>>>>> /usr/lib/debug//usr/local/apache2/bin/httpd...done.
>>>>>>>> done.
>>>>>>>> [Thread debugging using libthread_db enabled]
>>>>>>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>>>>>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>>>>>>>> Program terminated with signal SIGABRT, Aborted.
>>>>>>>> #0 0x00007f02724ea067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>>>> #0 0x00007f02724ea067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>>>> #1 0x00007f02724eb448 in abort () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>>>> #2 0x00007f02724e3266 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>>>> #3 0x00007f02724e3312 in __assert_fail () from
>>>>>>>> /lib/x86_64-linux-gnu/libc.so.6
>>>>>>>> #4 0x00007f027287062f in __pthread_tpp_change_priority ()
>>>>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>>>> #5 0x00007f0272865e9f in __pthread_mutex_lock_full ()
>>>>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>>>> #6 0x00007f0272a98db8 in allocator_alloc (in_size=in_size@entry=8032,
>>>>>>>> allocator=0x7f025c03d320) at memory/unix/apr_pools.c:244
>>>>>>>> #7 apr_allocator_alloc (allocator=0x7f025c03d320, size=size@entry=8032)
>>>>>>>> at memory/unix/apr_pools.c:438
>>>>>>>> #8 0x00007f0272cbcac4 in apr_bucket_alloc (size=8032, size@entry=8000,
>>>>>>>> list=0x7f02340008e8) at buckets/apr_buckets_alloc.c:157
>>>>>>>> #9 0x00007f0272cbdca2 in socket_bucket_read (a=0x7f0234000b08,
>>>>>>>> str=0x7f0260fa57b8, len=0x7f0260fa57c0, block=<optimized out>)
>>>>>>>> at buckets/apr_buckets_socket.c:34
>>>>>>>> #10 0x0000565098cf1fb1 in ap_core_input_filter (f=0x7f02340056b0,
>>>>>>>> b=0x7f0234005630, mode=<optimized out>, block=APR_NONBLOCK_READ,
>>>>>>>> readbytes=5) at core_filters.c:235
>>>>>>>> #11 0x0000565098d3aaca in logio_in_filter (f=<optimized out>,
>>>>>>>> bb=0x7f0234005630, mode=<optimized out>, block=<optimized out>,
>>>>>>>> readbytes=<optimized out>) at mod_logio.c:165
>>>>>>>> #12 0x0000565098d45b9a in bio_filter_in_read (bio=0x7f022801ff50,
>>>>>>>> in=0x7f02280345d3 "(\002\177", inlen=5) at ssl_engine_io.c:483
>>>>>>>> #13 0x00007f02736b014c in BIO_read ()
>>>>>>>> from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
>>>>>>>> #14 0x00007f0273a13c92 in ?? () from
>>>>>>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>>>>>>>> #15 0x00007f0273a1548d in ?? () from
>>>>>>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>>>>>>>> #16 0x00007f0273a12024 in ?? () from
>>>>>>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>>>>>>>> #17 0x0000565098d469f3 in ssl_io_input_read (inctx=0x7f02340035b8,
>>>>>>>> buf=0x7f0234003600 "", len=0x7f0260fa5b38) at ssl_engine_io.c:614
>>>>>>>> #18 0x0000565098d493ec in ssl_io_filter_input (f=0x7f0234005608,
>>>>>>>> bb=0x7f023400cd00, mode=<optimized out>, block=<optimized out>,
>>>>>>>> readbytes=8000) at ssl_engine_io.c:1474
>>>>>>>> #19 0x0000565098d5cd85 in h2_filter_core_input (f=0x7f0234005980,
>>>>>>>> brigade=0x7f023400cd00, mode=872467720, block=APR_NONBLOCK_READ,
>>>>>>>> readbytes=8000) at h2_filter.c:149
>>>>>>>> #20 0x0000565098d6809b in h2_session_read (session=0x7f023400c850,
>>>>>>>> block=<optimized out>) at h2_session.c:1440
>>>>>>>> #21 0x0000565098d6b97f in h2_session_process (session=0x7f023400c850,
>>>>>>>> async=4029) at h2_session.c:2074
>>>>>>>> #22 0x0000565098d5b84e in h2_conn_run (ctx=<optimized out>,
>>>>>>>> c=0x7f025c03d738)
>>>>>>>> at h2_conn.c:218
>>>>>>>> #23 0x0000565098d5e551 in h2_h2_process_conn (c=0x7f025c03d738) at
>>>>>>>> h2_h2.c:658
>>>>>>>> #24 0x0000565098d00fd0 in ap_run_process_connection (c=0x7f025c03d738)
>>>>>>>> at connection.c:42
>>>>>>>> #25 0x0000565098d9b6d0 in process_socket (my_thread_num=<optimized out>,
>>>>>>>> my_child_num=<optimized out>, cs=0x7f025c03d6a8, sock=<optimized out>,
>>>>>>>> p=<optimized out>, thd=<optimized out>) at event.c:1134
>>>>>>>> #26 worker_thread (thd=0xf9c, dummy=0xfbd) at event.c:2137
>>>>>>>> #27 0x00007f02728680a4 in start_thread ()
>>>>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>>>> #28 0x00007f027259d62d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>>>>
>>>>>>>> Stefan
>>>>>>>>
>>>>>>>>> Am 27.02.2017 um 19:44 schrieb Stefan Eissing:
>>>>>>>>> Damnit. Can you try the attached patch on top of mod_http2 v1.9.2? This one sets a mutex on the main connection allocator if missing and is pretty close to the version we ran successfully with on your site for days.
>>>>>>>>>
>>>>>>>>> Thanks again!
>>>>>>>>>
>>>>>>>>> -Stefan
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Am 27.02.2017 um 16:22 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>>>>>>>>
>>>>>>>>>> Hi Stefan,
>>>>>>>>>>
>>>>>>>>>> two new crashes here.
>>>>>>>>>>
>>>>>>>>>> Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
>>>>>>>>>> /usr/lib/debug//usr/local/apache2/bin/httpd...done.
>>>>>>>>>> done.
>>>>>>>>>> (gdb) (gdb) quit
>>>>>>>>>> Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
>>>>>>>>>> /usr/lib/debug//usr/local/apache2/bin/httpd...done.
>>>>>>>>>> done.
>>>>>>>>>> [Thread debugging using libthread_db enabled]
>>>>>>>>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>>>>>>>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>>>>>>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>>>>>>> #0 allocator_free (node=0x0, allocator=0x7f458005a320)
>>>>>>>>>> at memory/unix/apr_pools.c:381
>>>>>>>>>> #0 allocator_free (node=0x0, allocator=0x7f458005a320)
>>>>>>>>>> at memory/unix/apr_pools.c:381
>>>>>>>>>> #1 apr_pool_clear (pool=0x7f458004bec8) at memory/unix/apr_pools.c:793
>>>>>>>>>> #2 0x0000560ce83e5718 in ap_push_pool (queue_info=0x0,
>>>>>>>>>> pool_to_recycle=0x7f458005a328) at fdqueue.c:234
>>>>>>>>>> #3 0x0000560ce83e0a5a in process_lingering_close (cs=0x7f458004c158,
>>>>>>>>>> pfd=0x560ce8b8dda8) at event.c:1513
>>>>>>>>>> #4 0x0000560ce83e4590 in listener_thread (thd=0x0, dummy=0x5497f5242585c)
>>>>>>>>>> at event.c:1837
>>>>>>>>>> #5 0x00007f45967610a4 in start_thread ()
>>>>>>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>>>>>> #6 0x00007f459649662d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>>>>>> (gdb) (gdb) quit
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
>>>>>>>>>> /usr/lib/debug//usr/local/apache2/bin/httpd...done.
>>>>>>>>>> done.
>>>>>>>>>> [Thread debugging using libthread_db enabled]
>>>>>>>>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>>>>>>>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>>>>>>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>>>>>>> #0 allocator_free (node=0x0, allocator=0x7f4580062840)
>>>>>>>>>> at memory/unix/apr_pools.c:381
>>>>>>>>>> #0 allocator_free (node=0x0, allocator=0x7f4580062840)
>>>>>>>>>> at memory/unix/apr_pools.c:381
>>>>>>>>>> #1 apr_pool_clear (pool=0x7f4580069188) at memory/unix/apr_pools.c:793
>>>>>>>>>> #2 0x0000560ce83e5718 in ap_push_pool (queue_info=0x0,
>>>>>>>>>> pool_to_recycle=0x7f4580062848) at fdqueue.c:234
>>>>>>>>>> #3 0x0000560ce83e0a5a in process_lingering_close (cs=0x7f4580069418,
>>>>>>>>>> pfd=0x560ce8b8dda8) at event.c:1513
>>>>>>>>>> #4 0x0000560ce83e4590 in listener_thread (thd=0x0, dummy=0x549845279ce62)
>>>>>>>>>> at event.c:1837
>>>>>>>>>> #5 0x00007f45967610a4 in start_thread ()
>>>>>>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>>>>>> #6 0x00007f459649662d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>>>>>> (gdb) (gdb) quit
>>>>>>>>>>
>>>>>>>>>> Stefan
>>>>>>>>>>> Am 26.02.2017 um 19:46 schrieb Stefan Priebe - Profihost AG:
>>>>>>>>>>> Hi Stefan,
>>>>>>>>>>>
>>>>>>>>>>> currently everything fine. No segfaults.
>>>>>>>>>>>
>>>>>>>>>>> Stefan
>>>>>>>>>>>
>>>>>>>>>>>> Am 25.02.2017 um 20:40 schrieb Stefan Priebe - Profihost AG:
>>>>>>>>>>>> Hi Stefan,
>>>>>>>>>>>>> Am 25.02.2017 um 13:51 schrieb Stefan Eissing:
>>>>>>>>>>>>> Stefan,
>>>>>>>>>>>>>
>>>>>>>>>>>>> whenever you have time, please deploy https://github.com/icing/mod_h2/releases/tag/v1.9.2
>>>>>>>>>>>>>
>>>>>>>>>>>>> I added an own allocator with mutex to the http/2 main session. That is something of a middle-ground between placing that on the main conn (as we had in the crash free version) and 1.9.1 behaviour. Thanks!
>>>>>>>>>>>>
>>>>>>>>>>>> done. But please keep in mind that this crash might be very rare and
>>>>>>>>>>>> might even have happened with v1.9.0 if we've waited more time.
>>>>>>>>>>>>
>>>>>>>>>>>> Greets,
>>>>>>>>>>>> Stefan
>>>>>>>>>>>>
>>>>>>>>>>>>> -Stefan
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Am 24.02.2017 um 19:31 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Yann,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> here we go:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> (gdb) p *ipsub
>>>>>>>>>>>>>> $1 = {family = 2, sub = {1367753145, 0, 0, 0}, mask = {4294967295,
>>>>>>>>>>>>>> 4294967295, 4294967295, 4294967295}}
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> (gdb) p *sa
>>>>>>>>>>>>>> $2 = {pool = 0x10b500c4ff0b0a, hostname = 0x503040203030102 <error:
>>>>>>>>>>>>>> Cannot access memory at address 0x503040203030102>,
>>>>>>>>>>>>>> servname = 0x17d010000040405 <error: Cannot access memory at address
>>>>>>>>>>>>>> 0x17d010000040405>, port = 770, family = 554829073,
>>>>>>>>>>>>>> salen = 319177009, ipaddr_len = 570909009, addr_str_len = -2127424399,
>>>>>>>>>>>>>> ipaddr_ptr = 0x24f0d15215c1b142,
>>>>>>>>>>>>>> next = 0x17160a0982726233, sa = {sin = {sin_family = 6424, sin_port =
>>>>>>>>>>>>>> 9498, sin_addr = {s_addr = 690497318},
>>>>>>>>>>>>>> sin_zero = "*456789:"}, sin6 = {sin6_family = 6424, sin6_port =
>>>>>>>>>>>>>> 9498, sin6_flowinfo = 690497318, sin6_addr = {__in6_u = {
>>>>>>>>>>>>>> __u6_addr8 = "*456789:CDEFGHIJ", __u6_addr16 = {13354, 13877,
>>>>>>>>>>>>>> 14391, 14905, 17475, 17989, 18503, 19017}, __u6_addr32 = {
>>>>>>>>>>>>>> 909456426, 976828471, 1178944579, 1246316615}}},
>>>>>>>>>>>>>> sin6_scope_id = 1448432723}, sas = {ss_family = 6424,
>>>>>>>>>>>>>> __ss_align = 4195446337656140842,
>>>>>>>>>>>>>> __ss_padding =
>>>>>>>>>>>>>> "CDEFGHIJSTUVWXYZcdefghijstuvwxyz\203\204\205\206\207\210\211\212\222\223\224\225\226\227\230\231\232\242\243\244\245\246\247\250\251\252\262\263\264\265\266\267\270\271\272\302\303\304\305\306\307\310\311\312\322\323\324\325\326\327\330\331\332\341\342\343\344\345\346\347\350\351\352\361\362\363\364\365\366\367\370\371\372\377\304\000\037\001\000\003"}}}
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> (gdb) p *(struct in6_addr *)sa
>>>>>>>>>>>>>> $3 = {__in6_u = {__u6_addr8 =
>>>>>>>>>>>>>> "\n\v\377\304\000\265\020\000\002\001\003\003\002\004\003\005",
>>>>>>>>>>>>>> __u6_addr16 = {2826, 50431, 46336,
>>>>>>>>>>>>>> 16, 258, 771, 1026, 1283}, __u6_addr32 = {3305048842, 1094912,
>>>>>>>>>>>>>> 50528514, 84083714}}}
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Stefan
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Am 24.02.2017 um 14:18 schrieb Yann Ylavic:
>>>>>>>>>>>>>>> Hi Stefan (Priebe),
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Is IPv6 (really) involved in your network?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Could you please show up the gdb output of the below ?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Feb 24, 2017 at 2:07 PM, Yann Ylavic <yl...@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 1078 APR_DECLARE(int) apr_ipsubnet_test(apr_ipsubnet_t *ipsub,
>>>>>>>>>>>>>>>> apr_sockaddr_t *sa)
>>>>>>>>>>>>>>>> 1079 {
>>>>>>>>>>>>>>>> 1080 #if APR_HAVE_IPV6
>>>>>>>>>>>>>>>> 1081 /* XXX This line will segv on Win32 build with APR_HAVE_IPV6,
>>>>>>>>>>>>>>>> 1082 * but without the IPV6 drivers installed.
>>>>>>>>>>>>>>>> 1083 */
>>>>>>>>>>>>>>>> 1084 if (sa->family == AF_INET) {
>>>>>>>>>>>>>>>> 1085 if (ipsub->family == AF_INET &&
>>>>>>>>>>>>>>>> 1086 ((sa->sa.sin.sin_addr.s_addr & ipsub->mask[0]) ==
>>>>>>>>>>>>>>>> ipsub->sub[0])) {
>>>>>>>>>>>>>>>> 1087 return 1;
>>>>>>>>>>>>>>>> 1088 }
>>>>>>>>>>>>>>>> 1089 }
>>>>>>>>>>>>>>>> 1090 else if (IN6_IS_ADDR_V4MAPPED((struct in6_addr *)sa->ipaddr_ptr)) {
>>>>>>>>>>>>>>>> 1091 if (ipsub->family == AF_INET &&
>>>>>>>>>>>>>>>> 1092 (((apr_uint32_t *)sa->ipaddr_ptr)[3] &
>>>>>>>>>>>>>>>> ipsub->mask[0]) == ipsub->sub[0]) {
>>>>>>>>>>>>>>>> 1093 return 1;
>>>>>>>>>>>>>>>> 1094 }
>>>>>>>>>>>>>>>> 1095 }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> (gdb) p *ipsub
>>>>>>>>>>>>>>> (gdb) p *sa
>>>>>>>>>>>>>>> (gdb) p *(struct in6_addr *)sa
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> and possibly more to come...
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Yann.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Stefan Eissing
>>>>>>>>>>>>>
>>>>>>>>>>>>> <green/>bytes GmbH
>>>>>>>>>>>>> Hafenstrasse 16
>>>>>>>>>>>>> 48155 M�nster
>>>>>>>>>>>>> www.greenbytes.de
>>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Stefan Eissing
>>>>>>>>>
>>>>>>>>> <green/>bytes GmbH
>>>>>>>>> Hafenstrasse 16
>>>>>>>>> 48155 M�nster
>>>>>>>>> www.greenbytes.de
>>>>>>>>>
>>>>>>>
>>>>>>> Stefan Eissing
>>>>>>>
>>>>>>> <green/>bytes GmbH
>>>>>>> Hafenstrasse 16
>>>>>>> 48155 M�nster
>>>>>>> www.greenbytes.de
>>>>>>>
>>>>
>
> Stefan Eissing
>
> <green/>bytes GmbH
> Hafenstrasse 16
> 48155 M�nster
> www.greenbytes.de
>
Re: release v1.9.2
Posted by Stefan Eissing <st...@greenbytes.de>.
Thanks for running our patches with many changes! Not a mistake to have it running fine for four days. ;-)
Hmm, so we hunt a rare beast. After much effort from all sides we made it rare. So, nothing to be ashamed of. Need to keep looking...
> Am 09.03.2017 um 17:32 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>
> Hi Stefan,
>
> while doing longer testing i can say that also the version which was
> working for 4 days segfaults. So it was never solved ;-( Sorry about
> that. Testing was just too short.
>
> Greets,
> Stefan
>
> Am 05.03.2017 um 14:49 schrieb Stefan Priebe - Profihost AG:
>> Hi Stefan,
>>
>> yes this seems to correct BUT i'm not sure i the test was long enough
>> where i hadn't a segfault. I'll rerun a test with that version. To
>> verify this was no just a "fault".
>>
>> Greets,
>> Stefan
>> Am 05.03.2017 um 14:34 schrieb Stefan Eissing:
>>> Thanks. I try to summarize:
>>>
>>> - we did see this rarely with unpatched v1.9.2 and Yann's mpm patch v??
>>> - we still see this rarely with the patch adding mutex to the main conn allocator (so this seems not to change anything)
>>> - we did not see this with v1.9.2 and Yann's patch on ptrans version ??
>>>
>>> Is this correct?
>>>
>>>> Am 04.03.2017 um 22:34 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>>
>>>> Hi Stefan,
>>>>
>>>> current trace - not sure whether this is http2 related at all:
>>>>
>>>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>> #0 allocator_free (node=0x0, allocator=0x7fbf4c05a410)
>>>> at memory/unix/apr_pools.c:381
>>>> #0 allocator_free (node=0x0, allocator=0x7fbf4c05a410)
>>>> at memory/unix/apr_pools.c:381
>>>> #1 apr_pool_clear (pool=0x7fbf4c04f888) at memory/unix/apr_pools.c:793
>>>> #2 0x00005642878c2818 in ap_push_pool (queue_info=0x0,
>>>> pool_to_recycle=0x7fbf4c05a418) at fdqueue.c:234
>>>> #3 0x00005642878bdb5a in process_lingering_close (cs=0x7fbf4c04fb18,
>>>> pfd=0x5642886c7078) at event.c:1513
>>>> #4 0x00005642878c1690 in listener_thread (thd=0x0, dummy=0x549eb52311a6f)
>>>> at event.c:1837
>>>> #5 0x00007fbf5f2940a4 in start_thread ()
>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>> #6 0x00007fbf5efc962d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>
>>>> Stefan
>>>>
>>>>> Am 03.03.2017 um 21:02 schrieb Stefan Priebe - Profihost AG:
>>>>> Hi Stefan,
>>>>>
>>>>> live. Sorry for the late reply.
>>>>>
>>>>> Stefan
>>>>>
>>>>>> Am 28.02.2017 um 11:49 schrieb Stefan Eissing:
>>>>>> Meh, my mistake. Sorry about that. Did not cleanup properly. Dare you with an improved version:
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Am 28.02.2017 um 08:41 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>>>>>
>>>>>>> That one breaks everyting. Multiple crashes per second.
>>>>>>>
>>>>>>> [Thread debugging using libthread_db enabled]
>>>>>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>>>>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>>>>>>> Program terminated with signal SIGABRT, Aborted.
>>>>>>> #0 0x00007f02724ea067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>>> #0 0x00007f02724ea067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>>> #1 0x00007f02724eb448 in abort () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>>> #2 0x00007f02724e3266 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>>> #3 0x00007f02724e3312 in __assert_fail () from
>>>>>>> /lib/x86_64-linux-gnu/libc.so.6
>>>>>>> #4 0x00007f027287062f in __pthread_tpp_change_priority ()
>>>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>>> #5 0x00007f0272865e9f in __pthread_mutex_lock_full ()
>>>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>>> #6 0x00007f0272a98db8 in allocator_alloc (in_size=in_size@entry=8032,
>>>>>>> allocator=0x7f025c03f3c0) at memory/unix/apr_pools.c:244
>>>>>>> #7 apr_allocator_alloc (allocator=0x7f025c03f3c0, size=size@entry=8032)
>>>>>>> at memory/unix/apr_pools.c:438
>>>>>>> #8 0x00007f0272cbcac4 in apr_bucket_alloc (size=8032, size@entry=8000,
>>>>>>> list=0x7f022c0008e8) at buckets/apr_buckets_alloc.c:157
>>>>>>> #9 0x00007f0272cbdca2 in socket_bucket_read (a=0x7f022c000b08,
>>>>>>> str=0x7f02627a87b8, len=0x7f02627a87c0, block=<optimized out>)
>>>>>>> at buckets/apr_buckets_socket.c:34
>>>>>>> #10 0x0000565098cf1fb1 in ap_core_input_filter (f=0x7f022c0056b0,
>>>>>>> b=0x7f022c005630, mode=<optimized out>, block=APR_NONBLOCK_READ,
>>>>>>> readbytes=5) at core_filters.c:235
>>>>>>> #11 0x0000565098d3aaca in logio_in_filter (f=<optimized out>,
>>>>>>> bb=0x7f022c005630, mode=<optimized out>, block=<optimized out>,
>>>>>>> readbytes=<optimized out>) at mod_logio.c:165
>>>>>>> #12 0x0000565098d45b9a in bio_filter_in_read (bio=0x7f024800b460,
>>>>>>> in=0x7f02480492a3 "", inlen=5) at ssl_engine_io.c:483
>>>>>>> #13 0x00007f02736b014c in BIO_read ()
>>>>>>> from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
>>>>>>> #14 0x00007f0273a13c92 in ?? () from
>>>>>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>>>>>>> #15 0x00007f0273a1548d in ?? () from
>>>>>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>>>>>>> #16 0x00007f0273a12024 in ?? () from
>>>>>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>>>>>>> #17 0x0000565098d469f3 in ssl_io_input_read (inctx=0x7f022c0035b8,
>>>>>>> buf=0x7f022c003600 "GET
>>>>>>> /media/image/07/3c/c2/4612_13721_160192280322.jpeg HTTP/1.1\r\nHost:
>>>>>>> www.onlinestore.it\r\nUser-Agent: curl/7.15+ (x64-criteo) libcurl/7.15+
>>>>>>> OpenSSL zlib libidn\r\nAccept: */*\r\n\r\n", len=0x7f02627a8b38)
>>>>>>> at ssl_engine_io.c:614
>>>>>>> #18 0x0000565098d493ec in ssl_io_filter_input (f=0x7f022c005608,
>>>>>>> bb=0x7f022c006dd0, mode=<optimized out>, block=<optimized out>,
>>>>>>> readbytes=8000) at ssl_engine_io.c:1474
>>>>>>> #19 0x0000565098d5cd85 in h2_filter_core_input (f=0x7f022c005980,
>>>>>>> brigade=0x7f022c006dd0, mode=738225624, block=APR_NONBLOCK_READ,
>>>>>>> readbytes=8000) at h2_filter.c:149
>>>>>>> #20 0x0000565098d6809b in h2_session_read (session=0x7f022c006920,
>>>>>>> block=<optimized out>) at h2_session.c:1440
>>>>>>> #21 0x0000565098d6b97f in h2_session_process (session=0x7f022c006920,
>>>>>>> async=3964) at h2_session.c:2074
>>>>>>> #22 0x0000565098d5b84e in h2_conn_run (ctx=<optimized out>,
>>>>>>> c=0x7f025c03f7d8)
>>>>>>> at h2_conn.c:218
>>>>>>> #23 0x0000565098d5e551 in h2_h2_process_conn (c=0x7f025c03f7d8) at
>>>>>>> h2_h2.c:658
>>>>>>> #24 0x0000565098d00fd0 in ap_run_process_connection (c=0x7f025c03f7d8)
>>>>>>> at connection.c:42
>>>>>>> #25 0x0000565098d9b6d0 in process_socket (my_thread_num=<optimized out>,
>>>>>>> my_child_num=<optimized out>, cs=0x7f025c03f748, sock=<optimized out>,
>>>>>>> p=<optimized out>, thd=<optimized out>) at event.c:1134
>>>>>>> #26 worker_thread (thd=0xf5e, dummy=0xf7c) at event.c:2137
>>>>>>> #27 0x00007f02728680a4 in start_thread ()
>>>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>>> #28 0x00007f027259d62d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>>> (gdb) (gdb) quit
>>>>>>> Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
>>>>>>> /usr/lib/debug//usr/local/apache2/bin/httpd...done.
>>>>>>> done.
>>>>>>> [Thread debugging using libthread_db enabled]
>>>>>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>>>>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>>>>>>> Program terminated with signal SIGABRT, Aborted.
>>>>>>> #0 0x00007f02724ea067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>>> #0 0x00007f02724ea067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>>> #1 0x00007f02724eb448 in abort () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>>> #2 0x00007f02724e3266 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>>> #3 0x00007f02724e3312 in __assert_fail () from
>>>>>>> /lib/x86_64-linux-gnu/libc.so.6
>>>>>>> #4 0x00007f027287062f in __pthread_tpp_change_priority ()
>>>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>>> #5 0x00007f0272865e9f in __pthread_mutex_lock_full ()
>>>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>>> #6 0x00007f0272a98db8 in allocator_alloc (in_size=in_size@entry=8032,
>>>>>>> allocator=0x7f025c03d320) at memory/unix/apr_pools.c:244
>>>>>>> #7 apr_allocator_alloc (allocator=0x7f025c03d320, size=size@entry=8032)
>>>>>>> at memory/unix/apr_pools.c:438
>>>>>>> #8 0x00007f0272cbcac4 in apr_bucket_alloc (size=8032, size@entry=8000,
>>>>>>> list=0x7f02340008e8) at buckets/apr_buckets_alloc.c:157
>>>>>>> #9 0x00007f0272cbdca2 in socket_bucket_read (a=0x7f0234000b08,
>>>>>>> str=0x7f0260fa57b8, len=0x7f0260fa57c0, block=<optimized out>)
>>>>>>> at buckets/apr_buckets_socket.c:34
>>>>>>> #10 0x0000565098cf1fb1 in ap_core_input_filter (f=0x7f02340056b0,
>>>>>>> b=0x7f0234005630, mode=<optimized out>, block=APR_NONBLOCK_READ,
>>>>>>> readbytes=5) at core_filters.c:235
>>>>>>> #11 0x0000565098d3aaca in logio_in_filter (f=<optimized out>,
>>>>>>> bb=0x7f0234005630, mode=<optimized out>, block=<optimized out>,
>>>>>>> readbytes=<optimized out>) at mod_logio.c:165
>>>>>>> #12 0x0000565098d45b9a in bio_filter_in_read (bio=0x7f022801ff50,
>>>>>>> in=0x7f02280345d3 "(\002\177", inlen=5) at ssl_engine_io.c:483
>>>>>>> #13 0x00007f02736b014c in BIO_read ()
>>>>>>> from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
>>>>>>> #14 0x00007f0273a13c92 in ?? () from
>>>>>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>>>>>>> #15 0x00007f0273a1548d in ?? () from
>>>>>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>>>>>>> #16 0x00007f0273a12024 in ?? () from
>>>>>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>>>>>>> #17 0x0000565098d469f3 in ssl_io_input_read (inctx=0x7f02340035b8,
>>>>>>> buf=0x7f0234003600 "", len=0x7f0260fa5b38) at ssl_engine_io.c:614
>>>>>>> #18 0x0000565098d493ec in ssl_io_filter_input (f=0x7f0234005608,
>>>>>>> bb=0x7f023400cd00, mode=<optimized out>, block=<optimized out>,
>>>>>>> readbytes=8000) at ssl_engine_io.c:1474
>>>>>>> #19 0x0000565098d5cd85 in h2_filter_core_input (f=0x7f0234005980,
>>>>>>> brigade=0x7f023400cd00, mode=872467720, block=APR_NONBLOCK_READ,
>>>>>>> readbytes=8000) at h2_filter.c:149
>>>>>>> #20 0x0000565098d6809b in h2_session_read (session=0x7f023400c850,
>>>>>>> block=<optimized out>) at h2_session.c:1440
>>>>>>> #21 0x0000565098d6b97f in h2_session_process (session=0x7f023400c850,
>>>>>>> async=4029) at h2_session.c:2074
>>>>>>> #22 0x0000565098d5b84e in h2_conn_run (ctx=<optimized out>,
>>>>>>> c=0x7f025c03d738)
>>>>>>> at h2_conn.c:218
>>>>>>> #23 0x0000565098d5e551 in h2_h2_process_conn (c=0x7f025c03d738) at
>>>>>>> h2_h2.c:658
>>>>>>> #24 0x0000565098d00fd0 in ap_run_process_connection (c=0x7f025c03d738)
>>>>>>> at connection.c:42
>>>>>>> #25 0x0000565098d9b6d0 in process_socket (my_thread_num=<optimized out>,
>>>>>>> my_child_num=<optimized out>, cs=0x7f025c03d6a8, sock=<optimized out>,
>>>>>>> p=<optimized out>, thd=<optimized out>) at event.c:1134
>>>>>>> #26 worker_thread (thd=0xf9c, dummy=0xfbd) at event.c:2137
>>>>>>> #27 0x00007f02728680a4 in start_thread ()
>>>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>>> #28 0x00007f027259d62d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>>>
>>>>>>> Stefan
>>>>>>>
>>>>>>>> Am 27.02.2017 um 19:44 schrieb Stefan Eissing:
>>>>>>>> Damnit. Can you try the attached patch on top of mod_http2 v1.9.2? This one sets a mutex on the main connection allocator if missing and is pretty close to the version we ran successfully with on your site for days.
>>>>>>>>
>>>>>>>> Thanks again!
>>>>>>>>
>>>>>>>> -Stefan
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Am 27.02.2017 um 16:22 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>>>>>>>
>>>>>>>>> Hi Stefan,
>>>>>>>>>
>>>>>>>>> two new crashes here.
>>>>>>>>>
>>>>>>>>> Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
>>>>>>>>> /usr/lib/debug//usr/local/apache2/bin/httpd...done.
>>>>>>>>> done.
>>>>>>>>> (gdb) (gdb) quit
>>>>>>>>> Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
>>>>>>>>> /usr/lib/debug//usr/local/apache2/bin/httpd...done.
>>>>>>>>> done.
>>>>>>>>> [Thread debugging using libthread_db enabled]
>>>>>>>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>>>>>>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>>>>>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>>>>>> #0 allocator_free (node=0x0, allocator=0x7f458005a320)
>>>>>>>>> at memory/unix/apr_pools.c:381
>>>>>>>>> #0 allocator_free (node=0x0, allocator=0x7f458005a320)
>>>>>>>>> at memory/unix/apr_pools.c:381
>>>>>>>>> #1 apr_pool_clear (pool=0x7f458004bec8) at memory/unix/apr_pools.c:793
>>>>>>>>> #2 0x0000560ce83e5718 in ap_push_pool (queue_info=0x0,
>>>>>>>>> pool_to_recycle=0x7f458005a328) at fdqueue.c:234
>>>>>>>>> #3 0x0000560ce83e0a5a in process_lingering_close (cs=0x7f458004c158,
>>>>>>>>> pfd=0x560ce8b8dda8) at event.c:1513
>>>>>>>>> #4 0x0000560ce83e4590 in listener_thread (thd=0x0, dummy=0x5497f5242585c)
>>>>>>>>> at event.c:1837
>>>>>>>>> #5 0x00007f45967610a4 in start_thread ()
>>>>>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>>>>> #6 0x00007f459649662d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>>>>> (gdb) (gdb) quit
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
>>>>>>>>> /usr/lib/debug//usr/local/apache2/bin/httpd...done.
>>>>>>>>> done.
>>>>>>>>> [Thread debugging using libthread_db enabled]
>>>>>>>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>>>>>>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>>>>>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>>>>>> #0 allocator_free (node=0x0, allocator=0x7f4580062840)
>>>>>>>>> at memory/unix/apr_pools.c:381
>>>>>>>>> #0 allocator_free (node=0x0, allocator=0x7f4580062840)
>>>>>>>>> at memory/unix/apr_pools.c:381
>>>>>>>>> #1 apr_pool_clear (pool=0x7f4580069188) at memory/unix/apr_pools.c:793
>>>>>>>>> #2 0x0000560ce83e5718 in ap_push_pool (queue_info=0x0,
>>>>>>>>> pool_to_recycle=0x7f4580062848) at fdqueue.c:234
>>>>>>>>> #3 0x0000560ce83e0a5a in process_lingering_close (cs=0x7f4580069418,
>>>>>>>>> pfd=0x560ce8b8dda8) at event.c:1513
>>>>>>>>> #4 0x0000560ce83e4590 in listener_thread (thd=0x0, dummy=0x549845279ce62)
>>>>>>>>> at event.c:1837
>>>>>>>>> #5 0x00007f45967610a4 in start_thread ()
>>>>>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>>>>> #6 0x00007f459649662d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>>>>> (gdb) (gdb) quit
>>>>>>>>>
>>>>>>>>> Stefan
>>>>>>>>>> Am 26.02.2017 um 19:46 schrieb Stefan Priebe - Profihost AG:
>>>>>>>>>> Hi Stefan,
>>>>>>>>>>
>>>>>>>>>> currently everything fine. No segfaults.
>>>>>>>>>>
>>>>>>>>>> Stefan
>>>>>>>>>>
>>>>>>>>>>> Am 25.02.2017 um 20:40 schrieb Stefan Priebe - Profihost AG:
>>>>>>>>>>> Hi Stefan,
>>>>>>>>>>>> Am 25.02.2017 um 13:51 schrieb Stefan Eissing:
>>>>>>>>>>>> Stefan,
>>>>>>>>>>>>
>>>>>>>>>>>> whenever you have time, please deploy https://github.com/icing/mod_h2/releases/tag/v1.9.2
>>>>>>>>>>>>
>>>>>>>>>>>> I added an own allocator with mutex to the http/2 main session. That is something of a middle-ground between placing that on the main conn (as we had in the crash free version) and 1.9.1 behaviour. Thanks!
>>>>>>>>>>>
>>>>>>>>>>> done. But please keep in mind that this crash might be very rare and
>>>>>>>>>>> might even have happened with v1.9.0 if we've waited more time.
>>>>>>>>>>>
>>>>>>>>>>> Greets,
>>>>>>>>>>> Stefan
>>>>>>>>>>>
>>>>>>>>>>>> -Stefan
>>>>>>>>>>>>
>>>>>>>>>>>>> Am 24.02.2017 um 19:31 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Yann,
>>>>>>>>>>>>>
>>>>>>>>>>>>> here we go:
>>>>>>>>>>>>>
>>>>>>>>>>>>> (gdb) p *ipsub
>>>>>>>>>>>>> $1 = {family = 2, sub = {1367753145, 0, 0, 0}, mask = {4294967295,
>>>>>>>>>>>>> 4294967295, 4294967295, 4294967295}}
>>>>>>>>>>>>>
>>>>>>>>>>>>> (gdb) p *sa
>>>>>>>>>>>>> $2 = {pool = 0x10b500c4ff0b0a, hostname = 0x503040203030102 <error:
>>>>>>>>>>>>> Cannot access memory at address 0x503040203030102>,
>>>>>>>>>>>>> servname = 0x17d010000040405 <error: Cannot access memory at address
>>>>>>>>>>>>> 0x17d010000040405>, port = 770, family = 554829073,
>>>>>>>>>>>>> salen = 319177009, ipaddr_len = 570909009, addr_str_len = -2127424399,
>>>>>>>>>>>>> ipaddr_ptr = 0x24f0d15215c1b142,
>>>>>>>>>>>>> next = 0x17160a0982726233, sa = {sin = {sin_family = 6424, sin_port =
>>>>>>>>>>>>> 9498, sin_addr = {s_addr = 690497318},
>>>>>>>>>>>>> sin_zero = "*456789:"}, sin6 = {sin6_family = 6424, sin6_port =
>>>>>>>>>>>>> 9498, sin6_flowinfo = 690497318, sin6_addr = {__in6_u = {
>>>>>>>>>>>>> __u6_addr8 = "*456789:CDEFGHIJ", __u6_addr16 = {13354, 13877,
>>>>>>>>>>>>> 14391, 14905, 17475, 17989, 18503, 19017}, __u6_addr32 = {
>>>>>>>>>>>>> 909456426, 976828471, 1178944579, 1246316615}}},
>>>>>>>>>>>>> sin6_scope_id = 1448432723}, sas = {ss_family = 6424,
>>>>>>>>>>>>> __ss_align = 4195446337656140842,
>>>>>>>>>>>>> __ss_padding =
>>>>>>>>>>>>> "CDEFGHIJSTUVWXYZcdefghijstuvwxyz\203\204\205\206\207\210\211\212\222\223\224\225\226\227\230\231\232\242\243\244\245\246\247\250\251\252\262\263\264\265\266\267\270\271\272\302\303\304\305\306\307\310\311\312\322\323\324\325\326\327\330\331\332\341\342\343\344\345\346\347\350\351\352\361\362\363\364\365\366\367\370\371\372\377\304\000\037\001\000\003"}}}
>>>>>>>>>>>>>
>>>>>>>>>>>>> (gdb) p *(struct in6_addr *)sa
>>>>>>>>>>>>> $3 = {__in6_u = {__u6_addr8 =
>>>>>>>>>>>>> "\n\v\377\304\000\265\020\000\002\001\003\003\002\004\003\005",
>>>>>>>>>>>>> __u6_addr16 = {2826, 50431, 46336,
>>>>>>>>>>>>> 16, 258, 771, 1026, 1283}, __u6_addr32 = {3305048842, 1094912,
>>>>>>>>>>>>> 50528514, 84083714}}}
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Stefan
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Am 24.02.2017 um 14:18 schrieb Yann Ylavic:
>>>>>>>>>>>>>> Hi Stefan (Priebe),
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Is IPv6 (really) involved in your network?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Could you please show up the gdb output of the below ?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Feb 24, 2017 at 2:07 PM, Yann Ylavic <yl...@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 1078 APR_DECLARE(int) apr_ipsubnet_test(apr_ipsubnet_t *ipsub,
>>>>>>>>>>>>>>> apr_sockaddr_t *sa)
>>>>>>>>>>>>>>> 1079 {
>>>>>>>>>>>>>>> 1080 #if APR_HAVE_IPV6
>>>>>>>>>>>>>>> 1081 /* XXX This line will segv on Win32 build with APR_HAVE_IPV6,
>>>>>>>>>>>>>>> 1082 * but without the IPV6 drivers installed.
>>>>>>>>>>>>>>> 1083 */
>>>>>>>>>>>>>>> 1084 if (sa->family == AF_INET) {
>>>>>>>>>>>>>>> 1085 if (ipsub->family == AF_INET &&
>>>>>>>>>>>>>>> 1086 ((sa->sa.sin.sin_addr.s_addr & ipsub->mask[0]) ==
>>>>>>>>>>>>>>> ipsub->sub[0])) {
>>>>>>>>>>>>>>> 1087 return 1;
>>>>>>>>>>>>>>> 1088 }
>>>>>>>>>>>>>>> 1089 }
>>>>>>>>>>>>>>> 1090 else if (IN6_IS_ADDR_V4MAPPED((struct in6_addr *)sa->ipaddr_ptr)) {
>>>>>>>>>>>>>>> 1091 if (ipsub->family == AF_INET &&
>>>>>>>>>>>>>>> 1092 (((apr_uint32_t *)sa->ipaddr_ptr)[3] &
>>>>>>>>>>>>>>> ipsub->mask[0]) == ipsub->sub[0]) {
>>>>>>>>>>>>>>> 1093 return 1;
>>>>>>>>>>>>>>> 1094 }
>>>>>>>>>>>>>>> 1095 }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> (gdb) p *ipsub
>>>>>>>>>>>>>> (gdb) p *sa
>>>>>>>>>>>>>> (gdb) p *(struct in6_addr *)sa
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> and possibly more to come...
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Yann.
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Stefan Eissing
>>>>>>>>>>>>
>>>>>>>>>>>> <green/>bytes GmbH
>>>>>>>>>>>> Hafenstrasse 16
>>>>>>>>>>>> 48155 Münster
>>>>>>>>>>>> www.greenbytes.de
>>>>>>>>>>>>
>>>>>>>>
>>>>>>>> Stefan Eissing
>>>>>>>>
>>>>>>>> <green/>bytes GmbH
>>>>>>>> Hafenstrasse 16
>>>>>>>> 48155 Münster
>>>>>>>> www.greenbytes.de
>>>>>>>>
>>>>>>
>>>>>> Stefan Eissing
>>>>>>
>>>>>> <green/>bytes GmbH
>>>>>> Hafenstrasse 16
>>>>>> 48155 Münster
>>>>>> www.greenbytes.de
>>>>>>
>>>
Stefan Eissing
<green/>bytes GmbH
Hafenstrasse 16
48155 Münster
www.greenbytes.de
Re: release v1.9.2
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Hi Stefan,
while doing longer testing i can say that also the version which was
working for 4 days segfaults. So it was never solved ;-( Sorry about
that. Testing was just too short.
Greets,
Stefan
Am 05.03.2017 um 14:49 schrieb Stefan Priebe - Profihost AG:
> Hi Stefan,
>
> yes this seems to correct BUT i'm not sure i the test was long enough
> where i hadn't a segfault. I'll rerun a test with that version. To
> verify this was no just a "fault".
>
> Greets,
> Stefan
> Am 05.03.2017 um 14:34 schrieb Stefan Eissing:
>> Thanks. I try to summarize:
>>
>> - we did see this rarely with unpatched v1.9.2 and Yann's mpm patch v??
>> - we still see this rarely with the patch adding mutex to the main conn allocator (so this seems not to change anything)
>> - we did not see this with v1.9.2 and Yann's patch on ptrans version ??
>>
>> Is this correct?
>>
>>> Am 04.03.2017 um 22:34 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>
>>> Hi Stefan,
>>>
>>> current trace - not sure whether this is http2 related at all:
>>>
>>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>> #0 allocator_free (node=0x0, allocator=0x7fbf4c05a410)
>>> at memory/unix/apr_pools.c:381
>>> #0 allocator_free (node=0x0, allocator=0x7fbf4c05a410)
>>> at memory/unix/apr_pools.c:381
>>> #1 apr_pool_clear (pool=0x7fbf4c04f888) at memory/unix/apr_pools.c:793
>>> #2 0x00005642878c2818 in ap_push_pool (queue_info=0x0,
>>> pool_to_recycle=0x7fbf4c05a418) at fdqueue.c:234
>>> #3 0x00005642878bdb5a in process_lingering_close (cs=0x7fbf4c04fb18,
>>> pfd=0x5642886c7078) at event.c:1513
>>> #4 0x00005642878c1690 in listener_thread (thd=0x0, dummy=0x549eb52311a6f)
>>> at event.c:1837
>>> #5 0x00007fbf5f2940a4 in start_thread ()
>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>> #6 0x00007fbf5efc962d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>
>>> Stefan
>>>
>>>> Am 03.03.2017 um 21:02 schrieb Stefan Priebe - Profihost AG:
>>>> Hi Stefan,
>>>>
>>>> live. Sorry for the late reply.
>>>>
>>>> Stefan
>>>>
>>>>> Am 28.02.2017 um 11:49 schrieb Stefan Eissing:
>>>>> Meh, my mistake. Sorry about that. Did not cleanup properly. Dare you with an improved version:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> Am 28.02.2017 um 08:41 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>>>>
>>>>>> That one breaks everyting. Multiple crashes per second.
>>>>>>
>>>>>> [Thread debugging using libthread_db enabled]
>>>>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>>>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>>>>>> Program terminated with signal SIGABRT, Aborted.
>>>>>> #0 0x00007f02724ea067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>> #0 0x00007f02724ea067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>> #1 0x00007f02724eb448 in abort () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>> #2 0x00007f02724e3266 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>> #3 0x00007f02724e3312 in __assert_fail () from
>>>>>> /lib/x86_64-linux-gnu/libc.so.6
>>>>>> #4 0x00007f027287062f in __pthread_tpp_change_priority ()
>>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>> #5 0x00007f0272865e9f in __pthread_mutex_lock_full ()
>>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>> #6 0x00007f0272a98db8 in allocator_alloc (in_size=in_size@entry=8032,
>>>>>> allocator=0x7f025c03f3c0) at memory/unix/apr_pools.c:244
>>>>>> #7 apr_allocator_alloc (allocator=0x7f025c03f3c0, size=size@entry=8032)
>>>>>> at memory/unix/apr_pools.c:438
>>>>>> #8 0x00007f0272cbcac4 in apr_bucket_alloc (size=8032, size@entry=8000,
>>>>>> list=0x7f022c0008e8) at buckets/apr_buckets_alloc.c:157
>>>>>> #9 0x00007f0272cbdca2 in socket_bucket_read (a=0x7f022c000b08,
>>>>>> str=0x7f02627a87b8, len=0x7f02627a87c0, block=<optimized out>)
>>>>>> at buckets/apr_buckets_socket.c:34
>>>>>> #10 0x0000565098cf1fb1 in ap_core_input_filter (f=0x7f022c0056b0,
>>>>>> b=0x7f022c005630, mode=<optimized out>, block=APR_NONBLOCK_READ,
>>>>>> readbytes=5) at core_filters.c:235
>>>>>> #11 0x0000565098d3aaca in logio_in_filter (f=<optimized out>,
>>>>>> bb=0x7f022c005630, mode=<optimized out>, block=<optimized out>,
>>>>>> readbytes=<optimized out>) at mod_logio.c:165
>>>>>> #12 0x0000565098d45b9a in bio_filter_in_read (bio=0x7f024800b460,
>>>>>> in=0x7f02480492a3 "", inlen=5) at ssl_engine_io.c:483
>>>>>> #13 0x00007f02736b014c in BIO_read ()
>>>>>> from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
>>>>>> #14 0x00007f0273a13c92 in ?? () from
>>>>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>>>>>> #15 0x00007f0273a1548d in ?? () from
>>>>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>>>>>> #16 0x00007f0273a12024 in ?? () from
>>>>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>>>>>> #17 0x0000565098d469f3 in ssl_io_input_read (inctx=0x7f022c0035b8,
>>>>>> buf=0x7f022c003600 "GET
>>>>>> /media/image/07/3c/c2/4612_13721_160192280322.jpeg HTTP/1.1\r\nHost:
>>>>>> www.onlinestore.it\r\nUser-Agent: curl/7.15+ (x64-criteo) libcurl/7.15+
>>>>>> OpenSSL zlib libidn\r\nAccept: */*\r\n\r\n", len=0x7f02627a8b38)
>>>>>> at ssl_engine_io.c:614
>>>>>> #18 0x0000565098d493ec in ssl_io_filter_input (f=0x7f022c005608,
>>>>>> bb=0x7f022c006dd0, mode=<optimized out>, block=<optimized out>,
>>>>>> readbytes=8000) at ssl_engine_io.c:1474
>>>>>> #19 0x0000565098d5cd85 in h2_filter_core_input (f=0x7f022c005980,
>>>>>> brigade=0x7f022c006dd0, mode=738225624, block=APR_NONBLOCK_READ,
>>>>>> readbytes=8000) at h2_filter.c:149
>>>>>> #20 0x0000565098d6809b in h2_session_read (session=0x7f022c006920,
>>>>>> block=<optimized out>) at h2_session.c:1440
>>>>>> #21 0x0000565098d6b97f in h2_session_process (session=0x7f022c006920,
>>>>>> async=3964) at h2_session.c:2074
>>>>>> #22 0x0000565098d5b84e in h2_conn_run (ctx=<optimized out>,
>>>>>> c=0x7f025c03f7d8)
>>>>>> at h2_conn.c:218
>>>>>> #23 0x0000565098d5e551 in h2_h2_process_conn (c=0x7f025c03f7d8) at
>>>>>> h2_h2.c:658
>>>>>> #24 0x0000565098d00fd0 in ap_run_process_connection (c=0x7f025c03f7d8)
>>>>>> at connection.c:42
>>>>>> #25 0x0000565098d9b6d0 in process_socket (my_thread_num=<optimized out>,
>>>>>> my_child_num=<optimized out>, cs=0x7f025c03f748, sock=<optimized out>,
>>>>>> p=<optimized out>, thd=<optimized out>) at event.c:1134
>>>>>> #26 worker_thread (thd=0xf5e, dummy=0xf7c) at event.c:2137
>>>>>> #27 0x00007f02728680a4 in start_thread ()
>>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>> #28 0x00007f027259d62d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>> (gdb) (gdb) quit
>>>>>> Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
>>>>>> /usr/lib/debug//usr/local/apache2/bin/httpd...done.
>>>>>> done.
>>>>>> [Thread debugging using libthread_db enabled]
>>>>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>>>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>>>>>> Program terminated with signal SIGABRT, Aborted.
>>>>>> #0 0x00007f02724ea067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>> #0 0x00007f02724ea067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>> #1 0x00007f02724eb448 in abort () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>> #2 0x00007f02724e3266 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>> #3 0x00007f02724e3312 in __assert_fail () from
>>>>>> /lib/x86_64-linux-gnu/libc.so.6
>>>>>> #4 0x00007f027287062f in __pthread_tpp_change_priority ()
>>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>> #5 0x00007f0272865e9f in __pthread_mutex_lock_full ()
>>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>> #6 0x00007f0272a98db8 in allocator_alloc (in_size=in_size@entry=8032,
>>>>>> allocator=0x7f025c03d320) at memory/unix/apr_pools.c:244
>>>>>> #7 apr_allocator_alloc (allocator=0x7f025c03d320, size=size@entry=8032)
>>>>>> at memory/unix/apr_pools.c:438
>>>>>> #8 0x00007f0272cbcac4 in apr_bucket_alloc (size=8032, size@entry=8000,
>>>>>> list=0x7f02340008e8) at buckets/apr_buckets_alloc.c:157
>>>>>> #9 0x00007f0272cbdca2 in socket_bucket_read (a=0x7f0234000b08,
>>>>>> str=0x7f0260fa57b8, len=0x7f0260fa57c0, block=<optimized out>)
>>>>>> at buckets/apr_buckets_socket.c:34
>>>>>> #10 0x0000565098cf1fb1 in ap_core_input_filter (f=0x7f02340056b0,
>>>>>> b=0x7f0234005630, mode=<optimized out>, block=APR_NONBLOCK_READ,
>>>>>> readbytes=5) at core_filters.c:235
>>>>>> #11 0x0000565098d3aaca in logio_in_filter (f=<optimized out>,
>>>>>> bb=0x7f0234005630, mode=<optimized out>, block=<optimized out>,
>>>>>> readbytes=<optimized out>) at mod_logio.c:165
>>>>>> #12 0x0000565098d45b9a in bio_filter_in_read (bio=0x7f022801ff50,
>>>>>> in=0x7f02280345d3 "(\002\177", inlen=5) at ssl_engine_io.c:483
>>>>>> #13 0x00007f02736b014c in BIO_read ()
>>>>>> from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
>>>>>> #14 0x00007f0273a13c92 in ?? () from
>>>>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>>>>>> #15 0x00007f0273a1548d in ?? () from
>>>>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>>>>>> #16 0x00007f0273a12024 in ?? () from
>>>>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>>>>>> #17 0x0000565098d469f3 in ssl_io_input_read (inctx=0x7f02340035b8,
>>>>>> buf=0x7f0234003600 "", len=0x7f0260fa5b38) at ssl_engine_io.c:614
>>>>>> #18 0x0000565098d493ec in ssl_io_filter_input (f=0x7f0234005608,
>>>>>> bb=0x7f023400cd00, mode=<optimized out>, block=<optimized out>,
>>>>>> readbytes=8000) at ssl_engine_io.c:1474
>>>>>> #19 0x0000565098d5cd85 in h2_filter_core_input (f=0x7f0234005980,
>>>>>> brigade=0x7f023400cd00, mode=872467720, block=APR_NONBLOCK_READ,
>>>>>> readbytes=8000) at h2_filter.c:149
>>>>>> #20 0x0000565098d6809b in h2_session_read (session=0x7f023400c850,
>>>>>> block=<optimized out>) at h2_session.c:1440
>>>>>> #21 0x0000565098d6b97f in h2_session_process (session=0x7f023400c850,
>>>>>> async=4029) at h2_session.c:2074
>>>>>> #22 0x0000565098d5b84e in h2_conn_run (ctx=<optimized out>,
>>>>>> c=0x7f025c03d738)
>>>>>> at h2_conn.c:218
>>>>>> #23 0x0000565098d5e551 in h2_h2_process_conn (c=0x7f025c03d738) at
>>>>>> h2_h2.c:658
>>>>>> #24 0x0000565098d00fd0 in ap_run_process_connection (c=0x7f025c03d738)
>>>>>> at connection.c:42
>>>>>> #25 0x0000565098d9b6d0 in process_socket (my_thread_num=<optimized out>,
>>>>>> my_child_num=<optimized out>, cs=0x7f025c03d6a8, sock=<optimized out>,
>>>>>> p=<optimized out>, thd=<optimized out>) at event.c:1134
>>>>>> #26 worker_thread (thd=0xf9c, dummy=0xfbd) at event.c:2137
>>>>>> #27 0x00007f02728680a4 in start_thread ()
>>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>> #28 0x00007f027259d62d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>>
>>>>>> Stefan
>>>>>>
>>>>>>> Am 27.02.2017 um 19:44 schrieb Stefan Eissing:
>>>>>>> Damnit. Can you try the attached patch on top of mod_http2 v1.9.2? This one sets a mutex on the main connection allocator if missing and is pretty close to the version we ran successfully with on your site for days.
>>>>>>>
>>>>>>> Thanks again!
>>>>>>>
>>>>>>> -Stefan
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Am 27.02.2017 um 16:22 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>>>>>>
>>>>>>>> Hi Stefan,
>>>>>>>>
>>>>>>>> two new crashes here.
>>>>>>>>
>>>>>>>> Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
>>>>>>>> /usr/lib/debug//usr/local/apache2/bin/httpd...done.
>>>>>>>> done.
>>>>>>>> (gdb) (gdb) quit
>>>>>>>> Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
>>>>>>>> /usr/lib/debug//usr/local/apache2/bin/httpd...done.
>>>>>>>> done.
>>>>>>>> [Thread debugging using libthread_db enabled]
>>>>>>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>>>>>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>>>>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>>>>> #0 allocator_free (node=0x0, allocator=0x7f458005a320)
>>>>>>>> at memory/unix/apr_pools.c:381
>>>>>>>> #0 allocator_free (node=0x0, allocator=0x7f458005a320)
>>>>>>>> at memory/unix/apr_pools.c:381
>>>>>>>> #1 apr_pool_clear (pool=0x7f458004bec8) at memory/unix/apr_pools.c:793
>>>>>>>> #2 0x0000560ce83e5718 in ap_push_pool (queue_info=0x0,
>>>>>>>> pool_to_recycle=0x7f458005a328) at fdqueue.c:234
>>>>>>>> #3 0x0000560ce83e0a5a in process_lingering_close (cs=0x7f458004c158,
>>>>>>>> pfd=0x560ce8b8dda8) at event.c:1513
>>>>>>>> #4 0x0000560ce83e4590 in listener_thread (thd=0x0, dummy=0x5497f5242585c)
>>>>>>>> at event.c:1837
>>>>>>>> #5 0x00007f45967610a4 in start_thread ()
>>>>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>>>> #6 0x00007f459649662d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>>>> (gdb) (gdb) quit
>>>>>>>>
>>>>>>>>
>>>>>>>> Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
>>>>>>>> /usr/lib/debug//usr/local/apache2/bin/httpd...done.
>>>>>>>> done.
>>>>>>>> [Thread debugging using libthread_db enabled]
>>>>>>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>>>>>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>>>>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>>>>> #0 allocator_free (node=0x0, allocator=0x7f4580062840)
>>>>>>>> at memory/unix/apr_pools.c:381
>>>>>>>> #0 allocator_free (node=0x0, allocator=0x7f4580062840)
>>>>>>>> at memory/unix/apr_pools.c:381
>>>>>>>> #1 apr_pool_clear (pool=0x7f4580069188) at memory/unix/apr_pools.c:793
>>>>>>>> #2 0x0000560ce83e5718 in ap_push_pool (queue_info=0x0,
>>>>>>>> pool_to_recycle=0x7f4580062848) at fdqueue.c:234
>>>>>>>> #3 0x0000560ce83e0a5a in process_lingering_close (cs=0x7f4580069418,
>>>>>>>> pfd=0x560ce8b8dda8) at event.c:1513
>>>>>>>> #4 0x0000560ce83e4590 in listener_thread (thd=0x0, dummy=0x549845279ce62)
>>>>>>>> at event.c:1837
>>>>>>>> #5 0x00007f45967610a4 in start_thread ()
>>>>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>>>> #6 0x00007f459649662d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>>>> (gdb) (gdb) quit
>>>>>>>>
>>>>>>>> Stefan
>>>>>>>>> Am 26.02.2017 um 19:46 schrieb Stefan Priebe - Profihost AG:
>>>>>>>>> Hi Stefan,
>>>>>>>>>
>>>>>>>>> currently everything fine. No segfaults.
>>>>>>>>>
>>>>>>>>> Stefan
>>>>>>>>>
>>>>>>>>>> Am 25.02.2017 um 20:40 schrieb Stefan Priebe - Profihost AG:
>>>>>>>>>> Hi Stefan,
>>>>>>>>>>> Am 25.02.2017 um 13:51 schrieb Stefan Eissing:
>>>>>>>>>>> Stefan,
>>>>>>>>>>>
>>>>>>>>>>> whenever you have time, please deploy https://github.com/icing/mod_h2/releases/tag/v1.9.2
>>>>>>>>>>>
>>>>>>>>>>> I added an own allocator with mutex to the http/2 main session. That is something of a middle-ground between placing that on the main conn (as we had in the crash free version) and 1.9.1 behaviour. Thanks!
>>>>>>>>>>
>>>>>>>>>> done. But please keep in mind that this crash might be very rare and
>>>>>>>>>> might even have happened with v1.9.0 if we've waited more time.
>>>>>>>>>>
>>>>>>>>>> Greets,
>>>>>>>>>> Stefan
>>>>>>>>>>
>>>>>>>>>>> -Stefan
>>>>>>>>>>>
>>>>>>>>>>>> Am 24.02.2017 um 19:31 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Yann,
>>>>>>>>>>>>
>>>>>>>>>>>> here we go:
>>>>>>>>>>>>
>>>>>>>>>>>> (gdb) p *ipsub
>>>>>>>>>>>> $1 = {family = 2, sub = {1367753145, 0, 0, 0}, mask = {4294967295,
>>>>>>>>>>>> 4294967295, 4294967295, 4294967295}}
>>>>>>>>>>>>
>>>>>>>>>>>> (gdb) p *sa
>>>>>>>>>>>> $2 = {pool = 0x10b500c4ff0b0a, hostname = 0x503040203030102 <error:
>>>>>>>>>>>> Cannot access memory at address 0x503040203030102>,
>>>>>>>>>>>> servname = 0x17d010000040405 <error: Cannot access memory at address
>>>>>>>>>>>> 0x17d010000040405>, port = 770, family = 554829073,
>>>>>>>>>>>> salen = 319177009, ipaddr_len = 570909009, addr_str_len = -2127424399,
>>>>>>>>>>>> ipaddr_ptr = 0x24f0d15215c1b142,
>>>>>>>>>>>> next = 0x17160a0982726233, sa = {sin = {sin_family = 6424, sin_port =
>>>>>>>>>>>> 9498, sin_addr = {s_addr = 690497318},
>>>>>>>>>>>> sin_zero = "*456789:"}, sin6 = {sin6_family = 6424, sin6_port =
>>>>>>>>>>>> 9498, sin6_flowinfo = 690497318, sin6_addr = {__in6_u = {
>>>>>>>>>>>> __u6_addr8 = "*456789:CDEFGHIJ", __u6_addr16 = {13354, 13877,
>>>>>>>>>>>> 14391, 14905, 17475, 17989, 18503, 19017}, __u6_addr32 = {
>>>>>>>>>>>> 909456426, 976828471, 1178944579, 1246316615}}},
>>>>>>>>>>>> sin6_scope_id = 1448432723}, sas = {ss_family = 6424,
>>>>>>>>>>>> __ss_align = 4195446337656140842,
>>>>>>>>>>>> __ss_padding =
>>>>>>>>>>>> "CDEFGHIJSTUVWXYZcdefghijstuvwxyz\203\204\205\206\207\210\211\212\222\223\224\225\226\227\230\231\232\242\243\244\245\246\247\250\251\252\262\263\264\265\266\267\270\271\272\302\303\304\305\306\307\310\311\312\322\323\324\325\326\327\330\331\332\341\342\343\344\345\346\347\350\351\352\361\362\363\364\365\366\367\370\371\372\377\304\000\037\001\000\003"}}}
>>>>>>>>>>>>
>>>>>>>>>>>> (gdb) p *(struct in6_addr *)sa
>>>>>>>>>>>> $3 = {__in6_u = {__u6_addr8 =
>>>>>>>>>>>> "\n\v\377\304\000\265\020\000\002\001\003\003\002\004\003\005",
>>>>>>>>>>>> __u6_addr16 = {2826, 50431, 46336,
>>>>>>>>>>>> 16, 258, 771, 1026, 1283}, __u6_addr32 = {3305048842, 1094912,
>>>>>>>>>>>> 50528514, 84083714}}}
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Stefan
>>>>>>>>>>>>
>>>>>>>>>>>>> Am 24.02.2017 um 14:18 schrieb Yann Ylavic:
>>>>>>>>>>>>> Hi Stefan (Priebe),
>>>>>>>>>>>>>
>>>>>>>>>>>>> Is IPv6 (really) involved in your network?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Could you please show up the gdb output of the below ?
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Feb 24, 2017 at 2:07 PM, Yann Ylavic <yl...@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 1078 APR_DECLARE(int) apr_ipsubnet_test(apr_ipsubnet_t *ipsub,
>>>>>>>>>>>>>> apr_sockaddr_t *sa)
>>>>>>>>>>>>>> 1079 {
>>>>>>>>>>>>>> 1080 #if APR_HAVE_IPV6
>>>>>>>>>>>>>> 1081 /* XXX This line will segv on Win32 build with APR_HAVE_IPV6,
>>>>>>>>>>>>>> 1082 * but without the IPV6 drivers installed.
>>>>>>>>>>>>>> 1083 */
>>>>>>>>>>>>>> 1084 if (sa->family == AF_INET) {
>>>>>>>>>>>>>> 1085 if (ipsub->family == AF_INET &&
>>>>>>>>>>>>>> 1086 ((sa->sa.sin.sin_addr.s_addr & ipsub->mask[0]) ==
>>>>>>>>>>>>>> ipsub->sub[0])) {
>>>>>>>>>>>>>> 1087 return 1;
>>>>>>>>>>>>>> 1088 }
>>>>>>>>>>>>>> 1089 }
>>>>>>>>>>>>>> 1090 else if (IN6_IS_ADDR_V4MAPPED((struct in6_addr *)sa->ipaddr_ptr)) {
>>>>>>>>>>>>>> 1091 if (ipsub->family == AF_INET &&
>>>>>>>>>>>>>> 1092 (((apr_uint32_t *)sa->ipaddr_ptr)[3] &
>>>>>>>>>>>>>> ipsub->mask[0]) == ipsub->sub[0]) {
>>>>>>>>>>>>>> 1093 return 1;
>>>>>>>>>>>>>> 1094 }
>>>>>>>>>>>>>> 1095 }
>>>>>>>>>>>>>
>>>>>>>>>>>>> (gdb) p *ipsub
>>>>>>>>>>>>> (gdb) p *sa
>>>>>>>>>>>>> (gdb) p *(struct in6_addr *)sa
>>>>>>>>>>>>>
>>>>>>>>>>>>> and possibly more to come...
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Yann.
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Stefan Eissing
>>>>>>>>>>>
>>>>>>>>>>> <green/>bytes GmbH
>>>>>>>>>>> Hafenstrasse 16
>>>>>>>>>>> 48155 M�nster
>>>>>>>>>>> www.greenbytes.de
>>>>>>>>>>>
>>>>>>>
>>>>>>> Stefan Eissing
>>>>>>>
>>>>>>> <green/>bytes GmbH
>>>>>>> Hafenstrasse 16
>>>>>>> 48155 M�nster
>>>>>>> www.greenbytes.de
>>>>>>>
>>>>>
>>>>> Stefan Eissing
>>>>>
>>>>> <green/>bytes GmbH
>>>>> Hafenstrasse 16
>>>>> 48155 M�nster
>>>>> www.greenbytes.de
>>>>>
>>
Re: release v1.9.2
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Hi Stefan,
yes this seems to correct BUT i'm not sure i the test was long enough
where i hadn't a segfault. I'll rerun a test with that version. To
verify this was no just a "fault".
Greets,
Stefan
Am 05.03.2017 um 14:34 schrieb Stefan Eissing:
> Thanks. I try to summarize:
>
> - we did see this rarely with unpatched v1.9.2 and Yann's mpm patch v??
> - we still see this rarely with the patch adding mutex to the main conn allocator (so this seems not to change anything)
> - we did not see this with v1.9.2 and Yann's patch on ptrans version ??
>
> Is this correct?
>
>> Am 04.03.2017 um 22:34 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>
>> Hi Stefan,
>>
>> current trace - not sure whether this is http2 related at all:
>>
>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>> Program terminated with signal SIGSEGV, Segmentation fault.
>> #0 allocator_free (node=0x0, allocator=0x7fbf4c05a410)
>> at memory/unix/apr_pools.c:381
>> #0 allocator_free (node=0x0, allocator=0x7fbf4c05a410)
>> at memory/unix/apr_pools.c:381
>> #1 apr_pool_clear (pool=0x7fbf4c04f888) at memory/unix/apr_pools.c:793
>> #2 0x00005642878c2818 in ap_push_pool (queue_info=0x0,
>> pool_to_recycle=0x7fbf4c05a418) at fdqueue.c:234
>> #3 0x00005642878bdb5a in process_lingering_close (cs=0x7fbf4c04fb18,
>> pfd=0x5642886c7078) at event.c:1513
>> #4 0x00005642878c1690 in listener_thread (thd=0x0, dummy=0x549eb52311a6f)
>> at event.c:1837
>> #5 0x00007fbf5f2940a4 in start_thread ()
>> from /lib/x86_64-linux-gnu/libpthread.so.0
>> #6 0x00007fbf5efc962d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>
>> Stefan
>>
>>> Am 03.03.2017 um 21:02 schrieb Stefan Priebe - Profihost AG:
>>> Hi Stefan,
>>>
>>> live. Sorry for the late reply.
>>>
>>> Stefan
>>>
>>>> Am 28.02.2017 um 11:49 schrieb Stefan Eissing:
>>>> Meh, my mistake. Sorry about that. Did not cleanup properly. Dare you with an improved version:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> Am 28.02.2017 um 08:41 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>>>
>>>>> That one breaks everyting. Multiple crashes per second.
>>>>>
>>>>> [Thread debugging using libthread_db enabled]
>>>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>>>>> Program terminated with signal SIGABRT, Aborted.
>>>>> #0 0x00007f02724ea067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>>>>> #0 0x00007f02724ea067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>>>>> #1 0x00007f02724eb448 in abort () from /lib/x86_64-linux-gnu/libc.so.6
>>>>> #2 0x00007f02724e3266 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>>>>> #3 0x00007f02724e3312 in __assert_fail () from
>>>>> /lib/x86_64-linux-gnu/libc.so.6
>>>>> #4 0x00007f027287062f in __pthread_tpp_change_priority ()
>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>> #5 0x00007f0272865e9f in __pthread_mutex_lock_full ()
>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>> #6 0x00007f0272a98db8 in allocator_alloc (in_size=in_size@entry=8032,
>>>>> allocator=0x7f025c03f3c0) at memory/unix/apr_pools.c:244
>>>>> #7 apr_allocator_alloc (allocator=0x7f025c03f3c0, size=size@entry=8032)
>>>>> at memory/unix/apr_pools.c:438
>>>>> #8 0x00007f0272cbcac4 in apr_bucket_alloc (size=8032, size@entry=8000,
>>>>> list=0x7f022c0008e8) at buckets/apr_buckets_alloc.c:157
>>>>> #9 0x00007f0272cbdca2 in socket_bucket_read (a=0x7f022c000b08,
>>>>> str=0x7f02627a87b8, len=0x7f02627a87c0, block=<optimized out>)
>>>>> at buckets/apr_buckets_socket.c:34
>>>>> #10 0x0000565098cf1fb1 in ap_core_input_filter (f=0x7f022c0056b0,
>>>>> b=0x7f022c005630, mode=<optimized out>, block=APR_NONBLOCK_READ,
>>>>> readbytes=5) at core_filters.c:235
>>>>> #11 0x0000565098d3aaca in logio_in_filter (f=<optimized out>,
>>>>> bb=0x7f022c005630, mode=<optimized out>, block=<optimized out>,
>>>>> readbytes=<optimized out>) at mod_logio.c:165
>>>>> #12 0x0000565098d45b9a in bio_filter_in_read (bio=0x7f024800b460,
>>>>> in=0x7f02480492a3 "", inlen=5) at ssl_engine_io.c:483
>>>>> #13 0x00007f02736b014c in BIO_read ()
>>>>> from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
>>>>> #14 0x00007f0273a13c92 in ?? () from
>>>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>>>>> #15 0x00007f0273a1548d in ?? () from
>>>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>>>>> #16 0x00007f0273a12024 in ?? () from
>>>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>>>>> #17 0x0000565098d469f3 in ssl_io_input_read (inctx=0x7f022c0035b8,
>>>>> buf=0x7f022c003600 "GET
>>>>> /media/image/07/3c/c2/4612_13721_160192280322.jpeg HTTP/1.1\r\nHost:
>>>>> www.onlinestore.it\r\nUser-Agent: curl/7.15+ (x64-criteo) libcurl/7.15+
>>>>> OpenSSL zlib libidn\r\nAccept: */*\r\n\r\n", len=0x7f02627a8b38)
>>>>> at ssl_engine_io.c:614
>>>>> #18 0x0000565098d493ec in ssl_io_filter_input (f=0x7f022c005608,
>>>>> bb=0x7f022c006dd0, mode=<optimized out>, block=<optimized out>,
>>>>> readbytes=8000) at ssl_engine_io.c:1474
>>>>> #19 0x0000565098d5cd85 in h2_filter_core_input (f=0x7f022c005980,
>>>>> brigade=0x7f022c006dd0, mode=738225624, block=APR_NONBLOCK_READ,
>>>>> readbytes=8000) at h2_filter.c:149
>>>>> #20 0x0000565098d6809b in h2_session_read (session=0x7f022c006920,
>>>>> block=<optimized out>) at h2_session.c:1440
>>>>> #21 0x0000565098d6b97f in h2_session_process (session=0x7f022c006920,
>>>>> async=3964) at h2_session.c:2074
>>>>> #22 0x0000565098d5b84e in h2_conn_run (ctx=<optimized out>,
>>>>> c=0x7f025c03f7d8)
>>>>> at h2_conn.c:218
>>>>> #23 0x0000565098d5e551 in h2_h2_process_conn (c=0x7f025c03f7d8) at
>>>>> h2_h2.c:658
>>>>> #24 0x0000565098d00fd0 in ap_run_process_connection (c=0x7f025c03f7d8)
>>>>> at connection.c:42
>>>>> #25 0x0000565098d9b6d0 in process_socket (my_thread_num=<optimized out>,
>>>>> my_child_num=<optimized out>, cs=0x7f025c03f748, sock=<optimized out>,
>>>>> p=<optimized out>, thd=<optimized out>) at event.c:1134
>>>>> #26 worker_thread (thd=0xf5e, dummy=0xf7c) at event.c:2137
>>>>> #27 0x00007f02728680a4 in start_thread ()
>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>> #28 0x00007f027259d62d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>> (gdb) (gdb) quit
>>>>> Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
>>>>> /usr/lib/debug//usr/local/apache2/bin/httpd...done.
>>>>> done.
>>>>> [Thread debugging using libthread_db enabled]
>>>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>>>>> Program terminated with signal SIGABRT, Aborted.
>>>>> #0 0x00007f02724ea067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>>>>> #0 0x00007f02724ea067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>>>>> #1 0x00007f02724eb448 in abort () from /lib/x86_64-linux-gnu/libc.so.6
>>>>> #2 0x00007f02724e3266 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>>>>> #3 0x00007f02724e3312 in __assert_fail () from
>>>>> /lib/x86_64-linux-gnu/libc.so.6
>>>>> #4 0x00007f027287062f in __pthread_tpp_change_priority ()
>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>> #5 0x00007f0272865e9f in __pthread_mutex_lock_full ()
>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>> #6 0x00007f0272a98db8 in allocator_alloc (in_size=in_size@entry=8032,
>>>>> allocator=0x7f025c03d320) at memory/unix/apr_pools.c:244
>>>>> #7 apr_allocator_alloc (allocator=0x7f025c03d320, size=size@entry=8032)
>>>>> at memory/unix/apr_pools.c:438
>>>>> #8 0x00007f0272cbcac4 in apr_bucket_alloc (size=8032, size@entry=8000,
>>>>> list=0x7f02340008e8) at buckets/apr_buckets_alloc.c:157
>>>>> #9 0x00007f0272cbdca2 in socket_bucket_read (a=0x7f0234000b08,
>>>>> str=0x7f0260fa57b8, len=0x7f0260fa57c0, block=<optimized out>)
>>>>> at buckets/apr_buckets_socket.c:34
>>>>> #10 0x0000565098cf1fb1 in ap_core_input_filter (f=0x7f02340056b0,
>>>>> b=0x7f0234005630, mode=<optimized out>, block=APR_NONBLOCK_READ,
>>>>> readbytes=5) at core_filters.c:235
>>>>> #11 0x0000565098d3aaca in logio_in_filter (f=<optimized out>,
>>>>> bb=0x7f0234005630, mode=<optimized out>, block=<optimized out>,
>>>>> readbytes=<optimized out>) at mod_logio.c:165
>>>>> #12 0x0000565098d45b9a in bio_filter_in_read (bio=0x7f022801ff50,
>>>>> in=0x7f02280345d3 "(\002\177", inlen=5) at ssl_engine_io.c:483
>>>>> #13 0x00007f02736b014c in BIO_read ()
>>>>> from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
>>>>> #14 0x00007f0273a13c92 in ?? () from
>>>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>>>>> #15 0x00007f0273a1548d in ?? () from
>>>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>>>>> #16 0x00007f0273a12024 in ?? () from
>>>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>>>>> #17 0x0000565098d469f3 in ssl_io_input_read (inctx=0x7f02340035b8,
>>>>> buf=0x7f0234003600 "", len=0x7f0260fa5b38) at ssl_engine_io.c:614
>>>>> #18 0x0000565098d493ec in ssl_io_filter_input (f=0x7f0234005608,
>>>>> bb=0x7f023400cd00, mode=<optimized out>, block=<optimized out>,
>>>>> readbytes=8000) at ssl_engine_io.c:1474
>>>>> #19 0x0000565098d5cd85 in h2_filter_core_input (f=0x7f0234005980,
>>>>> brigade=0x7f023400cd00, mode=872467720, block=APR_NONBLOCK_READ,
>>>>> readbytes=8000) at h2_filter.c:149
>>>>> #20 0x0000565098d6809b in h2_session_read (session=0x7f023400c850,
>>>>> block=<optimized out>) at h2_session.c:1440
>>>>> #21 0x0000565098d6b97f in h2_session_process (session=0x7f023400c850,
>>>>> async=4029) at h2_session.c:2074
>>>>> #22 0x0000565098d5b84e in h2_conn_run (ctx=<optimized out>,
>>>>> c=0x7f025c03d738)
>>>>> at h2_conn.c:218
>>>>> #23 0x0000565098d5e551 in h2_h2_process_conn (c=0x7f025c03d738) at
>>>>> h2_h2.c:658
>>>>> #24 0x0000565098d00fd0 in ap_run_process_connection (c=0x7f025c03d738)
>>>>> at connection.c:42
>>>>> #25 0x0000565098d9b6d0 in process_socket (my_thread_num=<optimized out>,
>>>>> my_child_num=<optimized out>, cs=0x7f025c03d6a8, sock=<optimized out>,
>>>>> p=<optimized out>, thd=<optimized out>) at event.c:1134
>>>>> #26 worker_thread (thd=0xf9c, dummy=0xfbd) at event.c:2137
>>>>> #27 0x00007f02728680a4 in start_thread ()
>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>> #28 0x00007f027259d62d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>
>>>>> Stefan
>>>>>
>>>>>> Am 27.02.2017 um 19:44 schrieb Stefan Eissing:
>>>>>> Damnit. Can you try the attached patch on top of mod_http2 v1.9.2? This one sets a mutex on the main connection allocator if missing and is pretty close to the version we ran successfully with on your site for days.
>>>>>>
>>>>>> Thanks again!
>>>>>>
>>>>>> -Stefan
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Am 27.02.2017 um 16:22 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>>>>>
>>>>>>> Hi Stefan,
>>>>>>>
>>>>>>> two new crashes here.
>>>>>>>
>>>>>>> Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
>>>>>>> /usr/lib/debug//usr/local/apache2/bin/httpd...done.
>>>>>>> done.
>>>>>>> (gdb) (gdb) quit
>>>>>>> Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
>>>>>>> /usr/lib/debug//usr/local/apache2/bin/httpd...done.
>>>>>>> done.
>>>>>>> [Thread debugging using libthread_db enabled]
>>>>>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>>>>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>>>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>>>> #0 allocator_free (node=0x0, allocator=0x7f458005a320)
>>>>>>> at memory/unix/apr_pools.c:381
>>>>>>> #0 allocator_free (node=0x0, allocator=0x7f458005a320)
>>>>>>> at memory/unix/apr_pools.c:381
>>>>>>> #1 apr_pool_clear (pool=0x7f458004bec8) at memory/unix/apr_pools.c:793
>>>>>>> #2 0x0000560ce83e5718 in ap_push_pool (queue_info=0x0,
>>>>>>> pool_to_recycle=0x7f458005a328) at fdqueue.c:234
>>>>>>> #3 0x0000560ce83e0a5a in process_lingering_close (cs=0x7f458004c158,
>>>>>>> pfd=0x560ce8b8dda8) at event.c:1513
>>>>>>> #4 0x0000560ce83e4590 in listener_thread (thd=0x0, dummy=0x5497f5242585c)
>>>>>>> at event.c:1837
>>>>>>> #5 0x00007f45967610a4 in start_thread ()
>>>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>>> #6 0x00007f459649662d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>>> (gdb) (gdb) quit
>>>>>>>
>>>>>>>
>>>>>>> Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
>>>>>>> /usr/lib/debug//usr/local/apache2/bin/httpd...done.
>>>>>>> done.
>>>>>>> [Thread debugging using libthread_db enabled]
>>>>>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>>>>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>>>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>>>> #0 allocator_free (node=0x0, allocator=0x7f4580062840)
>>>>>>> at memory/unix/apr_pools.c:381
>>>>>>> #0 allocator_free (node=0x0, allocator=0x7f4580062840)
>>>>>>> at memory/unix/apr_pools.c:381
>>>>>>> #1 apr_pool_clear (pool=0x7f4580069188) at memory/unix/apr_pools.c:793
>>>>>>> #2 0x0000560ce83e5718 in ap_push_pool (queue_info=0x0,
>>>>>>> pool_to_recycle=0x7f4580062848) at fdqueue.c:234
>>>>>>> #3 0x0000560ce83e0a5a in process_lingering_close (cs=0x7f4580069418,
>>>>>>> pfd=0x560ce8b8dda8) at event.c:1513
>>>>>>> #4 0x0000560ce83e4590 in listener_thread (thd=0x0, dummy=0x549845279ce62)
>>>>>>> at event.c:1837
>>>>>>> #5 0x00007f45967610a4 in start_thread ()
>>>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>>> #6 0x00007f459649662d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>>> (gdb) (gdb) quit
>>>>>>>
>>>>>>> Stefan
>>>>>>>> Am 26.02.2017 um 19:46 schrieb Stefan Priebe - Profihost AG:
>>>>>>>> Hi Stefan,
>>>>>>>>
>>>>>>>> currently everything fine. No segfaults.
>>>>>>>>
>>>>>>>> Stefan
>>>>>>>>
>>>>>>>>> Am 25.02.2017 um 20:40 schrieb Stefan Priebe - Profihost AG:
>>>>>>>>> Hi Stefan,
>>>>>>>>>> Am 25.02.2017 um 13:51 schrieb Stefan Eissing:
>>>>>>>>>> Stefan,
>>>>>>>>>>
>>>>>>>>>> whenever you have time, please deploy https://github.com/icing/mod_h2/releases/tag/v1.9.2
>>>>>>>>>>
>>>>>>>>>> I added an own allocator with mutex to the http/2 main session. That is something of a middle-ground between placing that on the main conn (as we had in the crash free version) and 1.9.1 behaviour. Thanks!
>>>>>>>>>
>>>>>>>>> done. But please keep in mind that this crash might be very rare and
>>>>>>>>> might even have happened with v1.9.0 if we've waited more time.
>>>>>>>>>
>>>>>>>>> Greets,
>>>>>>>>> Stefan
>>>>>>>>>
>>>>>>>>>> -Stefan
>>>>>>>>>>
>>>>>>>>>>> Am 24.02.2017 um 19:31 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>>>>>>>>>
>>>>>>>>>>> Hi Yann,
>>>>>>>>>>>
>>>>>>>>>>> here we go:
>>>>>>>>>>>
>>>>>>>>>>> (gdb) p *ipsub
>>>>>>>>>>> $1 = {family = 2, sub = {1367753145, 0, 0, 0}, mask = {4294967295,
>>>>>>>>>>> 4294967295, 4294967295, 4294967295}}
>>>>>>>>>>>
>>>>>>>>>>> (gdb) p *sa
>>>>>>>>>>> $2 = {pool = 0x10b500c4ff0b0a, hostname = 0x503040203030102 <error:
>>>>>>>>>>> Cannot access memory at address 0x503040203030102>,
>>>>>>>>>>> servname = 0x17d010000040405 <error: Cannot access memory at address
>>>>>>>>>>> 0x17d010000040405>, port = 770, family = 554829073,
>>>>>>>>>>> salen = 319177009, ipaddr_len = 570909009, addr_str_len = -2127424399,
>>>>>>>>>>> ipaddr_ptr = 0x24f0d15215c1b142,
>>>>>>>>>>> next = 0x17160a0982726233, sa = {sin = {sin_family = 6424, sin_port =
>>>>>>>>>>> 9498, sin_addr = {s_addr = 690497318},
>>>>>>>>>>> sin_zero = "*456789:"}, sin6 = {sin6_family = 6424, sin6_port =
>>>>>>>>>>> 9498, sin6_flowinfo = 690497318, sin6_addr = {__in6_u = {
>>>>>>>>>>> __u6_addr8 = "*456789:CDEFGHIJ", __u6_addr16 = {13354, 13877,
>>>>>>>>>>> 14391, 14905, 17475, 17989, 18503, 19017}, __u6_addr32 = {
>>>>>>>>>>> 909456426, 976828471, 1178944579, 1246316615}}},
>>>>>>>>>>> sin6_scope_id = 1448432723}, sas = {ss_family = 6424,
>>>>>>>>>>> __ss_align = 4195446337656140842,
>>>>>>>>>>> __ss_padding =
>>>>>>>>>>> "CDEFGHIJSTUVWXYZcdefghijstuvwxyz\203\204\205\206\207\210\211\212\222\223\224\225\226\227\230\231\232\242\243\244\245\246\247\250\251\252\262\263\264\265\266\267\270\271\272\302\303\304\305\306\307\310\311\312\322\323\324\325\326\327\330\331\332\341\342\343\344\345\346\347\350\351\352\361\362\363\364\365\366\367\370\371\372\377\304\000\037\001\000\003"}}}
>>>>>>>>>>>
>>>>>>>>>>> (gdb) p *(struct in6_addr *)sa
>>>>>>>>>>> $3 = {__in6_u = {__u6_addr8 =
>>>>>>>>>>> "\n\v\377\304\000\265\020\000\002\001\003\003\002\004\003\005",
>>>>>>>>>>> __u6_addr16 = {2826, 50431, 46336,
>>>>>>>>>>> 16, 258, 771, 1026, 1283}, __u6_addr32 = {3305048842, 1094912,
>>>>>>>>>>> 50528514, 84083714}}}
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Stefan
>>>>>>>>>>>
>>>>>>>>>>>> Am 24.02.2017 um 14:18 schrieb Yann Ylavic:
>>>>>>>>>>>> Hi Stefan (Priebe),
>>>>>>>>>>>>
>>>>>>>>>>>> Is IPv6 (really) involved in your network?
>>>>>>>>>>>>
>>>>>>>>>>>> Could you please show up the gdb output of the below ?
>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Feb 24, 2017 at 2:07 PM, Yann Ylavic <yl...@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1078 APR_DECLARE(int) apr_ipsubnet_test(apr_ipsubnet_t *ipsub,
>>>>>>>>>>>>> apr_sockaddr_t *sa)
>>>>>>>>>>>>> 1079 {
>>>>>>>>>>>>> 1080 #if APR_HAVE_IPV6
>>>>>>>>>>>>> 1081 /* XXX This line will segv on Win32 build with APR_HAVE_IPV6,
>>>>>>>>>>>>> 1082 * but without the IPV6 drivers installed.
>>>>>>>>>>>>> 1083 */
>>>>>>>>>>>>> 1084 if (sa->family == AF_INET) {
>>>>>>>>>>>>> 1085 if (ipsub->family == AF_INET &&
>>>>>>>>>>>>> 1086 ((sa->sa.sin.sin_addr.s_addr & ipsub->mask[0]) ==
>>>>>>>>>>>>> ipsub->sub[0])) {
>>>>>>>>>>>>> 1087 return 1;
>>>>>>>>>>>>> 1088 }
>>>>>>>>>>>>> 1089 }
>>>>>>>>>>>>> 1090 else if (IN6_IS_ADDR_V4MAPPED((struct in6_addr *)sa->ipaddr_ptr)) {
>>>>>>>>>>>>> 1091 if (ipsub->family == AF_INET &&
>>>>>>>>>>>>> 1092 (((apr_uint32_t *)sa->ipaddr_ptr)[3] &
>>>>>>>>>>>>> ipsub->mask[0]) == ipsub->sub[0]) {
>>>>>>>>>>>>> 1093 return 1;
>>>>>>>>>>>>> 1094 }
>>>>>>>>>>>>> 1095 }
>>>>>>>>>>>>
>>>>>>>>>>>> (gdb) p *ipsub
>>>>>>>>>>>> (gdb) p *sa
>>>>>>>>>>>> (gdb) p *(struct in6_addr *)sa
>>>>>>>>>>>>
>>>>>>>>>>>> and possibly more to come...
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Yann.
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Stefan Eissing
>>>>>>>>>>
>>>>>>>>>> <green/>bytes GmbH
>>>>>>>>>> Hafenstrasse 16
>>>>>>>>>> 48155 M�nster
>>>>>>>>>> www.greenbytes.de
>>>>>>>>>>
>>>>>>
>>>>>> Stefan Eissing
>>>>>>
>>>>>> <green/>bytes GmbH
>>>>>> Hafenstrasse 16
>>>>>> 48155 M�nster
>>>>>> www.greenbytes.de
>>>>>>
>>>>
>>>> Stefan Eissing
>>>>
>>>> <green/>bytes GmbH
>>>> Hafenstrasse 16
>>>> 48155 M�nster
>>>> www.greenbytes.de
>>>>
>
Re: release v1.9.2
Posted by Stefan Eissing <st...@greenbytes.de>.
Thanks. I try to summarize:
- we did see this rarely with unpatched v1.9.2 and Yann's mpm patch v??
- we still see this rarely with the patch adding mutex to the main conn allocator (so this seems not to change anything)
- we did not see this with v1.9.2 and Yann's patch on ptrans version ??
Is this correct?
> Am 04.03.2017 um 22:34 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>
> Hi Stefan,
>
> current trace - not sure whether this is http2 related at all:
>
> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0 allocator_free (node=0x0, allocator=0x7fbf4c05a410)
> at memory/unix/apr_pools.c:381
> #0 allocator_free (node=0x0, allocator=0x7fbf4c05a410)
> at memory/unix/apr_pools.c:381
> #1 apr_pool_clear (pool=0x7fbf4c04f888) at memory/unix/apr_pools.c:793
> #2 0x00005642878c2818 in ap_push_pool (queue_info=0x0,
> pool_to_recycle=0x7fbf4c05a418) at fdqueue.c:234
> #3 0x00005642878bdb5a in process_lingering_close (cs=0x7fbf4c04fb18,
> pfd=0x5642886c7078) at event.c:1513
> #4 0x00005642878c1690 in listener_thread (thd=0x0, dummy=0x549eb52311a6f)
> at event.c:1837
> #5 0x00007fbf5f2940a4 in start_thread ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #6 0x00007fbf5efc962d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Stefan
>
>> Am 03.03.2017 um 21:02 schrieb Stefan Priebe - Profihost AG:
>> Hi Stefan,
>>
>> live. Sorry for the late reply.
>>
>> Stefan
>>
>>> Am 28.02.2017 um 11:49 schrieb Stefan Eissing:
>>> Meh, my mistake. Sorry about that. Did not cleanup properly. Dare you with an improved version:
>>>
>>>
>>>
>>>
>>>
>>>> Am 28.02.2017 um 08:41 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>>
>>>> That one breaks everyting. Multiple crashes per second.
>>>>
>>>> [Thread debugging using libthread_db enabled]
>>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>>>> Program terminated with signal SIGABRT, Aborted.
>>>> #0 0x00007f02724ea067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>>>> #0 0x00007f02724ea067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>>>> #1 0x00007f02724eb448 in abort () from /lib/x86_64-linux-gnu/libc.so.6
>>>> #2 0x00007f02724e3266 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>>>> #3 0x00007f02724e3312 in __assert_fail () from
>>>> /lib/x86_64-linux-gnu/libc.so.6
>>>> #4 0x00007f027287062f in __pthread_tpp_change_priority ()
>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>> #5 0x00007f0272865e9f in __pthread_mutex_lock_full ()
>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>> #6 0x00007f0272a98db8 in allocator_alloc (in_size=in_size@entry=8032,
>>>> allocator=0x7f025c03f3c0) at memory/unix/apr_pools.c:244
>>>> #7 apr_allocator_alloc (allocator=0x7f025c03f3c0, size=size@entry=8032)
>>>> at memory/unix/apr_pools.c:438
>>>> #8 0x00007f0272cbcac4 in apr_bucket_alloc (size=8032, size@entry=8000,
>>>> list=0x7f022c0008e8) at buckets/apr_buckets_alloc.c:157
>>>> #9 0x00007f0272cbdca2 in socket_bucket_read (a=0x7f022c000b08,
>>>> str=0x7f02627a87b8, len=0x7f02627a87c0, block=<optimized out>)
>>>> at buckets/apr_buckets_socket.c:34
>>>> #10 0x0000565098cf1fb1 in ap_core_input_filter (f=0x7f022c0056b0,
>>>> b=0x7f022c005630, mode=<optimized out>, block=APR_NONBLOCK_READ,
>>>> readbytes=5) at core_filters.c:235
>>>> #11 0x0000565098d3aaca in logio_in_filter (f=<optimized out>,
>>>> bb=0x7f022c005630, mode=<optimized out>, block=<optimized out>,
>>>> readbytes=<optimized out>) at mod_logio.c:165
>>>> #12 0x0000565098d45b9a in bio_filter_in_read (bio=0x7f024800b460,
>>>> in=0x7f02480492a3 "", inlen=5) at ssl_engine_io.c:483
>>>> #13 0x00007f02736b014c in BIO_read ()
>>>> from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
>>>> #14 0x00007f0273a13c92 in ?? () from
>>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>>>> #15 0x00007f0273a1548d in ?? () from
>>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>>>> #16 0x00007f0273a12024 in ?? () from
>>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>>>> #17 0x0000565098d469f3 in ssl_io_input_read (inctx=0x7f022c0035b8,
>>>> buf=0x7f022c003600 "GET
>>>> /media/image/07/3c/c2/4612_13721_160192280322.jpeg HTTP/1.1\r\nHost:
>>>> www.onlinestore.it\r\nUser-Agent: curl/7.15+ (x64-criteo) libcurl/7.15+
>>>> OpenSSL zlib libidn\r\nAccept: */*\r\n\r\n", len=0x7f02627a8b38)
>>>> at ssl_engine_io.c:614
>>>> #18 0x0000565098d493ec in ssl_io_filter_input (f=0x7f022c005608,
>>>> bb=0x7f022c006dd0, mode=<optimized out>, block=<optimized out>,
>>>> readbytes=8000) at ssl_engine_io.c:1474
>>>> #19 0x0000565098d5cd85 in h2_filter_core_input (f=0x7f022c005980,
>>>> brigade=0x7f022c006dd0, mode=738225624, block=APR_NONBLOCK_READ,
>>>> readbytes=8000) at h2_filter.c:149
>>>> #20 0x0000565098d6809b in h2_session_read (session=0x7f022c006920,
>>>> block=<optimized out>) at h2_session.c:1440
>>>> #21 0x0000565098d6b97f in h2_session_process (session=0x7f022c006920,
>>>> async=3964) at h2_session.c:2074
>>>> #22 0x0000565098d5b84e in h2_conn_run (ctx=<optimized out>,
>>>> c=0x7f025c03f7d8)
>>>> at h2_conn.c:218
>>>> #23 0x0000565098d5e551 in h2_h2_process_conn (c=0x7f025c03f7d8) at
>>>> h2_h2.c:658
>>>> #24 0x0000565098d00fd0 in ap_run_process_connection (c=0x7f025c03f7d8)
>>>> at connection.c:42
>>>> #25 0x0000565098d9b6d0 in process_socket (my_thread_num=<optimized out>,
>>>> my_child_num=<optimized out>, cs=0x7f025c03f748, sock=<optimized out>,
>>>> p=<optimized out>, thd=<optimized out>) at event.c:1134
>>>> #26 worker_thread (thd=0xf5e, dummy=0xf7c) at event.c:2137
>>>> #27 0x00007f02728680a4 in start_thread ()
>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>> #28 0x00007f027259d62d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>> (gdb) (gdb) quit
>>>> Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
>>>> /usr/lib/debug//usr/local/apache2/bin/httpd...done.
>>>> done.
>>>> [Thread debugging using libthread_db enabled]
>>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>>>> Program terminated with signal SIGABRT, Aborted.
>>>> #0 0x00007f02724ea067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>>>> #0 0x00007f02724ea067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>>>> #1 0x00007f02724eb448 in abort () from /lib/x86_64-linux-gnu/libc.so.6
>>>> #2 0x00007f02724e3266 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>>>> #3 0x00007f02724e3312 in __assert_fail () from
>>>> /lib/x86_64-linux-gnu/libc.so.6
>>>> #4 0x00007f027287062f in __pthread_tpp_change_priority ()
>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>> #5 0x00007f0272865e9f in __pthread_mutex_lock_full ()
>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>> #6 0x00007f0272a98db8 in allocator_alloc (in_size=in_size@entry=8032,
>>>> allocator=0x7f025c03d320) at memory/unix/apr_pools.c:244
>>>> #7 apr_allocator_alloc (allocator=0x7f025c03d320, size=size@entry=8032)
>>>> at memory/unix/apr_pools.c:438
>>>> #8 0x00007f0272cbcac4 in apr_bucket_alloc (size=8032, size@entry=8000,
>>>> list=0x7f02340008e8) at buckets/apr_buckets_alloc.c:157
>>>> #9 0x00007f0272cbdca2 in socket_bucket_read (a=0x7f0234000b08,
>>>> str=0x7f0260fa57b8, len=0x7f0260fa57c0, block=<optimized out>)
>>>> at buckets/apr_buckets_socket.c:34
>>>> #10 0x0000565098cf1fb1 in ap_core_input_filter (f=0x7f02340056b0,
>>>> b=0x7f0234005630, mode=<optimized out>, block=APR_NONBLOCK_READ,
>>>> readbytes=5) at core_filters.c:235
>>>> #11 0x0000565098d3aaca in logio_in_filter (f=<optimized out>,
>>>> bb=0x7f0234005630, mode=<optimized out>, block=<optimized out>,
>>>> readbytes=<optimized out>) at mod_logio.c:165
>>>> #12 0x0000565098d45b9a in bio_filter_in_read (bio=0x7f022801ff50,
>>>> in=0x7f02280345d3 "(\002\177", inlen=5) at ssl_engine_io.c:483
>>>> #13 0x00007f02736b014c in BIO_read ()
>>>> from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
>>>> #14 0x00007f0273a13c92 in ?? () from
>>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>>>> #15 0x00007f0273a1548d in ?? () from
>>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>>>> #16 0x00007f0273a12024 in ?? () from
>>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>>>> #17 0x0000565098d469f3 in ssl_io_input_read (inctx=0x7f02340035b8,
>>>> buf=0x7f0234003600 "", len=0x7f0260fa5b38) at ssl_engine_io.c:614
>>>> #18 0x0000565098d493ec in ssl_io_filter_input (f=0x7f0234005608,
>>>> bb=0x7f023400cd00, mode=<optimized out>, block=<optimized out>,
>>>> readbytes=8000) at ssl_engine_io.c:1474
>>>> #19 0x0000565098d5cd85 in h2_filter_core_input (f=0x7f0234005980,
>>>> brigade=0x7f023400cd00, mode=872467720, block=APR_NONBLOCK_READ,
>>>> readbytes=8000) at h2_filter.c:149
>>>> #20 0x0000565098d6809b in h2_session_read (session=0x7f023400c850,
>>>> block=<optimized out>) at h2_session.c:1440
>>>> #21 0x0000565098d6b97f in h2_session_process (session=0x7f023400c850,
>>>> async=4029) at h2_session.c:2074
>>>> #22 0x0000565098d5b84e in h2_conn_run (ctx=<optimized out>,
>>>> c=0x7f025c03d738)
>>>> at h2_conn.c:218
>>>> #23 0x0000565098d5e551 in h2_h2_process_conn (c=0x7f025c03d738) at
>>>> h2_h2.c:658
>>>> #24 0x0000565098d00fd0 in ap_run_process_connection (c=0x7f025c03d738)
>>>> at connection.c:42
>>>> #25 0x0000565098d9b6d0 in process_socket (my_thread_num=<optimized out>,
>>>> my_child_num=<optimized out>, cs=0x7f025c03d6a8, sock=<optimized out>,
>>>> p=<optimized out>, thd=<optimized out>) at event.c:1134
>>>> #26 worker_thread (thd=0xf9c, dummy=0xfbd) at event.c:2137
>>>> #27 0x00007f02728680a4 in start_thread ()
>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>> #28 0x00007f027259d62d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>
>>>> Stefan
>>>>
>>>>> Am 27.02.2017 um 19:44 schrieb Stefan Eissing:
>>>>> Damnit. Can you try the attached patch on top of mod_http2 v1.9.2? This one sets a mutex on the main connection allocator if missing and is pretty close to the version we ran successfully with on your site for days.
>>>>>
>>>>> Thanks again!
>>>>>
>>>>> -Stefan
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> Am 27.02.2017 um 16:22 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>>>>
>>>>>> Hi Stefan,
>>>>>>
>>>>>> two new crashes here.
>>>>>>
>>>>>> Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
>>>>>> /usr/lib/debug//usr/local/apache2/bin/httpd...done.
>>>>>> done.
>>>>>> (gdb) (gdb) quit
>>>>>> Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
>>>>>> /usr/lib/debug//usr/local/apache2/bin/httpd...done.
>>>>>> done.
>>>>>> [Thread debugging using libthread_db enabled]
>>>>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>>>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>>> #0 allocator_free (node=0x0, allocator=0x7f458005a320)
>>>>>> at memory/unix/apr_pools.c:381
>>>>>> #0 allocator_free (node=0x0, allocator=0x7f458005a320)
>>>>>> at memory/unix/apr_pools.c:381
>>>>>> #1 apr_pool_clear (pool=0x7f458004bec8) at memory/unix/apr_pools.c:793
>>>>>> #2 0x0000560ce83e5718 in ap_push_pool (queue_info=0x0,
>>>>>> pool_to_recycle=0x7f458005a328) at fdqueue.c:234
>>>>>> #3 0x0000560ce83e0a5a in process_lingering_close (cs=0x7f458004c158,
>>>>>> pfd=0x560ce8b8dda8) at event.c:1513
>>>>>> #4 0x0000560ce83e4590 in listener_thread (thd=0x0, dummy=0x5497f5242585c)
>>>>>> at event.c:1837
>>>>>> #5 0x00007f45967610a4 in start_thread ()
>>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>> #6 0x00007f459649662d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>> (gdb) (gdb) quit
>>>>>>
>>>>>>
>>>>>> Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
>>>>>> /usr/lib/debug//usr/local/apache2/bin/httpd...done.
>>>>>> done.
>>>>>> [Thread debugging using libthread_db enabled]
>>>>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>>>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>>> #0 allocator_free (node=0x0, allocator=0x7f4580062840)
>>>>>> at memory/unix/apr_pools.c:381
>>>>>> #0 allocator_free (node=0x0, allocator=0x7f4580062840)
>>>>>> at memory/unix/apr_pools.c:381
>>>>>> #1 apr_pool_clear (pool=0x7f4580069188) at memory/unix/apr_pools.c:793
>>>>>> #2 0x0000560ce83e5718 in ap_push_pool (queue_info=0x0,
>>>>>> pool_to_recycle=0x7f4580062848) at fdqueue.c:234
>>>>>> #3 0x0000560ce83e0a5a in process_lingering_close (cs=0x7f4580069418,
>>>>>> pfd=0x560ce8b8dda8) at event.c:1513
>>>>>> #4 0x0000560ce83e4590 in listener_thread (thd=0x0, dummy=0x549845279ce62)
>>>>>> at event.c:1837
>>>>>> #5 0x00007f45967610a4 in start_thread ()
>>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>> #6 0x00007f459649662d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>> (gdb) (gdb) quit
>>>>>>
>>>>>> Stefan
>>>>>>> Am 26.02.2017 um 19:46 schrieb Stefan Priebe - Profihost AG:
>>>>>>> Hi Stefan,
>>>>>>>
>>>>>>> currently everything fine. No segfaults.
>>>>>>>
>>>>>>> Stefan
>>>>>>>
>>>>>>>> Am 25.02.2017 um 20:40 schrieb Stefan Priebe - Profihost AG:
>>>>>>>> Hi Stefan,
>>>>>>>>> Am 25.02.2017 um 13:51 schrieb Stefan Eissing:
>>>>>>>>> Stefan,
>>>>>>>>>
>>>>>>>>> whenever you have time, please deploy https://github.com/icing/mod_h2/releases/tag/v1.9.2
>>>>>>>>>
>>>>>>>>> I added an own allocator with mutex to the http/2 main session. That is something of a middle-ground between placing that on the main conn (as we had in the crash free version) and 1.9.1 behaviour. Thanks!
>>>>>>>>
>>>>>>>> done. But please keep in mind that this crash might be very rare and
>>>>>>>> might even have happened with v1.9.0 if we've waited more time.
>>>>>>>>
>>>>>>>> Greets,
>>>>>>>> Stefan
>>>>>>>>
>>>>>>>>> -Stefan
>>>>>>>>>
>>>>>>>>>> Am 24.02.2017 um 19:31 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>>>>>>>>
>>>>>>>>>> Hi Yann,
>>>>>>>>>>
>>>>>>>>>> here we go:
>>>>>>>>>>
>>>>>>>>>> (gdb) p *ipsub
>>>>>>>>>> $1 = {family = 2, sub = {1367753145, 0, 0, 0}, mask = {4294967295,
>>>>>>>>>> 4294967295, 4294967295, 4294967295}}
>>>>>>>>>>
>>>>>>>>>> (gdb) p *sa
>>>>>>>>>> $2 = {pool = 0x10b500c4ff0b0a, hostname = 0x503040203030102 <error:
>>>>>>>>>> Cannot access memory at address 0x503040203030102>,
>>>>>>>>>> servname = 0x17d010000040405 <error: Cannot access memory at address
>>>>>>>>>> 0x17d010000040405>, port = 770, family = 554829073,
>>>>>>>>>> salen = 319177009, ipaddr_len = 570909009, addr_str_len = -2127424399,
>>>>>>>>>> ipaddr_ptr = 0x24f0d15215c1b142,
>>>>>>>>>> next = 0x17160a0982726233, sa = {sin = {sin_family = 6424, sin_port =
>>>>>>>>>> 9498, sin_addr = {s_addr = 690497318},
>>>>>>>>>> sin_zero = "*456789:"}, sin6 = {sin6_family = 6424, sin6_port =
>>>>>>>>>> 9498, sin6_flowinfo = 690497318, sin6_addr = {__in6_u = {
>>>>>>>>>> __u6_addr8 = "*456789:CDEFGHIJ", __u6_addr16 = {13354, 13877,
>>>>>>>>>> 14391, 14905, 17475, 17989, 18503, 19017}, __u6_addr32 = {
>>>>>>>>>> 909456426, 976828471, 1178944579, 1246316615}}},
>>>>>>>>>> sin6_scope_id = 1448432723}, sas = {ss_family = 6424,
>>>>>>>>>> __ss_align = 4195446337656140842,
>>>>>>>>>> __ss_padding =
>>>>>>>>>> "CDEFGHIJSTUVWXYZcdefghijstuvwxyz\203\204\205\206\207\210\211\212\222\223\224\225\226\227\230\231\232\242\243\244\245\246\247\250\251\252\262\263\264\265\266\267\270\271\272\302\303\304\305\306\307\310\311\312\322\323\324\325\326\327\330\331\332\341\342\343\344\345\346\347\350\351\352\361\362\363\364\365\366\367\370\371\372\377\304\000\037\001\000\003"}}}
>>>>>>>>>>
>>>>>>>>>> (gdb) p *(struct in6_addr *)sa
>>>>>>>>>> $3 = {__in6_u = {__u6_addr8 =
>>>>>>>>>> "\n\v\377\304\000\265\020\000\002\001\003\003\002\004\003\005",
>>>>>>>>>> __u6_addr16 = {2826, 50431, 46336,
>>>>>>>>>> 16, 258, 771, 1026, 1283}, __u6_addr32 = {3305048842, 1094912,
>>>>>>>>>> 50528514, 84083714}}}
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Stefan
>>>>>>>>>>
>>>>>>>>>>> Am 24.02.2017 um 14:18 schrieb Yann Ylavic:
>>>>>>>>>>> Hi Stefan (Priebe),
>>>>>>>>>>>
>>>>>>>>>>> Is IPv6 (really) involved in your network?
>>>>>>>>>>>
>>>>>>>>>>> Could you please show up the gdb output of the below ?
>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Feb 24, 2017 at 2:07 PM, Yann Ylavic <yl...@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> 1078 APR_DECLARE(int) apr_ipsubnet_test(apr_ipsubnet_t *ipsub,
>>>>>>>>>>>> apr_sockaddr_t *sa)
>>>>>>>>>>>> 1079 {
>>>>>>>>>>>> 1080 #if APR_HAVE_IPV6
>>>>>>>>>>>> 1081 /* XXX This line will segv on Win32 build with APR_HAVE_IPV6,
>>>>>>>>>>>> 1082 * but without the IPV6 drivers installed.
>>>>>>>>>>>> 1083 */
>>>>>>>>>>>> 1084 if (sa->family == AF_INET) {
>>>>>>>>>>>> 1085 if (ipsub->family == AF_INET &&
>>>>>>>>>>>> 1086 ((sa->sa.sin.sin_addr.s_addr & ipsub->mask[0]) ==
>>>>>>>>>>>> ipsub->sub[0])) {
>>>>>>>>>>>> 1087 return 1;
>>>>>>>>>>>> 1088 }
>>>>>>>>>>>> 1089 }
>>>>>>>>>>>> 1090 else if (IN6_IS_ADDR_V4MAPPED((struct in6_addr *)sa->ipaddr_ptr)) {
>>>>>>>>>>>> 1091 if (ipsub->family == AF_INET &&
>>>>>>>>>>>> 1092 (((apr_uint32_t *)sa->ipaddr_ptr)[3] &
>>>>>>>>>>>> ipsub->mask[0]) == ipsub->sub[0]) {
>>>>>>>>>>>> 1093 return 1;
>>>>>>>>>>>> 1094 }
>>>>>>>>>>>> 1095 }
>>>>>>>>>>>
>>>>>>>>>>> (gdb) p *ipsub
>>>>>>>>>>> (gdb) p *sa
>>>>>>>>>>> (gdb) p *(struct in6_addr *)sa
>>>>>>>>>>>
>>>>>>>>>>> and possibly more to come...
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Yann.
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Stefan Eissing
>>>>>>>>>
>>>>>>>>> <green/>bytes GmbH
>>>>>>>>> Hafenstrasse 16
>>>>>>>>> 48155 Münster
>>>>>>>>> www.greenbytes.de
>>>>>>>>>
>>>>>
>>>>> Stefan Eissing
>>>>>
>>>>> <green/>bytes GmbH
>>>>> Hafenstrasse 16
>>>>> 48155 Münster
>>>>> www.greenbytes.de
>>>>>
>>>
>>> Stefan Eissing
>>>
>>> <green/>bytes GmbH
>>> Hafenstrasse 16
>>> 48155 Münster
>>> www.greenbytes.de
>>>
Re: release v1.9.2
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Hi Stefan,
current trace - not sure whether this is http2 related at all:
Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 allocator_free (node=0x0, allocator=0x7fbf4c05a410)
at memory/unix/apr_pools.c:381
#0 allocator_free (node=0x0, allocator=0x7fbf4c05a410)
at memory/unix/apr_pools.c:381
#1 apr_pool_clear (pool=0x7fbf4c04f888) at memory/unix/apr_pools.c:793
#2 0x00005642878c2818 in ap_push_pool (queue_info=0x0,
pool_to_recycle=0x7fbf4c05a418) at fdqueue.c:234
#3 0x00005642878bdb5a in process_lingering_close (cs=0x7fbf4c04fb18,
pfd=0x5642886c7078) at event.c:1513
#4 0x00005642878c1690 in listener_thread (thd=0x0, dummy=0x549eb52311a6f)
at event.c:1837
#5 0x00007fbf5f2940a4 in start_thread ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007fbf5efc962d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Stefan
Am 03.03.2017 um 21:02 schrieb Stefan Priebe - Profihost AG:
> Hi Stefan,
>
> live. Sorry for the late reply.
>
> Stefan
>
> Am 28.02.2017 um 11:49 schrieb Stefan Eissing:
>> Meh, my mistake. Sorry about that. Did not cleanup properly. Dare you with an improved version:
>>
>>
>>
>>
>>
>>> Am 28.02.2017 um 08:41 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>
>>> That one breaks everyting. Multiple crashes per second.
>>>
>>> [Thread debugging using libthread_db enabled]
>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>>> Program terminated with signal SIGABRT, Aborted.
>>> #0 0x00007f02724ea067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>>> #0 0x00007f02724ea067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>>> #1 0x00007f02724eb448 in abort () from /lib/x86_64-linux-gnu/libc.so.6
>>> #2 0x00007f02724e3266 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>>> #3 0x00007f02724e3312 in __assert_fail () from
>>> /lib/x86_64-linux-gnu/libc.so.6
>>> #4 0x00007f027287062f in __pthread_tpp_change_priority ()
>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>> #5 0x00007f0272865e9f in __pthread_mutex_lock_full ()
>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>> #6 0x00007f0272a98db8 in allocator_alloc (in_size=in_size@entry=8032,
>>> allocator=0x7f025c03f3c0) at memory/unix/apr_pools.c:244
>>> #7 apr_allocator_alloc (allocator=0x7f025c03f3c0, size=size@entry=8032)
>>> at memory/unix/apr_pools.c:438
>>> #8 0x00007f0272cbcac4 in apr_bucket_alloc (size=8032, size@entry=8000,
>>> list=0x7f022c0008e8) at buckets/apr_buckets_alloc.c:157
>>> #9 0x00007f0272cbdca2 in socket_bucket_read (a=0x7f022c000b08,
>>> str=0x7f02627a87b8, len=0x7f02627a87c0, block=<optimized out>)
>>> at buckets/apr_buckets_socket.c:34
>>> #10 0x0000565098cf1fb1 in ap_core_input_filter (f=0x7f022c0056b0,
>>> b=0x7f022c005630, mode=<optimized out>, block=APR_NONBLOCK_READ,
>>> readbytes=5) at core_filters.c:235
>>> #11 0x0000565098d3aaca in logio_in_filter (f=<optimized out>,
>>> bb=0x7f022c005630, mode=<optimized out>, block=<optimized out>,
>>> readbytes=<optimized out>) at mod_logio.c:165
>>> #12 0x0000565098d45b9a in bio_filter_in_read (bio=0x7f024800b460,
>>> in=0x7f02480492a3 "", inlen=5) at ssl_engine_io.c:483
>>> #13 0x00007f02736b014c in BIO_read ()
>>> from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
>>> #14 0x00007f0273a13c92 in ?? () from
>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>>> #15 0x00007f0273a1548d in ?? () from
>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>>> #16 0x00007f0273a12024 in ?? () from
>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>>> #17 0x0000565098d469f3 in ssl_io_input_read (inctx=0x7f022c0035b8,
>>> buf=0x7f022c003600 "GET
>>> /media/image/07/3c/c2/4612_13721_160192280322.jpeg HTTP/1.1\r\nHost:
>>> www.onlinestore.it\r\nUser-Agent: curl/7.15+ (x64-criteo) libcurl/7.15+
>>> OpenSSL zlib libidn\r\nAccept: */*\r\n\r\n", len=0x7f02627a8b38)
>>> at ssl_engine_io.c:614
>>> #18 0x0000565098d493ec in ssl_io_filter_input (f=0x7f022c005608,
>>> bb=0x7f022c006dd0, mode=<optimized out>, block=<optimized out>,
>>> readbytes=8000) at ssl_engine_io.c:1474
>>> #19 0x0000565098d5cd85 in h2_filter_core_input (f=0x7f022c005980,
>>> brigade=0x7f022c006dd0, mode=738225624, block=APR_NONBLOCK_READ,
>>> readbytes=8000) at h2_filter.c:149
>>> #20 0x0000565098d6809b in h2_session_read (session=0x7f022c006920,
>>> block=<optimized out>) at h2_session.c:1440
>>> #21 0x0000565098d6b97f in h2_session_process (session=0x7f022c006920,
>>> async=3964) at h2_session.c:2074
>>> #22 0x0000565098d5b84e in h2_conn_run (ctx=<optimized out>,
>>> c=0x7f025c03f7d8)
>>> at h2_conn.c:218
>>> #23 0x0000565098d5e551 in h2_h2_process_conn (c=0x7f025c03f7d8) at
>>> h2_h2.c:658
>>> #24 0x0000565098d00fd0 in ap_run_process_connection (c=0x7f025c03f7d8)
>>> at connection.c:42
>>> #25 0x0000565098d9b6d0 in process_socket (my_thread_num=<optimized out>,
>>> my_child_num=<optimized out>, cs=0x7f025c03f748, sock=<optimized out>,
>>> p=<optimized out>, thd=<optimized out>) at event.c:1134
>>> #26 worker_thread (thd=0xf5e, dummy=0xf7c) at event.c:2137
>>> #27 0x00007f02728680a4 in start_thread ()
>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>> #28 0x00007f027259d62d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>> (gdb) (gdb) quit
>>> Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
>>> /usr/lib/debug//usr/local/apache2/bin/httpd...done.
>>> done.
>>> [Thread debugging using libthread_db enabled]
>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>>> Program terminated with signal SIGABRT, Aborted.
>>> #0 0x00007f02724ea067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>>> #0 0x00007f02724ea067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>>> #1 0x00007f02724eb448 in abort () from /lib/x86_64-linux-gnu/libc.so.6
>>> #2 0x00007f02724e3266 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>>> #3 0x00007f02724e3312 in __assert_fail () from
>>> /lib/x86_64-linux-gnu/libc.so.6
>>> #4 0x00007f027287062f in __pthread_tpp_change_priority ()
>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>> #5 0x00007f0272865e9f in __pthread_mutex_lock_full ()
>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>> #6 0x00007f0272a98db8 in allocator_alloc (in_size=in_size@entry=8032,
>>> allocator=0x7f025c03d320) at memory/unix/apr_pools.c:244
>>> #7 apr_allocator_alloc (allocator=0x7f025c03d320, size=size@entry=8032)
>>> at memory/unix/apr_pools.c:438
>>> #8 0x00007f0272cbcac4 in apr_bucket_alloc (size=8032, size@entry=8000,
>>> list=0x7f02340008e8) at buckets/apr_buckets_alloc.c:157
>>> #9 0x00007f0272cbdca2 in socket_bucket_read (a=0x7f0234000b08,
>>> str=0x7f0260fa57b8, len=0x7f0260fa57c0, block=<optimized out>)
>>> at buckets/apr_buckets_socket.c:34
>>> #10 0x0000565098cf1fb1 in ap_core_input_filter (f=0x7f02340056b0,
>>> b=0x7f0234005630, mode=<optimized out>, block=APR_NONBLOCK_READ,
>>> readbytes=5) at core_filters.c:235
>>> #11 0x0000565098d3aaca in logio_in_filter (f=<optimized out>,
>>> bb=0x7f0234005630, mode=<optimized out>, block=<optimized out>,
>>> readbytes=<optimized out>) at mod_logio.c:165
>>> #12 0x0000565098d45b9a in bio_filter_in_read (bio=0x7f022801ff50,
>>> in=0x7f02280345d3 "(\002\177", inlen=5) at ssl_engine_io.c:483
>>> #13 0x00007f02736b014c in BIO_read ()
>>> from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
>>> #14 0x00007f0273a13c92 in ?? () from
>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>>> #15 0x00007f0273a1548d in ?? () from
>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>>> #16 0x00007f0273a12024 in ?? () from
>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
>>> #17 0x0000565098d469f3 in ssl_io_input_read (inctx=0x7f02340035b8,
>>> buf=0x7f0234003600 "", len=0x7f0260fa5b38) at ssl_engine_io.c:614
>>> #18 0x0000565098d493ec in ssl_io_filter_input (f=0x7f0234005608,
>>> bb=0x7f023400cd00, mode=<optimized out>, block=<optimized out>,
>>> readbytes=8000) at ssl_engine_io.c:1474
>>> #19 0x0000565098d5cd85 in h2_filter_core_input (f=0x7f0234005980,
>>> brigade=0x7f023400cd00, mode=872467720, block=APR_NONBLOCK_READ,
>>> readbytes=8000) at h2_filter.c:149
>>> #20 0x0000565098d6809b in h2_session_read (session=0x7f023400c850,
>>> block=<optimized out>) at h2_session.c:1440
>>> #21 0x0000565098d6b97f in h2_session_process (session=0x7f023400c850,
>>> async=4029) at h2_session.c:2074
>>> #22 0x0000565098d5b84e in h2_conn_run (ctx=<optimized out>,
>>> c=0x7f025c03d738)
>>> at h2_conn.c:218
>>> #23 0x0000565098d5e551 in h2_h2_process_conn (c=0x7f025c03d738) at
>>> h2_h2.c:658
>>> #24 0x0000565098d00fd0 in ap_run_process_connection (c=0x7f025c03d738)
>>> at connection.c:42
>>> #25 0x0000565098d9b6d0 in process_socket (my_thread_num=<optimized out>,
>>> my_child_num=<optimized out>, cs=0x7f025c03d6a8, sock=<optimized out>,
>>> p=<optimized out>, thd=<optimized out>) at event.c:1134
>>> #26 worker_thread (thd=0xf9c, dummy=0xfbd) at event.c:2137
>>> #27 0x00007f02728680a4 in start_thread ()
>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>> #28 0x00007f027259d62d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>
>>> Stefan
>>>
>>> Am 27.02.2017 um 19:44 schrieb Stefan Eissing:
>>>> Damnit. Can you try the attached patch on top of mod_http2 v1.9.2? This one sets a mutex on the main connection allocator if missing and is pretty close to the version we ran successfully with on your site for days.
>>>>
>>>> Thanks again!
>>>>
>>>> -Stefan
>>>>
>>>>
>>>>
>>>>
>>>>> Am 27.02.2017 um 16:22 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>>>
>>>>> Hi Stefan,
>>>>>
>>>>> two new crashes here.
>>>>>
>>>>> Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
>>>>> /usr/lib/debug//usr/local/apache2/bin/httpd...done.
>>>>> done.
>>>>> (gdb) (gdb) quit
>>>>> Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
>>>>> /usr/lib/debug//usr/local/apache2/bin/httpd...done.
>>>>> done.
>>>>> [Thread debugging using libthread_db enabled]
>>>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>> #0 allocator_free (node=0x0, allocator=0x7f458005a320)
>>>>> at memory/unix/apr_pools.c:381
>>>>> #0 allocator_free (node=0x0, allocator=0x7f458005a320)
>>>>> at memory/unix/apr_pools.c:381
>>>>> #1 apr_pool_clear (pool=0x7f458004bec8) at memory/unix/apr_pools.c:793
>>>>> #2 0x0000560ce83e5718 in ap_push_pool (queue_info=0x0,
>>>>> pool_to_recycle=0x7f458005a328) at fdqueue.c:234
>>>>> #3 0x0000560ce83e0a5a in process_lingering_close (cs=0x7f458004c158,
>>>>> pfd=0x560ce8b8dda8) at event.c:1513
>>>>> #4 0x0000560ce83e4590 in listener_thread (thd=0x0, dummy=0x5497f5242585c)
>>>>> at event.c:1837
>>>>> #5 0x00007f45967610a4 in start_thread ()
>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>> #6 0x00007f459649662d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>> (gdb) (gdb) quit
>>>>>
>>>>>
>>>>> Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
>>>>> /usr/lib/debug//usr/local/apache2/bin/httpd...done.
>>>>> done.
>>>>> [Thread debugging using libthread_db enabled]
>>>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>> #0 allocator_free (node=0x0, allocator=0x7f4580062840)
>>>>> at memory/unix/apr_pools.c:381
>>>>> #0 allocator_free (node=0x0, allocator=0x7f4580062840)
>>>>> at memory/unix/apr_pools.c:381
>>>>> #1 apr_pool_clear (pool=0x7f4580069188) at memory/unix/apr_pools.c:793
>>>>> #2 0x0000560ce83e5718 in ap_push_pool (queue_info=0x0,
>>>>> pool_to_recycle=0x7f4580062848) at fdqueue.c:234
>>>>> #3 0x0000560ce83e0a5a in process_lingering_close (cs=0x7f4580069418,
>>>>> pfd=0x560ce8b8dda8) at event.c:1513
>>>>> #4 0x0000560ce83e4590 in listener_thread (thd=0x0, dummy=0x549845279ce62)
>>>>> at event.c:1837
>>>>> #5 0x00007f45967610a4 in start_thread ()
>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>> #6 0x00007f459649662d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>> (gdb) (gdb) quit
>>>>>
>>>>> Stefan
>>>>> Am 26.02.2017 um 19:46 schrieb Stefan Priebe - Profihost AG:
>>>>>> Hi Stefan,
>>>>>>
>>>>>> currently everything fine. No segfaults.
>>>>>>
>>>>>> Stefan
>>>>>>
>>>>>> Am 25.02.2017 um 20:40 schrieb Stefan Priebe - Profihost AG:
>>>>>>> Hi Stefan,
>>>>>>> Am 25.02.2017 um 13:51 schrieb Stefan Eissing:
>>>>>>>> Stefan,
>>>>>>>>
>>>>>>>> whenever you have time, please deploy https://github.com/icing/mod_h2/releases/tag/v1.9.2
>>>>>>>>
>>>>>>>> I added an own allocator with mutex to the http/2 main session. That is something of a middle-ground between placing that on the main conn (as we had in the crash free version) and 1.9.1 behaviour. Thanks!
>>>>>>>
>>>>>>> done. But please keep in mind that this crash might be very rare and
>>>>>>> might even have happened with v1.9.0 if we've waited more time.
>>>>>>>
>>>>>>> Greets,
>>>>>>> Stefan
>>>>>>>
>>>>>>>> -Stefan
>>>>>>>>
>>>>>>>>> Am 24.02.2017 um 19:31 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>>>>>>>
>>>>>>>>> Hi Yann,
>>>>>>>>>
>>>>>>>>> here we go:
>>>>>>>>>
>>>>>>>>> (gdb) p *ipsub
>>>>>>>>> $1 = {family = 2, sub = {1367753145, 0, 0, 0}, mask = {4294967295,
>>>>>>>>> 4294967295, 4294967295, 4294967295}}
>>>>>>>>>
>>>>>>>>> (gdb) p *sa
>>>>>>>>> $2 = {pool = 0x10b500c4ff0b0a, hostname = 0x503040203030102 <error:
>>>>>>>>> Cannot access memory at address 0x503040203030102>,
>>>>>>>>> servname = 0x17d010000040405 <error: Cannot access memory at address
>>>>>>>>> 0x17d010000040405>, port = 770, family = 554829073,
>>>>>>>>> salen = 319177009, ipaddr_len = 570909009, addr_str_len = -2127424399,
>>>>>>>>> ipaddr_ptr = 0x24f0d15215c1b142,
>>>>>>>>> next = 0x17160a0982726233, sa = {sin = {sin_family = 6424, sin_port =
>>>>>>>>> 9498, sin_addr = {s_addr = 690497318},
>>>>>>>>> sin_zero = "*456789:"}, sin6 = {sin6_family = 6424, sin6_port =
>>>>>>>>> 9498, sin6_flowinfo = 690497318, sin6_addr = {__in6_u = {
>>>>>>>>> __u6_addr8 = "*456789:CDEFGHIJ", __u6_addr16 = {13354, 13877,
>>>>>>>>> 14391, 14905, 17475, 17989, 18503, 19017}, __u6_addr32 = {
>>>>>>>>> 909456426, 976828471, 1178944579, 1246316615}}},
>>>>>>>>> sin6_scope_id = 1448432723}, sas = {ss_family = 6424,
>>>>>>>>> __ss_align = 4195446337656140842,
>>>>>>>>> __ss_padding =
>>>>>>>>> "CDEFGHIJSTUVWXYZcdefghijstuvwxyz\203\204\205\206\207\210\211\212\222\223\224\225\226\227\230\231\232\242\243\244\245\246\247\250\251\252\262\263\264\265\266\267\270\271\272\302\303\304\305\306\307\310\311\312\322\323\324\325\326\327\330\331\332\341\342\343\344\345\346\347\350\351\352\361\362\363\364\365\366\367\370\371\372\377\304\000\037\001\000\003"}}}
>>>>>>>>>
>>>>>>>>> (gdb) p *(struct in6_addr *)sa
>>>>>>>>> $3 = {__in6_u = {__u6_addr8 =
>>>>>>>>> "\n\v\377\304\000\265\020\000\002\001\003\003\002\004\003\005",
>>>>>>>>> __u6_addr16 = {2826, 50431, 46336,
>>>>>>>>> 16, 258, 771, 1026, 1283}, __u6_addr32 = {3305048842, 1094912,
>>>>>>>>> 50528514, 84083714}}}
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Stefan
>>>>>>>>>
>>>>>>>>> Am 24.02.2017 um 14:18 schrieb Yann Ylavic:
>>>>>>>>>> Hi Stefan (Priebe),
>>>>>>>>>>
>>>>>>>>>> Is IPv6 (really) involved in your network?
>>>>>>>>>>
>>>>>>>>>> Could you please show up the gdb output of the below ?
>>>>>>>>>>
>>>>>>>>>> On Fri, Feb 24, 2017 at 2:07 PM, Yann Ylavic <yl...@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> 1078 APR_DECLARE(int) apr_ipsubnet_test(apr_ipsubnet_t *ipsub,
>>>>>>>>>>> apr_sockaddr_t *sa)
>>>>>>>>>>> 1079 {
>>>>>>>>>>> 1080 #if APR_HAVE_IPV6
>>>>>>>>>>> 1081 /* XXX This line will segv on Win32 build with APR_HAVE_IPV6,
>>>>>>>>>>> 1082 * but without the IPV6 drivers installed.
>>>>>>>>>>> 1083 */
>>>>>>>>>>> 1084 if (sa->family == AF_INET) {
>>>>>>>>>>> 1085 if (ipsub->family == AF_INET &&
>>>>>>>>>>> 1086 ((sa->sa.sin.sin_addr.s_addr & ipsub->mask[0]) ==
>>>>>>>>>>> ipsub->sub[0])) {
>>>>>>>>>>> 1087 return 1;
>>>>>>>>>>> 1088 }
>>>>>>>>>>> 1089 }
>>>>>>>>>>> 1090 else if (IN6_IS_ADDR_V4MAPPED((struct in6_addr *)sa->ipaddr_ptr)) {
>>>>>>>>>>> 1091 if (ipsub->family == AF_INET &&
>>>>>>>>>>> 1092 (((apr_uint32_t *)sa->ipaddr_ptr)[3] &
>>>>>>>>>>> ipsub->mask[0]) == ipsub->sub[0]) {
>>>>>>>>>>> 1093 return 1;
>>>>>>>>>>> 1094 }
>>>>>>>>>>> 1095 }
>>>>>>>>>>
>>>>>>>>>> (gdb) p *ipsub
>>>>>>>>>> (gdb) p *sa
>>>>>>>>>> (gdb) p *(struct in6_addr *)sa
>>>>>>>>>>
>>>>>>>>>> and possibly more to come...
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Yann.
>>>>>>>>>>
>>>>>>>>
>>>>>>>> Stefan Eissing
>>>>>>>>
>>>>>>>> <green/>bytes GmbH
>>>>>>>> Hafenstrasse 16
>>>>>>>> 48155 M�nster
>>>>>>>> www.greenbytes.de
>>>>>>>>
>>>>
>>>> Stefan Eissing
>>>>
>>>> <green/>bytes GmbH
>>>> Hafenstrasse 16
>>>> 48155 M�nster
>>>> www.greenbytes.de
>>>>
>>
>> Stefan Eissing
>>
>> <green/>bytes GmbH
>> Hafenstrasse 16
>> 48155 M�nster
>> www.greenbytes.de
>>