You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Luca Toscano <to...@gmail.com> on 2016/09/14 16:47:52 UTC
Re: Frequent wake-ups for mpm_event
2016-08-25 16:45 GMT+02:00 Yann Ylavic <yl...@gmail.com>:
>
> Possibly others will test it too, and report...
>
>
Ping to everybody interested in the list to check Yann's patch in
https://bz.apache.org/bugzilla/show_bug.cgi?id=57399 and provide feedback :)
The end goal would be to reduce as much as possible the amount of wake-ups
done by the mpm-event's listener thread.
Thanks!
Regards,
Luca
Re: Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Hi Yann,
Am 18.09.2016 um 14:16 schrieb Yann Ylavic:
> On Sun, Sep 18, 2016 at 12:16 PM, Luca Toscano <to...@gmail.com> wrote:
>>
>> 2016-09-15 11:41 GMT+02:00 Stefan Priebe - Profihost AG
>> <s....@profihost.ag>:
>>>
>>> that sounds great. Is there a version available that applies to 2.4 ?
>>
>> I tried to study/port Yann's patch to 2.4.x. but it is not that
>> straightforward since there are a lot of differences between the two
>> branches, most of them afaics are related to the skiplist's handling and
>> usage.
>
> Patch against 2.4.x attached.
thanks a lot.
Stefan
Re: Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Thanks Yann. I'll wait a few days to check first if i'll see this on
other hosts too. i'll also try to grab the server-status but i'm not
sure if the server will answer to this request if the scoreboard is full.
Greets,
Stefan
Am 23.09.2016 um 16:53 schrieb Yann Ylavic:
> With the patch this time...
>
> On Fri, Sep 23, 2016 at 4:53 PM, Yann Ylavic <yl...@gmail.com> wrote:
>> On Fri, Sep 23, 2016 at 3:13 PM, Yann Ylavic <yl...@gmail.com> wrote:
>>>
>>> If you can try another (additive) patch on the faulty server, it would
>>> be awesome to test the one from [1], not sure it applies correctly
>>> with my 2.4.x patch though, possibly better with the trunk version
>>> over [1] (let me know otherwise).
>>
>> Not a big deal actually, the attached patch includes both [1] and [2] combined.
>>
>> Regards,
>> Yann.
>>
>> [1] https://bz.apache.org/bugzilla/show_bug.cgi?id=53555#c55
>> [2] https://bz.apache.org/bugzilla/show_bug.cgi?id=57399#c9
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Am 18.01.2017 um 22:53 schrieb Yann Ylavic:
> On Wed, Jan 18, 2017 at 10:50 PM, Yann Ylavic <yl...@gmail.com> wrote:
>> On Wed, Jan 18, 2017 at 10:44 PM, Yann Ylavic <yl...@gmail.com> wrote:
>>> On Wed, Jan 18, 2017 at 10:23 PM, Stefan Priebe - Profihost AG
>>> <s....@profihost.ag> wrote:
>>>>
>>>
>>> Also, do you confirm that:
>>> 1. it didn't crash with v5 + original (2.4.25's) mod_http2?
>>
>> Hmm, v5 was against 2.4.23, but still no crash happened with it right?
>>
>>> 2. it started crashing with v6 + original mod_http2 (before update to v1.8.8)?
>
> And of course, no crash in 2.4.25 with no patch at all?
Yes as well.
Stefan
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
On Thu, Jan 19, 2017 at 4:28 PM, Stefan Priebe - Profihost AG
<s....@profihost.ag> wrote:
>
> May this configure option have an influence?
> --enable-nonportable-atomics=yes
Sure, for speed, but not correctness ("yes" never caused an issue for
me/linux, AFAICT).
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
On Thu, Jan 19, 2017 at 4:45 PM, Stefan Priebe - Profihost AG
<s....@profihost.ag> wrote:
>
> Am 19.01.2017 um 16:34 schrieb Stefan Eissing:
>> Yann might already have asked this: any chance to compile with symbols and get a more readable stacktrace?
>
> Yes just tell me how ;-) i'm using dpkg-buildpackage and dh_strip. I've
> no idea why not all symbols are available.
dh_strip will strip the symbols, so you probably should not use it.
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
On Fri, Jan 20, 2017 at 12:49 PM, Stefan Priebe - Profihost AG
<s....@profihost.ag> wrote:
> might this be a debian bug? i can't reproduce with apr-included.
Did you build the included-apr (and httpd) with -fno-strict-aliasing ?
Not sure Debian sets it for its builds either, just a speculation (I
used to have issues in APR_RING without it).
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
On Wed, Jan 25, 2017 at 1:43 AM, Yann Ylavic <yl...@gmail.com> wrote:
> Hi,
>
> On Tue, Jan 24, 2017 at 4:09 PM, Stefan Priebe - Profihost AG
> <s....@profihost.ag> wrote:
>>
>> Thanks. Would currently wait if you had a look at stefans patch. Or
>> isn't that the same but "better"?
>
> Could you please give "h2_beams_cleanup_v3.diff" (from previous message) a try?
(if that wasn't clear enough, I meant on top of 1.8.9 of course)
>
> Thanks Stefan.
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
Hi,
On Tue, Jan 24, 2017 at 4:09 PM, Stefan Priebe - Profihost AG
<s....@profihost.ag> wrote:
>
> Thanks. Would currently wait if you had a look at stefans patch. Or
> isn't that the same but "better"?
Could you please give "h2_beams_cleanup_v3.diff" (from previous message) a try?
Thanks Stefan.
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Am 24.01.2017 um 14:22 schrieb Yann Ylavic:
> On Tue, Jan 24, 2017 at 10:02 AM, Stefan Priebe - Profihost AG
> <s....@profihost.ag> wrote:
>>
>> Am 23.01.2017 um 23:42 schrieb Yann Ylavic:
>>> On Mon, Jan 23, 2017 at 11:37 PM, Yann Ylavic <yl...@gmail.com> wrote:
>>>> Hi Stefan,
>>>>
>>>> On Mon, Jan 23, 2017 at 9:54 PM, Stefan Eissing
>>>> <st...@greenbytes.de> wrote:
>>>>>
>>>>> I am not aware of any special expectations now. Almost all is triggered by (parent) pool cleanups and is therefore more deterministic than before. The only explicit destroy is done on finished streams and slave connections no longer used. When the master conn disappears, all is deallocated as the force wills it.
>>>>
>>>> I wonder if the attached patch makes sense.
>>>> If beam_{recv,send}_cleanup() are to be executed on (parent) pool
>>>> destroy, there will be before beam_cleanup() itelf (which also calls
>>>> beam_send_cleanup() explicitly), so it should avoid potential a double
>>>> cleanup in this case.
>>>>
>>>> WDYT?
>>>
>>> Please ignore the last (garbage) hunk.
>>
>> last garbage hunk?
>>
>> This one?
>
> Yes, this change is not necessary/suitable.
Thanks. Would currently wait if you had a look at stefans patch. Or
isn't that the same but "better"?
Stefan
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Eissing <st...@greenbytes.de>.
> Am 20.02.2017 um 10:36 schrieb Stefan Priebe <s....@profihost.ag>:
>
> Hi,
>
> still not a single segfault with mod_http2 1.9.0 - good work!
Thanks! Our pleasure. Enjoy it until I break something in the next releases.
> @Yann
> Could you please tell me whether i should drop of your additional patches?
>
> Greets,
> Stefan
>
> Am 09.02.2017 um 11:55 schrieb Stefan Priebe - Profihost AG:
>> Hi Stefan,
>>
>> Am 09.02.2017 um 11:47 schrieb Stefan Eissing:
>>> Stefan,
>>>
>>> at this point and after several efforts to write the right patch, I reached the conclusion that I need to rethink the pool hierarchy and connection shutdown strategy in mod_http2. The current, organically grown implementation needs a refactor with the knowledge we have made so far.
>>
>> OK - thanks for your help and hard work.
>>
>>> So, a fix will not come today or tomorrow. But not too far away either. From your comments I assume that these crashed happen not too frequently. Hope this allows you to live with the current state for a while.
>>
>> Sure and yes it does not happen very often.
>>
>>> Btw. have the status 500 lines disappeared with the latest release? That was one point still open on my list...
>>
>> Yes sorry i missed to report back. It's working fine now.
>>
>>>
>>> Hope to give you something better to verify in your environment soon.
>>
>> Just tell me i'm ready to test.
>>
>> Greets,
>> Stefan
>>
>>> Cheers,
>>>
>>> Stefan
>>>
>>>> Am 07.02.2017 um 20:56 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>>
>>>> Hi,
>>>>
>>>> got this one today with both patches applied:
>>>>
>>>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>> #0 allocator_free (node=0x0, allocator=0x7f350405e030)
>>>> at memory/unix/apr_pools.c:381
>>>> #0 allocator_free (node=0x0, allocator=0x7f350405e030)
>>>> at memory/unix/apr_pools.c:381
>>>> #1 apr_pool_clear (pool=0x7f3504060468) at memory/unix/apr_pools.c:793
>>>> #2 0x000055d11256d688 in ap_push_pool (queue_info=0x7f35040604f8,
>>>> pool_to_recycle=0xffffffff) at fdqueue.c:234
>>>> #3 0x000055d11256899a in process_lingering_close (cs=0x7f3504060748,
>>>> pfd=0x55d1143fd7f8) at event.c:1513
>>>> #4 0x000055d11256c4d0 in listener_thread (thd=0x7f35040604f8,
>>>> dummy=0x547f569acd4a6) at event.c:1837
>>>> #5 0x00007f351757c0a4 in start_thread ()
>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>> #6 0x00007f35172b162d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>
>>>> Am 06.02.2017 um 22:24 schrieb Stefan Priebe - Profihost AG:
>>>>> Am 06.02.2017 um 22:22 schrieb Stefan Priebe - Profihost AG:
>>>>>> Hi Stefan,
>>>>>>
>>>>>> this one does not apply on top of yann's
>>>>>> mpm_event_listener_wakeup_bug57399_V7.patch patch.
>>>>>
>>>>> i've now used his patch to mpm and yours for mod_http2.
>>>>>
>>>>> Stefan
>>>>>
>>>>>>
>>>>>> Stefan
>>>>>> Am 06.02.2017 um 15:34 schrieb Stefan Eissing:
>>>>>>> Yes, that was mixing the order. The attached v4 compiles and runs the h2 tests for me without errors.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Am 06.02.2017 um 14:43 schrieb Yann Ylavic <yl...@gmail.com>:
>>>>>>>>
>>>>>>>> On Mon, Feb 6, 2017 at 2:31 PM, Stefan Eissing
>>>>>>>> <st...@greenbytes.de> wrote:
>>>>>>>>> Currently running some tests. Have crashes on the original patch in my test suite. Fixed one, hunting for the next...
>>>>>>>>
>>>>>>>> I think it comes from my change that creates slave connections from
>>>>>>>> master->pool (instead of mplx's), because now slave's pool is already
>>>>>>>> destroyed when h2_mplx_release_and_join()->task_destroy()->h2_slave_destroy()
>>>>>>>> is called (hence the crash).
>>>>>>>>
>>>>>>>> I restored your original code in this new (attached) patch.
>>>>>>>>
>>>>>>>> @s.priebe, would you test this one please?
>>>>>>>> <ptrans_and_slaves_allocator-v3.patch>
>>>>>>>
>>>>>>> Stefan Eissing
>>>>>>>
>>>>>>> <green/>bytes GmbH
>>>>>>> Hafenstrasse 16
>>>>>>> 48155 Münster
>>>>>>> www.greenbytes.de
>>>>>>>
>>>
>>> Stefan Eissing
>>>
>>> <green/>bytes GmbH
>>> Hafenstrasse 16
>>> 48155 Münster
>>> www.greenbytes.de
>>>
Stefan Eissing
<green/>bytes GmbH
Hafenstrasse 16
48155 Münster
www.greenbytes.de
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
On Mon, Feb 20, 2017 at 10:36 AM, Stefan Priebe <s....@profihost.ag> wrote:
>
> @Yann
> Could you please tell me whether i should drop of your additional patches?
Sorry Stefan, somehow I missed you message.
I'll reply (inline) on the other thread.
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe <s....@profihost.ag>.
Hi,
still not a single segfault with mod_http2 1.9.0 - good work!
@Yann
Could you please tell me whether i should drop of your additional patches?
Greets,
Stefan
Am 09.02.2017 um 11:55 schrieb Stefan Priebe - Profihost AG:
> Hi Stefan,
>
> Am 09.02.2017 um 11:47 schrieb Stefan Eissing:
>> Stefan,
>>
>> at this point and after several efforts to write the right patch, I reached the conclusion that I need to rethink the pool hierarchy and connection shutdown strategy in mod_http2. The current, organically grown implementation needs a refactor with the knowledge we have made so far.
>
> OK - thanks for your help and hard work.
>
>> So, a fix will not come today or tomorrow. But not too far away either. From your comments I assume that these crashed happen not too frequently. Hope this allows you to live with the current state for a while.
>
> Sure and yes it does not happen very often.
>
>> Btw. have the status 500 lines disappeared with the latest release? That was one point still open on my list...
>
> Yes sorry i missed to report back. It's working fine now.
>
>>
>> Hope to give you something better to verify in your environment soon.
>
> Just tell me i'm ready to test.
>
> Greets,
> Stefan
>
>> Cheers,
>>
>> Stefan
>>
>>> Am 07.02.2017 um 20:56 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>
>>> Hi,
>>>
>>> got this one today with both patches applied:
>>>
>>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>> #0 allocator_free (node=0x0, allocator=0x7f350405e030)
>>> at memory/unix/apr_pools.c:381
>>> #0 allocator_free (node=0x0, allocator=0x7f350405e030)
>>> at memory/unix/apr_pools.c:381
>>> #1 apr_pool_clear (pool=0x7f3504060468) at memory/unix/apr_pools.c:793
>>> #2 0x000055d11256d688 in ap_push_pool (queue_info=0x7f35040604f8,
>>> pool_to_recycle=0xffffffff) at fdqueue.c:234
>>> #3 0x000055d11256899a in process_lingering_close (cs=0x7f3504060748,
>>> pfd=0x55d1143fd7f8) at event.c:1513
>>> #4 0x000055d11256c4d0 in listener_thread (thd=0x7f35040604f8,
>>> dummy=0x547f569acd4a6) at event.c:1837
>>> #5 0x00007f351757c0a4 in start_thread ()
>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>> #6 0x00007f35172b162d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>
>>> Am 06.02.2017 um 22:24 schrieb Stefan Priebe - Profihost AG:
>>>> Am 06.02.2017 um 22:22 schrieb Stefan Priebe - Profihost AG:
>>>>> Hi Stefan,
>>>>>
>>>>> this one does not apply on top of yann's
>>>>> mpm_event_listener_wakeup_bug57399_V7.patch patch.
>>>>
>>>> i've now used his patch to mpm and yours for mod_http2.
>>>>
>>>> Stefan
>>>>
>>>>>
>>>>> Stefan
>>>>> Am 06.02.2017 um 15:34 schrieb Stefan Eissing:
>>>>>> Yes, that was mixing the order. The attached v4 compiles and runs the h2 tests for me without errors.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Am 06.02.2017 um 14:43 schrieb Yann Ylavic <yl...@gmail.com>:
>>>>>>>
>>>>>>> On Mon, Feb 6, 2017 at 2:31 PM, Stefan Eissing
>>>>>>> <st...@greenbytes.de> wrote:
>>>>>>>> Currently running some tests. Have crashes on the original patch in my test suite. Fixed one, hunting for the next...
>>>>>>>
>>>>>>> I think it comes from my change that creates slave connections from
>>>>>>> master->pool (instead of mplx's), because now slave's pool is already
>>>>>>> destroyed when h2_mplx_release_and_join()->task_destroy()->h2_slave_destroy()
>>>>>>> is called (hence the crash).
>>>>>>>
>>>>>>> I restored your original code in this new (attached) patch.
>>>>>>>
>>>>>>> @s.priebe, would you test this one please?
>>>>>>> <ptrans_and_slaves_allocator-v3.patch>
>>>>>>
>>>>>> Stefan Eissing
>>>>>>
>>>>>> <green/>bytes GmbH
>>>>>> Hafenstrasse 16
>>>>>> 48155 M�nster
>>>>>> www.greenbytes.de
>>>>>>
>>
>> Stefan Eissing
>>
>> <green/>bytes GmbH
>> Hafenstrasse 16
>> 48155 M�nster
>> www.greenbytes.de
>>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Hi Stefan,
Am 09.02.2017 um 11:47 schrieb Stefan Eissing:
> Stefan,
>
> at this point and after several efforts to write the right patch, I reached the conclusion that I need to rethink the pool hierarchy and connection shutdown strategy in mod_http2. The current, organically grown implementation needs a refactor with the knowledge we have made so far.
OK - thanks for your help and hard work.
> So, a fix will not come today or tomorrow. But not too far away either. From your comments I assume that these crashed happen not too frequently. Hope this allows you to live with the current state for a while.
Sure and yes it does not happen very often.
> Btw. have the status 500 lines disappeared with the latest release? That was one point still open on my list...
Yes sorry i missed to report back. It's working fine now.
>
> Hope to give you something better to verify in your environment soon.
Just tell me i'm ready to test.
Greets,
Stefan
> Cheers,
>
> Stefan
>
>> Am 07.02.2017 um 20:56 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>
>> Hi,
>>
>> got this one today with both patches applied:
>>
>> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
>> Program terminated with signal SIGSEGV, Segmentation fault.
>> #0 allocator_free (node=0x0, allocator=0x7f350405e030)
>> at memory/unix/apr_pools.c:381
>> #0 allocator_free (node=0x0, allocator=0x7f350405e030)
>> at memory/unix/apr_pools.c:381
>> #1 apr_pool_clear (pool=0x7f3504060468) at memory/unix/apr_pools.c:793
>> #2 0x000055d11256d688 in ap_push_pool (queue_info=0x7f35040604f8,
>> pool_to_recycle=0xffffffff) at fdqueue.c:234
>> #3 0x000055d11256899a in process_lingering_close (cs=0x7f3504060748,
>> pfd=0x55d1143fd7f8) at event.c:1513
>> #4 0x000055d11256c4d0 in listener_thread (thd=0x7f35040604f8,
>> dummy=0x547f569acd4a6) at event.c:1837
>> #5 0x00007f351757c0a4 in start_thread ()
>> from /lib/x86_64-linux-gnu/libpthread.so.0
>> #6 0x00007f35172b162d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>
>> Am 06.02.2017 um 22:24 schrieb Stefan Priebe - Profihost AG:
>>> Am 06.02.2017 um 22:22 schrieb Stefan Priebe - Profihost AG:
>>>> Hi Stefan,
>>>>
>>>> this one does not apply on top of yann's
>>>> mpm_event_listener_wakeup_bug57399_V7.patch patch.
>>>
>>> i've now used his patch to mpm and yours for mod_http2.
>>>
>>> Stefan
>>>
>>>>
>>>> Stefan
>>>> Am 06.02.2017 um 15:34 schrieb Stefan Eissing:
>>>>> Yes, that was mixing the order. The attached v4 compiles and runs the h2 tests for me without errors.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> Am 06.02.2017 um 14:43 schrieb Yann Ylavic <yl...@gmail.com>:
>>>>>>
>>>>>> On Mon, Feb 6, 2017 at 2:31 PM, Stefan Eissing
>>>>>> <st...@greenbytes.de> wrote:
>>>>>>> Currently running some tests. Have crashes on the original patch in my test suite. Fixed one, hunting for the next...
>>>>>>
>>>>>> I think it comes from my change that creates slave connections from
>>>>>> master->pool (instead of mplx's), because now slave's pool is already
>>>>>> destroyed when h2_mplx_release_and_join()->task_destroy()->h2_slave_destroy()
>>>>>> is called (hence the crash).
>>>>>>
>>>>>> I restored your original code in this new (attached) patch.
>>>>>>
>>>>>> @s.priebe, would you test this one please?
>>>>>> <ptrans_and_slaves_allocator-v3.patch>
>>>>>
>>>>> Stefan Eissing
>>>>>
>>>>> <green/>bytes GmbH
>>>>> Hafenstrasse 16
>>>>> 48155 M�nster
>>>>> www.greenbytes.de
>>>>>
>
> Stefan Eissing
>
> <green/>bytes GmbH
> Hafenstrasse 16
> 48155 M�nster
> www.greenbytes.de
>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Eissing <st...@greenbytes.de>.
Stefan,
at this point and after several efforts to write the right patch, I reached the conclusion that I need to rethink the pool hierarchy and connection shutdown strategy in mod_http2. The current, organically grown implementation needs a refactor with the knowledge we have made so far.
So, a fix will not come today or tomorrow. But not too far away either. From your comments I assume that these crashed happen not too frequently. Hope this allows you to live with the current state for a while.
Btw. have the status 500 lines disappeared with the latest release? That was one point still open on my list...
Hope to give you something better to verify in your environment soon.
Cheers,
Stefan
> Am 07.02.2017 um 20:56 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>
> Hi,
>
> got this one today with both patches applied:
>
> Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0 allocator_free (node=0x0, allocator=0x7f350405e030)
> at memory/unix/apr_pools.c:381
> #0 allocator_free (node=0x0, allocator=0x7f350405e030)
> at memory/unix/apr_pools.c:381
> #1 apr_pool_clear (pool=0x7f3504060468) at memory/unix/apr_pools.c:793
> #2 0x000055d11256d688 in ap_push_pool (queue_info=0x7f35040604f8,
> pool_to_recycle=0xffffffff) at fdqueue.c:234
> #3 0x000055d11256899a in process_lingering_close (cs=0x7f3504060748,
> pfd=0x55d1143fd7f8) at event.c:1513
> #4 0x000055d11256c4d0 in listener_thread (thd=0x7f35040604f8,
> dummy=0x547f569acd4a6) at event.c:1837
> #5 0x00007f351757c0a4 in start_thread ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #6 0x00007f35172b162d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Am 06.02.2017 um 22:24 schrieb Stefan Priebe - Profihost AG:
>> Am 06.02.2017 um 22:22 schrieb Stefan Priebe - Profihost AG:
>>> Hi Stefan,
>>>
>>> this one does not apply on top of yann's
>>> mpm_event_listener_wakeup_bug57399_V7.patch patch.
>>
>> i've now used his patch to mpm and yours for mod_http2.
>>
>> Stefan
>>
>>>
>>> Stefan
>>> Am 06.02.2017 um 15:34 schrieb Stefan Eissing:
>>>> Yes, that was mixing the order. The attached v4 compiles and runs the h2 tests for me without errors.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> Am 06.02.2017 um 14:43 schrieb Yann Ylavic <yl...@gmail.com>:
>>>>>
>>>>> On Mon, Feb 6, 2017 at 2:31 PM, Stefan Eissing
>>>>> <st...@greenbytes.de> wrote:
>>>>>> Currently running some tests. Have crashes on the original patch in my test suite. Fixed one, hunting for the next...
>>>>>
>>>>> I think it comes from my change that creates slave connections from
>>>>> master->pool (instead of mplx's), because now slave's pool is already
>>>>> destroyed when h2_mplx_release_and_join()->task_destroy()->h2_slave_destroy()
>>>>> is called (hence the crash).
>>>>>
>>>>> I restored your original code in this new (attached) patch.
>>>>>
>>>>> @s.priebe, would you test this one please?
>>>>> <ptrans_and_slaves_allocator-v3.patch>
>>>>
>>>> Stefan Eissing
>>>>
>>>> <green/>bytes GmbH
>>>> Hafenstrasse 16
>>>> 48155 Münster
>>>> www.greenbytes.de
>>>>
Stefan Eissing
<green/>bytes GmbH
Hafenstrasse 16
48155 Münster
www.greenbytes.de
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Hi,
got this one today with both patches applied:
Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 allocator_free (node=0x0, allocator=0x7f350405e030)
at memory/unix/apr_pools.c:381
#0 allocator_free (node=0x0, allocator=0x7f350405e030)
at memory/unix/apr_pools.c:381
#1 apr_pool_clear (pool=0x7f3504060468) at memory/unix/apr_pools.c:793
#2 0x000055d11256d688 in ap_push_pool (queue_info=0x7f35040604f8,
pool_to_recycle=0xffffffff) at fdqueue.c:234
#3 0x000055d11256899a in process_lingering_close (cs=0x7f3504060748,
pfd=0x55d1143fd7f8) at event.c:1513
#4 0x000055d11256c4d0 in listener_thread (thd=0x7f35040604f8,
dummy=0x547f569acd4a6) at event.c:1837
#5 0x00007f351757c0a4 in start_thread ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007f35172b162d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Am 06.02.2017 um 22:24 schrieb Stefan Priebe - Profihost AG:
> Am 06.02.2017 um 22:22 schrieb Stefan Priebe - Profihost AG:
>> Hi Stefan,
>>
>> this one does not apply on top of yann's
>> mpm_event_listener_wakeup_bug57399_V7.patch patch.
>
> i've now used his patch to mpm and yours for mod_http2.
>
> Stefan
>
>>
>> Stefan
>> Am 06.02.2017 um 15:34 schrieb Stefan Eissing:
>>> Yes, that was mixing the order. The attached v4 compiles and runs the h2 tests for me without errors.
>>>
>>>
>>>
>>>
>>>
>>>> Am 06.02.2017 um 14:43 schrieb Yann Ylavic <yl...@gmail.com>:
>>>>
>>>> On Mon, Feb 6, 2017 at 2:31 PM, Stefan Eissing
>>>> <st...@greenbytes.de> wrote:
>>>>> Currently running some tests. Have crashes on the original patch in my test suite. Fixed one, hunting for the next...
>>>>
>>>> I think it comes from my change that creates slave connections from
>>>> master->pool (instead of mplx's), because now slave's pool is already
>>>> destroyed when h2_mplx_release_and_join()->task_destroy()->h2_slave_destroy()
>>>> is called (hence the crash).
>>>>
>>>> I restored your original code in this new (attached) patch.
>>>>
>>>> @s.priebe, would you test this one please?
>>>> <ptrans_and_slaves_allocator-v3.patch>
>>>
>>> Stefan Eissing
>>>
>>> <green/>bytes GmbH
>>> Hafenstrasse 16
>>> 48155 M�nster
>>> www.greenbytes.de
>>>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Eissing <st...@greenbytes.de>.
Ok, thanks. I did not have the other one in place.
> Am 06.02.2017 um 22:24 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>
>> Am 06.02.2017 um 22:22 schrieb Stefan Priebe - Profihost AG:
>> Hi Stefan,
>>
>> this one does not apply on top of yann's
>> mpm_event_listener_wakeup_bug57399_V7.patch patch.
>
> i've now used his patch to mpm and yours for mod_http2.
>
> Stefan
>
>>
>> Stefan
>>> Am 06.02.2017 um 15:34 schrieb Stefan Eissing:
>>> Yes, that was mixing the order. The attached v4 compiles and runs the h2 tests for me without errors.
>>>
>>>
>>>
>>>
>>>
>>>> Am 06.02.2017 um 14:43 schrieb Yann Ylavic <yl...@gmail.com>:
>>>>
>>>> On Mon, Feb 6, 2017 at 2:31 PM, Stefan Eissing
>>>> <st...@greenbytes.de> wrote:
>>>>> Currently running some tests. Have crashes on the original patch in my test suite. Fixed one, hunting for the next...
>>>>
>>>> I think it comes from my change that creates slave connections from
>>>> master->pool (instead of mplx's), because now slave's pool is already
>>>> destroyed when h2_mplx_release_and_join()->task_destroy()->h2_slave_destroy()
>>>> is called (hence the crash).
>>>>
>>>> I restored your original code in this new (attached) patch.
>>>>
>>>> @s.priebe, would you test this one please?
>>>> <ptrans_and_slaves_allocator-v3.patch>
>>>
>>> Stefan Eissing
>>>
>>> <green/>bytes GmbH
>>> Hafenstrasse 16
>>> 48155 Münster
>>> www.greenbytes.de
>>>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Am 06.02.2017 um 22:22 schrieb Stefan Priebe - Profihost AG:
> Hi Stefan,
>
> this one does not apply on top of yann's
> mpm_event_listener_wakeup_bug57399_V7.patch patch.
i've now used his patch to mpm and yours for mod_http2.
Stefan
>
> Stefan
> Am 06.02.2017 um 15:34 schrieb Stefan Eissing:
>> Yes, that was mixing the order. The attached v4 compiles and runs the h2 tests for me without errors.
>>
>>
>>
>>
>>
>>> Am 06.02.2017 um 14:43 schrieb Yann Ylavic <yl...@gmail.com>:
>>>
>>> On Mon, Feb 6, 2017 at 2:31 PM, Stefan Eissing
>>> <st...@greenbytes.de> wrote:
>>>> Currently running some tests. Have crashes on the original patch in my test suite. Fixed one, hunting for the next...
>>>
>>> I think it comes from my change that creates slave connections from
>>> master->pool (instead of mplx's), because now slave's pool is already
>>> destroyed when h2_mplx_release_and_join()->task_destroy()->h2_slave_destroy()
>>> is called (hence the crash).
>>>
>>> I restored your original code in this new (attached) patch.
>>>
>>> @s.priebe, would you test this one please?
>>> <ptrans_and_slaves_allocator-v3.patch>
>>
>> Stefan Eissing
>>
>> <green/>bytes GmbH
>> Hafenstrasse 16
>> 48155 M�nster
>> www.greenbytes.de
>>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Hi Stefan,
this one does not apply on top of yann's
mpm_event_listener_wakeup_bug57399_V7.patch patch.
Stefan
Am 06.02.2017 um 15:34 schrieb Stefan Eissing:
> Yes, that was mixing the order. The attached v4 compiles and runs the h2 tests for me without errors.
>
>
>
>
>
>> Am 06.02.2017 um 14:43 schrieb Yann Ylavic <yl...@gmail.com>:
>>
>> On Mon, Feb 6, 2017 at 2:31 PM, Stefan Eissing
>> <st...@greenbytes.de> wrote:
>>> Currently running some tests. Have crashes on the original patch in my test suite. Fixed one, hunting for the next...
>>
>> I think it comes from my change that creates slave connections from
>> master->pool (instead of mplx's), because now slave's pool is already
>> destroyed when h2_mplx_release_and_join()->task_destroy()->h2_slave_destroy()
>> is called (hence the crash).
>>
>> I restored your original code in this new (attached) patch.
>>
>> @s.priebe, would you test this one please?
>> <ptrans_and_slaves_allocator-v3.patch>
>
> Stefan Eissing
>
> <green/>bytes GmbH
> Hafenstrasse 16
> 48155 M�nster
> www.greenbytes.de
>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Am 06.02.2017 um 11:56 schrieb Yann Ylavic:
> Hi Stefan,
>
> On Mon, Feb 6, 2017 at 9:57 AM, Stefan Priebe - Profihost AG
> <s....@profihost.ag> wrote:
>>
>> your last patch results in multiple crashes every second:
>
> Sorry about that, the changes in mpm_event were incorrect (the mutex
> was cleared with the pool when recycled, hence its pointer was
> dangling).
>
> New patch attached, this time tested with the httpd framework (where
> the previous patch segfaulted too).
Thanks but that one had crashed already once again.
This time:
error.log:
*** Error in `/usr/local/apache/bin/httpd': free(): invalid pointer:
0x00007f4bdc023dc0 ***
Core was generated by `/usr/local/apache/bin/httpd -DFOREGROUND'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 allocator_free (node=0x4480447f447e4466, allocator=0x7f4bc40008c0)
at memory/unix/apr_pools.c:381
#0 allocator_free (node=0x4480447f447e4466, allocator=0x7f4bc40008c0)
at memory/unix/apr_pools.c:381
#1 apr_pool_destroy (pool=0x7f4bc404a178) at memory/unix/apr_pools.c:856
#2 0x000055dab5531bff in task_destroy (m=0x7f4c4c00e8f8,
task=0x7f4bc404e210,
called_from_master=0) at h2_mplx.c:396
#3 0x000055dab5532e6b in task_done_iter (ctx=<optimized out>,
val=<optimized out>) at h2_mplx.c:1060
#4 0x00007f4c8b6965e6 in apr_hash_do (
comp=comp@entry=0x55dab5545140 <ihash_iter>,
rec=rec@entry=0x7f4c6bfe6480,
ht=<optimized out>) at tables/apr_hash.c:542
#5 0x000055dab5545b1f in h2_ihash_iter (ih=<optimized out>,
fn=fn@entry=0x55dab5532e60 <task_done_iter>,
ctx=ctx@entry=0x7f4c4c00e8f8)
at h2_util.c:315
#6 0x000055dab5533433 in h2_mplx_release_and_join (m=0x7f4c4c00e8f8,
wait=0x7f4c4c00e8a0) at h2_mplx.c:615
#7 0x000055dab5538ab4 in session_pool_cleanup (data=0x7f4c04005c78)
at h2_session.c:827
#8 0x00007f4c8b69f48e in run_cleanups (cref=0x7f4c4c00e878)
at memory/unix/apr_pools.c:2352
#9 apr_pool_destroy (pool=0x7f4c4c00e808) at memory/unix/apr_pools.c:804
#10 0x00007f4c8b69f745 in apr_pool_clear (pool=0x7f4c78058198)
at memory/unix/apr_pools.c:769
#11 0x000055dab5570668 in ap_push_pool (queue_info=0x7f4bc40008c0,
pool_to_recycle=0xc4041f91) at fdqueue.c:234
#12 0x000055dab556b99a in process_lingering_close (cs=0x7f4c78058478,
pfd=0x55dab6cf3fa8) at event.c:1513
#13 0x000055dab556f4d0 in listener_thread (thd=0x7f4bc40008c0,
dummy=0x547dbb6874f83) at event.c:1837
#14 0x00007f4c8b46e0a4 in start_thread ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#15 0x00007f4c8b1a362d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Greets,
Stefan
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Am 06.02.2017 um 16:06 schrieb Stefan Eissing:
> It does, at the end. Traversed the directories in different order it seems.
*arg* sorry
>
>> Am 06.02.2017 um 16:05 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>
>> Am 06.02.2017 um 16:03 schrieb Stefan Eissing:
>>> I took Yann's v3 patch and changed a tiny bit in the h2. I just added a var declaration in event.c without which it did not compile for me.
>>
>> But your patch does not contain the changes to the event mpm. That's why
>> i ask.
>>
>> Stefan
>>
>>> -Stefan
>>>
>>>> Am 06.02.2017 um 15:41 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>>
>>>> Hi,
>>>>
>>>> so i should test the mpm event part from Yann + your http2 part?
>>>>
>>>> Stefan
>>>>
>>>> Am 06.02.2017 um 15:34 schrieb Stefan Eissing:
>>>>> Yes, that was mixing the order. The attached v4 compiles and runs the h2 tests for me without errors.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> Am 06.02.2017 um 14:43 schrieb Yann Ylavic <yl...@gmail.com>:
>>>>>>
>>>>>> On Mon, Feb 6, 2017 at 2:31 PM, Stefan Eissing
>>>>>> <st...@greenbytes.de> wrote:
>>>>>>> Currently running some tests. Have crashes on the original patch in my test suite. Fixed one, hunting for the next...
>>>>>>
>>>>>> I think it comes from my change that creates slave connections from
>>>>>> master->pool (instead of mplx's), because now slave's pool is already
>>>>>> destroyed when h2_mplx_release_and_join()->task_destroy()->h2_slave_destroy()
>>>>>> is called (hence the crash).
>>>>>>
>>>>>> I restored your original code in this new (attached) patch.
>>>>>>
>>>>>> @s.priebe, would you test this one please?
>>>>>> <ptrans_and_slaves_allocator-v3.patch>
>>>>>
>>>>> Stefan Eissing
>>>>>
>>>>> <green/>bytes GmbH
>>>>> Hafenstrasse 16
>>>>> 48155 M�nster
>>>>> www.greenbytes.de
>>>>>
>>>
>>> Stefan Eissing
>>>
>>> <green/>bytes GmbH
>>> Hafenstrasse 16
>>> 48155 M�nster
>>> www.greenbytes.de
>>>
>
> Stefan Eissing
>
> <green/>bytes GmbH
> Hafenstrasse 16
> 48155 M�nster
> www.greenbytes.de
>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Eissing <st...@greenbytes.de>.
It does, at the end. Traversed the directories in different order it seems.
> Am 06.02.2017 um 16:05 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>
> Am 06.02.2017 um 16:03 schrieb Stefan Eissing:
>> I took Yann's v3 patch and changed a tiny bit in the h2. I just added a var declaration in event.c without which it did not compile for me.
>
> But your patch does not contain the changes to the event mpm. That's why
> i ask.
>
> Stefan
>
>> -Stefan
>>
>>> Am 06.02.2017 um 15:41 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>
>>> Hi,
>>>
>>> so i should test the mpm event part from Yann + your http2 part?
>>>
>>> Stefan
>>>
>>> Am 06.02.2017 um 15:34 schrieb Stefan Eissing:
>>>> Yes, that was mixing the order. The attached v4 compiles and runs the h2 tests for me without errors.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> Am 06.02.2017 um 14:43 schrieb Yann Ylavic <yl...@gmail.com>:
>>>>>
>>>>> On Mon, Feb 6, 2017 at 2:31 PM, Stefan Eissing
>>>>> <st...@greenbytes.de> wrote:
>>>>>> Currently running some tests. Have crashes on the original patch in my test suite. Fixed one, hunting for the next...
>>>>>
>>>>> I think it comes from my change that creates slave connections from
>>>>> master->pool (instead of mplx's), because now slave's pool is already
>>>>> destroyed when h2_mplx_release_and_join()->task_destroy()->h2_slave_destroy()
>>>>> is called (hence the crash).
>>>>>
>>>>> I restored your original code in this new (attached) patch.
>>>>>
>>>>> @s.priebe, would you test this one please?
>>>>> <ptrans_and_slaves_allocator-v3.patch>
>>>>
>>>> Stefan Eissing
>>>>
>>>> <green/>bytes GmbH
>>>> Hafenstrasse 16
>>>> 48155 Münster
>>>> www.greenbytes.de
>>>>
>>
>> Stefan Eissing
>>
>> <green/>bytes GmbH
>> Hafenstrasse 16
>> 48155 Münster
>> www.greenbytes.de
>>
Stefan Eissing
<green/>bytes GmbH
Hafenstrasse 16
48155 Münster
www.greenbytes.de
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Am 06.02.2017 um 16:03 schrieb Stefan Eissing:
> I took Yann's v3 patch and changed a tiny bit in the h2. I just added a var declaration in event.c without which it did not compile for me.
But your patch does not contain the changes to the event mpm. That's why
i ask.
Stefan
> -Stefan
>
>> Am 06.02.2017 um 15:41 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>
>> Hi,
>>
>> so i should test the mpm event part from Yann + your http2 part?
>>
>> Stefan
>>
>> Am 06.02.2017 um 15:34 schrieb Stefan Eissing:
>>> Yes, that was mixing the order. The attached v4 compiles and runs the h2 tests for me without errors.
>>>
>>>
>>>
>>>
>>>
>>>> Am 06.02.2017 um 14:43 schrieb Yann Ylavic <yl...@gmail.com>:
>>>>
>>>> On Mon, Feb 6, 2017 at 2:31 PM, Stefan Eissing
>>>> <st...@greenbytes.de> wrote:
>>>>> Currently running some tests. Have crashes on the original patch in my test suite. Fixed one, hunting for the next...
>>>>
>>>> I think it comes from my change that creates slave connections from
>>>> master->pool (instead of mplx's), because now slave's pool is already
>>>> destroyed when h2_mplx_release_and_join()->task_destroy()->h2_slave_destroy()
>>>> is called (hence the crash).
>>>>
>>>> I restored your original code in this new (attached) patch.
>>>>
>>>> @s.priebe, would you test this one please?
>>>> <ptrans_and_slaves_allocator-v3.patch>
>>>
>>> Stefan Eissing
>>>
>>> <green/>bytes GmbH
>>> Hafenstrasse 16
>>> 48155 M�nster
>>> www.greenbytes.de
>>>
>
> Stefan Eissing
>
> <green/>bytes GmbH
> Hafenstrasse 16
> 48155 M�nster
> www.greenbytes.de
>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Eissing <st...@greenbytes.de>.
I took Yann's v3 patch and changed a tiny bit in the h2. I just added a var declaration in event.c without which it did not compile for me.
-Stefan
> Am 06.02.2017 um 15:41 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>
> Hi,
>
> so i should test the mpm event part from Yann + your http2 part?
>
> Stefan
>
> Am 06.02.2017 um 15:34 schrieb Stefan Eissing:
>> Yes, that was mixing the order. The attached v4 compiles and runs the h2 tests for me without errors.
>>
>>
>>
>>
>>
>>> Am 06.02.2017 um 14:43 schrieb Yann Ylavic <yl...@gmail.com>:
>>>
>>> On Mon, Feb 6, 2017 at 2:31 PM, Stefan Eissing
>>> <st...@greenbytes.de> wrote:
>>>> Currently running some tests. Have crashes on the original patch in my test suite. Fixed one, hunting for the next...
>>>
>>> I think it comes from my change that creates slave connections from
>>> master->pool (instead of mplx's), because now slave's pool is already
>>> destroyed when h2_mplx_release_and_join()->task_destroy()->h2_slave_destroy()
>>> is called (hence the crash).
>>>
>>> I restored your original code in this new (attached) patch.
>>>
>>> @s.priebe, would you test this one please?
>>> <ptrans_and_slaves_allocator-v3.patch>
>>
>> Stefan Eissing
>>
>> <green/>bytes GmbH
>> Hafenstrasse 16
>> 48155 Münster
>> www.greenbytes.de
>>
Stefan Eissing
<green/>bytes GmbH
Hafenstrasse 16
48155 Münster
www.greenbytes.de
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Hi,
so i should test the mpm event part from Yann + your http2 part?
Stefan
Am 06.02.2017 um 15:34 schrieb Stefan Eissing:
> Yes, that was mixing the order. The attached v4 compiles and runs the h2 tests for me without errors.
>
>
>
>
>
>> Am 06.02.2017 um 14:43 schrieb Yann Ylavic <yl...@gmail.com>:
>>
>> On Mon, Feb 6, 2017 at 2:31 PM, Stefan Eissing
>> <st...@greenbytes.de> wrote:
>>> Currently running some tests. Have crashes on the original patch in my test suite. Fixed one, hunting for the next...
>>
>> I think it comes from my change that creates slave connections from
>> master->pool (instead of mplx's), because now slave's pool is already
>> destroyed when h2_mplx_release_and_join()->task_destroy()->h2_slave_destroy()
>> is called (hence the crash).
>>
>> I restored your original code in this new (attached) patch.
>>
>> @s.priebe, would you test this one please?
>> <ptrans_and_slaves_allocator-v3.patch>
>
> Stefan Eissing
>
> <green/>bytes GmbH
> Hafenstrasse 16
> 48155 M�nster
> www.greenbytes.de
>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Eissing <st...@greenbytes.de>.
Yes, that was mixing the order. The attached v4 compiles and runs the h2 tests for me without errors.
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
On Mon, Feb 6, 2017 at 2:31 PM, Stefan Eissing
<st...@greenbytes.de> wrote:
> Currently running some tests. Have crashes on the original patch in my test suite. Fixed one, hunting for the next...
I think it comes from my change that creates slave connections from
master->pool (instead of mplx's), because now slave's pool is already
destroyed when h2_mplx_release_and_join()->task_destroy()->h2_slave_destroy()
is called (hence the crash).
I restored your original code in this new (attached) patch.
@s.priebe, would you test this one please?
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Eissing <st...@greenbytes.de>.
Currently running some tests. Have crashes on the original patch in my test suite. Fixed one, hunting for the next...
> Am 06.02.2017 um 14:23 schrieb Ruediger Pluem <rp...@apache.org>:
>
>
>
> On 02/06/2017 01:51 PM, Yann Ylavic wrote:
>> On Mon, Feb 6, 2017 at 1:42 PM, Yann Ylavic <yl...@gmail.com> wrote:
>>> On Mon, Feb 6, 2017 at 1:29 PM, Ruediger Pluem <rp...@apache.org> wrote:
>>>>
>>>>
>>>> On 02/06/2017 11:56 AM, Yann Ylavic wrote:
>>>>> Hi Stefan,
>>>>>
>>>>> On Mon, Feb 6, 2017 at 9:57 AM, Stefan Priebe - Profihost AG
>>>>> <s....@profihost.ag> wrote:
>>>>>>
>>>>>> your last patch results in multiple crashes every second:
>>>>>
>>>>> Sorry about that, the changes in mpm_event were incorrect (the mutex
>>>>> was cleared with the pool when recycled, hence its pointer was
>>>>> dangling).
>>>>>
>>>>> New patch attached, this time tested with the httpd framework (where
>>>>> the previous patch segfaulted too).
>>>>>
>>>>> Thanks,
>>>>> Yann.
>>>>>
>>>>
>>>> Hmm, does it make sense performance wise to create the mutex over and over again?
>>>> Or is this planned to be improved once it is known to fix the crash issue?
>>>
>>> Yes, I'm thinking of it, but it's not easy because we need a pool to
>>> create the mutex.
>>> Using ptrans makes it cleared on recycle (hence re-created), and using
>>> the parent pool (pconf) requires synchronization.
>>>
>>> Possibly we could recycle both the pool (or the allocator) and its
>>> mutex, but ap_push/pop_pool() wouldn't be lockless anymore...
>>
>> Not sure it's really worth it either because apr_thread_mutex_create()
>> should boil down to "*mutex = PTHREAD_MUTEXT_INIT" on *nix, and
>> probably something equivalent for InitializeCriticalSection() on
>> windows...
>> We probably not spend many cycles here (compared to more synchronization).
>
> The question how much cycles this spends in GLIBC / kernel. I don't know. So maybe its not worth
> the effort. But if its not worth the effort it is worth documenting why :-)
>
>
> Regards
>
> Rüdiger
Stefan Eissing
<green/>bytes GmbH
Hafenstrasse 16
48155 Münster
www.greenbytes.de
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
On Mon, Feb 6, 2017 at 2:23 PM, Ruediger Pluem <rp...@apache.org> wrote:
>
> The question how much cycles this spends in GLIBC / kernel. I don't
> know. So maybe its not worth the effort. But if its not worth the
> effort it is worth documenting why :-)
Sure ;)
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Ruediger Pluem <rp...@apache.org>.
On 02/06/2017 01:51 PM, Yann Ylavic wrote:
> On Mon, Feb 6, 2017 at 1:42 PM, Yann Ylavic <yl...@gmail.com> wrote:
>> On Mon, Feb 6, 2017 at 1:29 PM, Ruediger Pluem <rp...@apache.org> wrote:
>>>
>>>
>>> On 02/06/2017 11:56 AM, Yann Ylavic wrote:
>>>> Hi Stefan,
>>>>
>>>> On Mon, Feb 6, 2017 at 9:57 AM, Stefan Priebe - Profihost AG
>>>> <s....@profihost.ag> wrote:
>>>>>
>>>>> your last patch results in multiple crashes every second:
>>>>
>>>> Sorry about that, the changes in mpm_event were incorrect (the mutex
>>>> was cleared with the pool when recycled, hence its pointer was
>>>> dangling).
>>>>
>>>> New patch attached, this time tested with the httpd framework (where
>>>> the previous patch segfaulted too).
>>>>
>>>> Thanks,
>>>> Yann.
>>>>
>>>
>>> Hmm, does it make sense performance wise to create the mutex over and over again?
>>> Or is this planned to be improved once it is known to fix the crash issue?
>>
>> Yes, I'm thinking of it, but it's not easy because we need a pool to
>> create the mutex.
>> Using ptrans makes it cleared on recycle (hence re-created), and using
>> the parent pool (pconf) requires synchronization.
>>
>> Possibly we could recycle both the pool (or the allocator) and its
>> mutex, but ap_push/pop_pool() wouldn't be lockless anymore...
>
> Not sure it's really worth it either because apr_thread_mutex_create()
> should boil down to "*mutex = PTHREAD_MUTEXT_INIT" on *nix, and
> probably something equivalent for InitializeCriticalSection() on
> windows...
> We probably not spend many cycles here (compared to more synchronization).
The question how much cycles this spends in GLIBC / kernel. I don't know. So maybe its not worth
the effort. But if its not worth the effort it is worth documenting why :-)
Regards
R�diger
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
On Mon, Feb 6, 2017 at 1:42 PM, Yann Ylavic <yl...@gmail.com> wrote:
> On Mon, Feb 6, 2017 at 1:29 PM, Ruediger Pluem <rp...@apache.org> wrote:
>>
>>
>> On 02/06/2017 11:56 AM, Yann Ylavic wrote:
>>> Hi Stefan,
>>>
>>> On Mon, Feb 6, 2017 at 9:57 AM, Stefan Priebe - Profihost AG
>>> <s....@profihost.ag> wrote:
>>>>
>>>> your last patch results in multiple crashes every second:
>>>
>>> Sorry about that, the changes in mpm_event were incorrect (the mutex
>>> was cleared with the pool when recycled, hence its pointer was
>>> dangling).
>>>
>>> New patch attached, this time tested with the httpd framework (where
>>> the previous patch segfaulted too).
>>>
>>> Thanks,
>>> Yann.
>>>
>>
>> Hmm, does it make sense performance wise to create the mutex over and over again?
>> Or is this planned to be improved once it is known to fix the crash issue?
>
> Yes, I'm thinking of it, but it's not easy because we need a pool to
> create the mutex.
> Using ptrans makes it cleared on recycle (hence re-created), and using
> the parent pool (pconf) requires synchronization.
>
> Possibly we could recycle both the pool (or the allocator) and its
> mutex, but ap_push/pop_pool() wouldn't be lockless anymore...
Not sure it's really worth it either because apr_thread_mutex_create()
should boil down to "*mutex = PTHREAD_MUTEXT_INIT" on *nix, and
probably something equivalent for InitializeCriticalSection() on
windows...
We probably not spend many cycles here (compared to more synchronization).
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
On Mon, Feb 6, 2017 at 1:29 PM, Ruediger Pluem <rp...@apache.org> wrote:
>
>
> On 02/06/2017 11:56 AM, Yann Ylavic wrote:
>> Hi Stefan,
>>
>> On Mon, Feb 6, 2017 at 9:57 AM, Stefan Priebe - Profihost AG
>> <s....@profihost.ag> wrote:
>>>
>>> your last patch results in multiple crashes every second:
>>
>> Sorry about that, the changes in mpm_event were incorrect (the mutex
>> was cleared with the pool when recycled, hence its pointer was
>> dangling).
>>
>> New patch attached, this time tested with the httpd framework (where
>> the previous patch segfaulted too).
>>
>> Thanks,
>> Yann.
>>
>
> Hmm, does it make sense performance wise to create the mutex over and over again?
> Or is this planned to be improved once it is known to fix the crash issue?
Yes, I'm thinking of it, but it's not easy because we need a pool to
create the mutex.
Using ptrans makes it cleared on recycle (hence re-created), and using
the parent pool (pconf) requires synchronization.
Possibly we could recycle both the pool (or the allocator) and its
mutex, but ap_push/pop_pool() wouldn't be lockless anymore...
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Ruediger Pluem <rp...@apache.org>.
On 02/06/2017 11:56 AM, Yann Ylavic wrote:
> Hi Stefan,
>
> On Mon, Feb 6, 2017 at 9:57 AM, Stefan Priebe - Profihost AG
> <s....@profihost.ag> wrote:
>>
>> your last patch results in multiple crashes every second:
>
> Sorry about that, the changes in mpm_event were incorrect (the mutex
> was cleared with the pool when recycled, hence its pointer was
> dangling).
>
> New patch attached, this time tested with the httpd framework (where
> the previous patch segfaulted too).
>
> Thanks,
> Yann.
>
Hmm, does it make sense performance wise to create the mutex over and over again?
Or is this planned to be improved once it is known to fix the crash issue?
Regards
Rdiger
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
Hi Stefan,
On Mon, Feb 6, 2017 at 9:57 AM, Stefan Priebe - Profihost AG
<s....@profihost.ag> wrote:
>
> your last patch results in multiple crashes every second:
Sorry about that, the changes in mpm_event were incorrect (the mutex
was cleared with the pool when recycled, hence its pointer was
dangling).
New patch attached, this time tested with the httpd framework (where
the previous patch segfaulted too).
Thanks,
Yann.
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Hi Yann,
your last patch results in multiple crashes every second:
Program terminated with signal SIGSEGV, Segmentation fault.
#0 apr_pool_cleanup_kill (p=0x7f2b00000000,
data=data@entry=0x7f2bb40478c0,
cleanup_fn=cleanup_fn@entry=0x7f2bc92285b0 <socket_cleanup>) at
memory/unix/apr_pools.c:2264
2264 memory/unix/apr_pools.c: No such file or directory.
#0 apr_pool_cleanup_kill (p=0x7f2b00000000,
data=data@entry=0x7f2bb40478c0,
cleanup_fn=cleanup_fn@entry=0x7f2bc92285b0 <socket_cleanup>) at
memory/unix/apr_pools.c:2264
#1 0x00007f2bc9224871 in apr_pool_cleanup_run (p=<optimized out>,
data=0x7f2bb40478c0, cleanup_fn=0x7f2bc92285b0 <socket_cleanup>)
at memory/unix/apr_pools.c:2342
#2 0x00007f2bc9228892 in apr_socket_close (thesocket=<optimized out>)
at network_io/unix/sockets.c:183
#3 0x0000555d4f07b99a in process_lingering_close (cs=0x7f2bb4047ac8,
pfd=0x555d4f75dfa8) at event.c:1510
#4 0x0000555d4f07f4e0 in listener_thread (thd=0x7f2b00000000,
dummy=0x547d8caf3534b) at event.c:1837
#5 0x00007f2bc8ff20a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007f2bc8d2762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Stefan
Am 06.02.2017 um 09:33 schrieb Stefan Priebe - Profihost AG:
> Hallo,
>
> up and running (apache 2.4.25 + mpm_event V7 + mpd_http2 v1.8.11 +
> mod_ssl patch + your new allocator patch). Will report back.
>
> Greets,
> Stefan
>
> Am 06.02.2017 um 01:19 schrieb Yann Ylavic:
>> Hi Stefan,
>>
>> On Sun, Feb 5, 2017 at 7:51 PM, Stefan Priebe - Profihost AG
>> <s....@profihost.ag> wrote:
>>>
>>> tested your patch against mod_ssl. I haven't seen any pool crashes again
>>> so it seems to fix this issue.
>>>
>>> But two new ones:
>>
>> Possibly a promising fix suggested (in another thread) by yet another
>> talented Stefan :)
>> He has spotted a possible issue in mpm_event which could affect mainly
>> mod_http2.
>> I applied his suggestion in the attached patch, and did some
>> extrapolation for similar code in mod_h2 (so all possible errors are
>> mine...).
>> Would you test this one please?
>>
>> @icing: I changed the parent pool of the slave connection from the
>> mplx pool to the master connection's (ptrans), just to simplify the
>> allocators in place for now.
>> I don't see it was needed from a concurrency POV, but if you wanted to
>> avoid locking when creating slaves' pool we can probably keep the
>> dedicated allocator finally (to reduce possible contention on
>> ptrans->mutex? No mutex needed I think since mplx doesn't seem to have
>> other subpools. Trade off with memory consumption, though).
>>
>>
>> Regards,
>> Yann.
>>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Hallo,
up and running (apache 2.4.25 + mpm_event V7 + mpd_http2 v1.8.11 +
mod_ssl patch + your new allocator patch). Will report back.
Greets,
Stefan
Am 06.02.2017 um 01:19 schrieb Yann Ylavic:
> Hi Stefan,
>
> On Sun, Feb 5, 2017 at 7:51 PM, Stefan Priebe - Profihost AG
> <s....@profihost.ag> wrote:
>>
>> tested your patch against mod_ssl. I haven't seen any pool crashes again
>> so it seems to fix this issue.
>>
>> But two new ones:
>
> Possibly a promising fix suggested (in another thread) by yet another
> talented Stefan :)
> He has spotted a possible issue in mpm_event which could affect mainly
> mod_http2.
> I applied his suggestion in the attached patch, and did some
> extrapolation for similar code in mod_h2 (so all possible errors are
> mine...).
> Would you test this one please?
>
> @icing: I changed the parent pool of the slave connection from the
> mplx pool to the master connection's (ptrans), just to simplify the
> allocators in place for now.
> I don't see it was needed from a concurrency POV, but if you wanted to
> avoid locking when creating slaves' pool we can probably keep the
> dedicated allocator finally (to reduce possible contention on
> ptrans->mutex? No mutex needed I think since mplx doesn't seem to have
> other subpools. Trade off with memory consumption, though).
>
>
> Regards,
> Yann.
>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Eissing <st...@greenbytes.de>.
Thanks Yann (the one and only) for quickly adressing this. This explains maybe some of my frustrations in the past with making pools/allocators work in the h2 environment.
-Stefan
> Am 06.02.2017 um 01:19 schrieb Yann Ylavic <yl...@gmail.com>:
>
> Hi Stefan,
>
> On Sun, Feb 5, 2017 at 7:51 PM, Stefan Priebe - Profihost AG
> <s....@profihost.ag> wrote:
>>
>> tested your patch against mod_ssl. I haven't seen any pool crashes again
>> so it seems to fix this issue.
>>
>> But two new ones:
>
> Possibly a promising fix suggested (in another thread) by yet another
> talented Stefan :)
> He has spotted a possible issue in mpm_event which could affect mainly
> mod_http2.
> I applied his suggestion in the attached patch, and did some
> extrapolation for similar code in mod_h2 (so all possible errors are
> mine...).
> Would you test this one please?
>
> @icing: I changed the parent pool of the slave connection from the
> mplx pool to the master connection's (ptrans), just to simplify the
> allocators in place for now.
> I don't see it was needed from a concurrency POV, but if you wanted to
> avoid locking when creating slaves' pool we can probably keep the
> dedicated allocator finally (to reduce possible contention on
> ptrans->mutex? No mutex needed I think since mplx doesn't seem to have
> other subpools. Trade off with memory consumption, though).
>
>
> Regards,
> Yann.
> <ptrans_and_slaves_allocator.patch>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
Hi Stefan,
On Sun, Feb 5, 2017 at 7:51 PM, Stefan Priebe - Profihost AG
<s....@profihost.ag> wrote:
>
> tested your patch against mod_ssl. I haven't seen any pool crashes again
> so it seems to fix this issue.
>
> But two new ones:
Possibly a promising fix suggested (in another thread) by yet another
talented Stefan :)
He has spotted a possible issue in mpm_event which could affect mainly
mod_http2.
I applied his suggestion in the attached patch, and did some
extrapolation for similar code in mod_h2 (so all possible errors are
mine...).
Would you test this one please?
@icing: I changed the parent pool of the slave connection from the
mplx pool to the master connection's (ptrans), just to simplify the
allocators in place for now.
I don't see it was needed from a concurrency POV, but if you wanted to
avoid locking when creating slaves' pool we can probably keep the
dedicated allocator finally (to reduce possible contention on
ptrans->mutex? No mutex needed I think since mplx doesn't seem to have
other subpools. Trade off with memory consumption, though).
Regards,
Yann.
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Eissing <st...@greenbytes.de>.
ok, I'll look at them.
> Am 05.02.2017 um 19:51 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>
> Hi Yann,
>
> tested your patch against mod_ssl. I haven't seen any pool crashes again
> so it seems to fix this issue.
>
> But two new ones:
>
> Program terminated with signal SIGABRT, Aborted.
> #0 0x00007ff523558067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
> #0 0x00007ff523558067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
> #1 0x00007ff523559448 in abort () from /lib/x86_64-linux-gnu/libc.so.6
> #2 0x00007ff5235961b4 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> #3 0x00007ff52359b98e in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> #4 0x00007ff52359c923 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> #5 0x00007ff523b06c83 in apr_allocator_destroy (allocator=0x7ff4d0000e50)
> at memory/unix/apr_pools.c:152
> #6 0x0000556e7986bbef in task_destroy (m=0x7ff46402a028,
> task=0x7ff4d0029f00,
> called_from_master=0) at h2_mplx.c:400
> #7 0x0000556e7986ce5b in task_done_iter (ctx=<optimized out>,
> val=<optimized out>) at h2_mplx.c:1064
> #8 0x00007ff523afe5e6 in apr_hash_do (
> comp=comp@entry=0x556e7987f180 <ihash_iter>,
> rec=rec@entry=0x7ff5037e5480,
> ht=<optimized out>) at tables/apr_hash.c:542
> #9 0x0000556e7987fb5f in h2_ihash_iter (ih=<optimized out>,
> fn=fn@entry=0x556e7986ce50 <task_done_iter>,
> ctx=ctx@entry=0x7ff46402a028)
> at h2_util.c:315
> #10 0x0000556e7986d463 in h2_mplx_release_and_join (m=0x7ff46402a028,
> wait=0x7ff464029fd0) at h2_mplx.c:619
> #11 0x0000556e79872ae4 in session_pool_cleanup (data=0x7ff464020318)
> at h2_session.c:827
> #12 0x00007ff523b0748e in run_cleanups (cref=0x7ff464029fa8)
> at memory/unix/apr_pools.c:2352
> #13 apr_pool_destroy (pool=0x7ff464029f38) at memory/unix/apr_pools.c:804
> #14 0x00007ff523b07745 in apr_pool_clear (pool=0x7ff4fc0150e8)
> at memory/unix/apr_pools.c:769
> #15 0x0000556e798aa698 in ap_push_pool (queue_info=0x1e2e,
> pool_to_recycle=0x1ebb) at fdqueue.c:234
> #16 0x0000556e798a59da in process_lingering_close (cs=0x7ff4fc015378,
> pfd=0x556e7b8bd888) at event.c:1513
> #17 0x0000556e798a9510 in listener_thread (thd=0x1e2e,
> dummy=0x547b44ea3e1b3)
> at event.c:1837
> #18 0x00007ff5238d60a4 in start_thread ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #19 0x00007ff52360b62d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> and
>
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0 apr_bucket_alloc (size=size@entry=64,
> list=list@entry=0x3fb50d4a7aeb1d49)
> at buckets/apr_buckets_alloc.c:128
> #0 apr_bucket_alloc (size=size@entry=64,
> list=list@entry=0x3fb50d4a7aeb1d49)
> at buckets/apr_buckets_alloc.c:128
> #1 0x00007ff523d2b1d3 in apr_bucket_heap_create (
> buf=0x7ff51003b3a8 "
> \311\021A\216y\034\347\034Wy\360\343R\275\226o\020iw\227r\337\377",
> length=1300, free_func=0x7ff523d2ab50 <apr_bucket_free>,
> list=0x3fb50d4a7aeb1d49) at buckets/apr_buckets_heap.c:81
> #2 0x0000556e79884f85 in append_scratch (io=0x7ff4440377c8)
> at h2_conn_io.c:165
> #3 0x0000556e79884ffa in assure_scratch_space (io=0x7ff4440377c8)
> at h2_conn_io.c:182
> #4 0x0000556e79885ce8 in h2_conn_io_pass (io=io@entry=0x7ff4440377c8,
> bb=0x7ff444133698) at h2_conn_io.c:393
> #5 0x0000556e798732be in on_send_data_cb (ngh2=<optimized out>,
> frame=<optimized out>, framehd=<optimized out>, length=1291,
> source=<optimized out>, userp=0x7ff444037780) at h2_session.c:648
> #6 0x00007ff5243dde95 in ?? () from
> /usr/lib/x86_64-linux-gnu/libnghttp2.so.14
> #7 0x00007ff5243deea9 in nghttp2_session_send ()
> from /usr/lib/x86_64-linux-gnu/libnghttp2.so.14
> #8 0x0000556e79875857 in h2_session_send (session=0x7ff444037780)
> at h2_session.c:1376
> #9 0x0000556e79878b6c in h2_session_process (session=0x7ff444037780,
> async=2062228809) at h2_session.c:2208
> #10 0x0000556e79867788 in h2_conn_run (ctx=0x7ff4440376b0, c=0x7ff51003b6a8)
> at h2_conn.c:214
> #11 0x0000556e7986a421 in h2_h2_process_conn (c=0x7ff51003b6a8) at
> h2_h2.c:658
> #12 0x0000556e7980d050 in ap_run_process_connection (c=0x7ff51003b6a8)
> at connection.c:42
> #13 0x0000556e798a7590 in process_socket (my_thread_num=<optimized out>,
> my_child_num=<optimized out>, cs=0x7ff51003b618, sock=<optimized out>,
> p=<optimized out>, thd=<optimized out>) at event.c:1134
> #14 worker_thread (thd=0x40, dummy=0x3fb50d4a7aeb1d49) at event.c:2137
> #15 0x00007ff5238d60a4 in start_thread ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #16 0x00007ff52360b62d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Greets,
> Stefan
>
>> Am 02.02.2017 um 11:09 schrieb Yann Ylavic:
>> Hi Stefan,
>>
>> On Tue, Jan 31, 2017 at 4:01 PM, Stefan Priebe - Profihost AG
>> <s....@profihost.ag> wrote:
>>>
>>> any ideas?
>>
>> I wonder if the attached patch (related to mod_ssl and proposed for
>> another segfault report) could help in your case.
>>
>> Would you mind give it a try?
>>
>>
>> Thanks,
>> Yann.
>>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Hi Yann,
tested your patch against mod_ssl. I haven't seen any pool crashes again
so it seems to fix this issue.
But two new ones:
Program terminated with signal SIGABRT, Aborted.
#0 0x00007ff523558067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#0 0x00007ff523558067 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ff523559448 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007ff5235961b4 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007ff52359b98e in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x00007ff52359c923 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x00007ff523b06c83 in apr_allocator_destroy (allocator=0x7ff4d0000e50)
at memory/unix/apr_pools.c:152
#6 0x0000556e7986bbef in task_destroy (m=0x7ff46402a028,
task=0x7ff4d0029f00,
called_from_master=0) at h2_mplx.c:400
#7 0x0000556e7986ce5b in task_done_iter (ctx=<optimized out>,
val=<optimized out>) at h2_mplx.c:1064
#8 0x00007ff523afe5e6 in apr_hash_do (
comp=comp@entry=0x556e7987f180 <ihash_iter>,
rec=rec@entry=0x7ff5037e5480,
ht=<optimized out>) at tables/apr_hash.c:542
#9 0x0000556e7987fb5f in h2_ihash_iter (ih=<optimized out>,
fn=fn@entry=0x556e7986ce50 <task_done_iter>,
ctx=ctx@entry=0x7ff46402a028)
at h2_util.c:315
#10 0x0000556e7986d463 in h2_mplx_release_and_join (m=0x7ff46402a028,
wait=0x7ff464029fd0) at h2_mplx.c:619
#11 0x0000556e79872ae4 in session_pool_cleanup (data=0x7ff464020318)
at h2_session.c:827
#12 0x00007ff523b0748e in run_cleanups (cref=0x7ff464029fa8)
at memory/unix/apr_pools.c:2352
#13 apr_pool_destroy (pool=0x7ff464029f38) at memory/unix/apr_pools.c:804
#14 0x00007ff523b07745 in apr_pool_clear (pool=0x7ff4fc0150e8)
at memory/unix/apr_pools.c:769
#15 0x0000556e798aa698 in ap_push_pool (queue_info=0x1e2e,
pool_to_recycle=0x1ebb) at fdqueue.c:234
#16 0x0000556e798a59da in process_lingering_close (cs=0x7ff4fc015378,
pfd=0x556e7b8bd888) at event.c:1513
#17 0x0000556e798a9510 in listener_thread (thd=0x1e2e,
dummy=0x547b44ea3e1b3)
at event.c:1837
#18 0x00007ff5238d60a4 in start_thread ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#19 0x00007ff52360b62d in clone () from /lib/x86_64-linux-gnu/libc.so.6
and
Program terminated with signal SIGSEGV, Segmentation fault.
#0 apr_bucket_alloc (size=size@entry=64,
list=list@entry=0x3fb50d4a7aeb1d49)
at buckets/apr_buckets_alloc.c:128
#0 apr_bucket_alloc (size=size@entry=64,
list=list@entry=0x3fb50d4a7aeb1d49)
at buckets/apr_buckets_alloc.c:128
#1 0x00007ff523d2b1d3 in apr_bucket_heap_create (
buf=0x7ff51003b3a8 "
\311\021A\216y\034\347\034Wy\360\343R\275\226o\020iw\227r\337\377",
length=1300, free_func=0x7ff523d2ab50 <apr_bucket_free>,
list=0x3fb50d4a7aeb1d49) at buckets/apr_buckets_heap.c:81
#2 0x0000556e79884f85 in append_scratch (io=0x7ff4440377c8)
at h2_conn_io.c:165
#3 0x0000556e79884ffa in assure_scratch_space (io=0x7ff4440377c8)
at h2_conn_io.c:182
#4 0x0000556e79885ce8 in h2_conn_io_pass (io=io@entry=0x7ff4440377c8,
bb=0x7ff444133698) at h2_conn_io.c:393
#5 0x0000556e798732be in on_send_data_cb (ngh2=<optimized out>,
frame=<optimized out>, framehd=<optimized out>, length=1291,
source=<optimized out>, userp=0x7ff444037780) at h2_session.c:648
#6 0x00007ff5243dde95 in ?? () from
/usr/lib/x86_64-linux-gnu/libnghttp2.so.14
#7 0x00007ff5243deea9 in nghttp2_session_send ()
from /usr/lib/x86_64-linux-gnu/libnghttp2.so.14
#8 0x0000556e79875857 in h2_session_send (session=0x7ff444037780)
at h2_session.c:1376
#9 0x0000556e79878b6c in h2_session_process (session=0x7ff444037780,
async=2062228809) at h2_session.c:2208
#10 0x0000556e79867788 in h2_conn_run (ctx=0x7ff4440376b0, c=0x7ff51003b6a8)
at h2_conn.c:214
#11 0x0000556e7986a421 in h2_h2_process_conn (c=0x7ff51003b6a8) at
h2_h2.c:658
#12 0x0000556e7980d050 in ap_run_process_connection (c=0x7ff51003b6a8)
at connection.c:42
#13 0x0000556e798a7590 in process_socket (my_thread_num=<optimized out>,
my_child_num=<optimized out>, cs=0x7ff51003b618, sock=<optimized out>,
p=<optimized out>, thd=<optimized out>) at event.c:1134
#14 worker_thread (thd=0x40, dummy=0x3fb50d4a7aeb1d49) at event.c:2137
#15 0x00007ff5238d60a4 in start_thread ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#16 0x00007ff52360b62d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Greets,
Stefan
Am 02.02.2017 um 11:09 schrieb Yann Ylavic:
> Hi Stefan,
>
> On Tue, Jan 31, 2017 at 4:01 PM, Stefan Priebe - Profihost AG
> <s....@profihost.ag> wrote:
>>
>> any ideas?
>
> I wonder if the attached patch (related to mod_ssl and proposed for
> another segfault report) could help in your case.
>
> Would you mind give it a try?
>
>
> Thanks,
> Yann.
>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Am 02.02.2017 um 11:09 schrieb Yann Ylavic:
> Hi Stefan,
>
> On Tue, Jan 31, 2017 at 4:01 PM, Stefan Priebe - Profihost AG
> <s....@profihost.ag> wrote:
>>
>> any ideas?
>
> I wonder if the attached patch (related to mod_ssl and proposed for
> another segfault report) could help in your case.
>
> Would you mind give it a try?
Up and running. It will take some time until i can report back as those
crashes are very rare.
Stefan
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Am 23.01.2017 um 21:54 schrieb Stefan Eissing:
>
>> Am 22.01.2017 um 22:22 schrieb Yann Ylavic <yl...@gmail.com>:
>>
>> @icing: Any special expectation in mod_h2 with regard to mpm workers
>> threads' lifetime (or keepalive connections that should stay alive for
>> the configured limit)?
>> I see that beam buckets make use of thread local storage/keys for
>> locking, and that they also handle the double cleanup like eoc buckets
>> did before 1.8.9, but can't follow all the paths yet.
>> Maybe something to look at there?
>
> - With your help, mod_http2 1.8.9 simplified cleanup by triggering it by the connection pool clean up only. That helped. The bucket beam still register for pool cleanup which could be done without, but I think it's a good failsafe.
> - The thread local is used for recursive locking and, once the outermost lock is released, are supposed to be NULL again. This I would like to eliminate one day.
> - the special handling for apr_files is gone as well. That was a work-around when shared file buckets were transported through a bucket beam. No more. Shared file buckets are copied now. Only file buckets with refcount == 1 are beamed now. That makes the beam the sole controller of the lifetime and makes setasides on the master connection work without special handling.
>
> I am not aware of any special expectations now. Almost all is triggered by (parent) pool cleanups and is therefore more deterministic than before. The only explicit destroy is done on finished streams and slave connections no longer used. When the master conn disappears, all is deallocated as the force wills it.
Thanks and + 1 - only two crashes in 48h. ,-) before i had around 2 per
hour. Need some time to debug them. Don't have dumps yet.
Is there an easier way to get core dumps than manually starting httpd
while using systemd?
ulimit -c unlimited
httpd -k start
Greets,
Stefan
>
> Stefan Eissing
>
> <green/>bytes GmbH
> Hafenstrasse 16
> 48155 M�nster
> www.greenbytes.de
>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Am 22.01.2017 um 22:22 schrieb Yann Ylavic:
> Hi Stefan,
>
> On Sun, Jan 22, 2017 at 8:00 PM, Stefan Priebe - Profihost AG
> <s....@profihost.ag> wrote:
>> Am 22.01.2017 um 18:02 schrieb Eric Covener:
>>> On Sun, Jan 22, 2017 at 11:21 AM, Stefan Priebe - Profihost AG
>>> <s....@profihost.ag> wrote:
>>>> Hi Stefan,
>>>>
>>>> no i was mistaken customer isn't using mod_proxy - but I think this is
>>>> the patch causing me problems:
>>>> https://github.com/apache/httpd/commit/a61a4bd02e483fb45d433343740d0130ee3d8d5d
>>>>
>>>> What do you think?
>>>>
>>>
>>> If that is the culprit, you could likely minimize it & help confirm with
>>>
>>> * Increase MaxSpareTheads to the current value of MaxClients (aka
>>> MaxRequestWorkers)
>>> * Make sure MaxRequestsPerChild is 0.
>>
>> Why not just revert that one?
>
> This commit is likely not the "culprit", but probably the one that
> brings up the issue.
>
> It makes so that mpm event's threads/keepalive connections get
> terminated more agressively/quickly on graceful restart, for the new
> generation of children processes to be able to handle new connections
> also more quickly.
>
> I agree with Eric that the next test would be to avoid "maintenance"
> graceful restarts by tunning MaxSpareTheads and MaxRequestsPerChild as
> suggested, mod_http2 should then expect the same behaviour from the
> mpm as with 2.4.23.
>
> You may still reproduce the crash with "explicit" graceful restarts
> (e.g. "apache[2]ctl -k graceful" or "/etc/init.d/apache2 reload"), but
> if it proves to be stable otherwise the issue is still about double
> cleanup/close when http2 pools/buckets are in place.
Thanks for this great explanation this makes sense. Currently i'm
waiting for a next crash. But didn't get one...
no idea which kind of clients are able todo them - but they were very
rare. 99% of the crashes are gone with mod_http2 v1.8.9.
May be i should retest your V7 patch?
Greets,
Stefan
> @icing: Any special expectation in mod_h2 with regard to mpm workers
> threads' lifetime (or keepalive connections that should stay alive for
> the configured limit)?
> I see that beam buckets make use of thread local storage/keys for
> locking, and that they also handle the double cleanup like eoc buckets
> did before 1.8.9, but can't follow all the paths yet.
> Maybe something to look at there?
>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Eissing <st...@greenbytes.de>.
> Am 25.01.2017 um 08:04 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>
> Hi Yann,
>
>> Am 25.01.2017 um 01:41 schrieb Yann Ylavic:
>> Hi Stefan,
>>
>> On Tue, Jan 24, 2017 at 1:37 PM, Stefan Eissing
>> <st...@greenbytes.de> wrote:
>>> Yann, thanks for the patch. I agree that the cleanups need to be killed in the right place. Not certain if it was wrong before, but that part is not easy to see for every combination.
>>>
>>> I did some rework and hope this makes it more readable. If you find the time to look at it, feedback welcome.
>>
>> I still fear that if beam->pool gets destroyed while both
>> beam_send_cleanup() and beam_cleanup() are registered, the former is
>> called twice.
>>
>> I'd change:
>> if (safe_send) {
>> if (beam->send_pool && beam->send_pool != beam->pool) {
>> apr_pool_cleanup_kill(beam->send_pool, beam, beam_send_cleanup);
>> }
>> status = beam_send_cleanup(beam);
>> }
>>
>> with:
>> if (safe_send) {
>> if (beam->send_pool) {
>> if (beam->send_pool != beam->pool) {
>> apr_pool_cleanup_kill(beam->send_pool, beam, beam_send_cleanup);
>> }
>> status = beam_send_cleanup(beam);
>> }
>> }
>>
>> since in the above case beam_send_cleanup is run first and sets send_pool=NULL.
>>
>> Attached v3 with this only change w.r.t. v2.
>> Otherwise, looks good to me, thanks!
>
> this one also contains changes to the even mpm, mod_proxy and
> mod_slotmem_shm. Is this expected?
Don't think so.
Yann is a busy man. :)
>
> Stefan
>>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Hi Yann,
Am 25.01.2017 um 01:41 schrieb Yann Ylavic:
> Hi Stefan,
>
> On Tue, Jan 24, 2017 at 1:37 PM, Stefan Eissing
> <st...@greenbytes.de> wrote:
>> Yann, thanks for the patch. I agree that the cleanups need to be killed in the right place. Not certain if it was wrong before, but that part is not easy to see for every combination.
>>
>> I did some rework and hope this makes it more readable. If you find the time to look at it, feedback welcome.
>
> I still fear that if beam->pool gets destroyed while both
> beam_send_cleanup() and beam_cleanup() are registered, the former is
> called twice.
>
> I'd change:
> if (safe_send) {
> if (beam->send_pool && beam->send_pool != beam->pool) {
> apr_pool_cleanup_kill(beam->send_pool, beam, beam_send_cleanup);
> }
> status = beam_send_cleanup(beam);
> }
>
> with:
> if (safe_send) {
> if (beam->send_pool) {
> if (beam->send_pool != beam->pool) {
> apr_pool_cleanup_kill(beam->send_pool, beam, beam_send_cleanup);
> }
> status = beam_send_cleanup(beam);
> }
> }
>
> since in the above case beam_send_cleanup is run first and sets send_pool=NULL.
>
> Attached v3 with this only change w.r.t. v2.
> Otherwise, looks good to me, thanks!
this one also contains changes to the even mpm, mod_proxy and
mod_slotmem_shm. Is this expected?
Stefan
>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe <s....@profihost.ag>.
Am 01.02.2017 um 06:36 schrieb Stefan Eissing:
> Stefan,
>
> did not have the time to look at it really. Will do, but very busy right now.
Thanks - no problem.
>
>> Am 31.01.2017 um 16:01 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>
>> Hi Yann,
>> Hi Stefan,
>>
>> any ideas?
>>
>> Stefan
>>
>>> Am 27.01.2017 um 20:30 schrieb Ruediger Pluem:
>>>
>>>
>>>> On 01/27/2017 08:10 PM, Stefan Priebe - Profihost AG wrote:
>>>> Hi Yann,
>>>>
>>>> this is now a segfault with mod_http2 + beam v4 patch + mpm v7 patch:
>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>> #0 allocator_free (node=0x0, allocator=0x7f8974080c90)
>>>> at memory/unix/apr_pools.c:381
>>>> #0 allocator_free (node=0x0, allocator=0x7f8974080c90)
>>>> at memory/unix/apr_pools.c:381
>>>> #1 apr_pool_clear (pool=0x7f89740830d8) at memory/unix/apr_pools.c:793
>>>
>>> pool from above != pool_to_recycle from below is strange.
>>>
>>>> #2 0x00005611421447a8 in ap_push_pool (queue_info=0x0,
>>>
>>> queue_info NULL? That looks bad.
>>>
>>>> pool_to_recycle=0x7f8974080c98) at fdqueue.c:234
>>>> #3 0x000056114213faea in process_lingering_close (cs=0x7f8974083368,
>>>> pfd=0x561142cc97f8) at event.c:1513
>>>> #4 0x0000561142143620 in listener_thread (thd=0x0, dummy=0x54716c3684bbc)
>>>> at event.c:1837
>>>> #5 0x00007f8989bb20a4 in start_thread ()
>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>> #6 0x00007f89898e762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>
>>> Regards
>>>
>>> R�diger
>>>
>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
Hi Stefan,
On Tue, Jan 31, 2017 at 4:01 PM, Stefan Priebe - Profihost AG
<s....@profihost.ag> wrote:
>
> any ideas?
I wonder if the attached patch (related to mod_ssl and proposed for
another segfault report) could help in your case.
Would you mind give it a try?
Thanks,
Yann.
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Eissing <st...@greenbytes.de>.
Stefan,
did not have the time to look at it really. Will do, but very busy right now.
> Am 31.01.2017 um 16:01 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>
> Hi Yann,
> Hi Stefan,
>
> any ideas?
>
> Stefan
>
>> Am 27.01.2017 um 20:30 schrieb Ruediger Pluem:
>>
>>
>>> On 01/27/2017 08:10 PM, Stefan Priebe - Profihost AG wrote:
>>> Hi Yann,
>>>
>>> this is now a segfault with mod_http2 + beam v4 patch + mpm v7 patch:
>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>> #0 allocator_free (node=0x0, allocator=0x7f8974080c90)
>>> at memory/unix/apr_pools.c:381
>>> #0 allocator_free (node=0x0, allocator=0x7f8974080c90)
>>> at memory/unix/apr_pools.c:381
>>> #1 apr_pool_clear (pool=0x7f89740830d8) at memory/unix/apr_pools.c:793
>>
>> pool from above != pool_to_recycle from below is strange.
>>
>>> #2 0x00005611421447a8 in ap_push_pool (queue_info=0x0,
>>
>> queue_info NULL? That looks bad.
>>
>>> pool_to_recycle=0x7f8974080c98) at fdqueue.c:234
>>> #3 0x000056114213faea in process_lingering_close (cs=0x7f8974083368,
>>> pfd=0x561142cc97f8) at event.c:1513
>>> #4 0x0000561142143620 in listener_thread (thd=0x0, dummy=0x54716c3684bbc)
>>> at event.c:1837
>>> #5 0x00007f8989bb20a4 in start_thread ()
>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>> #6 0x00007f89898e762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>
>> Regards
>>
>> Rüdiger
>>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Hi Yann,
Hi Stefan,
any ideas?
Stefan
Am 27.01.2017 um 20:30 schrieb Ruediger Pluem:
>
>
> On 01/27/2017 08:10 PM, Stefan Priebe - Profihost AG wrote:
>> Hi Yann,
>>
>> this is now a segfault with mod_http2 + beam v4 patch + mpm v7 patch:
>> Program terminated with signal SIGSEGV, Segmentation fault.
>> #0 allocator_free (node=0x0, allocator=0x7f8974080c90)
>> at memory/unix/apr_pools.c:381
>> #0 allocator_free (node=0x0, allocator=0x7f8974080c90)
>> at memory/unix/apr_pools.c:381
>> #1 apr_pool_clear (pool=0x7f89740830d8) at memory/unix/apr_pools.c:793
>
> pool from above != pool_to_recycle from below is strange.
>
>> #2 0x00005611421447a8 in ap_push_pool (queue_info=0x0,
>
> queue_info NULL? That looks bad.
>
>> pool_to_recycle=0x7f8974080c98) at fdqueue.c:234
>> #3 0x000056114213faea in process_lingering_close (cs=0x7f8974083368,
>> pfd=0x561142cc97f8) at event.c:1513
>> #4 0x0000561142143620 in listener_thread (thd=0x0, dummy=0x54716c3684bbc)
>> at event.c:1837
>> #5 0x00007f8989bb20a4 in start_thread ()
>> from /lib/x86_64-linux-gnu/libpthread.so.0
>> #6 0x00007f89898e762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Regards
>
> R�diger
>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Eissing <st...@greenbytes.de>.
As Rüdiger wrote, this stacktrace does not make any sense (to me), unless the pool cleanup messes up the stack.
Hmmm. This points to a major misconception of mine how things are supposed to happen. Will continue to look at this, but no easy solution in sight.
> Am 27.01.2017 um 20:30 schrieb Ruediger Pluem <rp...@apache.org>:
>
>
>
> On 01/27/2017 08:10 PM, Stefan Priebe - Profihost AG wrote:
>> Hi Yann,
>>
>> this is now a segfault with mod_http2 + beam v4 patch + mpm v7 patch:
>> Program terminated with signal SIGSEGV, Segmentation fault.
>> #0 allocator_free (node=0x0, allocator=0x7f8974080c90)
>> at memory/unix/apr_pools.c:381
>> #0 allocator_free (node=0x0, allocator=0x7f8974080c90)
>> at memory/unix/apr_pools.c:381
>> #1 apr_pool_clear (pool=0x7f89740830d8) at memory/unix/apr_pools.c:793
>
> pool from above != pool_to_recycle from below is strange.
There is no modification to that parameter in this function. Which would mean the cleanup function/child pool destruction somehow garbles the stack??? Which would explain the strangeness below:
>> #2 0x00005611421447a8 in ap_push_pool (queue_info=0x0,
>
> queue_info NULL? That looks bad.
And that should have crashed at fdqueue.c line 225 already if queue_info == NULLL.
>> pool_to_recycle=0x7f8974080c98) at fdqueue.c:234
>> #3 0x000056114213faea in process_lingering_close (cs=0x7f8974083368,
>> pfd=0x561142cc97f8) at event.c:1513
>> #4 0x0000561142143620 in listener_thread (thd=0x0, dummy=0x54716c3684bbc)
>> at event.c:1837
>> #5 0x00007f8989bb20a4 in start_thread ()
>> from /lib/x86_64-linux-gnu/libpthread.so.0
>> #6 0x00007f89898e762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Regards
>
> Rüdiger
>
Stefan Eissing
<green/>bytes GmbH
Hafenstrasse 16
48155 Münster
www.greenbytes.de
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Ruediger Pluem <rp...@apache.org>.
On 01/27/2017 08:10 PM, Stefan Priebe - Profihost AG wrote:
> Hi Yann,
>
> this is now a segfault with mod_http2 + beam v4 patch + mpm v7 patch:
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0 allocator_free (node=0x0, allocator=0x7f8974080c90)
> at memory/unix/apr_pools.c:381
> #0 allocator_free (node=0x0, allocator=0x7f8974080c90)
> at memory/unix/apr_pools.c:381
> #1 apr_pool_clear (pool=0x7f89740830d8) at memory/unix/apr_pools.c:793
pool from above != pool_to_recycle from below is strange.
> #2 0x00005611421447a8 in ap_push_pool (queue_info=0x0,
queue_info NULL? That looks bad.
> pool_to_recycle=0x7f8974080c98) at fdqueue.c:234
> #3 0x000056114213faea in process_lingering_close (cs=0x7f8974083368,
> pfd=0x561142cc97f8) at event.c:1513
> #4 0x0000561142143620 in listener_thread (thd=0x0, dummy=0x54716c3684bbc)
> at event.c:1837
> #5 0x00007f8989bb20a4 in start_thread ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #6 0x00007f89898e762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Regards
R�diger
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Hi Yann,
this is now a segfault with mod_http2 + beam v4 patch + mpm v7 patch:
Program terminated with signal SIGSEGV, Segmentation fault.
#0 allocator_free (node=0x0, allocator=0x7f8974080c90)
at memory/unix/apr_pools.c:381
#0 allocator_free (node=0x0, allocator=0x7f8974080c90)
at memory/unix/apr_pools.c:381
#1 apr_pool_clear (pool=0x7f89740830d8) at memory/unix/apr_pools.c:793
#2 0x00005611421447a8 in ap_push_pool (queue_info=0x0,
pool_to_recycle=0x7f8974080c98) at fdqueue.c:234
#3 0x000056114213faea in process_lingering_close (cs=0x7f8974083368,
pfd=0x561142cc97f8) at event.c:1513
#4 0x0000561142143620 in listener_thread (thd=0x0, dummy=0x54716c3684bbc)
at event.c:1837
#5 0x00007f8989bb20a4 in start_thread ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007f89898e762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Stefan
Am 25.01.2017 um 12:10 schrieb Stefan Priebe - Profihost AG:
> Am 25.01.2017 um 10:46 schrieb Stefan Eissing:
>> Thanks Yann!
>>
>> Stefan: here is the patch as committed to trunk:
>
> Up and running. Will report back.
>
>>
>>
>> Cheers, Stefan
>>
>>> Am 25.01.2017 um 01:41 schrieb Yann Ylavic <yl...@gmail.com>:
>>>
>>> Hi Stefan,
>>>
>>> On Tue, Jan 24, 2017 at 1:37 PM, Stefan Eissing
>>> <st...@greenbytes.de> wrote:
>>>> Yann, thanks for the patch. I agree that the cleanups need to be killed in the right place. Not certain if it was wrong before, but that part is not easy to see for every combination.
>>>>
>>>> I did some rework and hope this makes it more readable. If you find the time to look at it, feedback welcome.
>>>
>>> I still fear that if beam->pool gets destroyed while both
>>> beam_send_cleanup() and beam_cleanup() are registered, the former is
>>> called twice.
>>>
>>> I'd change:
>>> if (safe_send) {
>>> if (beam->send_pool && beam->send_pool != beam->pool) {
>>> apr_pool_cleanup_kill(beam->send_pool, beam, beam_send_cleanup);
>>> }
>>> status = beam_send_cleanup(beam);
>>> }
>>>
>>> with:
>>> if (safe_send) {
>>> if (beam->send_pool) {
>>> if (beam->send_pool != beam->pool) {
>>> apr_pool_cleanup_kill(beam->send_pool, beam, beam_send_cleanup);
>>> }
>>> status = beam_send_cleanup(beam);
>>> }
>>> }
>>>
>>> since in the above case beam_send_cleanup is run first and sets send_pool=NULL.
>>>
>>> Attached v3 with this only change w.r.t. v2.
>>> Otherwise, looks good to me, thanks!
>>> <h2_beams_cleanup_v3.diff>
>>
>> Stefan Eissing
>>
>> <green/>bytes GmbH
>> Hafenstrasse 16
>> 48155 M�nster
>> www.greenbytes.de
>>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Am 25.01.2017 um 10:46 schrieb Stefan Eissing:
> Thanks Yann!
>
> Stefan: here is the patch as committed to trunk:
Up and running. Will report back.
>
>
> Cheers, Stefan
>
>> Am 25.01.2017 um 01:41 schrieb Yann Ylavic <yl...@gmail.com>:
>>
>> Hi Stefan,
>>
>> On Tue, Jan 24, 2017 at 1:37 PM, Stefan Eissing
>> <st...@greenbytes.de> wrote:
>>> Yann, thanks for the patch. I agree that the cleanups need to be killed in the right place. Not certain if it was wrong before, but that part is not easy to see for every combination.
>>>
>>> I did some rework and hope this makes it more readable. If you find the time to look at it, feedback welcome.
>>
>> I still fear that if beam->pool gets destroyed while both
>> beam_send_cleanup() and beam_cleanup() are registered, the former is
>> called twice.
>>
>> I'd change:
>> if (safe_send) {
>> if (beam->send_pool && beam->send_pool != beam->pool) {
>> apr_pool_cleanup_kill(beam->send_pool, beam, beam_send_cleanup);
>> }
>> status = beam_send_cleanup(beam);
>> }
>>
>> with:
>> if (safe_send) {
>> if (beam->send_pool) {
>> if (beam->send_pool != beam->pool) {
>> apr_pool_cleanup_kill(beam->send_pool, beam, beam_send_cleanup);
>> }
>> status = beam_send_cleanup(beam);
>> }
>> }
>>
>> since in the above case beam_send_cleanup is run first and sets send_pool=NULL.
>>
>> Attached v3 with this only change w.r.t. v2.
>> Otherwise, looks good to me, thanks!
>> <h2_beams_cleanup_v3.diff>
>
> Stefan Eissing
>
> <green/>bytes GmbH
> Hafenstrasse 16
> 48155 M�nster
> www.greenbytes.de
>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Eissing <st...@greenbytes.de>.
Thanks Yann!
Stefan: here is the patch as committed to trunk:
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
Hi Stefan,
On Tue, Jan 24, 2017 at 1:37 PM, Stefan Eissing
<st...@greenbytes.de> wrote:
> Yann, thanks for the patch. I agree that the cleanups need to be killed in the right place. Not certain if it was wrong before, but that part is not easy to see for every combination.
>
> I did some rework and hope this makes it more readable. If you find the time to look at it, feedback welcome.
I still fear that if beam->pool gets destroyed while both
beam_send_cleanup() and beam_cleanup() are registered, the former is
called twice.
I'd change:
if (safe_send) {
if (beam->send_pool && beam->send_pool != beam->pool) {
apr_pool_cleanup_kill(beam->send_pool, beam, beam_send_cleanup);
}
status = beam_send_cleanup(beam);
}
with:
if (safe_send) {
if (beam->send_pool) {
if (beam->send_pool != beam->pool) {
apr_pool_cleanup_kill(beam->send_pool, beam, beam_send_cleanup);
}
status = beam_send_cleanup(beam);
}
}
since in the above case beam_send_cleanup is run first and sets send_pool=NULL.
Attached v3 with this only change w.r.t. v2.
Otherwise, looks good to me, thanks!
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Eissing <st...@greenbytes.de>.
Yann, thanks for the patch. I agree that the cleanups need to be killed in the right place. Not certain if it was wrong before, but that part is not easy to see for every combination.
I did some rework and hope this makes it more readable. If you find the time to look at it, feedback welcome.
Cheers, Stefan
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
On Tue, Jan 24, 2017 at 10:02 AM, Stefan Priebe - Profihost AG
<s....@profihost.ag> wrote:
>
> Am 23.01.2017 um 23:42 schrieb Yann Ylavic:
>> On Mon, Jan 23, 2017 at 11:37 PM, Yann Ylavic <yl...@gmail.com> wrote:
>>> Hi Stefan,
>>>
>>> On Mon, Jan 23, 2017 at 9:54 PM, Stefan Eissing
>>> <st...@greenbytes.de> wrote:
>>>>
>>>> I am not aware of any special expectations now. Almost all is triggered by (parent) pool cleanups and is therefore more deterministic than before. The only explicit destroy is done on finished streams and slave connections no longer used. When the master conn disappears, all is deallocated as the force wills it.
>>>
>>> I wonder if the attached patch makes sense.
>>> If beam_{recv,send}_cleanup() are to be executed on (parent) pool
>>> destroy, there will be before beam_cleanup() itelf (which also calls
>>> beam_send_cleanup() explicitly), so it should avoid potential a double
>>> cleanup in this case.
>>>
>>> WDYT?
>>
>> Please ignore the last (garbage) hunk.
>
> last garbage hunk?
>
> This one?
Yes, this change is not necessary/suitable.
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Am 23.01.2017 um 23:42 schrieb Yann Ylavic:
> On Mon, Jan 23, 2017 at 11:37 PM, Yann Ylavic <yl...@gmail.com> wrote:
>> Hi Stefan,
>>
>> On Mon, Jan 23, 2017 at 9:54 PM, Stefan Eissing
>> <st...@greenbytes.de> wrote:
>>>
>>> I am not aware of any special expectations now. Almost all is triggered by (parent) pool cleanups and is therefore more deterministic than before. The only explicit destroy is done on finished streams and slave connections no longer used. When the master conn disappears, all is deallocated as the force wills it.
>>
>> I wonder if the attached patch makes sense.
>> If beam_{recv,send}_cleanup() are to be executed on (parent) pool
>> destroy, there will be before beam_cleanup() itelf (which also calls
>> beam_send_cleanup() explicitly), so it should avoid potential a double
>> cleanup in this case.
>>
>> WDYT?
>
> Please ignore the last (garbage) hunk.
last garbage hunk?
This one?
@@ -520,8 +529,8 @@ static apr_status_t beam_cleanup(void *data)
* beam should have lost its mutex protection, meaning
* it is no longer used multi-threaded and we can safely
* purge all remaining sender buckets. */
- apr_pool_cleanup_kill(beam->send_pool, beam,
beam_send_cleanup);
ap_assert(!beam->m_enter);
+ beam_set_send_pool(beam, NULL);
beam_send_cleanup(beam);
}
ap_assert(H2_BPROXY_LIST_EMPTY(&beam->proxies));
Stefan
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
On Mon, Jan 23, 2017 at 11:37 PM, Yann Ylavic <yl...@gmail.com> wrote:
> Hi Stefan,
>
> On Mon, Jan 23, 2017 at 9:54 PM, Stefan Eissing
> <st...@greenbytes.de> wrote:
>>
>> I am not aware of any special expectations now. Almost all is triggered by (parent) pool cleanups and is therefore more deterministic than before. The only explicit destroy is done on finished streams and slave connections no longer used. When the master conn disappears, all is deallocated as the force wills it.
>
> I wonder if the attached patch makes sense.
> If beam_{recv,send}_cleanup() are to be executed on (parent) pool
> destroy, there will be before beam_cleanup() itelf (which also calls
> beam_send_cleanup() explicitly), so it should avoid potential a double
> cleanup in this case.
>
> WDYT?
Please ignore the last (garbage) hunk.
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
Hi Stefan,
On Mon, Jan 23, 2017 at 9:54 PM, Stefan Eissing
<st...@greenbytes.de> wrote:
>
> I am not aware of any special expectations now. Almost all is triggered by (parent) pool cleanups and is therefore more deterministic than before. The only explicit destroy is done on finished streams and slave connections no longer used. When the master conn disappears, all is deallocated as the force wills it.
I wonder if the attached patch makes sense.
If beam_{recv,send}_cleanup() are to be executed on (parent) pool
destroy, there will be before beam_cleanup() itelf (which also calls
beam_send_cleanup() explicitly), so it should avoid potential a double
cleanup in this case.
WDYT?
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Eissing <st...@greenbytes.de>.
> Am 22.01.2017 um 22:22 schrieb Yann Ylavic <yl...@gmail.com>:
>
> @icing: Any special expectation in mod_h2 with regard to mpm workers
> threads' lifetime (or keepalive connections that should stay alive for
> the configured limit)?
> I see that beam buckets make use of thread local storage/keys for
> locking, and that they also handle the double cleanup like eoc buckets
> did before 1.8.9, but can't follow all the paths yet.
> Maybe something to look at there?
- With your help, mod_http2 1.8.9 simplified cleanup by triggering it by the connection pool clean up only. That helped. The bucket beam still register for pool cleanup which could be done without, but I think it's a good failsafe.
- The thread local is used for recursive locking and, once the outermost lock is released, are supposed to be NULL again. This I would like to eliminate one day.
- the special handling for apr_files is gone as well. That was a work-around when shared file buckets were transported through a bucket beam. No more. Shared file buckets are copied now. Only file buckets with refcount == 1 are beamed now. That makes the beam the sole controller of the lifetime and makes setasides on the master connection work without special handling.
I am not aware of any special expectations now. Almost all is triggered by (parent) pool cleanups and is therefore more deterministic than before. The only explicit destroy is done on finished streams and slave connections no longer used. When the master conn disappears, all is deallocated as the force wills it.
Stefan Eissing
<green/>bytes GmbH
Hafenstrasse 16
48155 Münster
www.greenbytes.de
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
Hi Stefan,
On Sun, Jan 22, 2017 at 8:00 PM, Stefan Priebe - Profihost AG
<s....@profihost.ag> wrote:
> Am 22.01.2017 um 18:02 schrieb Eric Covener:
>> On Sun, Jan 22, 2017 at 11:21 AM, Stefan Priebe - Profihost AG
>> <s....@profihost.ag> wrote:
>>> Hi Stefan,
>>>
>>> no i was mistaken customer isn't using mod_proxy - but I think this is
>>> the patch causing me problems:
>>> https://github.com/apache/httpd/commit/a61a4bd02e483fb45d433343740d0130ee3d8d5d
>>>
>>> What do you think?
>>>
>>
>> If that is the culprit, you could likely minimize it & help confirm with
>>
>> * Increase MaxSpareTheads to the current value of MaxClients (aka
>> MaxRequestWorkers)
>> * Make sure MaxRequestsPerChild is 0.
>
> Why not just revert that one?
This commit is likely not the "culprit", but probably the one that
brings up the issue.
It makes so that mpm event's threads/keepalive connections get
terminated more agressively/quickly on graceful restart, for the new
generation of children processes to be able to handle new connections
also more quickly.
I agree with Eric that the next test would be to avoid "maintenance"
graceful restarts by tunning MaxSpareTheads and MaxRequestsPerChild as
suggested, mod_http2 should then expect the same behaviour from the
mpm as with 2.4.23.
You may still reproduce the crash with "explicit" graceful restarts
(e.g. "apache[2]ctl -k graceful" or "/etc/init.d/apache2 reload"), but
if it proves to be stable otherwise the issue is still about double
cleanup/close when http2 pools/buckets are in place.
@icing: Any special expectation in mod_h2 with regard to mpm workers
threads' lifetime (or keepalive connections that should stay alive for
the configured limit)?
I see that beam buckets make use of thread local storage/keys for
locking, and that they also handle the double cleanup like eoc buckets
did before 1.8.9, but can't follow all the paths yet.
Maybe something to look at there?
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Am 22.01.2017 um 18:02 schrieb Eric Covener:
> On Sun, Jan 22, 2017 at 11:21 AM, Stefan Priebe - Profihost AG
> <s....@profihost.ag> wrote:
>> Hi Stefan,
>>
>> no i was mistaken customer isn't using mod_proxy - but I think this is
>> the patch causing me problems:
>> https://github.com/apache/httpd/commit/a61a4bd02e483fb45d433343740d0130ee3d8d5d
>>
>> What do you think?
>>
>
> If that is the culprit, you could likely minimize it & help confirm with
>
> * Increase MaxSpareTheads to the current value of MaxClients (aka
> MaxRequestWorkers)
> * Make sure MaxRequestsPerChild is 0.
Why not just revert that one?
Greets,
Stefan
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Eissing <st...@greenbytes.de>.
Not my forté. Yann probably has a better idea here. Let's see if he has time to look at it tomorrow.
> Am 22.01.2017 um 17:21 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>
> Hi Stefan,
>
> no i was mistaken customer isn't using mod_proxy - but I think this is
> the patch causing me problems:
> https://github.com/apache/httpd/commit/a61a4bd02e483fb45d433343740d0130ee3d8d5d
>
> What do you think?
>
> Greets,
> Stefan
>
> Am 22.01.2017 um 17:17 schrieb Stefan Eissing:
>>
>>> Am 22.01.2017 um 17:14 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>
>>> *arg* it's just mod_proxy - just saw thread safety and apr bucket aloc.
>>
>> ??? Can you elaborate? Is your finding the known hcheck bug or something else?
>>
>>> Stefan
>>>
>>> Am 22.01.2017 um 17:06 schrieb Stefan Priebe - Profihost AG:
>>>> Looks like others have the same crashes too:
>>>> https://bz.apache.org/bugzilla/show_bug.cgi?id=60071
>>>> and
>>>> https://github.com/apache/httpd/commit/8e63c3c9372cd398f57357099aa941cbba695758
>>>>
>>>> So it looks like mod_http2 is running fine now. Thanks a lot Stefan.
>>>>
>>>> Yann i think i can start testing your mpm patch again after the
>>>> segfaults in 2.4 branch are fixed.
>>>>
>>>> Greets,
>>>> Stefan
>>>>
>>>> Am 22.01.2017 um 13:16 schrieb Stefan Priebe:
>>>>> Hi,
>>>>>
>>>>> and a new one but also in ap_start_lingering_close:
>>>>>
>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>> #0 apr_palloc (pool=pool@entry=0x7f455805e138, in_size=in_size@entry=32)
>>>>> at memory/unix/apr_pools.c:684
>>>>> #0 apr_palloc (pool=pool@entry=0x7f455805e138, in_size=in_size@entry=32)
>>>>> at memory/unix/apr_pools.c:684
>>>>> #1 0x00007f456bc5d8b4 in apr_brigade_create (p=0x7f455805e138,
>>>>> list=0x7f45040034e8) at buckets/apr_brigade.c:61
>>>>> #2 0x000055e165efa319 in ap_shutdown_conn (c=c@entry=0x7f455805e458,
>>>>> flush=flush@entry=1) at connection.c:76
>>>>> #3 0x000055e165efa40d in ap_flush_conn (c=0x7f455805e458) at
>>>>> connection.c:95
>>>>> #4 ap_start_lingering_close (c=0x7f455805e458) at connection.c:145
>>>>> #5 0x000055e165f942dd in start_lingering_close_blocking (cs=<optimized
>>>>> out>)
>>>>> at event.c:876
>>>>> #6 process_socket (my_thread_num=<optimized out>,
>>>>> my_child_num=<optimized out>, cs=0x7f455805e3c8, sock=<optimized out>,
>>>>> p=<optimized out>, thd=<optimized out>) at event.c:1153
>>>>> #7 worker_thread (thd=0x7f455805e138, dummy=0x20) at event.c:2001
>>>>> #8 0x00007f456b80a0a4 in start_thread ()
>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>> #9 0x00007f456b53f62d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>
>>>>> Stefan
>>>>>
>>>>> Am 21.01.2017 um 19:31 schrieb Stefan Priebe:
>>>>>> All last traces come from event, proces_longering_close ap_push_pool but
>>>>>> end in different functions. It looks like a race somewhere and it just
>>>>>> races at different function in the event of close and pool clear.
>>>>>>
>>>>>> Might there be two places where the same pool gets cleared?
>>>>>>
>>>>>> Stefan
>>>>>>
>>>>>> Am 21.01.2017 um 19:07 schrieb Stefan Priebe:
>>>>>>> Hi Stefan,
>>>>>>>
>>>>>>> thanks. No crashes where h2 comes up. But i still have these and no idea
>>>>>>> how to find out who and why they're crashing.
>>>>>>>
>>>>>>> Using host libthread_db library
>>>>>>> "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>>>>> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
>>>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>>>> #0 allocator_free (node=0x0, allocator=0x7f6e08066540)
>>>>>>> at memory/unix/apr_pools.c:381
>>>>>>> #0 allocator_free (node=0x0, allocator=0x7f6e08066540)
>>>>>>> at memory/unix/apr_pools.c:381
>>>>>>> #1 apr_pool_clear (pool=0x7f6e0808d238) at memory/unix/apr_pools.c:793
>>>>>>> #2 0x00000000004fe528 in ap_push_pool (queue_info=0x0,
>>>>>>> pool_to_recycle=0x7f6e08066548) at fdqueue.c:234
>>>>>>> #3 0x00000000004fa2c8 in process_lingering_close (cs=0x7f6e0808d4c8,
>>>>>>> pfd=0x1d3bf98) at event.c:1439
>>>>>>> #4 0x00000000004fd410 in listener_thread (thd=0x1d3cb70,
>>>>>>> dummy=0x7f6e0808d4c8)
>>>>>>> at event.c:1704
>>>>>>> #5 0x00007f6e1aed20a4 in start_thread ()
>>>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>>> #6 0x00007f6e1aa0362d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>>> (gdb) (gdb) quit
>>>>>>>
>>>>>>> Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
>>>>>>> /usr/lib/debug//usr/local/apache2/bin/httpd...done.
>>>>>>> done.
>>>>>>> [Thread debugging using libthread_db enabled]
>>>>>>> Using host libthread_db library
>>>>>>> "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>>>>> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
>>>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>>>> #0 allocator_free (node=0x0, allocator=0x7f6e08053ae0)
>>>>>>> at memory/unix/apr_pools.c:381
>>>>>>> #0 allocator_free (node=0x0, allocator=0x7f6e08053ae0)
>>>>>>> at memory/unix/apr_pools.c:381
>>>>>>> #1 apr_pool_clear (pool=0x7f6e08076bb8) at memory/unix/apr_pools.c:793
>>>>>>> #2 0x00000000004fe528 in ap_push_pool (queue_info=0x0,
>>>>>>> pool_to_recycle=0x7f6e08053ae8) at fdqueue.c:234
>>>>>>> #3 0x00000000004fa2c8 in process_lingering_close (cs=0x7f6e08076e48,
>>>>>>> pfd=0x1d3bf98) at event.c:1439
>>>>>>> #4 0x00000000004fd410 in listener_thread (thd=0x1d3cb70,
>>>>>>> dummy=0x7f6e08076e48)
>>>>>>> at event.c:1704
>>>>>>> #5 0x00007f6e1aed20a4 in start_thread ()
>>>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>>> #6 0x00007f6e1aa0362d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>>> (gdb) (gdb) quit
>>>>>>>
>>>>>>> Stefan
>>>>>>>
>>>>>>> Am 21.01.2017 um 17:03 schrieb Stefan Eissing:
>>>>>>>> Stefan,
>>>>>>>>
>>>>>>>> made a release at https://github.com/icing/mod_h2/releases/tag/v1.8.9
>>>>>>>> with all patches and improved (hopefully) on them a bit. If you dare
>>>>>>>> to drop that into your installation, that'd be great.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>>
>>>>>>>> Stefan
>>>>>>>>
>>>>>>>>> Am 21.01.2017 um 15:25 schrieb Stefan Priebe <s....@profihost.ag>:
>>>>>>>>>
>>>>>>>>> and i got another crash here:
>>>>>>>>>
>>>>>>>>> 2346 static void run_cleanups(cleanup_t **cref)
>>>>>>>>> 2347 {
>>>>>>>>> 2348 cleanup_t *c = *cref;
>>>>>>>>> 2349
>>>>>>>>> 2350 while (c) {
>>>>>>>>> 2351 *cref = c->next;
>>>>>>>>> 2352 (*c->plain_cleanup_fn)((void *)c->data); <== here
>>>>>>>>> 2353 c = *cref;
>>>>>>>>> 2354
>>>>>>>>>
>>>>>>>>> which looks similar to the other crash.
>>>>>>>>>
>>>>>>>>> #0 0x00007fe4bbd33e1b in run_cleanups (cref=<optimized out>) at
>>>>>>>>> memory/unix/apr_pools.c:2352
>>>>>>>>> #1 apr_pool_clear (pool=0x7fe4a804dac8) at
>>>>>>>>> memory/unix/apr_pools.c:772
>>>>>>>>> #2 0x00000000004feb38 in ap_push_pool
>>>>>>>>> (queue_info=0x6d616e79642d3733, pool_to_recycle=0x2) at fdqueue.c:234
>>>>>>>>> #3 0x00000000004fa8d8 in process_lingering_close (cs=0x7fe4a804dd58,
>>>>>>>>> pfd=0x25d3f98) at event.c:1439
>>>>>>>>>
>>>>>>>>> Details:
>>>>>>>>> (gdb) print c
>>>>>>>>> $1 = (cleanup_t *) 0x7fe4a804e9f0
>>>>>>>>> (gdb) print *c
>>>>>>>>> $2 = {next = 0x7fe4a804e870, data = 0x6d616e79642d3733,
>>>>>>>>> plain_cleanup_fn = 0x392d3734322e6369,
>>>>>>>>> child_cleanup_fn = 0x617465722e722d35}
>>>>>>>>> (gdb) print *c->data
>>>>>>>>> Attempt to dereference a generic pointer.
>>>>>>>>> (gdb) print *c->plain_cleanup_fn
>>>>>>>>> Cannot access memory at address 0x392d3734322e6369
>>>>>>>>> (gdb)
>>>>>>>>>
>>>>>>>>> Stefan
>>>>>>>>>
>>>>>>>>> Am 21.01.2017 um 15:18 schrieb Stefan Priebe:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> #0 apr_pool_cleanup_kill (p=0x7fe4a8072358,
>>>>>>>>>> data=data@entry=0x7fe4a80723e0,
>>>>>>>>>> cleanup_fn=cleanup_fn@entry=0x7fe4bbd38a40 <socket_cleanup>) at
>>>>>>>>>> memory/unix/apr_pools.c:2276
>>>>>>>>>>
>>>>>>>>>> it crashes here in apr:
>>>>>>>>>> 2276 if (c->data == data && c->plain_cleanup_fn ==
>>>>>>>>>> cleanup_fn) {
>>>>>>>>>>
>>>>>>>>>> some lines before c becomes this
>>>>>>>>>> 2264 c = p->cleanups;
>>>>>>>>>>
>>>>>>>>>> p is:
>>>>>>>>>> (gdb) print *p
>>>>>>>>>> $1 = {parent = 0x256f138, child = 0x7fe46c0751c8, sibling =
>>>>>>>>>> 0x7fe4a8096888, ref = 0x7fe4a8069fe8, cleanups = 0x7fe478159748,
>>>>>>>>>> free_cleanups = 0x7fe478159788, allocator = 0x7fe4a803b490,
>>>>>>>>>> subprocesses = 0x0, abort_fn = 0x43da00 <abort_on_oom>,
>>>>>>>>>> user_data = 0x0, tag = 0x502285 "transaction", active =
>>>>>>>>>> 0x7fe478158d70, self = 0x7fe4a8072330,
>>>>>>>>>> self_first_avail = 0x7fe4a80723d0 "X#\a\250\344\177", pre_cleanups =
>>>>>>>>>> 0x7fe4a8072ab8}
>>>>>>>>>>
>>>>>>>>>> wouldn't the error mean that p->cleanups is NULL?
>>>>>>>>>>
>>>>>>>>>> (gdb) print *p->cleanups
>>>>>>>>>> $2 = {next = 0x7fe478159628, data = 0x7fe478159648,
>>>>>>>>>> plain_cleanup_fn =
>>>>>>>>>> 0x7fe4bbd2ffd0 <apr_unix_file_cleanup>,
>>>>>>>>>> child_cleanup_fn = 0x7fe4bbd2ff70 <apr_unix_child_file_cleanup>}
>>>>>>>>>>
>>>>>>>>>> So p->cleanups->data is 0x7fe478159648 and data is 0x7fe4a80723e0?
>>>>>>>>>>
>>>>>>>>>> I don't get why it's segfaulting
>>>>>>>>>>
>>>>>>>>>> Stefan
>>>>>>>>>> Am 21.01.2017 um 09:50 schrieb Yann Ylavic:
>>>>>>>>>>> Hi Stefan,
>>>>>>>>>>>
>>>>>>>>>>> On Sat, Jan 21, 2017 at 9:45 AM, Stefan Priebe
>>>>>>>>>>> <s....@profihost.ag>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> after running the whole night. These are the only ones still
>>>>>>>>>>>> happening.
>>>>>>>>>>>> Should i revert the mpm patch to check whether it's the source?
>>>>>>>>>>>
>>>>>>>>>>> Yes please, we need to determine...
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Yann.
>>>>>>>>>>>
>>>>>>>>
>>>>>>>> Stefan Eissing
>>>>>>>>
>>>>>>>> <green/>bytes GmbH
>>>>>>>> Hafenstrasse 16
>>>>>>>> 48155 Münster
>>>>>>>> www.greenbytes.de
>>>>>>>>
>>
>> Stefan Eissing
>>
>> <green/>bytes GmbH
>> Hafenstrasse 16
>> 48155 Münster
>> www.greenbytes.de
>>
Stefan Eissing
<green/>bytes GmbH
Hafenstrasse 16
48155 Münster
www.greenbytes.de
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Eric Covener <co...@gmail.com>.
On Sun, Jan 22, 2017 at 11:21 AM, Stefan Priebe - Profihost AG
<s....@profihost.ag> wrote:
> Hi Stefan,
>
> no i was mistaken customer isn't using mod_proxy - but I think this is
> the patch causing me problems:
> https://github.com/apache/httpd/commit/a61a4bd02e483fb45d433343740d0130ee3d8d5d
>
> What do you think?
>
If that is the culprit, you could likely minimize it & help confirm with
* Increase MaxSpareTheads to the current value of MaxClients (aka
MaxRequestWorkers)
* Make sure MaxRequestsPerChild is 0.
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Hi Stefan,
no i was mistaken customer isn't using mod_proxy - but I think this is
the patch causing me problems:
https://github.com/apache/httpd/commit/a61a4bd02e483fb45d433343740d0130ee3d8d5d
What do you think?
Greets,
Stefan
Am 22.01.2017 um 17:17 schrieb Stefan Eissing:
>
>> Am 22.01.2017 um 17:14 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>
>> *arg* it's just mod_proxy - just saw thread safety and apr bucket aloc.
>
> ??? Can you elaborate? Is your finding the known hcheck bug or something else?
>
>> Stefan
>>
>> Am 22.01.2017 um 17:06 schrieb Stefan Priebe - Profihost AG:
>>> Looks like others have the same crashes too:
>>> https://bz.apache.org/bugzilla/show_bug.cgi?id=60071
>>> and
>>> https://github.com/apache/httpd/commit/8e63c3c9372cd398f57357099aa941cbba695758
>>>
>>> So it looks like mod_http2 is running fine now. Thanks a lot Stefan.
>>>
>>> Yann i think i can start testing your mpm patch again after the
>>> segfaults in 2.4 branch are fixed.
>>>
>>> Greets,
>>> Stefan
>>>
>>> Am 22.01.2017 um 13:16 schrieb Stefan Priebe:
>>>> Hi,
>>>>
>>>> and a new one but also in ap_start_lingering_close:
>>>>
>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>> #0 apr_palloc (pool=pool@entry=0x7f455805e138, in_size=in_size@entry=32)
>>>> at memory/unix/apr_pools.c:684
>>>> #0 apr_palloc (pool=pool@entry=0x7f455805e138, in_size=in_size@entry=32)
>>>> at memory/unix/apr_pools.c:684
>>>> #1 0x00007f456bc5d8b4 in apr_brigade_create (p=0x7f455805e138,
>>>> list=0x7f45040034e8) at buckets/apr_brigade.c:61
>>>> #2 0x000055e165efa319 in ap_shutdown_conn (c=c@entry=0x7f455805e458,
>>>> flush=flush@entry=1) at connection.c:76
>>>> #3 0x000055e165efa40d in ap_flush_conn (c=0x7f455805e458) at
>>>> connection.c:95
>>>> #4 ap_start_lingering_close (c=0x7f455805e458) at connection.c:145
>>>> #5 0x000055e165f942dd in start_lingering_close_blocking (cs=<optimized
>>>> out>)
>>>> at event.c:876
>>>> #6 process_socket (my_thread_num=<optimized out>,
>>>> my_child_num=<optimized out>, cs=0x7f455805e3c8, sock=<optimized out>,
>>>> p=<optimized out>, thd=<optimized out>) at event.c:1153
>>>> #7 worker_thread (thd=0x7f455805e138, dummy=0x20) at event.c:2001
>>>> #8 0x00007f456b80a0a4 in start_thread ()
>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>> #9 0x00007f456b53f62d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>
>>>> Stefan
>>>>
>>>> Am 21.01.2017 um 19:31 schrieb Stefan Priebe:
>>>>> All last traces come from event, proces_longering_close ap_push_pool but
>>>>> end in different functions. It looks like a race somewhere and it just
>>>>> races at different function in the event of close and pool clear.
>>>>>
>>>>> Might there be two places where the same pool gets cleared?
>>>>>
>>>>> Stefan
>>>>>
>>>>> Am 21.01.2017 um 19:07 schrieb Stefan Priebe:
>>>>>> Hi Stefan,
>>>>>>
>>>>>> thanks. No crashes where h2 comes up. But i still have these and no idea
>>>>>> how to find out who and why they're crashing.
>>>>>>
>>>>>> Using host libthread_db library
>>>>>> "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>>>> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
>>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>>> #0 allocator_free (node=0x0, allocator=0x7f6e08066540)
>>>>>> at memory/unix/apr_pools.c:381
>>>>>> #0 allocator_free (node=0x0, allocator=0x7f6e08066540)
>>>>>> at memory/unix/apr_pools.c:381
>>>>>> #1 apr_pool_clear (pool=0x7f6e0808d238) at memory/unix/apr_pools.c:793
>>>>>> #2 0x00000000004fe528 in ap_push_pool (queue_info=0x0,
>>>>>> pool_to_recycle=0x7f6e08066548) at fdqueue.c:234
>>>>>> #3 0x00000000004fa2c8 in process_lingering_close (cs=0x7f6e0808d4c8,
>>>>>> pfd=0x1d3bf98) at event.c:1439
>>>>>> #4 0x00000000004fd410 in listener_thread (thd=0x1d3cb70,
>>>>>> dummy=0x7f6e0808d4c8)
>>>>>> at event.c:1704
>>>>>> #5 0x00007f6e1aed20a4 in start_thread ()
>>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>> #6 0x00007f6e1aa0362d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>> (gdb) (gdb) quit
>>>>>>
>>>>>> Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
>>>>>> /usr/lib/debug//usr/local/apache2/bin/httpd...done.
>>>>>> done.
>>>>>> [Thread debugging using libthread_db enabled]
>>>>>> Using host libthread_db library
>>>>>> "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>>>> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
>>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>>> #0 allocator_free (node=0x0, allocator=0x7f6e08053ae0)
>>>>>> at memory/unix/apr_pools.c:381
>>>>>> #0 allocator_free (node=0x0, allocator=0x7f6e08053ae0)
>>>>>> at memory/unix/apr_pools.c:381
>>>>>> #1 apr_pool_clear (pool=0x7f6e08076bb8) at memory/unix/apr_pools.c:793
>>>>>> #2 0x00000000004fe528 in ap_push_pool (queue_info=0x0,
>>>>>> pool_to_recycle=0x7f6e08053ae8) at fdqueue.c:234
>>>>>> #3 0x00000000004fa2c8 in process_lingering_close (cs=0x7f6e08076e48,
>>>>>> pfd=0x1d3bf98) at event.c:1439
>>>>>> #4 0x00000000004fd410 in listener_thread (thd=0x1d3cb70,
>>>>>> dummy=0x7f6e08076e48)
>>>>>> at event.c:1704
>>>>>> #5 0x00007f6e1aed20a4 in start_thread ()
>>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>> #6 0x00007f6e1aa0362d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>> (gdb) (gdb) quit
>>>>>>
>>>>>> Stefan
>>>>>>
>>>>>> Am 21.01.2017 um 17:03 schrieb Stefan Eissing:
>>>>>>> Stefan,
>>>>>>>
>>>>>>> made a release at https://github.com/icing/mod_h2/releases/tag/v1.8.9
>>>>>>> with all patches and improved (hopefully) on them a bit. If you dare
>>>>>>> to drop that into your installation, that'd be great.
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Stefan
>>>>>>>
>>>>>>>> Am 21.01.2017 um 15:25 schrieb Stefan Priebe <s....@profihost.ag>:
>>>>>>>>
>>>>>>>> and i got another crash here:
>>>>>>>>
>>>>>>>> 2346 static void run_cleanups(cleanup_t **cref)
>>>>>>>> 2347 {
>>>>>>>> 2348 cleanup_t *c = *cref;
>>>>>>>> 2349
>>>>>>>> 2350 while (c) {
>>>>>>>> 2351 *cref = c->next;
>>>>>>>> 2352 (*c->plain_cleanup_fn)((void *)c->data); <== here
>>>>>>>> 2353 c = *cref;
>>>>>>>> 2354
>>>>>>>>
>>>>>>>> which looks similar to the other crash.
>>>>>>>>
>>>>>>>> #0 0x00007fe4bbd33e1b in run_cleanups (cref=<optimized out>) at
>>>>>>>> memory/unix/apr_pools.c:2352
>>>>>>>> #1 apr_pool_clear (pool=0x7fe4a804dac8) at
>>>>>>>> memory/unix/apr_pools.c:772
>>>>>>>> #2 0x00000000004feb38 in ap_push_pool
>>>>>>>> (queue_info=0x6d616e79642d3733, pool_to_recycle=0x2) at fdqueue.c:234
>>>>>>>> #3 0x00000000004fa8d8 in process_lingering_close (cs=0x7fe4a804dd58,
>>>>>>>> pfd=0x25d3f98) at event.c:1439
>>>>>>>>
>>>>>>>> Details:
>>>>>>>> (gdb) print c
>>>>>>>> $1 = (cleanup_t *) 0x7fe4a804e9f0
>>>>>>>> (gdb) print *c
>>>>>>>> $2 = {next = 0x7fe4a804e870, data = 0x6d616e79642d3733,
>>>>>>>> plain_cleanup_fn = 0x392d3734322e6369,
>>>>>>>> child_cleanup_fn = 0x617465722e722d35}
>>>>>>>> (gdb) print *c->data
>>>>>>>> Attempt to dereference a generic pointer.
>>>>>>>> (gdb) print *c->plain_cleanup_fn
>>>>>>>> Cannot access memory at address 0x392d3734322e6369
>>>>>>>> (gdb)
>>>>>>>>
>>>>>>>> Stefan
>>>>>>>>
>>>>>>>> Am 21.01.2017 um 15:18 schrieb Stefan Priebe:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> #0 apr_pool_cleanup_kill (p=0x7fe4a8072358,
>>>>>>>>> data=data@entry=0x7fe4a80723e0,
>>>>>>>>> cleanup_fn=cleanup_fn@entry=0x7fe4bbd38a40 <socket_cleanup>) at
>>>>>>>>> memory/unix/apr_pools.c:2276
>>>>>>>>>
>>>>>>>>> it crashes here in apr:
>>>>>>>>> 2276 if (c->data == data && c->plain_cleanup_fn ==
>>>>>>>>> cleanup_fn) {
>>>>>>>>>
>>>>>>>>> some lines before c becomes this
>>>>>>>>> 2264 c = p->cleanups;
>>>>>>>>>
>>>>>>>>> p is:
>>>>>>>>> (gdb) print *p
>>>>>>>>> $1 = {parent = 0x256f138, child = 0x7fe46c0751c8, sibling =
>>>>>>>>> 0x7fe4a8096888, ref = 0x7fe4a8069fe8, cleanups = 0x7fe478159748,
>>>>>>>>> free_cleanups = 0x7fe478159788, allocator = 0x7fe4a803b490,
>>>>>>>>> subprocesses = 0x0, abort_fn = 0x43da00 <abort_on_oom>,
>>>>>>>>> user_data = 0x0, tag = 0x502285 "transaction", active =
>>>>>>>>> 0x7fe478158d70, self = 0x7fe4a8072330,
>>>>>>>>> self_first_avail = 0x7fe4a80723d0 "X#\a\250\344\177", pre_cleanups =
>>>>>>>>> 0x7fe4a8072ab8}
>>>>>>>>>
>>>>>>>>> wouldn't the error mean that p->cleanups is NULL?
>>>>>>>>>
>>>>>>>>> (gdb) print *p->cleanups
>>>>>>>>> $2 = {next = 0x7fe478159628, data = 0x7fe478159648,
>>>>>>>>> plain_cleanup_fn =
>>>>>>>>> 0x7fe4bbd2ffd0 <apr_unix_file_cleanup>,
>>>>>>>>> child_cleanup_fn = 0x7fe4bbd2ff70 <apr_unix_child_file_cleanup>}
>>>>>>>>>
>>>>>>>>> So p->cleanups->data is 0x7fe478159648 and data is 0x7fe4a80723e0?
>>>>>>>>>
>>>>>>>>> I don't get why it's segfaulting
>>>>>>>>>
>>>>>>>>> Stefan
>>>>>>>>> Am 21.01.2017 um 09:50 schrieb Yann Ylavic:
>>>>>>>>>> Hi Stefan,
>>>>>>>>>>
>>>>>>>>>> On Sat, Jan 21, 2017 at 9:45 AM, Stefan Priebe
>>>>>>>>>> <s....@profihost.ag>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> after running the whole night. These are the only ones still
>>>>>>>>>>> happening.
>>>>>>>>>>> Should i revert the mpm patch to check whether it's the source?
>>>>>>>>>>
>>>>>>>>>> Yes please, we need to determine...
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Yann.
>>>>>>>>>>
>>>>>>>
>>>>>>> Stefan Eissing
>>>>>>>
>>>>>>> <green/>bytes GmbH
>>>>>>> Hafenstrasse 16
>>>>>>> 48155 M�nster
>>>>>>> www.greenbytes.de
>>>>>>>
>
> Stefan Eissing
>
> <green/>bytes GmbH
> Hafenstrasse 16
> 48155 M�nster
> www.greenbytes.de
>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Eissing <st...@greenbytes.de>.
> Am 22.01.2017 um 17:14 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>
> *arg* it's just mod_proxy - just saw thread safety and apr bucket aloc.
??? Can you elaborate? Is your finding the known hcheck bug or something else?
> Stefan
>
> Am 22.01.2017 um 17:06 schrieb Stefan Priebe - Profihost AG:
>> Looks like others have the same crashes too:
>> https://bz.apache.org/bugzilla/show_bug.cgi?id=60071
>> and
>> https://github.com/apache/httpd/commit/8e63c3c9372cd398f57357099aa941cbba695758
>>
>> So it looks like mod_http2 is running fine now. Thanks a lot Stefan.
>>
>> Yann i think i can start testing your mpm patch again after the
>> segfaults in 2.4 branch are fixed.
>>
>> Greets,
>> Stefan
>>
>> Am 22.01.2017 um 13:16 schrieb Stefan Priebe:
>>> Hi,
>>>
>>> and a new one but also in ap_start_lingering_close:
>>>
>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>> #0 apr_palloc (pool=pool@entry=0x7f455805e138, in_size=in_size@entry=32)
>>> at memory/unix/apr_pools.c:684
>>> #0 apr_palloc (pool=pool@entry=0x7f455805e138, in_size=in_size@entry=32)
>>> at memory/unix/apr_pools.c:684
>>> #1 0x00007f456bc5d8b4 in apr_brigade_create (p=0x7f455805e138,
>>> list=0x7f45040034e8) at buckets/apr_brigade.c:61
>>> #2 0x000055e165efa319 in ap_shutdown_conn (c=c@entry=0x7f455805e458,
>>> flush=flush@entry=1) at connection.c:76
>>> #3 0x000055e165efa40d in ap_flush_conn (c=0x7f455805e458) at
>>> connection.c:95
>>> #4 ap_start_lingering_close (c=0x7f455805e458) at connection.c:145
>>> #5 0x000055e165f942dd in start_lingering_close_blocking (cs=<optimized
>>> out>)
>>> at event.c:876
>>> #6 process_socket (my_thread_num=<optimized out>,
>>> my_child_num=<optimized out>, cs=0x7f455805e3c8, sock=<optimized out>,
>>> p=<optimized out>, thd=<optimized out>) at event.c:1153
>>> #7 worker_thread (thd=0x7f455805e138, dummy=0x20) at event.c:2001
>>> #8 0x00007f456b80a0a4 in start_thread ()
>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>> #9 0x00007f456b53f62d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>
>>> Stefan
>>>
>>> Am 21.01.2017 um 19:31 schrieb Stefan Priebe:
>>>> All last traces come from event, proces_longering_close ap_push_pool but
>>>> end in different functions. It looks like a race somewhere and it just
>>>> races at different function in the event of close and pool clear.
>>>>
>>>> Might there be two places where the same pool gets cleared?
>>>>
>>>> Stefan
>>>>
>>>> Am 21.01.2017 um 19:07 schrieb Stefan Priebe:
>>>>> Hi Stefan,
>>>>>
>>>>> thanks. No crashes where h2 comes up. But i still have these and no idea
>>>>> how to find out who and why they're crashing.
>>>>>
>>>>> Using host libthread_db library
>>>>> "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>>> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>> #0 allocator_free (node=0x0, allocator=0x7f6e08066540)
>>>>> at memory/unix/apr_pools.c:381
>>>>> #0 allocator_free (node=0x0, allocator=0x7f6e08066540)
>>>>> at memory/unix/apr_pools.c:381
>>>>> #1 apr_pool_clear (pool=0x7f6e0808d238) at memory/unix/apr_pools.c:793
>>>>> #2 0x00000000004fe528 in ap_push_pool (queue_info=0x0,
>>>>> pool_to_recycle=0x7f6e08066548) at fdqueue.c:234
>>>>> #3 0x00000000004fa2c8 in process_lingering_close (cs=0x7f6e0808d4c8,
>>>>> pfd=0x1d3bf98) at event.c:1439
>>>>> #4 0x00000000004fd410 in listener_thread (thd=0x1d3cb70,
>>>>> dummy=0x7f6e0808d4c8)
>>>>> at event.c:1704
>>>>> #5 0x00007f6e1aed20a4 in start_thread ()
>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>> #6 0x00007f6e1aa0362d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>> (gdb) (gdb) quit
>>>>>
>>>>> Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
>>>>> /usr/lib/debug//usr/local/apache2/bin/httpd...done.
>>>>> done.
>>>>> [Thread debugging using libthread_db enabled]
>>>>> Using host libthread_db library
>>>>> "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>>> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>> #0 allocator_free (node=0x0, allocator=0x7f6e08053ae0)
>>>>> at memory/unix/apr_pools.c:381
>>>>> #0 allocator_free (node=0x0, allocator=0x7f6e08053ae0)
>>>>> at memory/unix/apr_pools.c:381
>>>>> #1 apr_pool_clear (pool=0x7f6e08076bb8) at memory/unix/apr_pools.c:793
>>>>> #2 0x00000000004fe528 in ap_push_pool (queue_info=0x0,
>>>>> pool_to_recycle=0x7f6e08053ae8) at fdqueue.c:234
>>>>> #3 0x00000000004fa2c8 in process_lingering_close (cs=0x7f6e08076e48,
>>>>> pfd=0x1d3bf98) at event.c:1439
>>>>> #4 0x00000000004fd410 in listener_thread (thd=0x1d3cb70,
>>>>> dummy=0x7f6e08076e48)
>>>>> at event.c:1704
>>>>> #5 0x00007f6e1aed20a4 in start_thread ()
>>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>> #6 0x00007f6e1aa0362d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>> (gdb) (gdb) quit
>>>>>
>>>>> Stefan
>>>>>
>>>>> Am 21.01.2017 um 17:03 schrieb Stefan Eissing:
>>>>>> Stefan,
>>>>>>
>>>>>> made a release at https://github.com/icing/mod_h2/releases/tag/v1.8.9
>>>>>> with all patches and improved (hopefully) on them a bit. If you dare
>>>>>> to drop that into your installation, that'd be great.
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Stefan
>>>>>>
>>>>>>> Am 21.01.2017 um 15:25 schrieb Stefan Priebe <s....@profihost.ag>:
>>>>>>>
>>>>>>> and i got another crash here:
>>>>>>>
>>>>>>> 2346 static void run_cleanups(cleanup_t **cref)
>>>>>>> 2347 {
>>>>>>> 2348 cleanup_t *c = *cref;
>>>>>>> 2349
>>>>>>> 2350 while (c) {
>>>>>>> 2351 *cref = c->next;
>>>>>>> 2352 (*c->plain_cleanup_fn)((void *)c->data); <== here
>>>>>>> 2353 c = *cref;
>>>>>>> 2354
>>>>>>>
>>>>>>> which looks similar to the other crash.
>>>>>>>
>>>>>>> #0 0x00007fe4bbd33e1b in run_cleanups (cref=<optimized out>) at
>>>>>>> memory/unix/apr_pools.c:2352
>>>>>>> #1 apr_pool_clear (pool=0x7fe4a804dac8) at
>>>>>>> memory/unix/apr_pools.c:772
>>>>>>> #2 0x00000000004feb38 in ap_push_pool
>>>>>>> (queue_info=0x6d616e79642d3733, pool_to_recycle=0x2) at fdqueue.c:234
>>>>>>> #3 0x00000000004fa8d8 in process_lingering_close (cs=0x7fe4a804dd58,
>>>>>>> pfd=0x25d3f98) at event.c:1439
>>>>>>>
>>>>>>> Details:
>>>>>>> (gdb) print c
>>>>>>> $1 = (cleanup_t *) 0x7fe4a804e9f0
>>>>>>> (gdb) print *c
>>>>>>> $2 = {next = 0x7fe4a804e870, data = 0x6d616e79642d3733,
>>>>>>> plain_cleanup_fn = 0x392d3734322e6369,
>>>>>>> child_cleanup_fn = 0x617465722e722d35}
>>>>>>> (gdb) print *c->data
>>>>>>> Attempt to dereference a generic pointer.
>>>>>>> (gdb) print *c->plain_cleanup_fn
>>>>>>> Cannot access memory at address 0x392d3734322e6369
>>>>>>> (gdb)
>>>>>>>
>>>>>>> Stefan
>>>>>>>
>>>>>>> Am 21.01.2017 um 15:18 schrieb Stefan Priebe:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> #0 apr_pool_cleanup_kill (p=0x7fe4a8072358,
>>>>>>>> data=data@entry=0x7fe4a80723e0,
>>>>>>>> cleanup_fn=cleanup_fn@entry=0x7fe4bbd38a40 <socket_cleanup>) at
>>>>>>>> memory/unix/apr_pools.c:2276
>>>>>>>>
>>>>>>>> it crashes here in apr:
>>>>>>>> 2276 if (c->data == data && c->plain_cleanup_fn ==
>>>>>>>> cleanup_fn) {
>>>>>>>>
>>>>>>>> some lines before c becomes this
>>>>>>>> 2264 c = p->cleanups;
>>>>>>>>
>>>>>>>> p is:
>>>>>>>> (gdb) print *p
>>>>>>>> $1 = {parent = 0x256f138, child = 0x7fe46c0751c8, sibling =
>>>>>>>> 0x7fe4a8096888, ref = 0x7fe4a8069fe8, cleanups = 0x7fe478159748,
>>>>>>>> free_cleanups = 0x7fe478159788, allocator = 0x7fe4a803b490,
>>>>>>>> subprocesses = 0x0, abort_fn = 0x43da00 <abort_on_oom>,
>>>>>>>> user_data = 0x0, tag = 0x502285 "transaction", active =
>>>>>>>> 0x7fe478158d70, self = 0x7fe4a8072330,
>>>>>>>> self_first_avail = 0x7fe4a80723d0 "X#\a\250\344\177", pre_cleanups =
>>>>>>>> 0x7fe4a8072ab8}
>>>>>>>>
>>>>>>>> wouldn't the error mean that p->cleanups is NULL?
>>>>>>>>
>>>>>>>> (gdb) print *p->cleanups
>>>>>>>> $2 = {next = 0x7fe478159628, data = 0x7fe478159648,
>>>>>>>> plain_cleanup_fn =
>>>>>>>> 0x7fe4bbd2ffd0 <apr_unix_file_cleanup>,
>>>>>>>> child_cleanup_fn = 0x7fe4bbd2ff70 <apr_unix_child_file_cleanup>}
>>>>>>>>
>>>>>>>> So p->cleanups->data is 0x7fe478159648 and data is 0x7fe4a80723e0?
>>>>>>>>
>>>>>>>> I don't get why it's segfaulting
>>>>>>>>
>>>>>>>> Stefan
>>>>>>>> Am 21.01.2017 um 09:50 schrieb Yann Ylavic:
>>>>>>>>> Hi Stefan,
>>>>>>>>>
>>>>>>>>> On Sat, Jan 21, 2017 at 9:45 AM, Stefan Priebe
>>>>>>>>> <s....@profihost.ag>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> after running the whole night. These are the only ones still
>>>>>>>>>> happening.
>>>>>>>>>> Should i revert the mpm patch to check whether it's the source?
>>>>>>>>>
>>>>>>>>> Yes please, we need to determine...
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Yann.
>>>>>>>>>
>>>>>>
>>>>>> Stefan Eissing
>>>>>>
>>>>>> <green/>bytes GmbH
>>>>>> Hafenstrasse 16
>>>>>> 48155 Münster
>>>>>> www.greenbytes.de
>>>>>>
Stefan Eissing
<green/>bytes GmbH
Hafenstrasse 16
48155 Münster
www.greenbytes.de
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
*arg* it's just mod_proxy - just saw thread safety and apr bucket aloc.
Stefan
Am 22.01.2017 um 17:06 schrieb Stefan Priebe - Profihost AG:
> Looks like others have the same crashes too:
> https://bz.apache.org/bugzilla/show_bug.cgi?id=60071
> and
> https://github.com/apache/httpd/commit/8e63c3c9372cd398f57357099aa941cbba695758
>
> So it looks like mod_http2 is running fine now. Thanks a lot Stefan.
>
> Yann i think i can start testing your mpm patch again after the
> segfaults in 2.4 branch are fixed.
>
> Greets,
> Stefan
>
> Am 22.01.2017 um 13:16 schrieb Stefan Priebe:
>> Hi,
>>
>> and a new one but also in ap_start_lingering_close:
>>
>> Program terminated with signal SIGSEGV, Segmentation fault.
>> #0 apr_palloc (pool=pool@entry=0x7f455805e138, in_size=in_size@entry=32)
>> at memory/unix/apr_pools.c:684
>> #0 apr_palloc (pool=pool@entry=0x7f455805e138, in_size=in_size@entry=32)
>> at memory/unix/apr_pools.c:684
>> #1 0x00007f456bc5d8b4 in apr_brigade_create (p=0x7f455805e138,
>> list=0x7f45040034e8) at buckets/apr_brigade.c:61
>> #2 0x000055e165efa319 in ap_shutdown_conn (c=c@entry=0x7f455805e458,
>> flush=flush@entry=1) at connection.c:76
>> #3 0x000055e165efa40d in ap_flush_conn (c=0x7f455805e458) at
>> connection.c:95
>> #4 ap_start_lingering_close (c=0x7f455805e458) at connection.c:145
>> #5 0x000055e165f942dd in start_lingering_close_blocking (cs=<optimized
>> out>)
>> at event.c:876
>> #6 process_socket (my_thread_num=<optimized out>,
>> my_child_num=<optimized out>, cs=0x7f455805e3c8, sock=<optimized out>,
>> p=<optimized out>, thd=<optimized out>) at event.c:1153
>> #7 worker_thread (thd=0x7f455805e138, dummy=0x20) at event.c:2001
>> #8 0x00007f456b80a0a4 in start_thread ()
>> from /lib/x86_64-linux-gnu/libpthread.so.0
>> #9 0x00007f456b53f62d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>
>> Stefan
>>
>> Am 21.01.2017 um 19:31 schrieb Stefan Priebe:
>>> All last traces come from event, proces_longering_close ap_push_pool but
>>> end in different functions. It looks like a race somewhere and it just
>>> races at different function in the event of close and pool clear.
>>>
>>> Might there be two places where the same pool gets cleared?
>>>
>>> Stefan
>>>
>>> Am 21.01.2017 um 19:07 schrieb Stefan Priebe:
>>>> Hi Stefan,
>>>>
>>>> thanks. No crashes where h2 comes up. But i still have these and no idea
>>>> how to find out who and why they're crashing.
>>>>
>>>> Using host libthread_db library
>>>> "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>> #0 allocator_free (node=0x0, allocator=0x7f6e08066540)
>>>> at memory/unix/apr_pools.c:381
>>>> #0 allocator_free (node=0x0, allocator=0x7f6e08066540)
>>>> at memory/unix/apr_pools.c:381
>>>> #1 apr_pool_clear (pool=0x7f6e0808d238) at memory/unix/apr_pools.c:793
>>>> #2 0x00000000004fe528 in ap_push_pool (queue_info=0x0,
>>>> pool_to_recycle=0x7f6e08066548) at fdqueue.c:234
>>>> #3 0x00000000004fa2c8 in process_lingering_close (cs=0x7f6e0808d4c8,
>>>> pfd=0x1d3bf98) at event.c:1439
>>>> #4 0x00000000004fd410 in listener_thread (thd=0x1d3cb70,
>>>> dummy=0x7f6e0808d4c8)
>>>> at event.c:1704
>>>> #5 0x00007f6e1aed20a4 in start_thread ()
>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>> #6 0x00007f6e1aa0362d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>> (gdb) (gdb) quit
>>>>
>>>> Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
>>>> /usr/lib/debug//usr/local/apache2/bin/httpd...done.
>>>> done.
>>>> [Thread debugging using libthread_db enabled]
>>>> Using host libthread_db library
>>>> "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>> #0 allocator_free (node=0x0, allocator=0x7f6e08053ae0)
>>>> at memory/unix/apr_pools.c:381
>>>> #0 allocator_free (node=0x0, allocator=0x7f6e08053ae0)
>>>> at memory/unix/apr_pools.c:381
>>>> #1 apr_pool_clear (pool=0x7f6e08076bb8) at memory/unix/apr_pools.c:793
>>>> #2 0x00000000004fe528 in ap_push_pool (queue_info=0x0,
>>>> pool_to_recycle=0x7f6e08053ae8) at fdqueue.c:234
>>>> #3 0x00000000004fa2c8 in process_lingering_close (cs=0x7f6e08076e48,
>>>> pfd=0x1d3bf98) at event.c:1439
>>>> #4 0x00000000004fd410 in listener_thread (thd=0x1d3cb70,
>>>> dummy=0x7f6e08076e48)
>>>> at event.c:1704
>>>> #5 0x00007f6e1aed20a4 in start_thread ()
>>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>>> #6 0x00007f6e1aa0362d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>> (gdb) (gdb) quit
>>>>
>>>> Stefan
>>>>
>>>> Am 21.01.2017 um 17:03 schrieb Stefan Eissing:
>>>>> Stefan,
>>>>>
>>>>> made a release at https://github.com/icing/mod_h2/releases/tag/v1.8.9
>>>>> with all patches and improved (hopefully) on them a bit. If you dare
>>>>> to drop that into your installation, that'd be great.
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Stefan
>>>>>
>>>>>> Am 21.01.2017 um 15:25 schrieb Stefan Priebe <s....@profihost.ag>:
>>>>>>
>>>>>> and i got another crash here:
>>>>>>
>>>>>> 2346 static void run_cleanups(cleanup_t **cref)
>>>>>> 2347 {
>>>>>> 2348 cleanup_t *c = *cref;
>>>>>> 2349
>>>>>> 2350 while (c) {
>>>>>> 2351 *cref = c->next;
>>>>>> 2352 (*c->plain_cleanup_fn)((void *)c->data); <== here
>>>>>> 2353 c = *cref;
>>>>>> 2354
>>>>>>
>>>>>> which looks similar to the other crash.
>>>>>>
>>>>>> #0 0x00007fe4bbd33e1b in run_cleanups (cref=<optimized out>) at
>>>>>> memory/unix/apr_pools.c:2352
>>>>>> #1 apr_pool_clear (pool=0x7fe4a804dac8) at
>>>>>> memory/unix/apr_pools.c:772
>>>>>> #2 0x00000000004feb38 in ap_push_pool
>>>>>> (queue_info=0x6d616e79642d3733, pool_to_recycle=0x2) at fdqueue.c:234
>>>>>> #3 0x00000000004fa8d8 in process_lingering_close (cs=0x7fe4a804dd58,
>>>>>> pfd=0x25d3f98) at event.c:1439
>>>>>>
>>>>>> Details:
>>>>>> (gdb) print c
>>>>>> $1 = (cleanup_t *) 0x7fe4a804e9f0
>>>>>> (gdb) print *c
>>>>>> $2 = {next = 0x7fe4a804e870, data = 0x6d616e79642d3733,
>>>>>> plain_cleanup_fn = 0x392d3734322e6369,
>>>>>> child_cleanup_fn = 0x617465722e722d35}
>>>>>> (gdb) print *c->data
>>>>>> Attempt to dereference a generic pointer.
>>>>>> (gdb) print *c->plain_cleanup_fn
>>>>>> Cannot access memory at address 0x392d3734322e6369
>>>>>> (gdb)
>>>>>>
>>>>>> Stefan
>>>>>>
>>>>>> Am 21.01.2017 um 15:18 schrieb Stefan Priebe:
>>>>>>> Hi,
>>>>>>>
>>>>>>> #0 apr_pool_cleanup_kill (p=0x7fe4a8072358,
>>>>>>> data=data@entry=0x7fe4a80723e0,
>>>>>>> cleanup_fn=cleanup_fn@entry=0x7fe4bbd38a40 <socket_cleanup>) at
>>>>>>> memory/unix/apr_pools.c:2276
>>>>>>>
>>>>>>> it crashes here in apr:
>>>>>>> 2276 if (c->data == data && c->plain_cleanup_fn ==
>>>>>>> cleanup_fn) {
>>>>>>>
>>>>>>> some lines before c becomes this
>>>>>>> 2264 c = p->cleanups;
>>>>>>>
>>>>>>> p is:
>>>>>>> (gdb) print *p
>>>>>>> $1 = {parent = 0x256f138, child = 0x7fe46c0751c8, sibling =
>>>>>>> 0x7fe4a8096888, ref = 0x7fe4a8069fe8, cleanups = 0x7fe478159748,
>>>>>>> free_cleanups = 0x7fe478159788, allocator = 0x7fe4a803b490,
>>>>>>> subprocesses = 0x0, abort_fn = 0x43da00 <abort_on_oom>,
>>>>>>> user_data = 0x0, tag = 0x502285 "transaction", active =
>>>>>>> 0x7fe478158d70, self = 0x7fe4a8072330,
>>>>>>> self_first_avail = 0x7fe4a80723d0 "X#\a\250\344\177", pre_cleanups =
>>>>>>> 0x7fe4a8072ab8}
>>>>>>>
>>>>>>> wouldn't the error mean that p->cleanups is NULL?
>>>>>>>
>>>>>>> (gdb) print *p->cleanups
>>>>>>> $2 = {next = 0x7fe478159628, data = 0x7fe478159648,
>>>>>>> plain_cleanup_fn =
>>>>>>> 0x7fe4bbd2ffd0 <apr_unix_file_cleanup>,
>>>>>>> child_cleanup_fn = 0x7fe4bbd2ff70 <apr_unix_child_file_cleanup>}
>>>>>>>
>>>>>>> So p->cleanups->data is 0x7fe478159648 and data is 0x7fe4a80723e0?
>>>>>>>
>>>>>>> I don't get why it's segfaulting
>>>>>>>
>>>>>>> Stefan
>>>>>>> Am 21.01.2017 um 09:50 schrieb Yann Ylavic:
>>>>>>>> Hi Stefan,
>>>>>>>>
>>>>>>>> On Sat, Jan 21, 2017 at 9:45 AM, Stefan Priebe
>>>>>>>> <s....@profihost.ag>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> after running the whole night. These are the only ones still
>>>>>>>>> happening.
>>>>>>>>> Should i revert the mpm patch to check whether it's the source?
>>>>>>>>
>>>>>>>> Yes please, we need to determine...
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Yann.
>>>>>>>>
>>>>>
>>>>> Stefan Eissing
>>>>>
>>>>> <green/>bytes GmbH
>>>>> Hafenstrasse 16
>>>>> 48155 M�nster
>>>>> www.greenbytes.de
>>>>>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe <s....@profihost.ag>.
Am 21.01.2017 um 17:03 schrieb Stefan Eissing:
> Stefan,
>
> made a release at https://github.com/icing/mod_h2/releases/tag/v1.8.9 with all patches and improved (hopefully) on them a bit. If you dare to drop that into your installation, that'd be great.
thx it's running - will report back shortly.
Stefan
> Cheers,
>
> Stefan
>
>> Am 21.01.2017 um 15:25 schrieb Stefan Priebe <s....@profihost.ag>:
>>
>> and i got another crash here:
>>
>> 2346 static void run_cleanups(cleanup_t **cref)
>> 2347 {
>> 2348 cleanup_t *c = *cref;
>> 2349
>> 2350 while (c) {
>> 2351 *cref = c->next;
>> 2352 (*c->plain_cleanup_fn)((void *)c->data); <== here
>> 2353 c = *cref;
>> 2354
>>
>> which looks similar to the other crash.
>>
>> #0 0x00007fe4bbd33e1b in run_cleanups (cref=<optimized out>) at memory/unix/apr_pools.c:2352
>> #1 apr_pool_clear (pool=0x7fe4a804dac8) at memory/unix/apr_pools.c:772
>> #2 0x00000000004feb38 in ap_push_pool (queue_info=0x6d616e79642d3733, pool_to_recycle=0x2) at fdqueue.c:234
>> #3 0x00000000004fa8d8 in process_lingering_close (cs=0x7fe4a804dd58, pfd=0x25d3f98) at event.c:1439
>>
>> Details:
>> (gdb) print c
>> $1 = (cleanup_t *) 0x7fe4a804e9f0
>> (gdb) print *c
>> $2 = {next = 0x7fe4a804e870, data = 0x6d616e79642d3733, plain_cleanup_fn = 0x392d3734322e6369,
>> child_cleanup_fn = 0x617465722e722d35}
>> (gdb) print *c->data
>> Attempt to dereference a generic pointer.
>> (gdb) print *c->plain_cleanup_fn
>> Cannot access memory at address 0x392d3734322e6369
>> (gdb)
>>
>> Stefan
>>
>> Am 21.01.2017 um 15:18 schrieb Stefan Priebe:
>>> Hi,
>>>
>>> #0 apr_pool_cleanup_kill (p=0x7fe4a8072358,
>>> data=data@entry=0x7fe4a80723e0,
>>> cleanup_fn=cleanup_fn@entry=0x7fe4bbd38a40 <socket_cleanup>) at
>>> memory/unix/apr_pools.c:2276
>>>
>>> it crashes here in apr:
>>> 2276 if (c->data == data && c->plain_cleanup_fn == cleanup_fn) {
>>>
>>> some lines before c becomes this
>>> 2264 c = p->cleanups;
>>>
>>> p is:
>>> (gdb) print *p
>>> $1 = {parent = 0x256f138, child = 0x7fe46c0751c8, sibling =
>>> 0x7fe4a8096888, ref = 0x7fe4a8069fe8, cleanups = 0x7fe478159748,
>>> free_cleanups = 0x7fe478159788, allocator = 0x7fe4a803b490,
>>> subprocesses = 0x0, abort_fn = 0x43da00 <abort_on_oom>,
>>> user_data = 0x0, tag = 0x502285 "transaction", active =
>>> 0x7fe478158d70, self = 0x7fe4a8072330,
>>> self_first_avail = 0x7fe4a80723d0 "X#\a\250\344\177", pre_cleanups =
>>> 0x7fe4a8072ab8}
>>>
>>> wouldn't the error mean that p->cleanups is NULL?
>>>
>>> (gdb) print *p->cleanups
>>> $2 = {next = 0x7fe478159628, data = 0x7fe478159648, plain_cleanup_fn =
>>> 0x7fe4bbd2ffd0 <apr_unix_file_cleanup>,
>>> child_cleanup_fn = 0x7fe4bbd2ff70 <apr_unix_child_file_cleanup>}
>>>
>>> So p->cleanups->data is 0x7fe478159648 and data is 0x7fe4a80723e0?
>>>
>>> I don't get why it's segfaulting
>>>
>>> Stefan
>>> Am 21.01.2017 um 09:50 schrieb Yann Ylavic:
>>>> Hi Stefan,
>>>>
>>>> On Sat, Jan 21, 2017 at 9:45 AM, Stefan Priebe <s....@profihost.ag>
>>>> wrote:
>>>>>
>>>>> after running the whole night. These are the only ones still happening.
>>>>> Should i revert the mpm patch to check whether it's the source?
>>>>
>>>> Yes please, we need to determine...
>>>>
>>>> Thanks,
>>>> Yann.
>>>>
>
> Stefan Eissing
>
> <green/>bytes GmbH
> Hafenstrasse 16
> 48155 M�nster
> www.greenbytes.de
>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Looks like others have the same crashes too:
https://bz.apache.org/bugzilla/show_bug.cgi?id=60071
and
https://github.com/apache/httpd/commit/8e63c3c9372cd398f57357099aa941cbba695758
So it looks like mod_http2 is running fine now. Thanks a lot Stefan.
Yann i think i can start testing your mpm patch again after the
segfaults in 2.4 branch are fixed.
Greets,
Stefan
Am 22.01.2017 um 13:16 schrieb Stefan Priebe:
> Hi,
>
> and a new one but also in ap_start_lingering_close:
>
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0 apr_palloc (pool=pool@entry=0x7f455805e138, in_size=in_size@entry=32)
> at memory/unix/apr_pools.c:684
> #0 apr_palloc (pool=pool@entry=0x7f455805e138, in_size=in_size@entry=32)
> at memory/unix/apr_pools.c:684
> #1 0x00007f456bc5d8b4 in apr_brigade_create (p=0x7f455805e138,
> list=0x7f45040034e8) at buckets/apr_brigade.c:61
> #2 0x000055e165efa319 in ap_shutdown_conn (c=c@entry=0x7f455805e458,
> flush=flush@entry=1) at connection.c:76
> #3 0x000055e165efa40d in ap_flush_conn (c=0x7f455805e458) at
> connection.c:95
> #4 ap_start_lingering_close (c=0x7f455805e458) at connection.c:145
> #5 0x000055e165f942dd in start_lingering_close_blocking (cs=<optimized
> out>)
> at event.c:876
> #6 process_socket (my_thread_num=<optimized out>,
> my_child_num=<optimized out>, cs=0x7f455805e3c8, sock=<optimized out>,
> p=<optimized out>, thd=<optimized out>) at event.c:1153
> #7 worker_thread (thd=0x7f455805e138, dummy=0x20) at event.c:2001
> #8 0x00007f456b80a0a4 in start_thread ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #9 0x00007f456b53f62d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Stefan
>
> Am 21.01.2017 um 19:31 schrieb Stefan Priebe:
>> All last traces come from event, proces_longering_close ap_push_pool but
>> end in different functions. It looks like a race somewhere and it just
>> races at different function in the event of close and pool clear.
>>
>> Might there be two places where the same pool gets cleared?
>>
>> Stefan
>>
>> Am 21.01.2017 um 19:07 schrieb Stefan Priebe:
>>> Hi Stefan,
>>>
>>> thanks. No crashes where h2 comes up. But i still have these and no idea
>>> how to find out who and why they're crashing.
>>>
>>> Using host libthread_db library
>>> "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>> #0 allocator_free (node=0x0, allocator=0x7f6e08066540)
>>> at memory/unix/apr_pools.c:381
>>> #0 allocator_free (node=0x0, allocator=0x7f6e08066540)
>>> at memory/unix/apr_pools.c:381
>>> #1 apr_pool_clear (pool=0x7f6e0808d238) at memory/unix/apr_pools.c:793
>>> #2 0x00000000004fe528 in ap_push_pool (queue_info=0x0,
>>> pool_to_recycle=0x7f6e08066548) at fdqueue.c:234
>>> #3 0x00000000004fa2c8 in process_lingering_close (cs=0x7f6e0808d4c8,
>>> pfd=0x1d3bf98) at event.c:1439
>>> #4 0x00000000004fd410 in listener_thread (thd=0x1d3cb70,
>>> dummy=0x7f6e0808d4c8)
>>> at event.c:1704
>>> #5 0x00007f6e1aed20a4 in start_thread ()
>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>> #6 0x00007f6e1aa0362d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>> (gdb) (gdb) quit
>>>
>>> Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
>>> /usr/lib/debug//usr/local/apache2/bin/httpd...done.
>>> done.
>>> [Thread debugging using libthread_db enabled]
>>> Using host libthread_db library
>>> "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>> #0 allocator_free (node=0x0, allocator=0x7f6e08053ae0)
>>> at memory/unix/apr_pools.c:381
>>> #0 allocator_free (node=0x0, allocator=0x7f6e08053ae0)
>>> at memory/unix/apr_pools.c:381
>>> #1 apr_pool_clear (pool=0x7f6e08076bb8) at memory/unix/apr_pools.c:793
>>> #2 0x00000000004fe528 in ap_push_pool (queue_info=0x0,
>>> pool_to_recycle=0x7f6e08053ae8) at fdqueue.c:234
>>> #3 0x00000000004fa2c8 in process_lingering_close (cs=0x7f6e08076e48,
>>> pfd=0x1d3bf98) at event.c:1439
>>> #4 0x00000000004fd410 in listener_thread (thd=0x1d3cb70,
>>> dummy=0x7f6e08076e48)
>>> at event.c:1704
>>> #5 0x00007f6e1aed20a4 in start_thread ()
>>> from /lib/x86_64-linux-gnu/libpthread.so.0
>>> #6 0x00007f6e1aa0362d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>> (gdb) (gdb) quit
>>>
>>> Stefan
>>>
>>> Am 21.01.2017 um 17:03 schrieb Stefan Eissing:
>>>> Stefan,
>>>>
>>>> made a release at https://github.com/icing/mod_h2/releases/tag/v1.8.9
>>>> with all patches and improved (hopefully) on them a bit. If you dare
>>>> to drop that into your installation, that'd be great.
>>>>
>>>> Cheers,
>>>>
>>>> Stefan
>>>>
>>>>> Am 21.01.2017 um 15:25 schrieb Stefan Priebe <s....@profihost.ag>:
>>>>>
>>>>> and i got another crash here:
>>>>>
>>>>> 2346 static void run_cleanups(cleanup_t **cref)
>>>>> 2347 {
>>>>> 2348 cleanup_t *c = *cref;
>>>>> 2349
>>>>> 2350 while (c) {
>>>>> 2351 *cref = c->next;
>>>>> 2352 (*c->plain_cleanup_fn)((void *)c->data); <== here
>>>>> 2353 c = *cref;
>>>>> 2354
>>>>>
>>>>> which looks similar to the other crash.
>>>>>
>>>>> #0 0x00007fe4bbd33e1b in run_cleanups (cref=<optimized out>) at
>>>>> memory/unix/apr_pools.c:2352
>>>>> #1 apr_pool_clear (pool=0x7fe4a804dac8) at
>>>>> memory/unix/apr_pools.c:772
>>>>> #2 0x00000000004feb38 in ap_push_pool
>>>>> (queue_info=0x6d616e79642d3733, pool_to_recycle=0x2) at fdqueue.c:234
>>>>> #3 0x00000000004fa8d8 in process_lingering_close (cs=0x7fe4a804dd58,
>>>>> pfd=0x25d3f98) at event.c:1439
>>>>>
>>>>> Details:
>>>>> (gdb) print c
>>>>> $1 = (cleanup_t *) 0x7fe4a804e9f0
>>>>> (gdb) print *c
>>>>> $2 = {next = 0x7fe4a804e870, data = 0x6d616e79642d3733,
>>>>> plain_cleanup_fn = 0x392d3734322e6369,
>>>>> child_cleanup_fn = 0x617465722e722d35}
>>>>> (gdb) print *c->data
>>>>> Attempt to dereference a generic pointer.
>>>>> (gdb) print *c->plain_cleanup_fn
>>>>> Cannot access memory at address 0x392d3734322e6369
>>>>> (gdb)
>>>>>
>>>>> Stefan
>>>>>
>>>>> Am 21.01.2017 um 15:18 schrieb Stefan Priebe:
>>>>>> Hi,
>>>>>>
>>>>>> #0 apr_pool_cleanup_kill (p=0x7fe4a8072358,
>>>>>> data=data@entry=0x7fe4a80723e0,
>>>>>> cleanup_fn=cleanup_fn@entry=0x7fe4bbd38a40 <socket_cleanup>) at
>>>>>> memory/unix/apr_pools.c:2276
>>>>>>
>>>>>> it crashes here in apr:
>>>>>> 2276 if (c->data == data && c->plain_cleanup_fn ==
>>>>>> cleanup_fn) {
>>>>>>
>>>>>> some lines before c becomes this
>>>>>> 2264 c = p->cleanups;
>>>>>>
>>>>>> p is:
>>>>>> (gdb) print *p
>>>>>> $1 = {parent = 0x256f138, child = 0x7fe46c0751c8, sibling =
>>>>>> 0x7fe4a8096888, ref = 0x7fe4a8069fe8, cleanups = 0x7fe478159748,
>>>>>> free_cleanups = 0x7fe478159788, allocator = 0x7fe4a803b490,
>>>>>> subprocesses = 0x0, abort_fn = 0x43da00 <abort_on_oom>,
>>>>>> user_data = 0x0, tag = 0x502285 "transaction", active =
>>>>>> 0x7fe478158d70, self = 0x7fe4a8072330,
>>>>>> self_first_avail = 0x7fe4a80723d0 "X#\a\250\344\177", pre_cleanups =
>>>>>> 0x7fe4a8072ab8}
>>>>>>
>>>>>> wouldn't the error mean that p->cleanups is NULL?
>>>>>>
>>>>>> (gdb) print *p->cleanups
>>>>>> $2 = {next = 0x7fe478159628, data = 0x7fe478159648,
>>>>>> plain_cleanup_fn =
>>>>>> 0x7fe4bbd2ffd0 <apr_unix_file_cleanup>,
>>>>>> child_cleanup_fn = 0x7fe4bbd2ff70 <apr_unix_child_file_cleanup>}
>>>>>>
>>>>>> So p->cleanups->data is 0x7fe478159648 and data is 0x7fe4a80723e0?
>>>>>>
>>>>>> I don't get why it's segfaulting
>>>>>>
>>>>>> Stefan
>>>>>> Am 21.01.2017 um 09:50 schrieb Yann Ylavic:
>>>>>>> Hi Stefan,
>>>>>>>
>>>>>>> On Sat, Jan 21, 2017 at 9:45 AM, Stefan Priebe
>>>>>>> <s....@profihost.ag>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> after running the whole night. These are the only ones still
>>>>>>>> happening.
>>>>>>>> Should i revert the mpm patch to check whether it's the source?
>>>>>>>
>>>>>>> Yes please, we need to determine...
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Yann.
>>>>>>>
>>>>
>>>> Stefan Eissing
>>>>
>>>> <green/>bytes GmbH
>>>> Hafenstrasse 16
>>>> 48155 M�nster
>>>> www.greenbytes.de
>>>>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe <s....@profihost.ag>.
Hi,
and a new one but also in ap_start_lingering_close:
Program terminated with signal SIGSEGV, Segmentation fault.
#0 apr_palloc (pool=pool@entry=0x7f455805e138, in_size=in_size@entry=32)
at memory/unix/apr_pools.c:684
#0 apr_palloc (pool=pool@entry=0x7f455805e138, in_size=in_size@entry=32)
at memory/unix/apr_pools.c:684
#1 0x00007f456bc5d8b4 in apr_brigade_create (p=0x7f455805e138,
list=0x7f45040034e8) at buckets/apr_brigade.c:61
#2 0x000055e165efa319 in ap_shutdown_conn (c=c@entry=0x7f455805e458,
flush=flush@entry=1) at connection.c:76
#3 0x000055e165efa40d in ap_flush_conn (c=0x7f455805e458) at
connection.c:95
#4 ap_start_lingering_close (c=0x7f455805e458) at connection.c:145
#5 0x000055e165f942dd in start_lingering_close_blocking (cs=<optimized
out>)
at event.c:876
#6 process_socket (my_thread_num=<optimized out>,
my_child_num=<optimized out>, cs=0x7f455805e3c8, sock=<optimized out>,
p=<optimized out>, thd=<optimized out>) at event.c:1153
#7 worker_thread (thd=0x7f455805e138, dummy=0x20) at event.c:2001
#8 0x00007f456b80a0a4 in start_thread ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#9 0x00007f456b53f62d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Stefan
Am 21.01.2017 um 19:31 schrieb Stefan Priebe:
> All last traces come from event, proces_longering_close ap_push_pool but
> end in different functions. It looks like a race somewhere and it just
> races at different function in the event of close and pool clear.
>
> Might there be two places where the same pool gets cleared?
>
> Stefan
>
> Am 21.01.2017 um 19:07 schrieb Stefan Priebe:
>> Hi Stefan,
>>
>> thanks. No crashes where h2 comes up. But i still have these and no idea
>> how to find out who and why they're crashing.
>>
>> Using host libthread_db library
>> "/lib/x86_64-linux-gnu/libthread_db.so.1".
>> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
>> Program terminated with signal SIGSEGV, Segmentation fault.
>> #0 allocator_free (node=0x0, allocator=0x7f6e08066540)
>> at memory/unix/apr_pools.c:381
>> #0 allocator_free (node=0x0, allocator=0x7f6e08066540)
>> at memory/unix/apr_pools.c:381
>> #1 apr_pool_clear (pool=0x7f6e0808d238) at memory/unix/apr_pools.c:793
>> #2 0x00000000004fe528 in ap_push_pool (queue_info=0x0,
>> pool_to_recycle=0x7f6e08066548) at fdqueue.c:234
>> #3 0x00000000004fa2c8 in process_lingering_close (cs=0x7f6e0808d4c8,
>> pfd=0x1d3bf98) at event.c:1439
>> #4 0x00000000004fd410 in listener_thread (thd=0x1d3cb70,
>> dummy=0x7f6e0808d4c8)
>> at event.c:1704
>> #5 0x00007f6e1aed20a4 in start_thread ()
>> from /lib/x86_64-linux-gnu/libpthread.so.0
>> #6 0x00007f6e1aa0362d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> (gdb) (gdb) quit
>>
>> Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
>> /usr/lib/debug//usr/local/apache2/bin/httpd...done.
>> done.
>> [Thread debugging using libthread_db enabled]
>> Using host libthread_db library
>> "/lib/x86_64-linux-gnu/libthread_db.so.1".
>> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
>> Program terminated with signal SIGSEGV, Segmentation fault.
>> #0 allocator_free (node=0x0, allocator=0x7f6e08053ae0)
>> at memory/unix/apr_pools.c:381
>> #0 allocator_free (node=0x0, allocator=0x7f6e08053ae0)
>> at memory/unix/apr_pools.c:381
>> #1 apr_pool_clear (pool=0x7f6e08076bb8) at memory/unix/apr_pools.c:793
>> #2 0x00000000004fe528 in ap_push_pool (queue_info=0x0,
>> pool_to_recycle=0x7f6e08053ae8) at fdqueue.c:234
>> #3 0x00000000004fa2c8 in process_lingering_close (cs=0x7f6e08076e48,
>> pfd=0x1d3bf98) at event.c:1439
>> #4 0x00000000004fd410 in listener_thread (thd=0x1d3cb70,
>> dummy=0x7f6e08076e48)
>> at event.c:1704
>> #5 0x00007f6e1aed20a4 in start_thread ()
>> from /lib/x86_64-linux-gnu/libpthread.so.0
>> #6 0x00007f6e1aa0362d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> (gdb) (gdb) quit
>>
>> Stefan
>>
>> Am 21.01.2017 um 17:03 schrieb Stefan Eissing:
>>> Stefan,
>>>
>>> made a release at https://github.com/icing/mod_h2/releases/tag/v1.8.9
>>> with all patches and improved (hopefully) on them a bit. If you dare
>>> to drop that into your installation, that'd be great.
>>>
>>> Cheers,
>>>
>>> Stefan
>>>
>>>> Am 21.01.2017 um 15:25 schrieb Stefan Priebe <s....@profihost.ag>:
>>>>
>>>> and i got another crash here:
>>>>
>>>> 2346 static void run_cleanups(cleanup_t **cref)
>>>> 2347 {
>>>> 2348 cleanup_t *c = *cref;
>>>> 2349
>>>> 2350 while (c) {
>>>> 2351 *cref = c->next;
>>>> 2352 (*c->plain_cleanup_fn)((void *)c->data); <== here
>>>> 2353 c = *cref;
>>>> 2354
>>>>
>>>> which looks similar to the other crash.
>>>>
>>>> #0 0x00007fe4bbd33e1b in run_cleanups (cref=<optimized out>) at
>>>> memory/unix/apr_pools.c:2352
>>>> #1 apr_pool_clear (pool=0x7fe4a804dac8) at memory/unix/apr_pools.c:772
>>>> #2 0x00000000004feb38 in ap_push_pool
>>>> (queue_info=0x6d616e79642d3733, pool_to_recycle=0x2) at fdqueue.c:234
>>>> #3 0x00000000004fa8d8 in process_lingering_close (cs=0x7fe4a804dd58,
>>>> pfd=0x25d3f98) at event.c:1439
>>>>
>>>> Details:
>>>> (gdb) print c
>>>> $1 = (cleanup_t *) 0x7fe4a804e9f0
>>>> (gdb) print *c
>>>> $2 = {next = 0x7fe4a804e870, data = 0x6d616e79642d3733,
>>>> plain_cleanup_fn = 0x392d3734322e6369,
>>>> child_cleanup_fn = 0x617465722e722d35}
>>>> (gdb) print *c->data
>>>> Attempt to dereference a generic pointer.
>>>> (gdb) print *c->plain_cleanup_fn
>>>> Cannot access memory at address 0x392d3734322e6369
>>>> (gdb)
>>>>
>>>> Stefan
>>>>
>>>> Am 21.01.2017 um 15:18 schrieb Stefan Priebe:
>>>>> Hi,
>>>>>
>>>>> #0 apr_pool_cleanup_kill (p=0x7fe4a8072358,
>>>>> data=data@entry=0x7fe4a80723e0,
>>>>> cleanup_fn=cleanup_fn@entry=0x7fe4bbd38a40 <socket_cleanup>) at
>>>>> memory/unix/apr_pools.c:2276
>>>>>
>>>>> it crashes here in apr:
>>>>> 2276 if (c->data == data && c->plain_cleanup_fn ==
>>>>> cleanup_fn) {
>>>>>
>>>>> some lines before c becomes this
>>>>> 2264 c = p->cleanups;
>>>>>
>>>>> p is:
>>>>> (gdb) print *p
>>>>> $1 = {parent = 0x256f138, child = 0x7fe46c0751c8, sibling =
>>>>> 0x7fe4a8096888, ref = 0x7fe4a8069fe8, cleanups = 0x7fe478159748,
>>>>> free_cleanups = 0x7fe478159788, allocator = 0x7fe4a803b490,
>>>>> subprocesses = 0x0, abort_fn = 0x43da00 <abort_on_oom>,
>>>>> user_data = 0x0, tag = 0x502285 "transaction", active =
>>>>> 0x7fe478158d70, self = 0x7fe4a8072330,
>>>>> self_first_avail = 0x7fe4a80723d0 "X#\a\250\344\177", pre_cleanups =
>>>>> 0x7fe4a8072ab8}
>>>>>
>>>>> wouldn't the error mean that p->cleanups is NULL?
>>>>>
>>>>> (gdb) print *p->cleanups
>>>>> $2 = {next = 0x7fe478159628, data = 0x7fe478159648, plain_cleanup_fn =
>>>>> 0x7fe4bbd2ffd0 <apr_unix_file_cleanup>,
>>>>> child_cleanup_fn = 0x7fe4bbd2ff70 <apr_unix_child_file_cleanup>}
>>>>>
>>>>> So p->cleanups->data is 0x7fe478159648 and data is 0x7fe4a80723e0?
>>>>>
>>>>> I don't get why it's segfaulting
>>>>>
>>>>> Stefan
>>>>> Am 21.01.2017 um 09:50 schrieb Yann Ylavic:
>>>>>> Hi Stefan,
>>>>>>
>>>>>> On Sat, Jan 21, 2017 at 9:45 AM, Stefan Priebe
>>>>>> <s....@profihost.ag>
>>>>>> wrote:
>>>>>>>
>>>>>>> after running the whole night. These are the only ones still
>>>>>>> happening.
>>>>>>> Should i revert the mpm patch to check whether it's the source?
>>>>>>
>>>>>> Yes please, we need to determine...
>>>>>>
>>>>>> Thanks,
>>>>>> Yann.
>>>>>>
>>>
>>> Stefan Eissing
>>>
>>> <green/>bytes GmbH
>>> Hafenstrasse 16
>>> 48155 M�nster
>>> www.greenbytes.de
>>>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe <s....@profihost.ag>.
All last traces come from event, proces_longering_close ap_push_pool but
end in different functions. It looks like a race somewhere and it just
races at different function in the event of close and pool clear.
Might there be two places where the same pool gets cleared?
Stefan
Am 21.01.2017 um 19:07 schrieb Stefan Priebe:
> Hi Stefan,
>
> thanks. No crashes where h2 comes up. But i still have these and no idea
> how to find out who and why they're crashing.
>
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0 allocator_free (node=0x0, allocator=0x7f6e08066540)
> at memory/unix/apr_pools.c:381
> #0 allocator_free (node=0x0, allocator=0x7f6e08066540)
> at memory/unix/apr_pools.c:381
> #1 apr_pool_clear (pool=0x7f6e0808d238) at memory/unix/apr_pools.c:793
> #2 0x00000000004fe528 in ap_push_pool (queue_info=0x0,
> pool_to_recycle=0x7f6e08066548) at fdqueue.c:234
> #3 0x00000000004fa2c8 in process_lingering_close (cs=0x7f6e0808d4c8,
> pfd=0x1d3bf98) at event.c:1439
> #4 0x00000000004fd410 in listener_thread (thd=0x1d3cb70,
> dummy=0x7f6e0808d4c8)
> at event.c:1704
> #5 0x00007f6e1aed20a4 in start_thread ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #6 0x00007f6e1aa0362d in clone () from /lib/x86_64-linux-gnu/libc.so.6
> (gdb) (gdb) quit
>
> Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
> /usr/lib/debug//usr/local/apache2/bin/httpd...done.
> done.
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0 allocator_free (node=0x0, allocator=0x7f6e08053ae0)
> at memory/unix/apr_pools.c:381
> #0 allocator_free (node=0x0, allocator=0x7f6e08053ae0)
> at memory/unix/apr_pools.c:381
> #1 apr_pool_clear (pool=0x7f6e08076bb8) at memory/unix/apr_pools.c:793
> #2 0x00000000004fe528 in ap_push_pool (queue_info=0x0,
> pool_to_recycle=0x7f6e08053ae8) at fdqueue.c:234
> #3 0x00000000004fa2c8 in process_lingering_close (cs=0x7f6e08076e48,
> pfd=0x1d3bf98) at event.c:1439
> #4 0x00000000004fd410 in listener_thread (thd=0x1d3cb70,
> dummy=0x7f6e08076e48)
> at event.c:1704
> #5 0x00007f6e1aed20a4 in start_thread ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #6 0x00007f6e1aa0362d in clone () from /lib/x86_64-linux-gnu/libc.so.6
> (gdb) (gdb) quit
>
> Stefan
>
> Am 21.01.2017 um 17:03 schrieb Stefan Eissing:
>> Stefan,
>>
>> made a release at https://github.com/icing/mod_h2/releases/tag/v1.8.9
>> with all patches and improved (hopefully) on them a bit. If you dare
>> to drop that into your installation, that'd be great.
>>
>> Cheers,
>>
>> Stefan
>>
>>> Am 21.01.2017 um 15:25 schrieb Stefan Priebe <s....@profihost.ag>:
>>>
>>> and i got another crash here:
>>>
>>> 2346 static void run_cleanups(cleanup_t **cref)
>>> 2347 {
>>> 2348 cleanup_t *c = *cref;
>>> 2349
>>> 2350 while (c) {
>>> 2351 *cref = c->next;
>>> 2352 (*c->plain_cleanup_fn)((void *)c->data); <== here
>>> 2353 c = *cref;
>>> 2354
>>>
>>> which looks similar to the other crash.
>>>
>>> #0 0x00007fe4bbd33e1b in run_cleanups (cref=<optimized out>) at
>>> memory/unix/apr_pools.c:2352
>>> #1 apr_pool_clear (pool=0x7fe4a804dac8) at memory/unix/apr_pools.c:772
>>> #2 0x00000000004feb38 in ap_push_pool
>>> (queue_info=0x6d616e79642d3733, pool_to_recycle=0x2) at fdqueue.c:234
>>> #3 0x00000000004fa8d8 in process_lingering_close (cs=0x7fe4a804dd58,
>>> pfd=0x25d3f98) at event.c:1439
>>>
>>> Details:
>>> (gdb) print c
>>> $1 = (cleanup_t *) 0x7fe4a804e9f0
>>> (gdb) print *c
>>> $2 = {next = 0x7fe4a804e870, data = 0x6d616e79642d3733,
>>> plain_cleanup_fn = 0x392d3734322e6369,
>>> child_cleanup_fn = 0x617465722e722d35}
>>> (gdb) print *c->data
>>> Attempt to dereference a generic pointer.
>>> (gdb) print *c->plain_cleanup_fn
>>> Cannot access memory at address 0x392d3734322e6369
>>> (gdb)
>>>
>>> Stefan
>>>
>>> Am 21.01.2017 um 15:18 schrieb Stefan Priebe:
>>>> Hi,
>>>>
>>>> #0 apr_pool_cleanup_kill (p=0x7fe4a8072358,
>>>> data=data@entry=0x7fe4a80723e0,
>>>> cleanup_fn=cleanup_fn@entry=0x7fe4bbd38a40 <socket_cleanup>) at
>>>> memory/unix/apr_pools.c:2276
>>>>
>>>> it crashes here in apr:
>>>> 2276 if (c->data == data && c->plain_cleanup_fn ==
>>>> cleanup_fn) {
>>>>
>>>> some lines before c becomes this
>>>> 2264 c = p->cleanups;
>>>>
>>>> p is:
>>>> (gdb) print *p
>>>> $1 = {parent = 0x256f138, child = 0x7fe46c0751c8, sibling =
>>>> 0x7fe4a8096888, ref = 0x7fe4a8069fe8, cleanups = 0x7fe478159748,
>>>> free_cleanups = 0x7fe478159788, allocator = 0x7fe4a803b490,
>>>> subprocesses = 0x0, abort_fn = 0x43da00 <abort_on_oom>,
>>>> user_data = 0x0, tag = 0x502285 "transaction", active =
>>>> 0x7fe478158d70, self = 0x7fe4a8072330,
>>>> self_first_avail = 0x7fe4a80723d0 "X#\a\250\344\177", pre_cleanups =
>>>> 0x7fe4a8072ab8}
>>>>
>>>> wouldn't the error mean that p->cleanups is NULL?
>>>>
>>>> (gdb) print *p->cleanups
>>>> $2 = {next = 0x7fe478159628, data = 0x7fe478159648, plain_cleanup_fn =
>>>> 0x7fe4bbd2ffd0 <apr_unix_file_cleanup>,
>>>> child_cleanup_fn = 0x7fe4bbd2ff70 <apr_unix_child_file_cleanup>}
>>>>
>>>> So p->cleanups->data is 0x7fe478159648 and data is 0x7fe4a80723e0?
>>>>
>>>> I don't get why it's segfaulting
>>>>
>>>> Stefan
>>>> Am 21.01.2017 um 09:50 schrieb Yann Ylavic:
>>>>> Hi Stefan,
>>>>>
>>>>> On Sat, Jan 21, 2017 at 9:45 AM, Stefan Priebe <s....@profihost.ag>
>>>>> wrote:
>>>>>>
>>>>>> after running the whole night. These are the only ones still
>>>>>> happening.
>>>>>> Should i revert the mpm patch to check whether it's the source?
>>>>>
>>>>> Yes please, we need to determine...
>>>>>
>>>>> Thanks,
>>>>> Yann.
>>>>>
>>
>> Stefan Eissing
>>
>> <green/>bytes GmbH
>> Hafenstrasse 16
>> 48155 M�nster
>> www.greenbytes.de
>>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe <s....@profihost.ag>.
Hi Stefan,
thanks. No crashes where h2 comes up. But i still have these and no idea
how to find out who and why they're crashing.
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/local/apache2/bin/httpd -k start'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 allocator_free (node=0x0, allocator=0x7f6e08066540)
at memory/unix/apr_pools.c:381
#0 allocator_free (node=0x0, allocator=0x7f6e08066540)
at memory/unix/apr_pools.c:381
#1 apr_pool_clear (pool=0x7f6e0808d238) at memory/unix/apr_pools.c:793
#2 0x00000000004fe528 in ap_push_pool (queue_info=0x0,
pool_to_recycle=0x7f6e08066548) at fdqueue.c:234
#3 0x00000000004fa2c8 in process_lingering_close (cs=0x7f6e0808d4c8,
pfd=0x1d3bf98) at event.c:1439
#4 0x00000000004fd410 in listener_thread (thd=0x1d3cb70,
dummy=0x7f6e0808d4c8)
at event.c:1704
#5 0x00007f6e1aed20a4 in start_thread ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007f6e1aa0362d in clone () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) (gdb) quit
Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
/usr/lib/debug//usr/local/apache2/bin/httpd...done.
done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/local/apache2/bin/httpd -k start'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 allocator_free (node=0x0, allocator=0x7f6e08053ae0)
at memory/unix/apr_pools.c:381
#0 allocator_free (node=0x0, allocator=0x7f6e08053ae0)
at memory/unix/apr_pools.c:381
#1 apr_pool_clear (pool=0x7f6e08076bb8) at memory/unix/apr_pools.c:793
#2 0x00000000004fe528 in ap_push_pool (queue_info=0x0,
pool_to_recycle=0x7f6e08053ae8) at fdqueue.c:234
#3 0x00000000004fa2c8 in process_lingering_close (cs=0x7f6e08076e48,
pfd=0x1d3bf98) at event.c:1439
#4 0x00000000004fd410 in listener_thread (thd=0x1d3cb70,
dummy=0x7f6e08076e48)
at event.c:1704
#5 0x00007f6e1aed20a4 in start_thread ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007f6e1aa0362d in clone () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) (gdb) quit
Stefan
Am 21.01.2017 um 17:03 schrieb Stefan Eissing:
> Stefan,
>
> made a release at https://github.com/icing/mod_h2/releases/tag/v1.8.9 with all patches and improved (hopefully) on them a bit. If you dare to drop that into your installation, that'd be great.
>
> Cheers,
>
> Stefan
>
>> Am 21.01.2017 um 15:25 schrieb Stefan Priebe <s....@profihost.ag>:
>>
>> and i got another crash here:
>>
>> 2346 static void run_cleanups(cleanup_t **cref)
>> 2347 {
>> 2348 cleanup_t *c = *cref;
>> 2349
>> 2350 while (c) {
>> 2351 *cref = c->next;
>> 2352 (*c->plain_cleanup_fn)((void *)c->data); <== here
>> 2353 c = *cref;
>> 2354
>>
>> which looks similar to the other crash.
>>
>> #0 0x00007fe4bbd33e1b in run_cleanups (cref=<optimized out>) at memory/unix/apr_pools.c:2352
>> #1 apr_pool_clear (pool=0x7fe4a804dac8) at memory/unix/apr_pools.c:772
>> #2 0x00000000004feb38 in ap_push_pool (queue_info=0x6d616e79642d3733, pool_to_recycle=0x2) at fdqueue.c:234
>> #3 0x00000000004fa8d8 in process_lingering_close (cs=0x7fe4a804dd58, pfd=0x25d3f98) at event.c:1439
>>
>> Details:
>> (gdb) print c
>> $1 = (cleanup_t *) 0x7fe4a804e9f0
>> (gdb) print *c
>> $2 = {next = 0x7fe4a804e870, data = 0x6d616e79642d3733, plain_cleanup_fn = 0x392d3734322e6369,
>> child_cleanup_fn = 0x617465722e722d35}
>> (gdb) print *c->data
>> Attempt to dereference a generic pointer.
>> (gdb) print *c->plain_cleanup_fn
>> Cannot access memory at address 0x392d3734322e6369
>> (gdb)
>>
>> Stefan
>>
>> Am 21.01.2017 um 15:18 schrieb Stefan Priebe:
>>> Hi,
>>>
>>> #0 apr_pool_cleanup_kill (p=0x7fe4a8072358,
>>> data=data@entry=0x7fe4a80723e0,
>>> cleanup_fn=cleanup_fn@entry=0x7fe4bbd38a40 <socket_cleanup>) at
>>> memory/unix/apr_pools.c:2276
>>>
>>> it crashes here in apr:
>>> 2276 if (c->data == data && c->plain_cleanup_fn == cleanup_fn) {
>>>
>>> some lines before c becomes this
>>> 2264 c = p->cleanups;
>>>
>>> p is:
>>> (gdb) print *p
>>> $1 = {parent = 0x256f138, child = 0x7fe46c0751c8, sibling =
>>> 0x7fe4a8096888, ref = 0x7fe4a8069fe8, cleanups = 0x7fe478159748,
>>> free_cleanups = 0x7fe478159788, allocator = 0x7fe4a803b490,
>>> subprocesses = 0x0, abort_fn = 0x43da00 <abort_on_oom>,
>>> user_data = 0x0, tag = 0x502285 "transaction", active =
>>> 0x7fe478158d70, self = 0x7fe4a8072330,
>>> self_first_avail = 0x7fe4a80723d0 "X#\a\250\344\177", pre_cleanups =
>>> 0x7fe4a8072ab8}
>>>
>>> wouldn't the error mean that p->cleanups is NULL?
>>>
>>> (gdb) print *p->cleanups
>>> $2 = {next = 0x7fe478159628, data = 0x7fe478159648, plain_cleanup_fn =
>>> 0x7fe4bbd2ffd0 <apr_unix_file_cleanup>,
>>> child_cleanup_fn = 0x7fe4bbd2ff70 <apr_unix_child_file_cleanup>}
>>>
>>> So p->cleanups->data is 0x7fe478159648 and data is 0x7fe4a80723e0?
>>>
>>> I don't get why it's segfaulting
>>>
>>> Stefan
>>> Am 21.01.2017 um 09:50 schrieb Yann Ylavic:
>>>> Hi Stefan,
>>>>
>>>> On Sat, Jan 21, 2017 at 9:45 AM, Stefan Priebe <s....@profihost.ag>
>>>> wrote:
>>>>>
>>>>> after running the whole night. These are the only ones still happening.
>>>>> Should i revert the mpm patch to check whether it's the source?
>>>>
>>>> Yes please, we need to determine...
>>>>
>>>> Thanks,
>>>> Yann.
>>>>
>
> Stefan Eissing
>
> <green/>bytes GmbH
> Hafenstrasse 16
> 48155 M�nster
> www.greenbytes.de
>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Eissing <st...@greenbytes.de>.
Stefan,
made a release at https://github.com/icing/mod_h2/releases/tag/v1.8.9 with all patches and improved (hopefully) on them a bit. If you dare to drop that into your installation, that'd be great.
Cheers,
Stefan
> Am 21.01.2017 um 15:25 schrieb Stefan Priebe <s....@profihost.ag>:
>
> and i got another crash here:
>
> 2346 static void run_cleanups(cleanup_t **cref)
> 2347 {
> 2348 cleanup_t *c = *cref;
> 2349
> 2350 while (c) {
> 2351 *cref = c->next;
> 2352 (*c->plain_cleanup_fn)((void *)c->data); <== here
> 2353 c = *cref;
> 2354
>
> which looks similar to the other crash.
>
> #0 0x00007fe4bbd33e1b in run_cleanups (cref=<optimized out>) at memory/unix/apr_pools.c:2352
> #1 apr_pool_clear (pool=0x7fe4a804dac8) at memory/unix/apr_pools.c:772
> #2 0x00000000004feb38 in ap_push_pool (queue_info=0x6d616e79642d3733, pool_to_recycle=0x2) at fdqueue.c:234
> #3 0x00000000004fa8d8 in process_lingering_close (cs=0x7fe4a804dd58, pfd=0x25d3f98) at event.c:1439
>
> Details:
> (gdb) print c
> $1 = (cleanup_t *) 0x7fe4a804e9f0
> (gdb) print *c
> $2 = {next = 0x7fe4a804e870, data = 0x6d616e79642d3733, plain_cleanup_fn = 0x392d3734322e6369,
> child_cleanup_fn = 0x617465722e722d35}
> (gdb) print *c->data
> Attempt to dereference a generic pointer.
> (gdb) print *c->plain_cleanup_fn
> Cannot access memory at address 0x392d3734322e6369
> (gdb)
>
> Stefan
>
> Am 21.01.2017 um 15:18 schrieb Stefan Priebe:
>> Hi,
>>
>> #0 apr_pool_cleanup_kill (p=0x7fe4a8072358,
>> data=data@entry=0x7fe4a80723e0,
>> cleanup_fn=cleanup_fn@entry=0x7fe4bbd38a40 <socket_cleanup>) at
>> memory/unix/apr_pools.c:2276
>>
>> it crashes here in apr:
>> 2276 if (c->data == data && c->plain_cleanup_fn == cleanup_fn) {
>>
>> some lines before c becomes this
>> 2264 c = p->cleanups;
>>
>> p is:
>> (gdb) print *p
>> $1 = {parent = 0x256f138, child = 0x7fe46c0751c8, sibling =
>> 0x7fe4a8096888, ref = 0x7fe4a8069fe8, cleanups = 0x7fe478159748,
>> free_cleanups = 0x7fe478159788, allocator = 0x7fe4a803b490,
>> subprocesses = 0x0, abort_fn = 0x43da00 <abort_on_oom>,
>> user_data = 0x0, tag = 0x502285 "transaction", active =
>> 0x7fe478158d70, self = 0x7fe4a8072330,
>> self_first_avail = 0x7fe4a80723d0 "X#\a\250\344\177", pre_cleanups =
>> 0x7fe4a8072ab8}
>>
>> wouldn't the error mean that p->cleanups is NULL?
>>
>> (gdb) print *p->cleanups
>> $2 = {next = 0x7fe478159628, data = 0x7fe478159648, plain_cleanup_fn =
>> 0x7fe4bbd2ffd0 <apr_unix_file_cleanup>,
>> child_cleanup_fn = 0x7fe4bbd2ff70 <apr_unix_child_file_cleanup>}
>>
>> So p->cleanups->data is 0x7fe478159648 and data is 0x7fe4a80723e0?
>>
>> I don't get why it's segfaulting
>>
>> Stefan
>> Am 21.01.2017 um 09:50 schrieb Yann Ylavic:
>>> Hi Stefan,
>>>
>>> On Sat, Jan 21, 2017 at 9:45 AM, Stefan Priebe <s....@profihost.ag>
>>> wrote:
>>>>
>>>> after running the whole night. These are the only ones still happening.
>>>> Should i revert the mpm patch to check whether it's the source?
>>>
>>> Yes please, we need to determine...
>>>
>>> Thanks,
>>> Yann.
>>>
Stefan Eissing
<green/>bytes GmbH
Hafenstrasse 16
48155 Münster
www.greenbytes.de
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe <s....@profihost.ag>.
and i got another crash here:
2346 static void run_cleanups(cleanup_t **cref)
2347 {
2348 cleanup_t *c = *cref;
2349
2350 while (c) {
2351 *cref = c->next;
2352 (*c->plain_cleanup_fn)((void *)c->data); <== here
2353 c = *cref;
2354
which looks similar to the other crash.
#0 0x00007fe4bbd33e1b in run_cleanups (cref=<optimized out>) at
memory/unix/apr_pools.c:2352
#1 apr_pool_clear (pool=0x7fe4a804dac8) at memory/unix/apr_pools.c:772
#2 0x00000000004feb38 in ap_push_pool (queue_info=0x6d616e79642d3733,
pool_to_recycle=0x2) at fdqueue.c:234
#3 0x00000000004fa8d8 in process_lingering_close (cs=0x7fe4a804dd58,
pfd=0x25d3f98) at event.c:1439
Details:
(gdb) print c
$1 = (cleanup_t *) 0x7fe4a804e9f0
(gdb) print *c
$2 = {next = 0x7fe4a804e870, data = 0x6d616e79642d3733, plain_cleanup_fn
= 0x392d3734322e6369,
child_cleanup_fn = 0x617465722e722d35}
(gdb) print *c->data
Attempt to dereference a generic pointer.
(gdb) print *c->plain_cleanup_fn
Cannot access memory at address 0x392d3734322e6369
(gdb)
Stefan
Am 21.01.2017 um 15:18 schrieb Stefan Priebe:
> Hi,
>
> #0 apr_pool_cleanup_kill (p=0x7fe4a8072358,
> data=data@entry=0x7fe4a80723e0,
> cleanup_fn=cleanup_fn@entry=0x7fe4bbd38a40 <socket_cleanup>) at
> memory/unix/apr_pools.c:2276
>
> it crashes here in apr:
> 2276 if (c->data == data && c->plain_cleanup_fn == cleanup_fn) {
>
> some lines before c becomes this
> 2264 c = p->cleanups;
>
> p is:
> (gdb) print *p
> $1 = {parent = 0x256f138, child = 0x7fe46c0751c8, sibling =
> 0x7fe4a8096888, ref = 0x7fe4a8069fe8, cleanups = 0x7fe478159748,
> free_cleanups = 0x7fe478159788, allocator = 0x7fe4a803b490,
> subprocesses = 0x0, abort_fn = 0x43da00 <abort_on_oom>,
> user_data = 0x0, tag = 0x502285 "transaction", active =
> 0x7fe478158d70, self = 0x7fe4a8072330,
> self_first_avail = 0x7fe4a80723d0 "X#\a\250\344\177", pre_cleanups =
> 0x7fe4a8072ab8}
>
> wouldn't the error mean that p->cleanups is NULL?
>
> (gdb) print *p->cleanups
> $2 = {next = 0x7fe478159628, data = 0x7fe478159648, plain_cleanup_fn =
> 0x7fe4bbd2ffd0 <apr_unix_file_cleanup>,
> child_cleanup_fn = 0x7fe4bbd2ff70 <apr_unix_child_file_cleanup>}
>
> So p->cleanups->data is 0x7fe478159648 and data is 0x7fe4a80723e0?
>
> I don't get why it's segfaulting
>
> Stefan
> Am 21.01.2017 um 09:50 schrieb Yann Ylavic:
>> Hi Stefan,
>>
>> On Sat, Jan 21, 2017 at 9:45 AM, Stefan Priebe <s....@profihost.ag>
>> wrote:
>>>
>>> after running the whole night. These are the only ones still happening.
>>> Should i revert the mpm patch to check whether it's the source?
>>
>> Yes please, we need to determine...
>>
>> Thanks,
>> Yann.
>>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe <s....@profihost.ag>.
Hi,
#0 apr_pool_cleanup_kill (p=0x7fe4a8072358,
data=data@entry=0x7fe4a80723e0,
cleanup_fn=cleanup_fn@entry=0x7fe4bbd38a40 <socket_cleanup>) at
memory/unix/apr_pools.c:2276
it crashes here in apr:
2276 if (c->data == data && c->plain_cleanup_fn == cleanup_fn) {
some lines before c becomes this
2264 c = p->cleanups;
p is:
(gdb) print *p
$1 = {parent = 0x256f138, child = 0x7fe46c0751c8, sibling =
0x7fe4a8096888, ref = 0x7fe4a8069fe8, cleanups = 0x7fe478159748,
free_cleanups = 0x7fe478159788, allocator = 0x7fe4a803b490,
subprocesses = 0x0, abort_fn = 0x43da00 <abort_on_oom>,
user_data = 0x0, tag = 0x502285 "transaction", active =
0x7fe478158d70, self = 0x7fe4a8072330,
self_first_avail = 0x7fe4a80723d0 "X#\a\250\344\177", pre_cleanups =
0x7fe4a8072ab8}
wouldn't the error mean that p->cleanups is NULL?
(gdb) print *p->cleanups
$2 = {next = 0x7fe478159628, data = 0x7fe478159648, plain_cleanup_fn =
0x7fe4bbd2ffd0 <apr_unix_file_cleanup>,
child_cleanup_fn = 0x7fe4bbd2ff70 <apr_unix_child_file_cleanup>}
So p->cleanups->data is 0x7fe478159648 and data is 0x7fe4a80723e0?
I don't get why it's segfaulting
Stefan
Am 21.01.2017 um 09:50 schrieb Yann Ylavic:
> Hi Stefan,
>
> On Sat, Jan 21, 2017 at 9:45 AM, Stefan Priebe <s....@profihost.ag> wrote:
>>
>> after running the whole night. These are the only ones still happening.
>> Should i revert the mpm patch to check whether it's the source?
>
> Yes please, we need to determine...
>
> Thanks,
> Yann.
>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe <s....@profihost.ag>.
Hi Yann,
looks better so far. This is the only one i got without mpm patch:
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/local/apache2/bin/httpd -k start'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 apr_pool_cleanup_kill (p=0x7fe4a808b538,
data=data@entry=0x7fe4a808b5c0,
cleanup_fn=cleanup_fn@entry=0x7fe4bbd38a40 <socket_cleanup>)
at memory/unix/apr_pools.c:2276
#0 apr_pool_cleanup_kill (p=0x7fe4a808b538,
data=data@entry=0x7fe4a808b5c0,
cleanup_fn=cleanup_fn@entry=0x7fe4bbd38a40 <socket_cleanup>)
at memory/unix/apr_pools.c:2276
#1 0x00007fe4bbd34e91 in apr_pool_cleanup_run (p=<optimized out>,
data=0x7fe4a808b5c0, cleanup_fn=0x7fe4bbd38a40 <socket_cleanup>)
at memory/unix/apr_pools.c:2342
#2 0x00007fe4bbd38d22 in apr_socket_close (thesocket=<optimized out>)
at network_io/unix/sockets.c:183
#3 0x00000000004fa88f in process_lingering_close (cs=0x7fe4a808b7c8,
pfd=0x25d3f98) at event.c:1432
#4 0x00000000004fda20 in listener_thread (thd=0x25d4b70,
dummy=0x7fe4a808b7c8)
at event.c:1704
#5 0x00007fe4bb4bf0a4 in start_thread ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007fe4baff062d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Stefan
Am 21.01.2017 um 09:50 schrieb Yann Ylavic:
> Hi Stefan,
>
> On Sat, Jan 21, 2017 at 9:45 AM, Stefan Priebe <s....@profihost.ag> wrote:
>>
>> after running the whole night. These are the only ones still happening.
>> Should i revert the mpm patch to check whether it's the source?
>
> Yes please, we need to determine...
>
> Thanks,
> Yann.
>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
Hi Stefan,
On Sat, Jan 21, 2017 at 9:45 AM, Stefan Priebe <s....@profihost.ag> wrote:
>
> after running the whole night. These are the only ones still happening.
> Should i revert the mpm patch to check whether it's the source?
Yes please, we need to determine...
Thanks,
Yann.
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe <s....@profihost.ag>.
Hi,
after running the whole night. These are the only ones still happening.
Should i revert the mpm patch to check whether it's the source? I'm only
seeing one related to mod_http2.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/local/apache2/bin/httpd -k start'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 apr_pool_cleanup_kill (p=0x7f128c039378,
data=data@entry=0x7f128c039400,
cleanup_fn=cleanup_fn@entry=0x7f129fe89a40 <socket_cleanup>)
at memory/unix/apr_pools.c:2276
#0 apr_pool_cleanup_kill (p=0x7f128c039378,
data=data@entry=0x7f128c039400,
cleanup_fn=cleanup_fn@entry=0x7f129fe89a40 <socket_cleanup>)
at memory/unix/apr_pools.c:2276
#1 0x00007f129fe85e91 in apr_pool_cleanup_run (p=<optimized out>,
data=0x7f128c039400, cleanup_fn=0x7f129fe89a40 <socket_cleanup>)
at memory/unix/apr_pools.c:2342
#2 0x00007f129fe89d22 in apr_socket_close (thesocket=<optimized out>)
at network_io/unix/sockets.c:183
#3 0x00000000004fa54e in process_lingering_close (cs=0x7f128c039608,
pfd=0x1a4afa8) at event.c:1510
#4 0x00000000004fdc30 in listener_thread (thd=0x7f128c039378,
dummy=0x7f128c039608) at event.c:1837
#5 0x00007f129f6100a4 in start_thread ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007f129f14162d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
/usr/lib/debug//usr/local/apache2/bin/httpd...done.
done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/local/apache2/bin/httpd -k start'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 apr_pool_cleanup_kill (p=0x7f128c075f88,
data=data@entry=0x7f128c076010,
cleanup_fn=cleanup_fn@entry=0x7f129fe89a40 <socket_cleanup>)
at memory/unix/apr_pools.c:2276
#0 apr_pool_cleanup_kill (p=0x7f128c075f88,
data=data@entry=0x7f128c076010,
cleanup_fn=cleanup_fn@entry=0x7f129fe89a40 <socket_cleanup>)
at memory/unix/apr_pools.c:2276
#1 0x00007f129fe85e91 in apr_pool_cleanup_run (p=<optimized out>,
data=0x7f128c076010, cleanup_fn=0x7f129fe89a40 <socket_cleanup>)
at memory/unix/apr_pools.c:2342
#2 0x00007f129fe89d22 in apr_socket_close (thesocket=<optimized out>)
at network_io/unix/sockets.c:183
#3 0x00000000004fa54e in process_lingering_close (cs=0x7f128c076218,
pfd=0x1a4afa8) at event.c:1510
#4 0x00000000004fdc30 in listener_thread (thd=0x7f128c075f88,
dummy=0x7f128c076218) at event.c:1837
#5 0x00007f129f6100a4 in start_thread ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007f129f14162d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
/usr/lib/debug//usr/local/apache2/bin/httpd...done.
done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/local/apache2/bin/httpd -k start'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f12a02ce832 in apr_bucket_free (mem=0x7f11d802d838)
at buckets/apr_buckets_alloc.c:200
#0 0x00007f12a02ce832 in apr_bucket_free (mem=0x7f11d802d838)
at buckets/apr_buckets_alloc.c:200
#1 0x00007f12a02ced97 in heap_bucket_destroy (data=0x7f11d00769c8)
at buckets/apr_buckets_heap.c:36
#2 0x000000000045b9d3 in writev_nonblocking (s=0x7f128c086b20,
vec=0x7f12837ed890, nvec=4, bb=0x7f128c087378,
cumulative_bytes_written=0x7f11d007f7e8, c=0x7f128c086db8)
at core_filters.c:796
#3 0x000000000045bba1 in send_brigade_nonblocking (s=0x7f128c086b20,
bb=0x7f128c087378, bytes_written=0x76ce00 <apr_bucket_type_heap>,
c=0x7f128c087380) at core_filters.c:704
#4 0x000000000045c996 in ap_core_output_filter (f=0x7f11d802d838,
new_bb=0x7f128c087378) at core_filters.c:556
#5 0x00000000004aac18 in bio_filter_out_pass (
outctx=outctx@entry=0x7f128c087358) at ssl_engine_io.c:139
#6 0x00000000004aacbe in bio_filter_out_write (bio=<optimized out>,
in=0x7f11d803b8a3 "\027\003\003\005,\f\037h<5-O\005\272", inl=1329)
at ssl_engine_io.c:234
#7 0x00007f12a0cc124c in BIO_write ()
from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
#8 0x00007f12a1024fe2 in ?? () from
/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
#9 0x00007f12a10256c5 in ?? () from
/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
#10 0x00000000004ae04a in ssl_filter_write (f=<optimized out>,
f=<optimized out>, len=<optimized out>, data=<optimized out>)
at ssl_engine_io.c:793
#11 ssl_io_filter_output (f=0x7f128c087330, bb=0x7f11d007f898)
at ssl_engine_io.c:1746
#12 0x00000000004ab30a in ssl_io_filter_coalesce (f=0x7f11d802d838,
bb=0x7f11d007f898) at ssl_engine_io.c:1663
#13 0x00000000004db9ed in pass_output (io=0x7f11d0026148, session_eoc=0x0,
flush=<optimized out>) at h2_conn_io.c:296
#14 0x00000000004dbf20 in h2_conn_io_flush (io=0x7f11d0026148)
at h2_conn_io.c:346
#15 0x00000000004d012d in h2_session_process (session=0x7f11d0026100,
async=1)
at h2_session.c:2248
#16 0x00000000004bf641 in h2_conn_run (ctx=0x7f11d007f7a0, c=0x7f128c086db8)
at h2_conn.c:214
#17 0x00000000004c202a in h2_h2_process_conn (c=0x7f11d802d838) at
h2_h2.c:658
#18 0x000000000046a450 in ap_run_process_connection (c=0x7f128c086db8)
at connection.c:42
#19 0x00000000004fbf10 in process_socket (my_thread_num=<optimized out>,
my_child_num=<optimized out>, cs=0x7f128c086d28, sock=<optimized out>,
p=<optimized out>, thd=<optimized out>) at event.c:1134
#20 worker_thread (thd=0x7f11d802d838, dummy=0x382d35312e63696d)
at event.c:2137
#21 0x00007f129f6100a4 in start_thread ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#22 0x00007f129f14162d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
/usr/lib/debug//usr/local/apache2/bin/httpd...done.
done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/local/apache2/bin/httpd -k start'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 apr_pool_cleanup_kill (p=0x7f128c06e658,
data=data@entry=0x7f128c06e6e0,
cleanup_fn=cleanup_fn@entry=0x7f129fe89a40 <socket_cleanup>)
at memory/unix/apr_pools.c:2276
#0 apr_pool_cleanup_kill (p=0x7f128c06e658,
data=data@entry=0x7f128c06e6e0,
cleanup_fn=cleanup_fn@entry=0x7f129fe89a40 <socket_cleanup>)
at memory/unix/apr_pools.c:2276
#1 0x00007f129fe85e91 in apr_pool_cleanup_run (p=<optimized out>,
data=0x7f128c06e6e0, cleanup_fn=0x7f129fe89a40 <socket_cleanup>)
at memory/unix/apr_pools.c:2342
#2 0x00007f129fe89d22 in apr_socket_close (thesocket=<optimized out>)
at network_io/unix/sockets.c:183
#3 0x00000000004fa54e in process_lingering_close (cs=0x7f128c06e8e8,
pfd=0x1a4afa8) at event.c:1510
#4 0x00000000004fdc30 in listener_thread (thd=0x7f128c06e658,
dummy=0x7f128c06e8e8) at event.c:1837
#5 0x00007f129f6100a4 in start_thread ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007f129f14162d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
/usr/lib/debug//usr/local/apache2/bin/httpd...done.
done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/local/apache2/bin/httpd -k start'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 apr_pool_cleanup_kill (p=0x7f128c078ee8,
data=data@entry=0x7f128c078f70,
cleanup_fn=cleanup_fn@entry=0x7f129fe89a40 <socket_cleanup>)
at memory/unix/apr_pools.c:2276
#0 apr_pool_cleanup_kill (p=0x7f128c078ee8,
data=data@entry=0x7f128c078f70,
cleanup_fn=cleanup_fn@entry=0x7f129fe89a40 <socket_cleanup>)
at memory/unix/apr_pools.c:2276
#1 0x00007f129fe85e91 in apr_pool_cleanup_run (p=<optimized out>,
data=0x7f128c078f70, cleanup_fn=0x7f129fe89a40 <socket_cleanup>)
at memory/unix/apr_pools.c:2342
#2 0x00007f129fe89d22 in apr_socket_close (thesocket=<optimized out>)
at network_io/unix/sockets.c:183
#3 0x00000000004fa54e in process_lingering_close (cs=0x7f128c079178,
pfd=0x1a4afa8) at event.c:1510
#4 0x00000000004fdc30 in listener_thread (thd=0x7f128c078ee8,
dummy=0x7f128c079178) at event.c:1837
#5 0x00007f129f6100a4 in start_thread ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007f129f14162d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
/usr/lib/debug//usr/local/apache2/bin/httpd...done.
done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/local/apache2/bin/httpd -k start'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 apr_pool_cleanup_kill (p=0x7f128c0623c8,
data=data@entry=0x7f128c062450,
cleanup_fn=cleanup_fn@entry=0x7f129fe89a40 <socket_cleanup>)
at memory/unix/apr_pools.c:2276
#0 apr_pool_cleanup_kill (p=0x7f128c0623c8,
data=data@entry=0x7f128c062450,
cleanup_fn=cleanup_fn@entry=0x7f129fe89a40 <socket_cleanup>)
at memory/unix/apr_pools.c:2276
#1 0x00007f129fe85e91 in apr_pool_cleanup_run (p=<optimized out>,
data=0x7f128c062450, cleanup_fn=0x7f129fe89a40 <socket_cleanup>)
at memory/unix/apr_pools.c:2342
#2 0x00007f129fe89d22 in apr_socket_close (thesocket=<optimized out>)
at network_io/unix/sockets.c:183
#3 0x00000000004fa54e in process_lingering_close (cs=0x7f128c062658,
pfd=0x1a4afa8) at event.c:1510
#4 0x00000000004fdc30 in listener_thread (thd=0x7f128c0623c8,
dummy=0x7f128c062658) at event.c:1837
#5 0x00007f129f6100a4 in start_thread ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007f129f14162d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Reading symbols from /usr/local/apache/bin/httpd...Reading symbols from
/usr/lib/debug//usr/local/apache2/bin/httpd...done.
done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/local/apache2/bin/httpd -k start'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f129f612274 in pthread_mutex_lock ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#0 0x00007f129f612274 in pthread_mutex_lock ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f129fe83bc9 in apr_thread_mutex_lock (mutex=<optimized out>)
at locks/unix/thread_mutex.c:92
#2 0x00007f129fe82712 in apr_file_seek
(thefile=thefile@entry=0x7f120c0670c0,
where=where@entry=0, offset=offset@entry=0x7f12897f9b60)
at file_io/unix/seek.c:62
#3 0x00000000004dc413 in read_to_scratch (b=0x7f11c807ea08,
io=0x7f11c80806d8)
at h2_conn_io.c:220
#4 h2_conn_io_pass (io=io@entry=0x7f11c80806d8, bb=0x7f11c8080a08)
at h2_conn_io.c:434
#5 0x00000000004ca3ae in on_send_data_cb (ngh2=<optimized out>,
frame=<optimized out>, framehd=<optimized out>, length=1291,
source=<optimized out>, userp=0x7f11c8080690) at h2_session.c:649
#6 0x00007f12a0980e95 in ?? () from
/usr/lib/x86_64-linux-gnu/libnghttp2.so.14
#7 0x00007f12a0981ea9 in nghttp2_session_send ()
from /usr/lib/x86_64-linux-gnu/libnghttp2.so.14
#8 0x00000000004cca89 in h2_session_send (session=0x7f11c8080690)
at h2_session.c:1414
#9 0x00000000004cfc8c in h2_session_process (session=0x7f11c8080690,
async=1)
at h2_session.c:2246
#10 0x00000000004bf641 in h2_conn_run (ctx=0x7f120c066730, c=0x7f128c06cd98)
at h2_conn.c:214
#11 0x00000000004c202a in h2_h2_process_conn (c=0x3732323535395f39)
at h2_h2.c:658
#12 0x000000000046a450 in ap_run_process_connection (c=0x7f128c06cd98)
at connection.c:42
#13 0x00000000004fbf10 in process_socket (my_thread_num=<optimized out>,
my_child_num=<optimized out>, cs=0x7f128c06cd08, sock=<optimized out>,
p=<optimized out>, thd=<optimized out>) at event.c:1134
#14 worker_thread (thd=0x3732323535395f39, dummy=0x0) at event.c:2137
#15 0x00007f129f6100a4 in start_thread ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#16 0x00007f129f14162d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Stefan
Am 20.01.2017 um 23:28 schrieb Stefan Priebe - Profihost AG:
> Hi,
>
> this patch solved the file_close segfault. Never saw that again.
>
> Right now i got this one:
>
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0 0x00007f129f612274 in pthread_mutex_lock () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> (gdb) bt
> #0 0x00007f129f612274 in pthread_mutex_lock () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00007f129fe83bc9 in apr_thread_mutex_lock (mutex=<optimized out>)
> at locks/unix/thread_mutex.c:92
> #2 0x00007f129fe82712 in apr_file_seek
> (thefile=thefile@entry=0x7f120c0670c0, where=where@entry=0,
> offset=offset@entry=0x7f12897f9b60) at file_io/unix/seek.c:62
> #3 0x00000000004dc413 in read_to_scratch (b=0x7f11c807ea08,
> io=0x7f11c80806d8) at h2_conn_io.c:220
> #4 h2_conn_io_pass (io=io@entry=0x7f11c80806d8, bb=0x7f11c8080a08) at
> h2_conn_io.c:434
> #5 0x00000000004ca3ae in on_send_data_cb (ngh2=<optimized out>,
> frame=<optimized out>, framehd=<optimized out>, length=1291,
> source=<optimized out>, userp=0x7f11c8080690) at h2_session.c:649
> #6 0x00007f12a0980e95 in ?? () from
> /usr/lib/x86_64-linux-gnu/libnghttp2.so.14
> #7 0x00007f12a0981ea9 in nghttp2_session_send () from
> /usr/lib/x86_64-linux-gnu/libnghttp2.so.14
> #8 0x00000000004cca89 in h2_session_send (session=0x7f11c8080690) at
> h2_session.c:1414
> #9 0x00000000004cfc8c in h2_session_process (session=0x7f11c8080690,
> async=1) at h2_session.c:2246
> #10 0x00000000004bf641 in h2_conn_run (ctx=0x7f120c066730,
> c=0x7f128c06cd98) at h2_conn.c:214
> #11 0x00000000004c202a in h2_h2_process_conn (c=0x3732323535395f39) at
> h2_h2.c:658
> #12 0x000000000046a450 in ap_run_process_connection (c=0x7f128c06cd98)
> at connection.c:42
> #13 0x00000000004fbf10 in process_socket (my_thread_num=<optimized out>,
> my_child_num=<optimized out>, cs=0x7f128c06cd08,
> sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at
> event.c:1134
> #14 worker_thread (thd=0x3732323535395f39, dummy=0x0) at event.c:2137
> #15 0x00007f129f6100a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #16 0x00007f129f14162d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Stefan
>
> Am 20.01.2017 um 18:17 schrieb Yann Ylavic:
>> On Fri, Jan 20, 2017 at 5:51 PM, Stefan Priebe - Profihost AG
>> <s....@profihost.ag> wrote:
>>> Yes. Until now I got 4 traces. But all are the same pointing to apr kill
>>> pool. Not like before where i got many different ones.
>>
>> Could you try this new patch on mod_http2 please?
>>
>> Thanks,
>> Yann.
>>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Hi,
this patch solved the file_close segfault. Never saw that again.
Right now i got this one:
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f129f612274 in pthread_mutex_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
(gdb) bt
#0 0x00007f129f612274 in pthread_mutex_lock () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f129fe83bc9 in apr_thread_mutex_lock (mutex=<optimized out>)
at locks/unix/thread_mutex.c:92
#2 0x00007f129fe82712 in apr_file_seek
(thefile=thefile@entry=0x7f120c0670c0, where=where@entry=0,
offset=offset@entry=0x7f12897f9b60) at file_io/unix/seek.c:62
#3 0x00000000004dc413 in read_to_scratch (b=0x7f11c807ea08,
io=0x7f11c80806d8) at h2_conn_io.c:220
#4 h2_conn_io_pass (io=io@entry=0x7f11c80806d8, bb=0x7f11c8080a08) at
h2_conn_io.c:434
#5 0x00000000004ca3ae in on_send_data_cb (ngh2=<optimized out>,
frame=<optimized out>, framehd=<optimized out>, length=1291,
source=<optimized out>, userp=0x7f11c8080690) at h2_session.c:649
#6 0x00007f12a0980e95 in ?? () from
/usr/lib/x86_64-linux-gnu/libnghttp2.so.14
#7 0x00007f12a0981ea9 in nghttp2_session_send () from
/usr/lib/x86_64-linux-gnu/libnghttp2.so.14
#8 0x00000000004cca89 in h2_session_send (session=0x7f11c8080690) at
h2_session.c:1414
#9 0x00000000004cfc8c in h2_session_process (session=0x7f11c8080690,
async=1) at h2_session.c:2246
#10 0x00000000004bf641 in h2_conn_run (ctx=0x7f120c066730,
c=0x7f128c06cd98) at h2_conn.c:214
#11 0x00000000004c202a in h2_h2_process_conn (c=0x3732323535395f39) at
h2_h2.c:658
#12 0x000000000046a450 in ap_run_process_connection (c=0x7f128c06cd98)
at connection.c:42
#13 0x00000000004fbf10 in process_socket (my_thread_num=<optimized out>,
my_child_num=<optimized out>, cs=0x7f128c06cd08,
sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at
event.c:1134
#14 worker_thread (thd=0x3732323535395f39, dummy=0x0) at event.c:2137
#15 0x00007f129f6100a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#16 0x00007f129f14162d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Stefan
Am 20.01.2017 um 18:17 schrieb Yann Ylavic:
> On Fri, Jan 20, 2017 at 5:51 PM, Stefan Priebe - Profihost AG
> <s....@profihost.ag> wrote:
>> Yes. Until now I got 4 traces. But all are the same pointing to apr kill
>> pool. Not like before where i got many different ones.
>
> Could you try this new patch on mod_http2 please?
>
> Thanks,
> Yann.
>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Am 20.01.2017 um 18:17 schrieb Yann Ylavic:
> On Fri, Jan 20, 2017 at 5:51 PM, Stefan Priebe - Profihost AG
> <s....@profihost.ag> wrote:
>> Yes. Until now I got 4 traces. But all are the same pointing to apr kill
>> pool. Not like before where i got many different ones.
>
> Could you try this new patch on mod_http2 please?
status ist still used in the log_cerror function. compilation fails. For
now i just commented the ap_log_cerror call.
Stefan
>
> Thanks,
> Yann.
>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Eissing <st...@greenbytes.de>.
Good catch, Yann.
I have local modifications that throw most of the manual cleanup out and rely on auto pool cleanup as much as possible. Curious if that will solve Stefan's crashes.
> Am 20.01.2017 um 18:17 schrieb Yann Ylavic <yl...@gmail.com>:
>
> On Fri, Jan 20, 2017 at 5:51 PM, Stefan Priebe - Profihost AG
> <s....@profihost.ag> wrote:
>> Yes. Until now I got 4 traces. But all are the same pointing to apr kill
>> pool. Not like before where i got many different ones.
>
> Could you try this new patch on mod_http2 please?
>
> Thanks,
> Yann.
> <stream_pool_cleanup.patch>
Stefan Eissing
<green/>bytes GmbH
Hafenstrasse 16
48155 Münster
www.greenbytes.de
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
On Fri, Jan 20, 2017 at 5:51 PM, Stefan Priebe - Profihost AG
<s....@profihost.ag> wrote:
> Yes. Until now I got 4 traces. But all are the same pointing to apr kill
> pool. Not like before where i got many different ones.
Could you try this new patch on mod_http2 please?
Thanks,
Yann.
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Yes. Until now I got 4 traces. But all are the same pointing to apr kill pool. Not like before where i got many different ones.
Greets,
Stefan
Excuse my typo sent from my mobile phone.
> Am 20.01.2017 um 17:44 schrieb Yann Ylavic <yl...@gmail.com>:
>
> On Fri, Jan 20, 2017 at 5:41 PM, Stefan Priebe - Profihost AG
> <s....@profihost.ag> wrote:
>> Sorry misstyped it's V7 with mod_http 1.8.8 unpatched.
>
> Unpatched but still compiled with -fno-strict-aliasing (along with APR)?
>
> Thanks,
> Yann.
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
On Fri, Jan 20, 2017 at 5:41 PM, Stefan Priebe - Profihost AG
<s....@profihost.ag> wrote:
> Sorry misstyped it's V7 with mod_http 1.8.8 unpatched.
Unpatched but still compiled with -fno-strict-aliasing (along with APR)?
Thanks,
Yann.
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Sorry misstyped it's V7 with mod_http 1.8.8 unpatched.
Stefan
Excuse my typo sent from my mobile phone.
> Am 20.01.2017 um 16:23 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>
> Hi,
>
> it crashed again with V6 and plain mod_http2 v1.8.8.
>
> This is the crash incl. line numbers:
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0 apr_pool_cleanup_kill (p=0x2d392d3333322d32,
> data=data@entry=0x7f330006bc70,
> cleanup_fn=cleanup_fn@entry=0x7f33155b0f90 <apr_unix_file_cleanup>)
> at memory/unix/apr_pools.c:2264
> 2264 memory/unix/apr_pools.c: No such file or directory.
> (gdb) bt
> #0 apr_pool_cleanup_kill (p=0x2d392d3333322d32,
> data=data@entry=0x7f330006bc70,
> cleanup_fn=cleanup_fn@entry=0x7f33155b0f90 <apr_unix_file_cleanup>)
> at memory/unix/apr_pools.c:2264
> #1 0x00007f33155b5e51 in apr_pool_cleanup_run (p=<optimized out>,
> data=0x7f330006bc70,
> cleanup_fn=0x7f33155b0f90 <apr_unix_file_cleanup>) at
> memory/unix/apr_pools.c:2342
> #2 0x00007f33155b1322 in apr_file_close (file=<optimized out>) at
> file_io/unix/open.c:255
> #3 0x00000000004d0012 in stream_pool_cleanup (ctx=0x7f3294022480) at
> h2_stream.c:182
> #4 0x00007f33155b4b1e in run_cleanups (cref=<optimized out>) at
> memory/unix/apr_pools.c:2352
> #5 apr_pool_destroy (pool=0x7f3294022408) at memory/unix/apr_pools.c:814
> #6 0x00000000004d0786 in h2_stream_destroy (stream=<optimized out>) at
> h2_stream.c:249
> #7 0x00000000004c334c in stream_done (m=<optimized out>,
> stream=<optimized out>, rst_error=<optimized out>) at h2_mplx.c:470
> #8 0x00000000004c335b in stream_done_iter (ctx=<optimized out>,
> val=<optimized out>) at h2_mplx.c:475
> #9 0x00007f33155ac156 in apr_hash_do (comp=comp@entry=0x4d5880
> <ihash_iter>, rec=rec@entry=0x7f32f7feea50, ht=<optimized out>)
> at tables/apr_hash.c:542
> #10 0x00000000004d623d in h2_ihash_iter (ih=<optimized out>,
> fn=fn@entry=0x4c3350 <stream_done_iter>, ctx=ctx@entry=0x7f3294034c88)
> at h2_util.c:315
> #11 0x00000000004c4649 in h2_mplx_release_and_join (m=0x7f3294034c88,
> wait=0x7f3294034c30) at h2_mplx.c:579
> #12 0x00000000004ca7cf in h2_session_destroy (session=0x7f3294034a50) at
> h2_session.c:739
> #13 0x000000000045b726 in remove_empty_buckets (bb=0x7f330006af18) at
> core_filters.c:720
> #14 0x000000000045be28 in setaside_remaining_output (f=0x7f32ec0c5b88,
> ctx=0x7f32ec0c5e48, bb=0x7f330006af18, c=<optimized out>,
> c=<optimized out>) at core_filters.c:584
> #15 0x000000000045c896 in ap_core_output_filter (f=0x2d392d3333322d32,
> new_bb=0x7f330006af18) at core_filters.c:568
> #16 0x00000000004ad932 in ssl_io_filter_output (f=0x7f330006aed0,
> bb=0x7f32ec0c5f10) at ssl_engine_io.c:1716
> #17 0x00000000004aad0a in ssl_io_filter_coalesce (f=0x2d392d3333322d32,
> bb=0x7f32ec0c5f10) at ssl_engine_io.c:1663
> #18 0x00000000004db543 in pass_output (io=0x7f3294034a98,
> session_eoc=0x7f3294034a50, flush=<optimized out>) at h2_conn_io.c:311
> #19 0x00000000004cf50a in h2_session_process (session=0x7f3294034a50,
> async=1) at h2_session.c:2347
> #20 0x00000000004befb2 in h2_conn_run (ctx=0x7f32ec0c5e18,
> c=0x7f330006a958) at h2_conn.c:214
> #21 0x00000000004c198a in h2_h2_process_conn (c=0x2d392d3333322d32) at
> h2_h2.c:658
> #22 0x000000000046a2b0 in ap_run_process_connection (c=0x7f330006a958)
> at connection.c:42
> #23 0x00000000004fb890 in process_socket (my_thread_num=<optimized out>,
> my_child_num=<optimized out>, cs=0x7f330006a8c8,
> sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at
> event.c:1134
> #24 worker_thread (thd=0x2d392d3333322d32, dummy=0x7f330006bc70) at
> event.c:2137
> #25 0x00007f3314d400a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #26 0x00007f331487162d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Stefan
>> Am 20.01.2017 um 13:20 schrieb Stefan Eissing:
>> Please without. Then I least know if that version behaves. Planning on more extensive changes for a 1.9.0 now. Thanks!
>>
>> -Stefan
>>
>>> Am 20.01.2017 um 13:18 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>
>>> Will start retesting V6 patch. Should I use mod_http2 1.8.8 with or without patches?
>>>
>>> Greets,
>>> Stefan
>>>
>>> Excuse my typo sent from my mobile phone.
>>>
>>>> Am 20.01.2017 um 13:04 schrieb Stefan Eissing <st...@greenbytes.de>:
>>>>
>>>> Different apr versions? Might there have been a bugfix affecting us?
>>>>
>>>>> Am 20.01.2017 um 12:49 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>>>
>>>>> might this be a debian bug? i can't reproduce with apr-included.
>>>>>
>>>>>> Am 20.01.2017 um 08:20 schrieb Yann Ylavic:
>>>>>> Hi,
>>>>>>
>>>>>> On Fri, Jan 20, 2017 at 8:03 AM, Stefan Priebe - Profihost AG
>>>>>> <s....@profihost.ag> wrote:
>>>>>>> Hi Stefan,
>>>>>>>
>>>>>>>> Am 19.01.2017 um 22:44 schrieb Stefan Eissing:
>>>>>>>> this seems to be a tough bone to chew. Therefore we need to go deeper:
>>>>>>>> - can you compile the module so that we see line numbers in the trace?
>>>>>>>
>>>>>>> Do you have any idea how to arrange this? I've no idea how to pass the
>>>>>>> -ggdb option through Apache.
>>>>>>
>>>>>> DEB_CFLAGS_SET="-O2 -ggdb -fno-strict-aliasing ..." dpkg-buildpackage ...
>>>>>>
>>>>>>>
>>>>>>>> - which apr version are you using?
>>>>>>> this one:
>>>>>>> https://packages.debian.org/jessie/libapr1
>>>>>>
>>>>>> Could you also build libapr1 with this same flags?
>>>>>>
>>>>
>>>> Stefan Eissing
>>>>
>>>> <green/>bytes GmbH
>>>> Hafenstrasse 16
>>>> 48155 Münster
>>>> www.greenbytes.de
>>>>
>>
>> Stefan Eissing
>>
>> <green/>bytes GmbH
>> Hafenstrasse 16
>> 48155 Münster
>> www.greenbytes.de
>>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Hi,
it crashed again with V6 and plain mod_http2 v1.8.8.
This is the crash incl. line numbers:
Program terminated with signal SIGSEGV, Segmentation fault.
#0 apr_pool_cleanup_kill (p=0x2d392d3333322d32,
data=data@entry=0x7f330006bc70,
cleanup_fn=cleanup_fn@entry=0x7f33155b0f90 <apr_unix_file_cleanup>)
at memory/unix/apr_pools.c:2264
2264 memory/unix/apr_pools.c: No such file or directory.
(gdb) bt
#0 apr_pool_cleanup_kill (p=0x2d392d3333322d32,
data=data@entry=0x7f330006bc70,
cleanup_fn=cleanup_fn@entry=0x7f33155b0f90 <apr_unix_file_cleanup>)
at memory/unix/apr_pools.c:2264
#1 0x00007f33155b5e51 in apr_pool_cleanup_run (p=<optimized out>,
data=0x7f330006bc70,
cleanup_fn=0x7f33155b0f90 <apr_unix_file_cleanup>) at
memory/unix/apr_pools.c:2342
#2 0x00007f33155b1322 in apr_file_close (file=<optimized out>) at
file_io/unix/open.c:255
#3 0x00000000004d0012 in stream_pool_cleanup (ctx=0x7f3294022480) at
h2_stream.c:182
#4 0x00007f33155b4b1e in run_cleanups (cref=<optimized out>) at
memory/unix/apr_pools.c:2352
#5 apr_pool_destroy (pool=0x7f3294022408) at memory/unix/apr_pools.c:814
#6 0x00000000004d0786 in h2_stream_destroy (stream=<optimized out>) at
h2_stream.c:249
#7 0x00000000004c334c in stream_done (m=<optimized out>,
stream=<optimized out>, rst_error=<optimized out>) at h2_mplx.c:470
#8 0x00000000004c335b in stream_done_iter (ctx=<optimized out>,
val=<optimized out>) at h2_mplx.c:475
#9 0x00007f33155ac156 in apr_hash_do (comp=comp@entry=0x4d5880
<ihash_iter>, rec=rec@entry=0x7f32f7feea50, ht=<optimized out>)
at tables/apr_hash.c:542
#10 0x00000000004d623d in h2_ihash_iter (ih=<optimized out>,
fn=fn@entry=0x4c3350 <stream_done_iter>, ctx=ctx@entry=0x7f3294034c88)
at h2_util.c:315
#11 0x00000000004c4649 in h2_mplx_release_and_join (m=0x7f3294034c88,
wait=0x7f3294034c30) at h2_mplx.c:579
#12 0x00000000004ca7cf in h2_session_destroy (session=0x7f3294034a50) at
h2_session.c:739
#13 0x000000000045b726 in remove_empty_buckets (bb=0x7f330006af18) at
core_filters.c:720
#14 0x000000000045be28 in setaside_remaining_output (f=0x7f32ec0c5b88,
ctx=0x7f32ec0c5e48, bb=0x7f330006af18, c=<optimized out>,
c=<optimized out>) at core_filters.c:584
#15 0x000000000045c896 in ap_core_output_filter (f=0x2d392d3333322d32,
new_bb=0x7f330006af18) at core_filters.c:568
#16 0x00000000004ad932 in ssl_io_filter_output (f=0x7f330006aed0,
bb=0x7f32ec0c5f10) at ssl_engine_io.c:1716
#17 0x00000000004aad0a in ssl_io_filter_coalesce (f=0x2d392d3333322d32,
bb=0x7f32ec0c5f10) at ssl_engine_io.c:1663
#18 0x00000000004db543 in pass_output (io=0x7f3294034a98,
session_eoc=0x7f3294034a50, flush=<optimized out>) at h2_conn_io.c:311
#19 0x00000000004cf50a in h2_session_process (session=0x7f3294034a50,
async=1) at h2_session.c:2347
#20 0x00000000004befb2 in h2_conn_run (ctx=0x7f32ec0c5e18,
c=0x7f330006a958) at h2_conn.c:214
#21 0x00000000004c198a in h2_h2_process_conn (c=0x2d392d3333322d32) at
h2_h2.c:658
#22 0x000000000046a2b0 in ap_run_process_connection (c=0x7f330006a958)
at connection.c:42
#23 0x00000000004fb890 in process_socket (my_thread_num=<optimized out>,
my_child_num=<optimized out>, cs=0x7f330006a8c8,
sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at
event.c:1134
#24 worker_thread (thd=0x2d392d3333322d32, dummy=0x7f330006bc70) at
event.c:2137
#25 0x00007f3314d400a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#26 0x00007f331487162d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Stefan
Am 20.01.2017 um 13:20 schrieb Stefan Eissing:
> Please without. Then I least know if that version behaves. Planning on more extensive changes for a 1.9.0 now. Thanks!
>
> -Stefan
>
>> Am 20.01.2017 um 13:18 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>
>> Will start retesting V6 patch. Should I use mod_http2 1.8.8 with or without patches?
>>
>> Greets,
>> Stefan
>>
>> Excuse my typo sent from my mobile phone.
>>
>> Am 20.01.2017 um 13:04 schrieb Stefan Eissing <st...@greenbytes.de>:
>>
>>> Different apr versions? Might there have been a bugfix affecting us?
>>>
>>>> Am 20.01.2017 um 12:49 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>>
>>>> might this be a debian bug? i can't reproduce with apr-included.
>>>>
>>>> Am 20.01.2017 um 08:20 schrieb Yann Ylavic:
>>>>> Hi,
>>>>>
>>>>> On Fri, Jan 20, 2017 at 8:03 AM, Stefan Priebe - Profihost AG
>>>>> <s....@profihost.ag> wrote:
>>>>>> Hi Stefan,
>>>>>>
>>>>>> Am 19.01.2017 um 22:44 schrieb Stefan Eissing:
>>>>>>> this seems to be a tough bone to chew. Therefore we need to go deeper:
>>>>>>> - can you compile the module so that we see line numbers in the trace?
>>>>>>
>>>>>> Do you have any idea how to arrange this? I've no idea how to pass the
>>>>>> -ggdb option through Apache.
>>>>>
>>>>> DEB_CFLAGS_SET="-O2 -ggdb -fno-strict-aliasing ..." dpkg-buildpackage ...
>>>>>
>>>>>>
>>>>>>> - which apr version are you using?
>>>>>> this one:
>>>>>> https://packages.debian.org/jessie/libapr1
>>>>>
>>>>> Could you also build libapr1 with this same flags?
>>>>>
>>>
>>> Stefan Eissing
>>>
>>> <green/>bytes GmbH
>>> Hafenstrasse 16
>>> 48155 M�nster
>>> www.greenbytes.de
>>>
>
> Stefan Eissing
>
> <green/>bytes GmbH
> Hafenstrasse 16
> 48155 M�nster
> www.greenbytes.de
>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
Please use V7 too, I don't think I'll revert it in trunk (it makes sense).
On Fri, Jan 20, 2017 at 1:20 PM, Stefan Eissing
<st...@greenbytes.de> wrote:
> Please without. Then I least know if that version behaves. Planning on more extensive changes for a 1.9.0 now. Thanks!
>
> -Stefan
>
>> Am 20.01.2017 um 13:18 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>
>> Will start retesting V6 patch. Should I use mod_http2 1.8.8 with or without patches?
>>
>> Greets,
>> Stefan
>>
>> Excuse my typo sent from my mobile phone.
>>
>> Am 20.01.2017 um 13:04 schrieb Stefan Eissing <st...@greenbytes.de>:
>>
>>> Different apr versions? Might there have been a bugfix affecting us?
>>>
>>>> Am 20.01.2017 um 12:49 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>>
>>>> might this be a debian bug? i can't reproduce with apr-included.
>>>>
>>>> Am 20.01.2017 um 08:20 schrieb Yann Ylavic:
>>>>> Hi,
>>>>>
>>>>> On Fri, Jan 20, 2017 at 8:03 AM, Stefan Priebe - Profihost AG
>>>>> <s....@profihost.ag> wrote:
>>>>>> Hi Stefan,
>>>>>>
>>>>>> Am 19.01.2017 um 22:44 schrieb Stefan Eissing:
>>>>>>> this seems to be a tough bone to chew. Therefore we need to go deeper:
>>>>>>> - can you compile the module so that we see line numbers in the trace?
>>>>>>
>>>>>> Do you have any idea how to arrange this? I've no idea how to pass the
>>>>>> -ggdb option through Apache.
>>>>>
>>>>> DEB_CFLAGS_SET="-O2 -ggdb -fno-strict-aliasing ..." dpkg-buildpackage ...
>>>>>
>>>>>>
>>>>>>> - which apr version are you using?
>>>>>> this one:
>>>>>> https://packages.debian.org/jessie/libapr1
>>>>>
>>>>> Could you also build libapr1 with this same flags?
>>>>>
>>>
>>> Stefan Eissing
>>>
>>> <green/>bytes GmbH
>>> Hafenstrasse 16
>>> 48155 Münster
>>> www.greenbytes.de
>>>
>
> Stefan Eissing
>
> <green/>bytes GmbH
> Hafenstrasse 16
> 48155 Münster
> www.greenbytes.de
>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Eissing <st...@greenbytes.de>.
Please without. Then I least know if that version behaves. Planning on more extensive changes for a 1.9.0 now. Thanks!
-Stefan
> Am 20.01.2017 um 13:18 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>
> Will start retesting V6 patch. Should I use mod_http2 1.8.8 with or without patches?
>
> Greets,
> Stefan
>
> Excuse my typo sent from my mobile phone.
>
> Am 20.01.2017 um 13:04 schrieb Stefan Eissing <st...@greenbytes.de>:
>
>> Different apr versions? Might there have been a bugfix affecting us?
>>
>>> Am 20.01.2017 um 12:49 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>
>>> might this be a debian bug? i can't reproduce with apr-included.
>>>
>>> Am 20.01.2017 um 08:20 schrieb Yann Ylavic:
>>>> Hi,
>>>>
>>>> On Fri, Jan 20, 2017 at 8:03 AM, Stefan Priebe - Profihost AG
>>>> <s....@profihost.ag> wrote:
>>>>> Hi Stefan,
>>>>>
>>>>> Am 19.01.2017 um 22:44 schrieb Stefan Eissing:
>>>>>> this seems to be a tough bone to chew. Therefore we need to go deeper:
>>>>>> - can you compile the module so that we see line numbers in the trace?
>>>>>
>>>>> Do you have any idea how to arrange this? I've no idea how to pass the
>>>>> -ggdb option through Apache.
>>>>
>>>> DEB_CFLAGS_SET="-O2 -ggdb -fno-strict-aliasing ..." dpkg-buildpackage ...
>>>>
>>>>>
>>>>>> - which apr version are you using?
>>>>> this one:
>>>>> https://packages.debian.org/jessie/libapr1
>>>>
>>>> Could you also build libapr1 with this same flags?
>>>>
>>
>> Stefan Eissing
>>
>> <green/>bytes GmbH
>> Hafenstrasse 16
>> 48155 Münster
>> www.greenbytes.de
>>
Stefan Eissing
<green/>bytes GmbH
Hafenstrasse 16
48155 Münster
www.greenbytes.de
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Will start retesting V6 patch. Should I use mod_http2 1.8.8 with or without patches?
Greets,
Stefan
Excuse my typo sent from my mobile phone.
> Am 20.01.2017 um 13:04 schrieb Stefan Eissing <st...@greenbytes.de>:
>
> Different apr versions? Might there have been a bugfix affecting us?
>
>> Am 20.01.2017 um 12:49 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>
>> might this be a debian bug? i can't reproduce with apr-included.
>>
>>> Am 20.01.2017 um 08:20 schrieb Yann Ylavic:
>>> Hi,
>>>
>>> On Fri, Jan 20, 2017 at 8:03 AM, Stefan Priebe - Profihost AG
>>> <s....@profihost.ag> wrote:
>>>> Hi Stefan,
>>>>
>>>>> Am 19.01.2017 um 22:44 schrieb Stefan Eissing:
>>>>> this seems to be a tough bone to chew. Therefore we need to go deeper:
>>>>> - can you compile the module so that we see line numbers in the trace?
>>>>
>>>> Do you have any idea how to arrange this? I've no idea how to pass the
>>>> -ggdb option through Apache.
>>>
>>> DEB_CFLAGS_SET="-O2 -ggdb -fno-strict-aliasing ..." dpkg-buildpackage ...
>>>
>>>>
>>>>> - which apr version are you using?
>>>> this one:
>>>> https://packages.debian.org/jessie/libapr1
>>>
>>> Could you also build libapr1 with this same flags?
>>>
>
> Stefan Eissing
>
> <green/>bytes GmbH
> Hafenstrasse 16
> 48155 Münster
> www.greenbytes.de
>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Eissing <st...@greenbytes.de>.
Different apr versions? Might there have been a bugfix affecting us?
> Am 20.01.2017 um 12:49 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>
> might this be a debian bug? i can't reproduce with apr-included.
>
> Am 20.01.2017 um 08:20 schrieb Yann Ylavic:
>> Hi,
>>
>> On Fri, Jan 20, 2017 at 8:03 AM, Stefan Priebe - Profihost AG
>> <s....@profihost.ag> wrote:
>>> Hi Stefan,
>>>
>>> Am 19.01.2017 um 22:44 schrieb Stefan Eissing:
>>>> this seems to be a tough bone to chew. Therefore we need to go deeper:
>>>> - can you compile the module so that we see line numbers in the trace?
>>>
>>> Do you have any idea how to arrange this? I've no idea how to pass the
>>> -ggdb option through Apache.
>>
>> DEB_CFLAGS_SET="-O2 -ggdb -fno-strict-aliasing ..." dpkg-buildpackage ...
>>
>>>
>>>> - which apr version are you using?
>>> this one:
>>> https://packages.debian.org/jessie/libapr1
>>
>> Could you also build libapr1 with this same flags?
>>
Stefan Eissing
<green/>bytes GmbH
Hafenstrasse 16
48155 Münster
www.greenbytes.de
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
might this be a debian bug? i can't reproduce with apr-included.
Am 20.01.2017 um 08:20 schrieb Yann Ylavic:
> Hi,
>
> On Fri, Jan 20, 2017 at 8:03 AM, Stefan Priebe - Profihost AG
> <s....@profihost.ag> wrote:
>> Hi Stefan,
>>
>> Am 19.01.2017 um 22:44 schrieb Stefan Eissing:
>>> this seems to be a tough bone to chew. Therefore we need to go deeper:
>>> - can you compile the module so that we see line numbers in the trace?
>>
>> Do you have any idea how to arrange this? I've no idea how to pass the
>> -ggdb option through Apache.
>
> DEB_CFLAGS_SET="-O2 -ggdb -fno-strict-aliasing ..." dpkg-buildpackage ...
>
>>
>>> - which apr version are you using?
>> this one:
>> https://packages.debian.org/jessie/libapr1
>
> Could you also build libapr1 with this same flags?
>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Am 20.01.2017 um 08:20 schrieb Yann Ylavic:
> Hi,
>
> On Fri, Jan 20, 2017 at 8:03 AM, Stefan Priebe - Profihost AG
> <s....@profihost.ag> wrote:
>> Hi Stefan,
>>
>> Am 19.01.2017 um 22:44 schrieb Stefan Eissing:
>>> this seems to be a tough bone to chew. Therefore we need to go deeper:
>>> - can you compile the module so that we see line numbers in the trace?
>>
>> Do you have any idea how to arrange this? I've no idea how to pass the
>> -ggdb option through Apache.
>
> DEB_CFLAGS_SET="-O2 -ggdb -fno-strict-aliasing ..." dpkg-buildpackage ...
OK i switched now to included apr and included apr-util. And i added
those CFLAGS to my configure line.
Have to wait for a new crash.
Stefan
>
>>
>>> - which apr version are you using?
>> this one:
>> https://packages.debian.org/jessie/libapr1
>
> Could you also build libapr1 with this same flags?
>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
Hi,
On Fri, Jan 20, 2017 at 8:03 AM, Stefan Priebe - Profihost AG
<s....@profihost.ag> wrote:
> Hi Stefan,
>
> Am 19.01.2017 um 22:44 schrieb Stefan Eissing:
>> this seems to be a tough bone to chew. Therefore we need to go deeper:
>> - can you compile the module so that we see line numbers in the trace?
>
> Do you have any idea how to arrange this? I've no idea how to pass the
> -ggdb option through Apache.
DEB_CFLAGS_SET="-O2 -ggdb -fno-strict-aliasing ..." dpkg-buildpackage ...
>
>> - which apr version are you using?
> this one:
> https://packages.debian.org/jessie/libapr1
Could you also build libapr1 with this same flags?
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Hi Stefan,
Am 19.01.2017 um 22:44 schrieb Stefan Eissing:
> this seems to be a tough bone to chew. Therefore we need to go deeper:
> - can you compile the module so that we see line numbers in the trace?
Do you have any idea how to arrange this? I've no idea how to pass the
-ggdb option through Apache. I'm compiling mod_http2 as a static module
included in httpd.
> - which apr version are you using?
this one:
https://packages.debian.org/jessie/libapr1
> - can you reproduce this at will? How? Which client? Just a GET or something more sophisticated?
No i can't ;-) i've just a bunch of live webshops producing this. So
just real users - most probably GET + POST.
Stefan
>
> Thanks for the help!
>
> Cheers,
>
> Stefan
>
>> Am 19.01.2017 um 22:01 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>
>> new segfault with both patches on top of v1.8.8:
>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
>> Program terminated with signal SIGSEGV, Segmentation fault.
>> #0 0x00007fbefce0d014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>> #0 0x00007fbefce0d014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>> #1 0x00007fbefd2a0036 in ?? () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>> #2 0x00007fbefd2a046f in apr_hash_set () from
>> /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>> #3 0x000000000052a26b in h2_ihash_remove ()
>> #4 0x0000000000506b24 in purge_stream ()
>> #5 0x000000000052a1c2 in ihash_iter ()
>> #6 0x00007fbefd2a08a6 in apr_hash_do () from
>> /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>> #7 0x000000000052a200 in h2_ihash_iter ()
>> #8 0x0000000000506b65 in purge_streams ()
>> #9 0x00000000005082dd in h2_mplx_release_and_join ()
>> #10 0x00000000005158f9 in h2_session_cleanup ()
>> #11 0x00000000005164a4 in session_pool_cleanup ()
>> #12 0x00007fbefd2a9976 in apr_pool_destroy () from
>> /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>> #13 0x00007fbefd2a9c55 in apr_pool_clear () from
>> /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>> #14 0x0000000000566acd in ap_push_pool ()
>> #15 0x000000000056008a in process_lingering_close ()
>> #16 0x0000000000560ced in listener_thread ()
>> #17 0x00007fbefd0780a4 in start_thread () from
>> /lib/x86_64-linux-gnu/libpthread.so.0
>> #18 0x00007fbefcdad62d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>
>> Stefan
>>
>> Am 19.01.2017 um 21:48 schrieb Stefan Eissing:
>>> On top please. There is only one way: forward!
>>>
>>>> Am 19.01.2017 um 21:47 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>>
>>>>
>>>> Am 19.01.2017 um 21:39 schrieb Stefan Eissing:
>>>>> Thanks, Stefan. Can you given the attached Patch a try?
>>>>
>>>> sure. On top of the last one? Or should i drop it?
>>>>
>>>>
>>>> Stefan
>>>>
>>>>>> Am 19.01.2017 um 19:33 schrieb Stefan Priebe <s....@profihost.ag>:
>>>>>>
>>>>>> Here some more segfaults from 2.4.25 no mpm patch but http2 v1.8.8:
>>>>>>
>>>>>> #################################################################
>>>>>>
>>>>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>>>> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
>>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>>> #0 0x00007f61f673b014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>> #0 0x00007f61f673b014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>> #1 0x00007f61f6bce036 in ?? () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>>>>> #2 0x00007f61f6bce46f in apr_hash_set () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>>>>> #3 0x000000000052a26d in h2_ihash_remove ()
>>>>>> #4 0x0000000000506b24 in purge_stream ()
>>>>>> #5 0x000000000052a1c4 in ihash_iter ()
>>>>>> #6 0x00007f61f6bce8a6 in apr_hash_do () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>>>>> #7 0x000000000052a202 in h2_ihash_iter ()
>>>>>> #8 0x0000000000506b65 in purge_streams ()
>>>>>> #9 0x00000000005082df in h2_mplx_release_and_join ()
>>>>>> #10 0x00000000005158fb in h2_session_cleanup ()
>>>>>> #11 0x00000000005164a6 in session_pool_cleanup ()
>>>>>> #12 0x00007f61f6bd7976 in apr_pool_destroy () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>>>>> #13 0x00007f61f6bd7c55 in apr_pool_clear () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>>>>> #14 0x0000000000566acf in ap_push_pool ()
>>>>>> #15 0x000000000056008c in process_lingering_close ()
>>>>>> #16 0x0000000000560cef in listener_thread ()
>>>>>> #17 0x00007f61f69a60a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>> #18 0x00007f61f66db62d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>>
>>>>>> #################################################################
>>>>>>
>>>>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>>>> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
>>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>>> #0 0x00007f61f673b014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>> #0 0x00007f61f673b014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>> #1 0x00007f61f6bce036 in ?? () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>>>>> #2 0x00007f61f6bce46f in apr_hash_set () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>>>>> #3 0x000000000052a26d in h2_ihash_remove ()
>>>>>> #4 0x0000000000506b24 in purge_stream ()
>>>>>> #5 0x000000000052a1c4 in ihash_iter ()
>>>>>> #6 0x00007f61f6bce8a6 in apr_hash_do () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>>>>> #7 0x000000000052a202 in h2_ihash_iter ()
>>>>>> #8 0x0000000000506b65 in purge_streams ()
>>>>>> #9 0x00000000005082df in h2_mplx_release_and_join ()
>>>>>> #10 0x00000000005158fb in h2_session_cleanup ()
>>>>>> #11 0x00000000005164a6 in session_pool_cleanup ()
>>>>>> #12 0x00007f61f6bd7976 in apr_pool_destroy () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>>>>> #13 0x00007f61f6bd7c55 in apr_pool_clear () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>>>>> #14 0x0000000000566acf in ap_push_pool ()
>>>>>> #15 0x000000000056008c in process_lingering_close ()
>>>>>> #16 0x0000000000560cef in listener_thread ()
>>>>>> #17 0x00007f61f69a60a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>> #18 0x00007f61f66db62d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>>
>>>>>> #################################################################
>>>>>>
>>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>>> #0 0x00007f204f922d63 in apr_pool_cleanup_kill () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>>>>> (gdb) bt
>>>>>> #0 0x00007f204f922d63 in apr_pool_cleanup_kill () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>>>>> #1 0x00007f204f922e21 in apr_pool_cleanup_run () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>>>>> #2 0x000000000055ffe9 in process_lingering_close ()
>>>>>> #3 0x0000000000560cef in listener_thread ()
>>>>>> #4 0x00007f204f6f00a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>> #5 0x00007f204f42562d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>>
>>>>>> full bt:
>>>>>> http://pastebin.com/raw/bP1vaYjw
>>>>>>
>>>>>> #################################################################
>>>>>>
>>>>>> Greets,
>>>>>> Stefan
>>>>>>
>>>>>> Am 19.01.2017 um 16:47 schrieb Stefan Priebe - Profihost AG:
>>>>>>> arg sorry my fault.
>>>>>>>
>>>>>>> Here is a complete trace:
>>>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>>>> #0 0x00007fc1c23e0f23 in apr_brigade_length () from
>>>>>>> /usr/lib/x86_64-linux-gnu/libaprutil-1.so.0
>>>>>>> (gdb) bt
>>>>>>> #0 0x00007fc1c23e0f23 in apr_brigade_length () from
>>>>>>> /usr/lib/x86_64-linux-gnu/libaprutil-1.so.0
>>>>>>> #1 0x000000000052a8f1 in h2_util_bb_avail ()
>>>>>>> #2 0x0000000000521eaa in h2_stream_out_prepare ()
>>>>>>> #3 0x000000000051a55a in on_stream_resume ()
>>>>>>> #4 0x000000000050bdff in h2_mplx_dispatch_master_events ()
>>>>>>> #5 0x000000000051dc32 in h2_session_process ()
>>>>>>> #6 0x0000000000500115 in h2_conn_run ()
>>>>>>> #7 0x0000000000504b51 in h2_h2_process_conn ()
>>>>>>> #8 0x000000000047cb19 in ap_run_process_connection ()
>>>>>>> #9 0x000000000055e755 in process_socket ()
>>>>>>> #10 0x0000000000560f5c in worker_thread ()
>>>>>>> #11 0x00007fc1c1f8d0a4 in start_thread () from
>>>>>>> /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>>> #12 0x00007fc1c1cc262d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>>>
>>>>>>> Stefan
>>>>>>>
>>>>>>> Am 19.01.2017 um 16:45 schrieb Stefan Priebe - Profihost AG:
>>>>>>>>
>>>>>>>> Am 19.01.2017 um 16:34 schrieb Stefan Eissing:
>>>>>>>>> Yann might already have asked this: any chance to compile with symbols and get a more readable stacktrace?
>>>>>>>>
>>>>>>>> Yes just tell me how ;-) i'm using dpkg-buildpackage and dh_strip. I've
>>>>>>>> no idea why not all symbols are available.
>>>>>>>>
>>>>>>>> Do i need to pass a specific option to configure
>>>>>>>>
>>>>>>>> Stefan
>>>>>>>>
>>>>>>>>>> Am 19.01.2017 um 16:30 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>>>>>>>>
>>>>>>>>>> With stock 2.4.25 + patch i'm getting this one again:
>>>>>>>>>> (gdb) bt
>>>>>>>>>> #0 0x0000000000521dcd in h2_stream_out_prepare ()
>>>>>>>>>> #1 0x00007fc1a2feca80 in ?? ()
>>>>>>>>>> #2 0x00007fc1a2feca8c in ?? ()
>>>>>>>>>> #3 0x00007fc1a2feca90 in ?? ()
>>>>>>>>>> #4 0x00007fc1a057c0a0 in ?? ()
>>>>>>>>>> #5 0x00007fc1a057cdd8 in ?? ()
>>>>>>>>>> #6 0x00007fc1a2fecac0 in ?? ()
>>>>>>>>>> #7 0x0000000000000000 in ?? ()
>>>>>>>>>>
>>>>>>>>>> Stefan
>>>>>>>>>>
>>>>>>>>>> Am 19.01.2017 um 16:28 schrieb Stefan Priebe - Profihost AG:
>>>>>>>>>>> I'm now testing stock 2.4.25 + patch.
>>>>>>>>>>>
>>>>>>>>>>> May this configure option have an influence?
>>>>>>>>>>> --enable-nonportable-atomics=yes
>>>>>>>>>>>
>>>>>>>>>>> Greets,
>>>>>>>>>>> Stefan
>>>>>>>>>>>
>>>>>>>>>>> Am 19.01.2017 um 15:35 schrieb Yann Ylavic:
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Jan 19, 2017 at 3:00 PM, Stefan Priebe - Profihost AG
>>>>>>>>>>>> <s....@profihost.ag> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> @Yann:
>>>>>>>>>>>>> should i use V7 or V6?
>>>>>>>>>>>>
>>>>>>>>>>>> I'd prefer you'd use none (such that we can verify the patch with
>>>>>>>>>>>> stock 2.4.25, modulo mod_http2), but if it's easier for you to
>>>>>>>>>>>> reproduce with an event's patch, please use the v6 (and if it fails
>>>>>>>>>>>> then v7, and if it fails then no patch, really).
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Stefan Eissing
>>>>>>>>>
>>>>>>>>> <green/>bytes GmbH
>>>>>>>>> Hafenstrasse 16
>>>>>>>>> 48155 M�nster
>>>>>>>>> www.greenbytes.de
>>>>>>>>>
>>>>>
>>>>> Stefan Eissing
>>>>>
>>>>> <green/>bytes GmbH
>>>>> Hafenstrasse 16
>>>>> 48155 M�nster
>>>>> www.greenbytes.de
>>>>>
>>>
>>> Stefan Eissing
>>>
>>> <green/>bytes GmbH
>>> Hafenstrasse 16
>>> 48155 M�nster
>>> www.greenbytes.de
>>>
>
> Stefan Eissing
>
> <green/>bytes GmbH
> Hafenstrasse 16
> 48155 M�nster
> www.greenbytes.de
>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Eissing <st...@greenbytes.de>.
Stefan,
this seems to be a tough bone to chew. Therefore we need to go deeper:
- can you compile the module so that we see line numbers in the trace?
- which apr version are you using?
- can you reproduce this at will? How? Which client? Just a GET or something more sophisticated?
Thanks for the help!
Cheers,
Stefan
> Am 19.01.2017 um 22:01 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>
> new segfault with both patches on top of v1.8.8:
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0 0x00007fbefce0d014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> #0 0x00007fbefce0d014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> #1 0x00007fbefd2a0036 in ?? () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> #2 0x00007fbefd2a046f in apr_hash_set () from
> /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> #3 0x000000000052a26b in h2_ihash_remove ()
> #4 0x0000000000506b24 in purge_stream ()
> #5 0x000000000052a1c2 in ihash_iter ()
> #6 0x00007fbefd2a08a6 in apr_hash_do () from
> /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> #7 0x000000000052a200 in h2_ihash_iter ()
> #8 0x0000000000506b65 in purge_streams ()
> #9 0x00000000005082dd in h2_mplx_release_and_join ()
> #10 0x00000000005158f9 in h2_session_cleanup ()
> #11 0x00000000005164a4 in session_pool_cleanup ()
> #12 0x00007fbefd2a9976 in apr_pool_destroy () from
> /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> #13 0x00007fbefd2a9c55 in apr_pool_clear () from
> /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> #14 0x0000000000566acd in ap_push_pool ()
> #15 0x000000000056008a in process_lingering_close ()
> #16 0x0000000000560ced in listener_thread ()
> #17 0x00007fbefd0780a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #18 0x00007fbefcdad62d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Stefan
>
> Am 19.01.2017 um 21:48 schrieb Stefan Eissing:
>> On top please. There is only one way: forward!
>>
>>> Am 19.01.2017 um 21:47 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>
>>>
>>> Am 19.01.2017 um 21:39 schrieb Stefan Eissing:
>>>> Thanks, Stefan. Can you given the attached Patch a try?
>>>
>>> sure. On top of the last one? Or should i drop it?
>>>
>>>
>>> Stefan
>>>
>>>>> Am 19.01.2017 um 19:33 schrieb Stefan Priebe <s....@profihost.ag>:
>>>>>
>>>>> Here some more segfaults from 2.4.25 no mpm patch but http2 v1.8.8:
>>>>>
>>>>> #################################################################
>>>>>
>>>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>>> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>> #0 0x00007f61f673b014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>>>>> #0 0x00007f61f673b014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>>>>> #1 0x00007f61f6bce036 in ?? () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>>>> #2 0x00007f61f6bce46f in apr_hash_set () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>>>> #3 0x000000000052a26d in h2_ihash_remove ()
>>>>> #4 0x0000000000506b24 in purge_stream ()
>>>>> #5 0x000000000052a1c4 in ihash_iter ()
>>>>> #6 0x00007f61f6bce8a6 in apr_hash_do () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>>>> #7 0x000000000052a202 in h2_ihash_iter ()
>>>>> #8 0x0000000000506b65 in purge_streams ()
>>>>> #9 0x00000000005082df in h2_mplx_release_and_join ()
>>>>> #10 0x00000000005158fb in h2_session_cleanup ()
>>>>> #11 0x00000000005164a6 in session_pool_cleanup ()
>>>>> #12 0x00007f61f6bd7976 in apr_pool_destroy () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>>>> #13 0x00007f61f6bd7c55 in apr_pool_clear () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>>>> #14 0x0000000000566acf in ap_push_pool ()
>>>>> #15 0x000000000056008c in process_lingering_close ()
>>>>> #16 0x0000000000560cef in listener_thread ()
>>>>> #17 0x00007f61f69a60a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>> #18 0x00007f61f66db62d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>
>>>>> #################################################################
>>>>>
>>>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>>> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>> #0 0x00007f61f673b014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>>>>> #0 0x00007f61f673b014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>>>>> #1 0x00007f61f6bce036 in ?? () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>>>> #2 0x00007f61f6bce46f in apr_hash_set () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>>>> #3 0x000000000052a26d in h2_ihash_remove ()
>>>>> #4 0x0000000000506b24 in purge_stream ()
>>>>> #5 0x000000000052a1c4 in ihash_iter ()
>>>>> #6 0x00007f61f6bce8a6 in apr_hash_do () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>>>> #7 0x000000000052a202 in h2_ihash_iter ()
>>>>> #8 0x0000000000506b65 in purge_streams ()
>>>>> #9 0x00000000005082df in h2_mplx_release_and_join ()
>>>>> #10 0x00000000005158fb in h2_session_cleanup ()
>>>>> #11 0x00000000005164a6 in session_pool_cleanup ()
>>>>> #12 0x00007f61f6bd7976 in apr_pool_destroy () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>>>> #13 0x00007f61f6bd7c55 in apr_pool_clear () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>>>> #14 0x0000000000566acf in ap_push_pool ()
>>>>> #15 0x000000000056008c in process_lingering_close ()
>>>>> #16 0x0000000000560cef in listener_thread ()
>>>>> #17 0x00007f61f69a60a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>> #18 0x00007f61f66db62d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>
>>>>> #################################################################
>>>>>
>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>> #0 0x00007f204f922d63 in apr_pool_cleanup_kill () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>>>> (gdb) bt
>>>>> #0 0x00007f204f922d63 in apr_pool_cleanup_kill () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>>>> #1 0x00007f204f922e21 in apr_pool_cleanup_run () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>>>> #2 0x000000000055ffe9 in process_lingering_close ()
>>>>> #3 0x0000000000560cef in listener_thread ()
>>>>> #4 0x00007f204f6f00a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>> #5 0x00007f204f42562d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>
>>>>> full bt:
>>>>> http://pastebin.com/raw/bP1vaYjw
>>>>>
>>>>> #################################################################
>>>>>
>>>>> Greets,
>>>>> Stefan
>>>>>
>>>>> Am 19.01.2017 um 16:47 schrieb Stefan Priebe - Profihost AG:
>>>>>> arg sorry my fault.
>>>>>>
>>>>>> Here is a complete trace:
>>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>>> #0 0x00007fc1c23e0f23 in apr_brigade_length () from
>>>>>> /usr/lib/x86_64-linux-gnu/libaprutil-1.so.0
>>>>>> (gdb) bt
>>>>>> #0 0x00007fc1c23e0f23 in apr_brigade_length () from
>>>>>> /usr/lib/x86_64-linux-gnu/libaprutil-1.so.0
>>>>>> #1 0x000000000052a8f1 in h2_util_bb_avail ()
>>>>>> #2 0x0000000000521eaa in h2_stream_out_prepare ()
>>>>>> #3 0x000000000051a55a in on_stream_resume ()
>>>>>> #4 0x000000000050bdff in h2_mplx_dispatch_master_events ()
>>>>>> #5 0x000000000051dc32 in h2_session_process ()
>>>>>> #6 0x0000000000500115 in h2_conn_run ()
>>>>>> #7 0x0000000000504b51 in h2_h2_process_conn ()
>>>>>> #8 0x000000000047cb19 in ap_run_process_connection ()
>>>>>> #9 0x000000000055e755 in process_socket ()
>>>>>> #10 0x0000000000560f5c in worker_thread ()
>>>>>> #11 0x00007fc1c1f8d0a4 in start_thread () from
>>>>>> /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>> #12 0x00007fc1c1cc262d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>>
>>>>>> Stefan
>>>>>>
>>>>>> Am 19.01.2017 um 16:45 schrieb Stefan Priebe - Profihost AG:
>>>>>>>
>>>>>>> Am 19.01.2017 um 16:34 schrieb Stefan Eissing:
>>>>>>>> Yann might already have asked this: any chance to compile with symbols and get a more readable stacktrace?
>>>>>>>
>>>>>>> Yes just tell me how ;-) i'm using dpkg-buildpackage and dh_strip. I've
>>>>>>> no idea why not all symbols are available.
>>>>>>>
>>>>>>> Do i need to pass a specific option to configure
>>>>>>>
>>>>>>> Stefan
>>>>>>>
>>>>>>>>> Am 19.01.2017 um 16:30 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>>>>>>>
>>>>>>>>> With stock 2.4.25 + patch i'm getting this one again:
>>>>>>>>> (gdb) bt
>>>>>>>>> #0 0x0000000000521dcd in h2_stream_out_prepare ()
>>>>>>>>> #1 0x00007fc1a2feca80 in ?? ()
>>>>>>>>> #2 0x00007fc1a2feca8c in ?? ()
>>>>>>>>> #3 0x00007fc1a2feca90 in ?? ()
>>>>>>>>> #4 0x00007fc1a057c0a0 in ?? ()
>>>>>>>>> #5 0x00007fc1a057cdd8 in ?? ()
>>>>>>>>> #6 0x00007fc1a2fecac0 in ?? ()
>>>>>>>>> #7 0x0000000000000000 in ?? ()
>>>>>>>>>
>>>>>>>>> Stefan
>>>>>>>>>
>>>>>>>>> Am 19.01.2017 um 16:28 schrieb Stefan Priebe - Profihost AG:
>>>>>>>>>> I'm now testing stock 2.4.25 + patch.
>>>>>>>>>>
>>>>>>>>>> May this configure option have an influence?
>>>>>>>>>> --enable-nonportable-atomics=yes
>>>>>>>>>>
>>>>>>>>>> Greets,
>>>>>>>>>> Stefan
>>>>>>>>>>
>>>>>>>>>> Am 19.01.2017 um 15:35 schrieb Yann Ylavic:
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Jan 19, 2017 at 3:00 PM, Stefan Priebe - Profihost AG
>>>>>>>>>>> <s....@profihost.ag> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> @Yann:
>>>>>>>>>>>> should i use V7 or V6?
>>>>>>>>>>>
>>>>>>>>>>> I'd prefer you'd use none (such that we can verify the patch with
>>>>>>>>>>> stock 2.4.25, modulo mod_http2), but if it's easier for you to
>>>>>>>>>>> reproduce with an event's patch, please use the v6 (and if it fails
>>>>>>>>>>> then v7, and if it fails then no patch, really).
>>>>>>>>>>>
>>>>>>>>
>>>>>>>> Stefan Eissing
>>>>>>>>
>>>>>>>> <green/>bytes GmbH
>>>>>>>> Hafenstrasse 16
>>>>>>>> 48155 Münster
>>>>>>>> www.greenbytes.de
>>>>>>>>
>>>>
>>>> Stefan Eissing
>>>>
>>>> <green/>bytes GmbH
>>>> Hafenstrasse 16
>>>> 48155 Münster
>>>> www.greenbytes.de
>>>>
>>
>> Stefan Eissing
>>
>> <green/>bytes GmbH
>> Hafenstrasse 16
>> 48155 Münster
>> www.greenbytes.de
>>
Stefan Eissing
<green/>bytes GmbH
Hafenstrasse 16
48155 Münster
www.greenbytes.de
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
new segfault with both patches on top of v1.8.8:
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/local/apache2/bin/httpd -k start'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007fbefce0d014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#0 0x00007fbefce0d014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fbefd2a0036 in ?? () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
#2 0x00007fbefd2a046f in apr_hash_set () from
/usr/lib/x86_64-linux-gnu/libapr-1.so.0
#3 0x000000000052a26b in h2_ihash_remove ()
#4 0x0000000000506b24 in purge_stream ()
#5 0x000000000052a1c2 in ihash_iter ()
#6 0x00007fbefd2a08a6 in apr_hash_do () from
/usr/lib/x86_64-linux-gnu/libapr-1.so.0
#7 0x000000000052a200 in h2_ihash_iter ()
#8 0x0000000000506b65 in purge_streams ()
#9 0x00000000005082dd in h2_mplx_release_and_join ()
#10 0x00000000005158f9 in h2_session_cleanup ()
#11 0x00000000005164a4 in session_pool_cleanup ()
#12 0x00007fbefd2a9976 in apr_pool_destroy () from
/usr/lib/x86_64-linux-gnu/libapr-1.so.0
#13 0x00007fbefd2a9c55 in apr_pool_clear () from
/usr/lib/x86_64-linux-gnu/libapr-1.so.0
#14 0x0000000000566acd in ap_push_pool ()
#15 0x000000000056008a in process_lingering_close ()
#16 0x0000000000560ced in listener_thread ()
#17 0x00007fbefd0780a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#18 0x00007fbefcdad62d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Stefan
Am 19.01.2017 um 21:48 schrieb Stefan Eissing:
> On top please. There is only one way: forward!
>
>> Am 19.01.2017 um 21:47 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>
>>
>> Am 19.01.2017 um 21:39 schrieb Stefan Eissing:
>>> Thanks, Stefan. Can you given the attached Patch a try?
>>
>> sure. On top of the last one? Or should i drop it?
>>
>>
>> Stefan
>>
>>>> Am 19.01.2017 um 19:33 schrieb Stefan Priebe <s....@profihost.ag>:
>>>>
>>>> Here some more segfaults from 2.4.25 no mpm patch but http2 v1.8.8:
>>>>
>>>> #################################################################
>>>>
>>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>> #0 0x00007f61f673b014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>>>> #0 0x00007f61f673b014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>>>> #1 0x00007f61f6bce036 in ?? () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>>> #2 0x00007f61f6bce46f in apr_hash_set () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>>> #3 0x000000000052a26d in h2_ihash_remove ()
>>>> #4 0x0000000000506b24 in purge_stream ()
>>>> #5 0x000000000052a1c4 in ihash_iter ()
>>>> #6 0x00007f61f6bce8a6 in apr_hash_do () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>>> #7 0x000000000052a202 in h2_ihash_iter ()
>>>> #8 0x0000000000506b65 in purge_streams ()
>>>> #9 0x00000000005082df in h2_mplx_release_and_join ()
>>>> #10 0x00000000005158fb in h2_session_cleanup ()
>>>> #11 0x00000000005164a6 in session_pool_cleanup ()
>>>> #12 0x00007f61f6bd7976 in apr_pool_destroy () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>>> #13 0x00007f61f6bd7c55 in apr_pool_clear () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>>> #14 0x0000000000566acf in ap_push_pool ()
>>>> #15 0x000000000056008c in process_lingering_close ()
>>>> #16 0x0000000000560cef in listener_thread ()
>>>> #17 0x00007f61f69a60a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
>>>> #18 0x00007f61f66db62d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>
>>>> #################################################################
>>>>
>>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>> #0 0x00007f61f673b014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>>>> #0 0x00007f61f673b014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>>>> #1 0x00007f61f6bce036 in ?? () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>>> #2 0x00007f61f6bce46f in apr_hash_set () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>>> #3 0x000000000052a26d in h2_ihash_remove ()
>>>> #4 0x0000000000506b24 in purge_stream ()
>>>> #5 0x000000000052a1c4 in ihash_iter ()
>>>> #6 0x00007f61f6bce8a6 in apr_hash_do () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>>> #7 0x000000000052a202 in h2_ihash_iter ()
>>>> #8 0x0000000000506b65 in purge_streams ()
>>>> #9 0x00000000005082df in h2_mplx_release_and_join ()
>>>> #10 0x00000000005158fb in h2_session_cleanup ()
>>>> #11 0x00000000005164a6 in session_pool_cleanup ()
>>>> #12 0x00007f61f6bd7976 in apr_pool_destroy () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>>> #13 0x00007f61f6bd7c55 in apr_pool_clear () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>>> #14 0x0000000000566acf in ap_push_pool ()
>>>> #15 0x000000000056008c in process_lingering_close ()
>>>> #16 0x0000000000560cef in listener_thread ()
>>>> #17 0x00007f61f69a60a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
>>>> #18 0x00007f61f66db62d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>
>>>> #################################################################
>>>>
>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>> #0 0x00007f204f922d63 in apr_pool_cleanup_kill () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>>> (gdb) bt
>>>> #0 0x00007f204f922d63 in apr_pool_cleanup_kill () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>>> #1 0x00007f204f922e21 in apr_pool_cleanup_run () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>>> #2 0x000000000055ffe9 in process_lingering_close ()
>>>> #3 0x0000000000560cef in listener_thread ()
>>>> #4 0x00007f204f6f00a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
>>>> #5 0x00007f204f42562d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>
>>>> full bt:
>>>> http://pastebin.com/raw/bP1vaYjw
>>>>
>>>> #################################################################
>>>>
>>>> Greets,
>>>> Stefan
>>>>
>>>> Am 19.01.2017 um 16:47 schrieb Stefan Priebe - Profihost AG:
>>>>> arg sorry my fault.
>>>>>
>>>>> Here is a complete trace:
>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>> #0 0x00007fc1c23e0f23 in apr_brigade_length () from
>>>>> /usr/lib/x86_64-linux-gnu/libaprutil-1.so.0
>>>>> (gdb) bt
>>>>> #0 0x00007fc1c23e0f23 in apr_brigade_length () from
>>>>> /usr/lib/x86_64-linux-gnu/libaprutil-1.so.0
>>>>> #1 0x000000000052a8f1 in h2_util_bb_avail ()
>>>>> #2 0x0000000000521eaa in h2_stream_out_prepare ()
>>>>> #3 0x000000000051a55a in on_stream_resume ()
>>>>> #4 0x000000000050bdff in h2_mplx_dispatch_master_events ()
>>>>> #5 0x000000000051dc32 in h2_session_process ()
>>>>> #6 0x0000000000500115 in h2_conn_run ()
>>>>> #7 0x0000000000504b51 in h2_h2_process_conn ()
>>>>> #8 0x000000000047cb19 in ap_run_process_connection ()
>>>>> #9 0x000000000055e755 in process_socket ()
>>>>> #10 0x0000000000560f5c in worker_thread ()
>>>>> #11 0x00007fc1c1f8d0a4 in start_thread () from
>>>>> /lib/x86_64-linux-gnu/libpthread.so.0
>>>>> #12 0x00007fc1c1cc262d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>>
>>>>> Stefan
>>>>>
>>>>> Am 19.01.2017 um 16:45 schrieb Stefan Priebe - Profihost AG:
>>>>>>
>>>>>> Am 19.01.2017 um 16:34 schrieb Stefan Eissing:
>>>>>>> Yann might already have asked this: any chance to compile with symbols and get a more readable stacktrace?
>>>>>>
>>>>>> Yes just tell me how ;-) i'm using dpkg-buildpackage and dh_strip. I've
>>>>>> no idea why not all symbols are available.
>>>>>>
>>>>>> Do i need to pass a specific option to configure
>>>>>>
>>>>>> Stefan
>>>>>>
>>>>>>>> Am 19.01.2017 um 16:30 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>>>>>>
>>>>>>>> With stock 2.4.25 + patch i'm getting this one again:
>>>>>>>> (gdb) bt
>>>>>>>> #0 0x0000000000521dcd in h2_stream_out_prepare ()
>>>>>>>> #1 0x00007fc1a2feca80 in ?? ()
>>>>>>>> #2 0x00007fc1a2feca8c in ?? ()
>>>>>>>> #3 0x00007fc1a2feca90 in ?? ()
>>>>>>>> #4 0x00007fc1a057c0a0 in ?? ()
>>>>>>>> #5 0x00007fc1a057cdd8 in ?? ()
>>>>>>>> #6 0x00007fc1a2fecac0 in ?? ()
>>>>>>>> #7 0x0000000000000000 in ?? ()
>>>>>>>>
>>>>>>>> Stefan
>>>>>>>>
>>>>>>>> Am 19.01.2017 um 16:28 schrieb Stefan Priebe - Profihost AG:
>>>>>>>>> I'm now testing stock 2.4.25 + patch.
>>>>>>>>>
>>>>>>>>> May this configure option have an influence?
>>>>>>>>> --enable-nonportable-atomics=yes
>>>>>>>>>
>>>>>>>>> Greets,
>>>>>>>>> Stefan
>>>>>>>>>
>>>>>>>>> Am 19.01.2017 um 15:35 schrieb Yann Ylavic:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> On Thu, Jan 19, 2017 at 3:00 PM, Stefan Priebe - Profihost AG
>>>>>>>>>> <s....@profihost.ag> wrote:
>>>>>>>>>>>
>>>>>>>>>>> @Yann:
>>>>>>>>>>> should i use V7 or V6?
>>>>>>>>>>
>>>>>>>>>> I'd prefer you'd use none (such that we can verify the patch with
>>>>>>>>>> stock 2.4.25, modulo mod_http2), but if it's easier for you to
>>>>>>>>>> reproduce with an event's patch, please use the v6 (and if it fails
>>>>>>>>>> then v7, and if it fails then no patch, really).
>>>>>>>>>>
>>>>>>>
>>>>>>> Stefan Eissing
>>>>>>>
>>>>>>> <green/>bytes GmbH
>>>>>>> Hafenstrasse 16
>>>>>>> 48155 M�nster
>>>>>>> www.greenbytes.de
>>>>>>>
>>>
>>> Stefan Eissing
>>>
>>> <green/>bytes GmbH
>>> Hafenstrasse 16
>>> 48155 M�nster
>>> www.greenbytes.de
>>>
>
> Stefan Eissing
>
> <green/>bytes GmbH
> Hafenstrasse 16
> 48155 M�nster
> www.greenbytes.de
>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Eissing <st...@greenbytes.de>.
On top please. There is only one way: forward!
> Am 19.01.2017 um 21:47 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>
>
> Am 19.01.2017 um 21:39 schrieb Stefan Eissing:
>> Thanks, Stefan. Can you given the attached Patch a try?
>
> sure. On top of the last one? Or should i drop it?
>
>
> Stefan
>
>>> Am 19.01.2017 um 19:33 schrieb Stefan Priebe <s....@profihost.ag>:
>>>
>>> Here some more segfaults from 2.4.25 no mpm patch but http2 v1.8.8:
>>>
>>> #################################################################
>>>
>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>> #0 0x00007f61f673b014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>>> #0 0x00007f61f673b014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>>> #1 0x00007f61f6bce036 in ?? () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>> #2 0x00007f61f6bce46f in apr_hash_set () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>> #3 0x000000000052a26d in h2_ihash_remove ()
>>> #4 0x0000000000506b24 in purge_stream ()
>>> #5 0x000000000052a1c4 in ihash_iter ()
>>> #6 0x00007f61f6bce8a6 in apr_hash_do () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>> #7 0x000000000052a202 in h2_ihash_iter ()
>>> #8 0x0000000000506b65 in purge_streams ()
>>> #9 0x00000000005082df in h2_mplx_release_and_join ()
>>> #10 0x00000000005158fb in h2_session_cleanup ()
>>> #11 0x00000000005164a6 in session_pool_cleanup ()
>>> #12 0x00007f61f6bd7976 in apr_pool_destroy () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>> #13 0x00007f61f6bd7c55 in apr_pool_clear () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>> #14 0x0000000000566acf in ap_push_pool ()
>>> #15 0x000000000056008c in process_lingering_close ()
>>> #16 0x0000000000560cef in listener_thread ()
>>> #17 0x00007f61f69a60a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
>>> #18 0x00007f61f66db62d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>
>>> #################################################################
>>>
>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>> #0 0x00007f61f673b014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>>> #0 0x00007f61f673b014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>>> #1 0x00007f61f6bce036 in ?? () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>> #2 0x00007f61f6bce46f in apr_hash_set () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>> #3 0x000000000052a26d in h2_ihash_remove ()
>>> #4 0x0000000000506b24 in purge_stream ()
>>> #5 0x000000000052a1c4 in ihash_iter ()
>>> #6 0x00007f61f6bce8a6 in apr_hash_do () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>> #7 0x000000000052a202 in h2_ihash_iter ()
>>> #8 0x0000000000506b65 in purge_streams ()
>>> #9 0x00000000005082df in h2_mplx_release_and_join ()
>>> #10 0x00000000005158fb in h2_session_cleanup ()
>>> #11 0x00000000005164a6 in session_pool_cleanup ()
>>> #12 0x00007f61f6bd7976 in apr_pool_destroy () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>> #13 0x00007f61f6bd7c55 in apr_pool_clear () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>> #14 0x0000000000566acf in ap_push_pool ()
>>> #15 0x000000000056008c in process_lingering_close ()
>>> #16 0x0000000000560cef in listener_thread ()
>>> #17 0x00007f61f69a60a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
>>> #18 0x00007f61f66db62d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>
>>> #################################################################
>>>
>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>> #0 0x00007f204f922d63 in apr_pool_cleanup_kill () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>> (gdb) bt
>>> #0 0x00007f204f922d63 in apr_pool_cleanup_kill () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>> #1 0x00007f204f922e21 in apr_pool_cleanup_run () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>>> #2 0x000000000055ffe9 in process_lingering_close ()
>>> #3 0x0000000000560cef in listener_thread ()
>>> #4 0x00007f204f6f00a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
>>> #5 0x00007f204f42562d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>
>>> full bt:
>>> http://pastebin.com/raw/bP1vaYjw
>>>
>>> #################################################################
>>>
>>> Greets,
>>> Stefan
>>>
>>> Am 19.01.2017 um 16:47 schrieb Stefan Priebe - Profihost AG:
>>>> arg sorry my fault.
>>>>
>>>> Here is a complete trace:
>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>> #0 0x00007fc1c23e0f23 in apr_brigade_length () from
>>>> /usr/lib/x86_64-linux-gnu/libaprutil-1.so.0
>>>> (gdb) bt
>>>> #0 0x00007fc1c23e0f23 in apr_brigade_length () from
>>>> /usr/lib/x86_64-linux-gnu/libaprutil-1.so.0
>>>> #1 0x000000000052a8f1 in h2_util_bb_avail ()
>>>> #2 0x0000000000521eaa in h2_stream_out_prepare ()
>>>> #3 0x000000000051a55a in on_stream_resume ()
>>>> #4 0x000000000050bdff in h2_mplx_dispatch_master_events ()
>>>> #5 0x000000000051dc32 in h2_session_process ()
>>>> #6 0x0000000000500115 in h2_conn_run ()
>>>> #7 0x0000000000504b51 in h2_h2_process_conn ()
>>>> #8 0x000000000047cb19 in ap_run_process_connection ()
>>>> #9 0x000000000055e755 in process_socket ()
>>>> #10 0x0000000000560f5c in worker_thread ()
>>>> #11 0x00007fc1c1f8d0a4 in start_thread () from
>>>> /lib/x86_64-linux-gnu/libpthread.so.0
>>>> #12 0x00007fc1c1cc262d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>>
>>>> Stefan
>>>>
>>>> Am 19.01.2017 um 16:45 schrieb Stefan Priebe - Profihost AG:
>>>>>
>>>>> Am 19.01.2017 um 16:34 schrieb Stefan Eissing:
>>>>>> Yann might already have asked this: any chance to compile with symbols and get a more readable stacktrace?
>>>>>
>>>>> Yes just tell me how ;-) i'm using dpkg-buildpackage and dh_strip. I've
>>>>> no idea why not all symbols are available.
>>>>>
>>>>> Do i need to pass a specific option to configure
>>>>>
>>>>> Stefan
>>>>>
>>>>>>> Am 19.01.2017 um 16:30 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>>>>>
>>>>>>> With stock 2.4.25 + patch i'm getting this one again:
>>>>>>> (gdb) bt
>>>>>>> #0 0x0000000000521dcd in h2_stream_out_prepare ()
>>>>>>> #1 0x00007fc1a2feca80 in ?? ()
>>>>>>> #2 0x00007fc1a2feca8c in ?? ()
>>>>>>> #3 0x00007fc1a2feca90 in ?? ()
>>>>>>> #4 0x00007fc1a057c0a0 in ?? ()
>>>>>>> #5 0x00007fc1a057cdd8 in ?? ()
>>>>>>> #6 0x00007fc1a2fecac0 in ?? ()
>>>>>>> #7 0x0000000000000000 in ?? ()
>>>>>>>
>>>>>>> Stefan
>>>>>>>
>>>>>>> Am 19.01.2017 um 16:28 schrieb Stefan Priebe - Profihost AG:
>>>>>>>> I'm now testing stock 2.4.25 + patch.
>>>>>>>>
>>>>>>>> May this configure option have an influence?
>>>>>>>> --enable-nonportable-atomics=yes
>>>>>>>>
>>>>>>>> Greets,
>>>>>>>> Stefan
>>>>>>>>
>>>>>>>> Am 19.01.2017 um 15:35 schrieb Yann Ylavic:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> On Thu, Jan 19, 2017 at 3:00 PM, Stefan Priebe - Profihost AG
>>>>>>>>> <s....@profihost.ag> wrote:
>>>>>>>>>>
>>>>>>>>>> @Yann:
>>>>>>>>>> should i use V7 or V6?
>>>>>>>>>
>>>>>>>>> I'd prefer you'd use none (such that we can verify the patch with
>>>>>>>>> stock 2.4.25, modulo mod_http2), but if it's easier for you to
>>>>>>>>> reproduce with an event's patch, please use the v6 (and if it fails
>>>>>>>>> then v7, and if it fails then no patch, really).
>>>>>>>>>
>>>>>>
>>>>>> Stefan Eissing
>>>>>>
>>>>>> <green/>bytes GmbH
>>>>>> Hafenstrasse 16
>>>>>> 48155 Münster
>>>>>> www.greenbytes.de
>>>>>>
>>
>> Stefan Eissing
>>
>> <green/>bytes GmbH
>> Hafenstrasse 16
>> 48155 Münster
>> www.greenbytes.de
>>
Stefan Eissing
<green/>bytes GmbH
Hafenstrasse 16
48155 Münster
www.greenbytes.de
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Am 19.01.2017 um 21:39 schrieb Stefan Eissing:
> Thanks, Stefan. Can you given the attached Patch a try?
sure. On top of the last one? Or should i drop it?
Stefan
>> Am 19.01.2017 um 19:33 schrieb Stefan Priebe <s....@profihost.ag>:
>>
>> Here some more segfaults from 2.4.25 no mpm patch but http2 v1.8.8:
>>
>> #################################################################
>>
>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
>> Program terminated with signal SIGSEGV, Segmentation fault.
>> #0 0x00007f61f673b014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>> #0 0x00007f61f673b014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>> #1 0x00007f61f6bce036 in ?? () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>> #2 0x00007f61f6bce46f in apr_hash_set () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>> #3 0x000000000052a26d in h2_ihash_remove ()
>> #4 0x0000000000506b24 in purge_stream ()
>> #5 0x000000000052a1c4 in ihash_iter ()
>> #6 0x00007f61f6bce8a6 in apr_hash_do () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>> #7 0x000000000052a202 in h2_ihash_iter ()
>> #8 0x0000000000506b65 in purge_streams ()
>> #9 0x00000000005082df in h2_mplx_release_and_join ()
>> #10 0x00000000005158fb in h2_session_cleanup ()
>> #11 0x00000000005164a6 in session_pool_cleanup ()
>> #12 0x00007f61f6bd7976 in apr_pool_destroy () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>> #13 0x00007f61f6bd7c55 in apr_pool_clear () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>> #14 0x0000000000566acf in ap_push_pool ()
>> #15 0x000000000056008c in process_lingering_close ()
>> #16 0x0000000000560cef in listener_thread ()
>> #17 0x00007f61f69a60a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
>> #18 0x00007f61f66db62d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>
>> #################################################################
>>
>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
>> Program terminated with signal SIGSEGV, Segmentation fault.
>> #0 0x00007f61f673b014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>> #0 0x00007f61f673b014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>> #1 0x00007f61f6bce036 in ?? () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>> #2 0x00007f61f6bce46f in apr_hash_set () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>> #3 0x000000000052a26d in h2_ihash_remove ()
>> #4 0x0000000000506b24 in purge_stream ()
>> #5 0x000000000052a1c4 in ihash_iter ()
>> #6 0x00007f61f6bce8a6 in apr_hash_do () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>> #7 0x000000000052a202 in h2_ihash_iter ()
>> #8 0x0000000000506b65 in purge_streams ()
>> #9 0x00000000005082df in h2_mplx_release_and_join ()
>> #10 0x00000000005158fb in h2_session_cleanup ()
>> #11 0x00000000005164a6 in session_pool_cleanup ()
>> #12 0x00007f61f6bd7976 in apr_pool_destroy () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>> #13 0x00007f61f6bd7c55 in apr_pool_clear () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>> #14 0x0000000000566acf in ap_push_pool ()
>> #15 0x000000000056008c in process_lingering_close ()
>> #16 0x0000000000560cef in listener_thread ()
>> #17 0x00007f61f69a60a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
>> #18 0x00007f61f66db62d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>
>> #################################################################
>>
>> Program terminated with signal SIGSEGV, Segmentation fault.
>> #0 0x00007f204f922d63 in apr_pool_cleanup_kill () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>> (gdb) bt
>> #0 0x00007f204f922d63 in apr_pool_cleanup_kill () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>> #1 0x00007f204f922e21 in apr_pool_cleanup_run () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>> #2 0x000000000055ffe9 in process_lingering_close ()
>> #3 0x0000000000560cef in listener_thread ()
>> #4 0x00007f204f6f00a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
>> #5 0x00007f204f42562d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>
>> full bt:
>> http://pastebin.com/raw/bP1vaYjw
>>
>> #################################################################
>>
>> Greets,
>> Stefan
>>
>> Am 19.01.2017 um 16:47 schrieb Stefan Priebe - Profihost AG:
>>> arg sorry my fault.
>>>
>>> Here is a complete trace:
>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>> #0 0x00007fc1c23e0f23 in apr_brigade_length () from
>>> /usr/lib/x86_64-linux-gnu/libaprutil-1.so.0
>>> (gdb) bt
>>> #0 0x00007fc1c23e0f23 in apr_brigade_length () from
>>> /usr/lib/x86_64-linux-gnu/libaprutil-1.so.0
>>> #1 0x000000000052a8f1 in h2_util_bb_avail ()
>>> #2 0x0000000000521eaa in h2_stream_out_prepare ()
>>> #3 0x000000000051a55a in on_stream_resume ()
>>> #4 0x000000000050bdff in h2_mplx_dispatch_master_events ()
>>> #5 0x000000000051dc32 in h2_session_process ()
>>> #6 0x0000000000500115 in h2_conn_run ()
>>> #7 0x0000000000504b51 in h2_h2_process_conn ()
>>> #8 0x000000000047cb19 in ap_run_process_connection ()
>>> #9 0x000000000055e755 in process_socket ()
>>> #10 0x0000000000560f5c in worker_thread ()
>>> #11 0x00007fc1c1f8d0a4 in start_thread () from
>>> /lib/x86_64-linux-gnu/libpthread.so.0
>>> #12 0x00007fc1c1cc262d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>>
>>> Stefan
>>>
>>> Am 19.01.2017 um 16:45 schrieb Stefan Priebe - Profihost AG:
>>>>
>>>> Am 19.01.2017 um 16:34 schrieb Stefan Eissing:
>>>>> Yann might already have asked this: any chance to compile with symbols and get a more readable stacktrace?
>>>>
>>>> Yes just tell me how ;-) i'm using dpkg-buildpackage and dh_strip. I've
>>>> no idea why not all symbols are available.
>>>>
>>>> Do i need to pass a specific option to configure
>>>>
>>>> Stefan
>>>>
>>>>>> Am 19.01.2017 um 16:30 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>>>>
>>>>>> With stock 2.4.25 + patch i'm getting this one again:
>>>>>> (gdb) bt
>>>>>> #0 0x0000000000521dcd in h2_stream_out_prepare ()
>>>>>> #1 0x00007fc1a2feca80 in ?? ()
>>>>>> #2 0x00007fc1a2feca8c in ?? ()
>>>>>> #3 0x00007fc1a2feca90 in ?? ()
>>>>>> #4 0x00007fc1a057c0a0 in ?? ()
>>>>>> #5 0x00007fc1a057cdd8 in ?? ()
>>>>>> #6 0x00007fc1a2fecac0 in ?? ()
>>>>>> #7 0x0000000000000000 in ?? ()
>>>>>>
>>>>>> Stefan
>>>>>>
>>>>>> Am 19.01.2017 um 16:28 schrieb Stefan Priebe - Profihost AG:
>>>>>>> I'm now testing stock 2.4.25 + patch.
>>>>>>>
>>>>>>> May this configure option have an influence?
>>>>>>> --enable-nonportable-atomics=yes
>>>>>>>
>>>>>>> Greets,
>>>>>>> Stefan
>>>>>>>
>>>>>>> Am 19.01.2017 um 15:35 schrieb Yann Ylavic:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> On Thu, Jan 19, 2017 at 3:00 PM, Stefan Priebe - Profihost AG
>>>>>>>> <s....@profihost.ag> wrote:
>>>>>>>>>
>>>>>>>>> @Yann:
>>>>>>>>> should i use V7 or V6?
>>>>>>>>
>>>>>>>> I'd prefer you'd use none (such that we can verify the patch with
>>>>>>>> stock 2.4.25, modulo mod_http2), but if it's easier for you to
>>>>>>>> reproduce with an event's patch, please use the v6 (and if it fails
>>>>>>>> then v7, and if it fails then no patch, really).
>>>>>>>>
>>>>>
>>>>> Stefan Eissing
>>>>>
>>>>> <green/>bytes GmbH
>>>>> Hafenstrasse 16
>>>>> 48155 M�nster
>>>>> www.greenbytes.de
>>>>>
>
> Stefan Eissing
>
> <green/>bytes GmbH
> Hafenstrasse 16
> 48155 M�nster
> www.greenbytes.de
>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Eissing <st...@greenbytes.de>.
Thanks, Stefan. Can you given the attached Patch a try?
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe <s....@profihost.ag>.
Here some more segfaults from 2.4.25 no mpm patch but http2 v1.8.8:
#################################################################
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/local/apache2/bin/httpd -k start'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f61f673b014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#0 0x00007f61f673b014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f61f6bce036 in ?? () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
#2 0x00007f61f6bce46f in apr_hash_set () from
/usr/lib/x86_64-linux-gnu/libapr-1.so.0
#3 0x000000000052a26d in h2_ihash_remove ()
#4 0x0000000000506b24 in purge_stream ()
#5 0x000000000052a1c4 in ihash_iter ()
#6 0x00007f61f6bce8a6 in apr_hash_do () from
/usr/lib/x86_64-linux-gnu/libapr-1.so.0
#7 0x000000000052a202 in h2_ihash_iter ()
#8 0x0000000000506b65 in purge_streams ()
#9 0x00000000005082df in h2_mplx_release_and_join ()
#10 0x00000000005158fb in h2_session_cleanup ()
#11 0x00000000005164a6 in session_pool_cleanup ()
#12 0x00007f61f6bd7976 in apr_pool_destroy () from
/usr/lib/x86_64-linux-gnu/libapr-1.so.0
#13 0x00007f61f6bd7c55 in apr_pool_clear () from
/usr/lib/x86_64-linux-gnu/libapr-1.so.0
#14 0x0000000000566acf in ap_push_pool ()
#15 0x000000000056008c in process_lingering_close ()
#16 0x0000000000560cef in listener_thread ()
#17 0x00007f61f69a60a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#18 0x00007f61f66db62d in clone () from /lib/x86_64-linux-gnu/libc.so.6
#################################################################
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/local/apache2/bin/httpd -k start'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f61f673b014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#0 0x00007f61f673b014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f61f6bce036 in ?? () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
#2 0x00007f61f6bce46f in apr_hash_set () from
/usr/lib/x86_64-linux-gnu/libapr-1.so.0
#3 0x000000000052a26d in h2_ihash_remove ()
#4 0x0000000000506b24 in purge_stream ()
#5 0x000000000052a1c4 in ihash_iter ()
#6 0x00007f61f6bce8a6 in apr_hash_do () from
/usr/lib/x86_64-linux-gnu/libapr-1.so.0
#7 0x000000000052a202 in h2_ihash_iter ()
#8 0x0000000000506b65 in purge_streams ()
#9 0x00000000005082df in h2_mplx_release_and_join ()
#10 0x00000000005158fb in h2_session_cleanup ()
#11 0x00000000005164a6 in session_pool_cleanup ()
#12 0x00007f61f6bd7976 in apr_pool_destroy () from
/usr/lib/x86_64-linux-gnu/libapr-1.so.0
#13 0x00007f61f6bd7c55 in apr_pool_clear () from
/usr/lib/x86_64-linux-gnu/libapr-1.so.0
#14 0x0000000000566acf in ap_push_pool ()
#15 0x000000000056008c in process_lingering_close ()
#16 0x0000000000560cef in listener_thread ()
#17 0x00007f61f69a60a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#18 0x00007f61f66db62d in clone () from /lib/x86_64-linux-gnu/libc.so.6
#################################################################
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f204f922d63 in apr_pool_cleanup_kill () from
/usr/lib/x86_64-linux-gnu/libapr-1.so.0
(gdb) bt
#0 0x00007f204f922d63 in apr_pool_cleanup_kill () from
/usr/lib/x86_64-linux-gnu/libapr-1.so.0
#1 0x00007f204f922e21 in apr_pool_cleanup_run () from
/usr/lib/x86_64-linux-gnu/libapr-1.so.0
#2 0x000000000055ffe9 in process_lingering_close ()
#3 0x0000000000560cef in listener_thread ()
#4 0x00007f204f6f00a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007f204f42562d in clone () from /lib/x86_64-linux-gnu/libc.so.6
full bt:
http://pastebin.com/raw/bP1vaYjw
#################################################################
Greets,
Stefan
Am 19.01.2017 um 16:47 schrieb Stefan Priebe - Profihost AG:
> arg sorry my fault.
>
> Here is a complete trace:
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0 0x00007fc1c23e0f23 in apr_brigade_length () from
> /usr/lib/x86_64-linux-gnu/libaprutil-1.so.0
> (gdb) bt
> #0 0x00007fc1c23e0f23 in apr_brigade_length () from
> /usr/lib/x86_64-linux-gnu/libaprutil-1.so.0
> #1 0x000000000052a8f1 in h2_util_bb_avail ()
> #2 0x0000000000521eaa in h2_stream_out_prepare ()
> #3 0x000000000051a55a in on_stream_resume ()
> #4 0x000000000050bdff in h2_mplx_dispatch_master_events ()
> #5 0x000000000051dc32 in h2_session_process ()
> #6 0x0000000000500115 in h2_conn_run ()
> #7 0x0000000000504b51 in h2_h2_process_conn ()
> #8 0x000000000047cb19 in ap_run_process_connection ()
> #9 0x000000000055e755 in process_socket ()
> #10 0x0000000000560f5c in worker_thread ()
> #11 0x00007fc1c1f8d0a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #12 0x00007fc1c1cc262d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Stefan
>
> Am 19.01.2017 um 16:45 schrieb Stefan Priebe - Profihost AG:
>>
>> Am 19.01.2017 um 16:34 schrieb Stefan Eissing:
>>> Yann might already have asked this: any chance to compile with symbols and get a more readable stacktrace?
>>
>> Yes just tell me how ;-) i'm using dpkg-buildpackage and dh_strip. I've
>> no idea why not all symbols are available.
>>
>> Do i need to pass a specific option to configure
>>
>> Stefan
>>
>>>> Am 19.01.2017 um 16:30 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>>
>>>> With stock 2.4.25 + patch i'm getting this one again:
>>>> (gdb) bt
>>>> #0 0x0000000000521dcd in h2_stream_out_prepare ()
>>>> #1 0x00007fc1a2feca80 in ?? ()
>>>> #2 0x00007fc1a2feca8c in ?? ()
>>>> #3 0x00007fc1a2feca90 in ?? ()
>>>> #4 0x00007fc1a057c0a0 in ?? ()
>>>> #5 0x00007fc1a057cdd8 in ?? ()
>>>> #6 0x00007fc1a2fecac0 in ?? ()
>>>> #7 0x0000000000000000 in ?? ()
>>>>
>>>> Stefan
>>>>
>>>> Am 19.01.2017 um 16:28 schrieb Stefan Priebe - Profihost AG:
>>>>> I'm now testing stock 2.4.25 + patch.
>>>>>
>>>>> May this configure option have an influence?
>>>>> --enable-nonportable-atomics=yes
>>>>>
>>>>> Greets,
>>>>> Stefan
>>>>>
>>>>> Am 19.01.2017 um 15:35 schrieb Yann Ylavic:
>>>>>> Hi,
>>>>>>
>>>>>> On Thu, Jan 19, 2017 at 3:00 PM, Stefan Priebe - Profihost AG
>>>>>> <s....@profihost.ag> wrote:
>>>>>>>
>>>>>>> @Yann:
>>>>>>> should i use V7 or V6?
>>>>>>
>>>>>> I'd prefer you'd use none (such that we can verify the patch with
>>>>>> stock 2.4.25, modulo mod_http2), but if it's easier for you to
>>>>>> reproduce with an event's patch, please use the v6 (and if it fails
>>>>>> then v7, and if it fails then no patch, really).
>>>>>>
>>>
>>> Stefan Eissing
>>>
>>> <green/>bytes GmbH
>>> Hafenstrasse 16
>>> 48155 M�nster
>>> www.greenbytes.de
>>>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
arg sorry my fault.
Here is a complete trace:
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007fc1c23e0f23 in apr_brigade_length () from
/usr/lib/x86_64-linux-gnu/libaprutil-1.so.0
(gdb) bt
#0 0x00007fc1c23e0f23 in apr_brigade_length () from
/usr/lib/x86_64-linux-gnu/libaprutil-1.so.0
#1 0x000000000052a8f1 in h2_util_bb_avail ()
#2 0x0000000000521eaa in h2_stream_out_prepare ()
#3 0x000000000051a55a in on_stream_resume ()
#4 0x000000000050bdff in h2_mplx_dispatch_master_events ()
#5 0x000000000051dc32 in h2_session_process ()
#6 0x0000000000500115 in h2_conn_run ()
#7 0x0000000000504b51 in h2_h2_process_conn ()
#8 0x000000000047cb19 in ap_run_process_connection ()
#9 0x000000000055e755 in process_socket ()
#10 0x0000000000560f5c in worker_thread ()
#11 0x00007fc1c1f8d0a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#12 0x00007fc1c1cc262d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Stefan
Am 19.01.2017 um 16:45 schrieb Stefan Priebe - Profihost AG:
>
> Am 19.01.2017 um 16:34 schrieb Stefan Eissing:
>> Yann might already have asked this: any chance to compile with symbols and get a more readable stacktrace?
>
> Yes just tell me how ;-) i'm using dpkg-buildpackage and dh_strip. I've
> no idea why not all symbols are available.
>
> Do i need to pass a specific option to configure
>
> Stefan
>
>>> Am 19.01.2017 um 16:30 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>
>>> With stock 2.4.25 + patch i'm getting this one again:
>>> (gdb) bt
>>> #0 0x0000000000521dcd in h2_stream_out_prepare ()
>>> #1 0x00007fc1a2feca80 in ?? ()
>>> #2 0x00007fc1a2feca8c in ?? ()
>>> #3 0x00007fc1a2feca90 in ?? ()
>>> #4 0x00007fc1a057c0a0 in ?? ()
>>> #5 0x00007fc1a057cdd8 in ?? ()
>>> #6 0x00007fc1a2fecac0 in ?? ()
>>> #7 0x0000000000000000 in ?? ()
>>>
>>> Stefan
>>>
>>> Am 19.01.2017 um 16:28 schrieb Stefan Priebe - Profihost AG:
>>>> I'm now testing stock 2.4.25 + patch.
>>>>
>>>> May this configure option have an influence?
>>>> --enable-nonportable-atomics=yes
>>>>
>>>> Greets,
>>>> Stefan
>>>>
>>>> Am 19.01.2017 um 15:35 schrieb Yann Ylavic:
>>>>> Hi,
>>>>>
>>>>> On Thu, Jan 19, 2017 at 3:00 PM, Stefan Priebe - Profihost AG
>>>>> <s....@profihost.ag> wrote:
>>>>>>
>>>>>> @Yann:
>>>>>> should i use V7 or V6?
>>>>>
>>>>> I'd prefer you'd use none (such that we can verify the patch with
>>>>> stock 2.4.25, modulo mod_http2), but if it's easier for you to
>>>>> reproduce with an event's patch, please use the v6 (and if it fails
>>>>> then v7, and if it fails then no patch, really).
>>>>>
>>
>> Stefan Eissing
>>
>> <green/>bytes GmbH
>> Hafenstrasse 16
>> 48155 M�nster
>> www.greenbytes.de
>>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Am 19.01.2017 um 16:34 schrieb Stefan Eissing:
> Yann might already have asked this: any chance to compile with symbols and get a more readable stacktrace?
Yes just tell me how ;-) i'm using dpkg-buildpackage and dh_strip. I've
no idea why not all symbols are available.
Do i need to pass a specific option to configure
Stefan
>> Am 19.01.2017 um 16:30 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>
>> With stock 2.4.25 + patch i'm getting this one again:
>> (gdb) bt
>> #0 0x0000000000521dcd in h2_stream_out_prepare ()
>> #1 0x00007fc1a2feca80 in ?? ()
>> #2 0x00007fc1a2feca8c in ?? ()
>> #3 0x00007fc1a2feca90 in ?? ()
>> #4 0x00007fc1a057c0a0 in ?? ()
>> #5 0x00007fc1a057cdd8 in ?? ()
>> #6 0x00007fc1a2fecac0 in ?? ()
>> #7 0x0000000000000000 in ?? ()
>>
>> Stefan
>>
>> Am 19.01.2017 um 16:28 schrieb Stefan Priebe - Profihost AG:
>>> I'm now testing stock 2.4.25 + patch.
>>>
>>> May this configure option have an influence?
>>> --enable-nonportable-atomics=yes
>>>
>>> Greets,
>>> Stefan
>>>
>>> Am 19.01.2017 um 15:35 schrieb Yann Ylavic:
>>>> Hi,
>>>>
>>>> On Thu, Jan 19, 2017 at 3:00 PM, Stefan Priebe - Profihost AG
>>>> <s....@profihost.ag> wrote:
>>>>>
>>>>> @Yann:
>>>>> should i use V7 or V6?
>>>>
>>>> I'd prefer you'd use none (such that we can verify the patch with
>>>> stock 2.4.25, modulo mod_http2), but if it's easier for you to
>>>> reproduce with an event's patch, please use the v6 (and if it fails
>>>> then v7, and if it fails then no patch, really).
>>>>
>
> Stefan Eissing
>
> <green/>bytes GmbH
> Hafenstrasse 16
> 48155 M�nster
> www.greenbytes.de
>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Eissing <st...@greenbytes.de>.
Yann might already have asked this: any chance to compile with symbols and get a more readable stacktrace?
> Am 19.01.2017 um 16:30 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>
> With stock 2.4.25 + patch i'm getting this one again:
> (gdb) bt
> #0 0x0000000000521dcd in h2_stream_out_prepare ()
> #1 0x00007fc1a2feca80 in ?? ()
> #2 0x00007fc1a2feca8c in ?? ()
> #3 0x00007fc1a2feca90 in ?? ()
> #4 0x00007fc1a057c0a0 in ?? ()
> #5 0x00007fc1a057cdd8 in ?? ()
> #6 0x00007fc1a2fecac0 in ?? ()
> #7 0x0000000000000000 in ?? ()
>
> Stefan
>
> Am 19.01.2017 um 16:28 schrieb Stefan Priebe - Profihost AG:
>> I'm now testing stock 2.4.25 + patch.
>>
>> May this configure option have an influence?
>> --enable-nonportable-atomics=yes
>>
>> Greets,
>> Stefan
>>
>> Am 19.01.2017 um 15:35 schrieb Yann Ylavic:
>>> Hi,
>>>
>>> On Thu, Jan 19, 2017 at 3:00 PM, Stefan Priebe - Profihost AG
>>> <s....@profihost.ag> wrote:
>>>>
>>>> @Yann:
>>>> should i use V7 or V6?
>>>
>>> I'd prefer you'd use none (such that we can verify the patch with
>>> stock 2.4.25, modulo mod_http2), but if it's easier for you to
>>> reproduce with an event's patch, please use the v6 (and if it fails
>>> then v7, and if it fails then no patch, really).
>>>
Stefan Eissing
<green/>bytes GmbH
Hafenstrasse 16
48155 Münster
www.greenbytes.de
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
With stock 2.4.25 + patch i'm getting this one again:
(gdb) bt
#0 0x0000000000521dcd in h2_stream_out_prepare ()
#1 0x00007fc1a2feca80 in ?? ()
#2 0x00007fc1a2feca8c in ?? ()
#3 0x00007fc1a2feca90 in ?? ()
#4 0x00007fc1a057c0a0 in ?? ()
#5 0x00007fc1a057cdd8 in ?? ()
#6 0x00007fc1a2fecac0 in ?? ()
#7 0x0000000000000000 in ?? ()
Stefan
Am 19.01.2017 um 16:28 schrieb Stefan Priebe - Profihost AG:
> I'm now testing stock 2.4.25 + patch.
>
> May this configure option have an influence?
> --enable-nonportable-atomics=yes
>
> Greets,
> Stefan
>
> Am 19.01.2017 um 15:35 schrieb Yann Ylavic:
>> Hi,
>>
>> On Thu, Jan 19, 2017 at 3:00 PM, Stefan Priebe - Profihost AG
>> <s....@profihost.ag> wrote:
>>>
>>> @Yann:
>>> should i use V7 or V6?
>>
>> I'd prefer you'd use none (such that we can verify the patch with
>> stock 2.4.25, modulo mod_http2), but if it's easier for you to
>> reproduce with an event's patch, please use the v6 (and if it fails
>> then v7, and if it fails then no patch, really).
>>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
I'm now testing stock 2.4.25 + patch.
May this configure option have an influence?
--enable-nonportable-atomics=yes
Greets,
Stefan
Am 19.01.2017 um 15:35 schrieb Yann Ylavic:
> Hi,
>
> On Thu, Jan 19, 2017 at 3:00 PM, Stefan Priebe - Profihost AG
> <s....@profihost.ag> wrote:
>>
>> @Yann:
>> should i use V7 or V6?
>
> I'd prefer you'd use none (such that we can verify the patch with
> stock 2.4.25, modulo mod_http2), but if it's easier for you to
> reproduce with an event's patch, please use the v6 (and if it fails
> then v7, and if it fails then no patch, really).
>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
This one is with mod_http2 v1.8.8 + http2 patch - no mpm patch.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f1aa4c45d63 in apr_pool_cleanup_kill () from
/usr/lib/x86_64-linux-gnu/libapr-1.so.0
(gdb) bt
#0 0x00007f1aa4c45d63 in apr_pool_cleanup_kill () from
/usr/lib/x86_64-linux-gnu/libapr-1.so.0
#1 0x00007f1aa4c45e21 in apr_pool_cleanup_run () from
/usr/lib/x86_64-linux-gnu/libapr-1.so.0
#2 0x000000000051f29d in stream_pool_cleanup ()
#3 0x00007f1a85fce8b0 in ?? ()
#4 0x00007f1a8462a0a0 in ?? ()
#5 0x00007f1a85fce8d4 in ?? ()
#6 0x00007f1a84636cc0 in ?? ()
#7 0x00007f1a8462a0a0 in ?? ()
#8 0x000000008462a028 in ?? ()
#9 0x00007f1a8462a028 in ?? ()
#10 0x00007f1aa4c449be in apr_pool_destroy () from
/usr/lib/x86_64-linux-gnu/libapr-1.so.0
#11 0x000000000051fe96 in h2_stream_destroy ()
#12 0x000000078460c118 in ?? ()
#13 0x00007f1a8462a0a0 in ?? ()
#14 0x00007f1a85fce940 in ?? ()
#15 0x0000000000507ab7 in stream_done ()
#16 0x00007f1aa6025060 in ?? () from /lib64/ld-linux-x86-64.so.2
#17 0x0000000000506394 in enter_mutex ()
#18 0x00007f1a8462a0a0 in ?? ()
#19 0x00007f1a8460c118 in ?? ()
#20 0x0000000000000000 in ?? ()
Stefan
Am 19.01.2017 um 15:35 schrieb Yann Ylavic:
> Hi,
>
> On Thu, Jan 19, 2017 at 3:00 PM, Stefan Priebe - Profihost AG
> <s....@profihost.ag> wrote:
>>
>> @Yann:
>> should i use V7 or V6?
>
> I'd prefer you'd use none (such that we can verify the patch with
> stock 2.4.25, modulo mod_http2), but if it's easier for you to
> reproduce with an event's patch, please use the v6 (and if it fails
> then v7, and if it fails then no patch, really).
>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
Hi,
On Thu, Jan 19, 2017 at 3:00 PM, Stefan Priebe - Profihost AG
<s....@profihost.ag> wrote:
>
> @Yann:
> should i use V7 or V6?
I'd prefer you'd use none (such that we can verify the patch with
stock 2.4.25, modulo mod_http2), but if it's easier for you to
reproduce with an event's patch, please use the v6 (and if it fails
then v7, and if it fails then no patch, really).
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Am 19.01.2017 um 15:05 schrieb Stefan Eissing:
> I prefer on top of v1.8.8, but it's your installation. Should also work on older versions.
got this one:
(gdb) bt
#0 0x00007f4c424ec014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f4c4297f036 in ?? () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
#2 0x00007f4c4297f46f in apr_hash_set () from
/usr/lib/x86_64-linux-gnu/libapr-1.so.0
#3 0x000000000052a26d in h2_ihash_remove ()
#4 0x0000005b00000000 in ?? ()
#5 0x00007f4c2217b328 in ?? ()
#6 0x00007f4c22fec300 in ?? ()
#7 0x0000000000506b24 in purge_stream ()
#8 0x00007f4c206f50a0 in ?? ()
#9 0x00007f4c209eb118 in ?? ()
#10 0x00007f4c216f70a0 in ?? ()
#11 0x0000005b00000000 in ?? ()
#12 0x00007f4c206f50a0 in ?? ()
#13 0x00007f4c209eb118 in ?? ()
#14 0x00007f4c22fec340 in ?? ()
#15 0x000000000052a1c4 in ihash_iter ()
#16 0x00007f4c206f50a0 in ?? ()
#17 0x0000000000000004 in ?? ()
#18 0x00007f4c206f50a0 in ?? ()
#19 0x00007f4c22fec3c0 in ?? ()
#20 0x0000000000000000 in ?? ()
i've now removed any mpm_event patch and try to look again at v1.8.8 +
your patch.
Stefan
>
>> Am 19.01.2017 um 15:00 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>
>> Hi,
>>
>> on top of v1.8.8?
>>
>> @Yann:
>> should i use V7 or V6?
>>
>> Stefan
>>
>> Am 19.01.2017 um 14:56 schrieb Stefan Eissing:
>>> Stefan,
>>>
>>> could you check if the attached patch mitigates the problem at least to some extend? Was suggested to me by Yann, all faults are mine. Thanks!
>>>
>>> Cheers, Stefan
>>>
>>>
>>>
>>>
>>>
>>>> Am 19.01.2017 um 12:45 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>>
>>>> Am 19.01.2017 um 11:56 schrieb Stefan Eissing:
>>>>> Stefan,
>>>>>
>>>>> yes, that is a known one that was addressed in v1.8.7. But I think it is related to the others. There is some mistake in my thinking about session pool cleanup that is more often uncovered by Yann's recent patches. I'll need some deep dive into this one.
>>>>>
>>>>> For you, that means v1.8.8 is the best right now. Hopefully we'll find this soon.
>>>>
>>>> But v1.8.8 is def. more often crashing than v1.8.3 which is included in
>>>> apache 2.4.25.
>>>>
>>>> With v.8.8 i see crashes like these which i don't have with 1.8.3:
>>>> (gdb) bt
>>>> #0 0x00007f6b5f9fef23 in apr_brigade_length () from
>>>> /usr/lib/x86_64-linux-gnu/libaprutil-1.so.0
>>>> #1 0x000000000052afb6 in h2_util_bb_avail ()
>>>> #2 0x00000000407e7a10 in ?? ()
>>>> #3 0x00007f6b407e7a8c in ?? ()
>>>> #4 0x00007f6b407e7a90 in ?? ()
>>>> #5 0x00007f6b3e608668 in ?? ()
>>>> #6 0x0000000000000000 in ?? ()
>>>>
>>>> Greets,
>>>> Stefan
>>>>
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Stefan
>>>>>
>>>>>> Am 19.01.2017 um 11:39 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> with V7 and mod_http2 core i'm always seeing exactly THIS trace and
>>>>>> instead of every few minutes just once an hour:
>>>>>>
>>>>>> [Thread debugging using libthread_db enabled]
>>>>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>>>> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
>>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>>> #0 0x0000000000521d98 in h2_stream_out_prepare ()
>>>>>> (gdb) bt
>>>>>> #0 0x0000000000521d98 in h2_stream_out_prepare ()
>>>>>> #1 0x00007fb65bfeea80 in ?? ()
>>>>>> #2 0x00007fb65bfeea8c in ?? ()
>>>>>> #3 0x00007fb65bfeea90 in ?? ()
>>>>>> #4 0x00007fb6599100a0 in ?? ()
>>>>>> #5 0x00007fb659910f70 in ?? ()
>>>>>> #6 0x00007fb65bfeeac0 in ?? ()
>>>>>> #7 0x0000000000000000 in ?? ()
>>>>>>
>>>>>> not all the others like with v1.8.8. So may this be a different one?
>>>>>>
>>>>>> Stefan
>>>>>>
>>>>>> Am 19.01.2017 um 09:21 schrieb Stefan Priebe - Profihost AG:
>>>>>>> And another one:
>>>>>>>
>>>>>>> #0 0x00007f7d81ba1f23 in apr_brigade_length () from
>>>>>>> /usr/lib/x86_64-linux-gnu/libaprutil-1.so.0
>>>>>>> (gdb) bt
>>>>>>> #0 0x00007f7d81ba1f23 in apr_brigade_length () from
>>>>>>> /usr/lib/x86_64-linux-gnu/libaprutil-1.so.0
>>>>>>> #1 0x000000000052a8bc in h2_util_bb_avail ()
>>>>>>> #2 0x000000007112ca10 in ?? ()
>>>>>>> #3 0x00007f7d7112ca8c in ?? ()
>>>>>>> #4 0x00007f7d7112ca90 in ?? ()
>>>>>>> #5 0x00007f7d60a4bad0 in ?? ()
>>>>>>> #6 0x0000000000000000 in ?? ()
>>>>>>> Am 19.01.2017 um 09:20 schrieb Stefan Priebe - Profihost AG:
>>>>>>>> Hi,
>>>>>>>> Am 19.01.2017 um 09:11 schrieb Yann Ylavic:
>>>>>>>>> On Thu, Jan 19, 2017 at 9:05 AM, Stefan Priebe - Profihost AG
>>>>>>>>> <s....@profihost.ag> wrote:
>>>>>>>>>> With a vanilla apache 2.4.25 i got this one:
>>>>>>>>>>
>>>>>>>>>> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
>>>>>>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>>>>>>> #0 0x00007fe2bb7b8add in read () from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>>>>>> (gdb) bt
>>>>>>>>>> #0 0x00007fe2bb7b8add in read () from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>>>>>> #1 0x000000000048043a in ap_mpm_podx_check ()
>>>>>>>>>> #2 <signal handler called>
>>>>>>>>>> #3 0x000000004b351bd0 in ?? ()
>>>>>>>>>> Cannot access memory at address 0x0
>>>>>>>>>
>>>>>>>>> This is probably not the faulting thread, could you find it with
>>>>>>>>> "thread apply all bt" or paste the whole output please?
>>>>>>>>
>>>>>>>> ah i found it via thread apply all bt:
>>>>>>>>
>>>>>>>> (gdb) bt
>>>>>>>> #0 0x0000000000521d98 in h2_stream_out_prepare ()
>>>>>>>> #1 0x00007f5cbdfeaa80 in ?? ()
>>>>>>>> #2 0x00007f5cbdfeaa8c in ?? ()
>>>>>>>> #3 0x00007f5cbdfeaa90 in ?? ()
>>>>>>>> #4 0x00007f5cb97ec0a0 in ?? ()
>>>>>>>> #5 0x00007f5cb97ec988 in ?? ()
>>>>>>>> #6 0x00007f5cbdfeaac0 in ?? ()
>>>>>>>> #7 0x0000000000000000 in ?? ()
>>>>>>>>
>>>>>>>> this is with a vanilla 2.4.25.
>>>>>>>>
>>>>>>>> Greets,
>>>>>>>> Stefan
>>>>>>>>
>>>>>
>>>>> Stefan Eissing
>>>>>
>>>>> <green/>bytes GmbH
>>>>> Hafenstrasse 16
>>>>> 48155 M�nster
>>>>> www.greenbytes.de
>>>>>
>>>
>>> Stefan Eissing
>>>
>>> <green/>bytes GmbH
>>> Hafenstrasse 16
>>> 48155 M�nster
>>> www.greenbytes.de
>>>
>
> Stefan Eissing
>
> <green/>bytes GmbH
> Hafenstrasse 16
> 48155 M�nster
> www.greenbytes.de
>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Eissing <st...@greenbytes.de>.
I prefer on top of v1.8.8, but it's your installation. Should also work on older versions.
> Am 19.01.2017 um 15:00 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>
> Hi,
>
> on top of v1.8.8?
>
> @Yann:
> should i use V7 or V6?
>
> Stefan
>
> Am 19.01.2017 um 14:56 schrieb Stefan Eissing:
>> Stefan,
>>
>> could you check if the attached patch mitigates the problem at least to some extend? Was suggested to me by Yann, all faults are mine. Thanks!
>>
>> Cheers, Stefan
>>
>>
>>
>>
>>
>>> Am 19.01.2017 um 12:45 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>
>>> Am 19.01.2017 um 11:56 schrieb Stefan Eissing:
>>>> Stefan,
>>>>
>>>> yes, that is a known one that was addressed in v1.8.7. But I think it is related to the others. There is some mistake in my thinking about session pool cleanup that is more often uncovered by Yann's recent patches. I'll need some deep dive into this one.
>>>>
>>>> For you, that means v1.8.8 is the best right now. Hopefully we'll find this soon.
>>>
>>> But v1.8.8 is def. more often crashing than v1.8.3 which is included in
>>> apache 2.4.25.
>>>
>>> With v.8.8 i see crashes like these which i don't have with 1.8.3:
>>> (gdb) bt
>>> #0 0x00007f6b5f9fef23 in apr_brigade_length () from
>>> /usr/lib/x86_64-linux-gnu/libaprutil-1.so.0
>>> #1 0x000000000052afb6 in h2_util_bb_avail ()
>>> #2 0x00000000407e7a10 in ?? ()
>>> #3 0x00007f6b407e7a8c in ?? ()
>>> #4 0x00007f6b407e7a90 in ?? ()
>>> #5 0x00007f6b3e608668 in ?? ()
>>> #6 0x0000000000000000 in ?? ()
>>>
>>> Greets,
>>> Stefan
>>>
>>>>
>>>> Cheers,
>>>>
>>>> Stefan
>>>>
>>>>> Am 19.01.2017 um 11:39 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>>>
>>>>> Hi,
>>>>>
>>>>> with V7 and mod_http2 core i'm always seeing exactly THIS trace and
>>>>> instead of every few minutes just once an hour:
>>>>>
>>>>> [Thread debugging using libthread_db enabled]
>>>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>>> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>> #0 0x0000000000521d98 in h2_stream_out_prepare ()
>>>>> (gdb) bt
>>>>> #0 0x0000000000521d98 in h2_stream_out_prepare ()
>>>>> #1 0x00007fb65bfeea80 in ?? ()
>>>>> #2 0x00007fb65bfeea8c in ?? ()
>>>>> #3 0x00007fb65bfeea90 in ?? ()
>>>>> #4 0x00007fb6599100a0 in ?? ()
>>>>> #5 0x00007fb659910f70 in ?? ()
>>>>> #6 0x00007fb65bfeeac0 in ?? ()
>>>>> #7 0x0000000000000000 in ?? ()
>>>>>
>>>>> not all the others like with v1.8.8. So may this be a different one?
>>>>>
>>>>> Stefan
>>>>>
>>>>> Am 19.01.2017 um 09:21 schrieb Stefan Priebe - Profihost AG:
>>>>>> And another one:
>>>>>>
>>>>>> #0 0x00007f7d81ba1f23 in apr_brigade_length () from
>>>>>> /usr/lib/x86_64-linux-gnu/libaprutil-1.so.0
>>>>>> (gdb) bt
>>>>>> #0 0x00007f7d81ba1f23 in apr_brigade_length () from
>>>>>> /usr/lib/x86_64-linux-gnu/libaprutil-1.so.0
>>>>>> #1 0x000000000052a8bc in h2_util_bb_avail ()
>>>>>> #2 0x000000007112ca10 in ?? ()
>>>>>> #3 0x00007f7d7112ca8c in ?? ()
>>>>>> #4 0x00007f7d7112ca90 in ?? ()
>>>>>> #5 0x00007f7d60a4bad0 in ?? ()
>>>>>> #6 0x0000000000000000 in ?? ()
>>>>>> Am 19.01.2017 um 09:20 schrieb Stefan Priebe - Profihost AG:
>>>>>>> Hi,
>>>>>>> Am 19.01.2017 um 09:11 schrieb Yann Ylavic:
>>>>>>>> On Thu, Jan 19, 2017 at 9:05 AM, Stefan Priebe - Profihost AG
>>>>>>>> <s....@profihost.ag> wrote:
>>>>>>>>> With a vanilla apache 2.4.25 i got this one:
>>>>>>>>>
>>>>>>>>> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
>>>>>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>>>>>> #0 0x00007fe2bb7b8add in read () from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>>>>> (gdb) bt
>>>>>>>>> #0 0x00007fe2bb7b8add in read () from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>>>>> #1 0x000000000048043a in ap_mpm_podx_check ()
>>>>>>>>> #2 <signal handler called>
>>>>>>>>> #3 0x000000004b351bd0 in ?? ()
>>>>>>>>> Cannot access memory at address 0x0
>>>>>>>>
>>>>>>>> This is probably not the faulting thread, could you find it with
>>>>>>>> "thread apply all bt" or paste the whole output please?
>>>>>>>
>>>>>>> ah i found it via thread apply all bt:
>>>>>>>
>>>>>>> (gdb) bt
>>>>>>> #0 0x0000000000521d98 in h2_stream_out_prepare ()
>>>>>>> #1 0x00007f5cbdfeaa80 in ?? ()
>>>>>>> #2 0x00007f5cbdfeaa8c in ?? ()
>>>>>>> #3 0x00007f5cbdfeaa90 in ?? ()
>>>>>>> #4 0x00007f5cb97ec0a0 in ?? ()
>>>>>>> #5 0x00007f5cb97ec988 in ?? ()
>>>>>>> #6 0x00007f5cbdfeaac0 in ?? ()
>>>>>>> #7 0x0000000000000000 in ?? ()
>>>>>>>
>>>>>>> this is with a vanilla 2.4.25.
>>>>>>>
>>>>>>> Greets,
>>>>>>> Stefan
>>>>>>>
>>>>
>>>> Stefan Eissing
>>>>
>>>> <green/>bytes GmbH
>>>> Hafenstrasse 16
>>>> 48155 Münster
>>>> www.greenbytes.de
>>>>
>>
>> Stefan Eissing
>>
>> <green/>bytes GmbH
>> Hafenstrasse 16
>> 48155 Münster
>> www.greenbytes.de
>>
Stefan Eissing
<green/>bytes GmbH
Hafenstrasse 16
48155 Münster
www.greenbytes.de
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Hi,
on top of v1.8.8?
@Yann:
should i use V7 or V6?
Stefan
Am 19.01.2017 um 14:56 schrieb Stefan Eissing:
> Stefan,
>
> could you check if the attached patch mitigates the problem at least to some extend? Was suggested to me by Yann, all faults are mine. Thanks!
>
> Cheers, Stefan
>
>
>
>
>
>> Am 19.01.2017 um 12:45 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>
>> Am 19.01.2017 um 11:56 schrieb Stefan Eissing:
>>> Stefan,
>>>
>>> yes, that is a known one that was addressed in v1.8.7. But I think it is related to the others. There is some mistake in my thinking about session pool cleanup that is more often uncovered by Yann's recent patches. I'll need some deep dive into this one.
>>>
>>> For you, that means v1.8.8 is the best right now. Hopefully we'll find this soon.
>>
>> But v1.8.8 is def. more often crashing than v1.8.3 which is included in
>> apache 2.4.25.
>>
>> With v.8.8 i see crashes like these which i don't have with 1.8.3:
>> (gdb) bt
>> #0 0x00007f6b5f9fef23 in apr_brigade_length () from
>> /usr/lib/x86_64-linux-gnu/libaprutil-1.so.0
>> #1 0x000000000052afb6 in h2_util_bb_avail ()
>> #2 0x00000000407e7a10 in ?? ()
>> #3 0x00007f6b407e7a8c in ?? ()
>> #4 0x00007f6b407e7a90 in ?? ()
>> #5 0x00007f6b3e608668 in ?? ()
>> #6 0x0000000000000000 in ?? ()
>>
>> Greets,
>> Stefan
>>
>>>
>>> Cheers,
>>>
>>> Stefan
>>>
>>>> Am 19.01.2017 um 11:39 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>>>
>>>> Hi,
>>>>
>>>> with V7 and mod_http2 core i'm always seeing exactly THIS trace and
>>>> instead of every few minutes just once an hour:
>>>>
>>>> [Thread debugging using libthread_db enabled]
>>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>> #0 0x0000000000521d98 in h2_stream_out_prepare ()
>>>> (gdb) bt
>>>> #0 0x0000000000521d98 in h2_stream_out_prepare ()
>>>> #1 0x00007fb65bfeea80 in ?? ()
>>>> #2 0x00007fb65bfeea8c in ?? ()
>>>> #3 0x00007fb65bfeea90 in ?? ()
>>>> #4 0x00007fb6599100a0 in ?? ()
>>>> #5 0x00007fb659910f70 in ?? ()
>>>> #6 0x00007fb65bfeeac0 in ?? ()
>>>> #7 0x0000000000000000 in ?? ()
>>>>
>>>> not all the others like with v1.8.8. So may this be a different one?
>>>>
>>>> Stefan
>>>>
>>>> Am 19.01.2017 um 09:21 schrieb Stefan Priebe - Profihost AG:
>>>>> And another one:
>>>>>
>>>>> #0 0x00007f7d81ba1f23 in apr_brigade_length () from
>>>>> /usr/lib/x86_64-linux-gnu/libaprutil-1.so.0
>>>>> (gdb) bt
>>>>> #0 0x00007f7d81ba1f23 in apr_brigade_length () from
>>>>> /usr/lib/x86_64-linux-gnu/libaprutil-1.so.0
>>>>> #1 0x000000000052a8bc in h2_util_bb_avail ()
>>>>> #2 0x000000007112ca10 in ?? ()
>>>>> #3 0x00007f7d7112ca8c in ?? ()
>>>>> #4 0x00007f7d7112ca90 in ?? ()
>>>>> #5 0x00007f7d60a4bad0 in ?? ()
>>>>> #6 0x0000000000000000 in ?? ()
>>>>> Am 19.01.2017 um 09:20 schrieb Stefan Priebe - Profihost AG:
>>>>>> Hi,
>>>>>> Am 19.01.2017 um 09:11 schrieb Yann Ylavic:
>>>>>>> On Thu, Jan 19, 2017 at 9:05 AM, Stefan Priebe - Profihost AG
>>>>>>> <s....@profihost.ag> wrote:
>>>>>>>> With a vanilla apache 2.4.25 i got this one:
>>>>>>>>
>>>>>>>> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
>>>>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>>>>> #0 0x00007fe2bb7b8add in read () from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>>>> (gdb) bt
>>>>>>>> #0 0x00007fe2bb7b8add in read () from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>>>> #1 0x000000000048043a in ap_mpm_podx_check ()
>>>>>>>> #2 <signal handler called>
>>>>>>>> #3 0x000000004b351bd0 in ?? ()
>>>>>>>> Cannot access memory at address 0x0
>>>>>>>
>>>>>>> This is probably not the faulting thread, could you find it with
>>>>>>> "thread apply all bt" or paste the whole output please?
>>>>>>
>>>>>> ah i found it via thread apply all bt:
>>>>>>
>>>>>> (gdb) bt
>>>>>> #0 0x0000000000521d98 in h2_stream_out_prepare ()
>>>>>> #1 0x00007f5cbdfeaa80 in ?? ()
>>>>>> #2 0x00007f5cbdfeaa8c in ?? ()
>>>>>> #3 0x00007f5cbdfeaa90 in ?? ()
>>>>>> #4 0x00007f5cb97ec0a0 in ?? ()
>>>>>> #5 0x00007f5cb97ec988 in ?? ()
>>>>>> #6 0x00007f5cbdfeaac0 in ?? ()
>>>>>> #7 0x0000000000000000 in ?? ()
>>>>>>
>>>>>> this is with a vanilla 2.4.25.
>>>>>>
>>>>>> Greets,
>>>>>> Stefan
>>>>>>
>>>
>>> Stefan Eissing
>>>
>>> <green/>bytes GmbH
>>> Hafenstrasse 16
>>> 48155 M�nster
>>> www.greenbytes.de
>>>
>
> Stefan Eissing
>
> <green/>bytes GmbH
> Hafenstrasse 16
> 48155 M�nster
> www.greenbytes.de
>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Eissing <st...@greenbytes.de>.
Stefan,
could you check if the attached patch mitigates the problem at least to some extend? Was suggested to me by Yann, all faults are mine. Thanks!
Cheers, Stefan
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Am 19.01.2017 um 11:56 schrieb Stefan Eissing:
> Stefan,
>
> yes, that is a known one that was addressed in v1.8.7. But I think it is related to the others. There is some mistake in my thinking about session pool cleanup that is more often uncovered by Yann's recent patches. I'll need some deep dive into this one.
>
> For you, that means v1.8.8 is the best right now. Hopefully we'll find this soon.
But v1.8.8 is def. more often crashing than v1.8.3 which is included in
apache 2.4.25.
With v.8.8 i see crashes like these which i don't have with 1.8.3:
(gdb) bt
#0 0x00007f6b5f9fef23 in apr_brigade_length () from
/usr/lib/x86_64-linux-gnu/libaprutil-1.so.0
#1 0x000000000052afb6 in h2_util_bb_avail ()
#2 0x00000000407e7a10 in ?? ()
#3 0x00007f6b407e7a8c in ?? ()
#4 0x00007f6b407e7a90 in ?? ()
#5 0x00007f6b3e608668 in ?? ()
#6 0x0000000000000000 in ?? ()
Greets,
Stefan
>
> Cheers,
>
> Stefan
>
>> Am 19.01.2017 um 11:39 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>
>> Hi,
>>
>> with V7 and mod_http2 core i'm always seeing exactly THIS trace and
>> instead of every few minutes just once an hour:
>>
>> [Thread debugging using libthread_db enabled]
>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
>> Program terminated with signal SIGSEGV, Segmentation fault.
>> #0 0x0000000000521d98 in h2_stream_out_prepare ()
>> (gdb) bt
>> #0 0x0000000000521d98 in h2_stream_out_prepare ()
>> #1 0x00007fb65bfeea80 in ?? ()
>> #2 0x00007fb65bfeea8c in ?? ()
>> #3 0x00007fb65bfeea90 in ?? ()
>> #4 0x00007fb6599100a0 in ?? ()
>> #5 0x00007fb659910f70 in ?? ()
>> #6 0x00007fb65bfeeac0 in ?? ()
>> #7 0x0000000000000000 in ?? ()
>>
>> not all the others like with v1.8.8. So may this be a different one?
>>
>> Stefan
>>
>> Am 19.01.2017 um 09:21 schrieb Stefan Priebe - Profihost AG:
>>> And another one:
>>>
>>> #0 0x00007f7d81ba1f23 in apr_brigade_length () from
>>> /usr/lib/x86_64-linux-gnu/libaprutil-1.so.0
>>> (gdb) bt
>>> #0 0x00007f7d81ba1f23 in apr_brigade_length () from
>>> /usr/lib/x86_64-linux-gnu/libaprutil-1.so.0
>>> #1 0x000000000052a8bc in h2_util_bb_avail ()
>>> #2 0x000000007112ca10 in ?? ()
>>> #3 0x00007f7d7112ca8c in ?? ()
>>> #4 0x00007f7d7112ca90 in ?? ()
>>> #5 0x00007f7d60a4bad0 in ?? ()
>>> #6 0x0000000000000000 in ?? ()
>>> Am 19.01.2017 um 09:20 schrieb Stefan Priebe - Profihost AG:
>>>> Hi,
>>>> Am 19.01.2017 um 09:11 schrieb Yann Ylavic:
>>>>> On Thu, Jan 19, 2017 at 9:05 AM, Stefan Priebe - Profihost AG
>>>>> <s....@profihost.ag> wrote:
>>>>>> With a vanilla apache 2.4.25 i got this one:
>>>>>>
>>>>>> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
>>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>>> #0 0x00007fe2bb7b8add in read () from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>> (gdb) bt
>>>>>> #0 0x00007fe2bb7b8add in read () from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>>> #1 0x000000000048043a in ap_mpm_podx_check ()
>>>>>> #2 <signal handler called>
>>>>>> #3 0x000000004b351bd0 in ?? ()
>>>>>> Cannot access memory at address 0x0
>>>>>
>>>>> This is probably not the faulting thread, could you find it with
>>>>> "thread apply all bt" or paste the whole output please?
>>>>
>>>> ah i found it via thread apply all bt:
>>>>
>>>> (gdb) bt
>>>> #0 0x0000000000521d98 in h2_stream_out_prepare ()
>>>> #1 0x00007f5cbdfeaa80 in ?? ()
>>>> #2 0x00007f5cbdfeaa8c in ?? ()
>>>> #3 0x00007f5cbdfeaa90 in ?? ()
>>>> #4 0x00007f5cb97ec0a0 in ?? ()
>>>> #5 0x00007f5cb97ec988 in ?? ()
>>>> #6 0x00007f5cbdfeaac0 in ?? ()
>>>> #7 0x0000000000000000 in ?? ()
>>>>
>>>> this is with a vanilla 2.4.25.
>>>>
>>>> Greets,
>>>> Stefan
>>>>
>
> Stefan Eissing
>
> <green/>bytes GmbH
> Hafenstrasse 16
> 48155 M�nster
> www.greenbytes.de
>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Eissing <st...@greenbytes.de>.
Stefan,
yes, that is a known one that was addressed in v1.8.7. But I think it is related to the others. There is some mistake in my thinking about session pool cleanup that is more often uncovered by Yann's recent patches. I'll need some deep dive into this one.
For you, that means v1.8.8 is the best right now. Hopefully we'll find this soon.
Cheers,
Stefan
> Am 19.01.2017 um 11:39 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>
> Hi,
>
> with V7 and mod_http2 core i'm always seeing exactly THIS trace and
> instead of every few minutes just once an hour:
>
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0 0x0000000000521d98 in h2_stream_out_prepare ()
> (gdb) bt
> #0 0x0000000000521d98 in h2_stream_out_prepare ()
> #1 0x00007fb65bfeea80 in ?? ()
> #2 0x00007fb65bfeea8c in ?? ()
> #3 0x00007fb65bfeea90 in ?? ()
> #4 0x00007fb6599100a0 in ?? ()
> #5 0x00007fb659910f70 in ?? ()
> #6 0x00007fb65bfeeac0 in ?? ()
> #7 0x0000000000000000 in ?? ()
>
> not all the others like with v1.8.8. So may this be a different one?
>
> Stefan
>
> Am 19.01.2017 um 09:21 schrieb Stefan Priebe - Profihost AG:
>> And another one:
>>
>> #0 0x00007f7d81ba1f23 in apr_brigade_length () from
>> /usr/lib/x86_64-linux-gnu/libaprutil-1.so.0
>> (gdb) bt
>> #0 0x00007f7d81ba1f23 in apr_brigade_length () from
>> /usr/lib/x86_64-linux-gnu/libaprutil-1.so.0
>> #1 0x000000000052a8bc in h2_util_bb_avail ()
>> #2 0x000000007112ca10 in ?? ()
>> #3 0x00007f7d7112ca8c in ?? ()
>> #4 0x00007f7d7112ca90 in ?? ()
>> #5 0x00007f7d60a4bad0 in ?? ()
>> #6 0x0000000000000000 in ?? ()
>> Am 19.01.2017 um 09:20 schrieb Stefan Priebe - Profihost AG:
>>> Hi,
>>> Am 19.01.2017 um 09:11 schrieb Yann Ylavic:
>>>> On Thu, Jan 19, 2017 at 9:05 AM, Stefan Priebe - Profihost AG
>>>> <s....@profihost.ag> wrote:
>>>>> With a vanilla apache 2.4.25 i got this one:
>>>>>
>>>>> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>> #0 0x00007fe2bb7b8add in read () from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>> (gdb) bt
>>>>> #0 0x00007fe2bb7b8add in read () from /lib/x86_64-linux-gnu/libpthread.so.0
>>>>> #1 0x000000000048043a in ap_mpm_podx_check ()
>>>>> #2 <signal handler called>
>>>>> #3 0x000000004b351bd0 in ?? ()
>>>>> Cannot access memory at address 0x0
>>>>
>>>> This is probably not the faulting thread, could you find it with
>>>> "thread apply all bt" or paste the whole output please?
>>>
>>> ah i found it via thread apply all bt:
>>>
>>> (gdb) bt
>>> #0 0x0000000000521d98 in h2_stream_out_prepare ()
>>> #1 0x00007f5cbdfeaa80 in ?? ()
>>> #2 0x00007f5cbdfeaa8c in ?? ()
>>> #3 0x00007f5cbdfeaa90 in ?? ()
>>> #4 0x00007f5cb97ec0a0 in ?? ()
>>> #5 0x00007f5cb97ec988 in ?? ()
>>> #6 0x00007f5cbdfeaac0 in ?? ()
>>> #7 0x0000000000000000 in ?? ()
>>>
>>> this is with a vanilla 2.4.25.
>>>
>>> Greets,
>>> Stefan
>>>
Stefan Eissing
<green/>bytes GmbH
Hafenstrasse 16
48155 Münster
www.greenbytes.de
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Hi,
with V7 and mod_http2 core i'm always seeing exactly THIS trace and
instead of every few minutes just once an hour:
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/local/apache2/bin/httpd -k start'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000000000521d98 in h2_stream_out_prepare ()
(gdb) bt
#0 0x0000000000521d98 in h2_stream_out_prepare ()
#1 0x00007fb65bfeea80 in ?? ()
#2 0x00007fb65bfeea8c in ?? ()
#3 0x00007fb65bfeea90 in ?? ()
#4 0x00007fb6599100a0 in ?? ()
#5 0x00007fb659910f70 in ?? ()
#6 0x00007fb65bfeeac0 in ?? ()
#7 0x0000000000000000 in ?? ()
not all the others like with v1.8.8. So may this be a different one?
Stefan
Am 19.01.2017 um 09:21 schrieb Stefan Priebe - Profihost AG:
> And another one:
>
> #0 0x00007f7d81ba1f23 in apr_brigade_length () from
> /usr/lib/x86_64-linux-gnu/libaprutil-1.so.0
> (gdb) bt
> #0 0x00007f7d81ba1f23 in apr_brigade_length () from
> /usr/lib/x86_64-linux-gnu/libaprutil-1.so.0
> #1 0x000000000052a8bc in h2_util_bb_avail ()
> #2 0x000000007112ca10 in ?? ()
> #3 0x00007f7d7112ca8c in ?? ()
> #4 0x00007f7d7112ca90 in ?? ()
> #5 0x00007f7d60a4bad0 in ?? ()
> #6 0x0000000000000000 in ?? ()
> Am 19.01.2017 um 09:20 schrieb Stefan Priebe - Profihost AG:
>> Hi,
>> Am 19.01.2017 um 09:11 schrieb Yann Ylavic:
>>> On Thu, Jan 19, 2017 at 9:05 AM, Stefan Priebe - Profihost AG
>>> <s....@profihost.ag> wrote:
>>>> With a vanilla apache 2.4.25 i got this one:
>>>>
>>>> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>> #0 0x00007fe2bb7b8add in read () from /lib/x86_64-linux-gnu/libpthread.so.0
>>>> (gdb) bt
>>>> #0 0x00007fe2bb7b8add in read () from /lib/x86_64-linux-gnu/libpthread.so.0
>>>> #1 0x000000000048043a in ap_mpm_podx_check ()
>>>> #2 <signal handler called>
>>>> #3 0x000000004b351bd0 in ?? ()
>>>> Cannot access memory at address 0x0
>>>
>>> This is probably not the faulting thread, could you find it with
>>> "thread apply all bt" or paste the whole output please?
>>
>> ah i found it via thread apply all bt:
>>
>> (gdb) bt
>> #0 0x0000000000521d98 in h2_stream_out_prepare ()
>> #1 0x00007f5cbdfeaa80 in ?? ()
>> #2 0x00007f5cbdfeaa8c in ?? ()
>> #3 0x00007f5cbdfeaa90 in ?? ()
>> #4 0x00007f5cb97ec0a0 in ?? ()
>> #5 0x00007f5cb97ec988 in ?? ()
>> #6 0x00007f5cbdfeaac0 in ?? ()
>> #7 0x0000000000000000 in ?? ()
>>
>> this is with a vanilla 2.4.25.
>>
>> Greets,
>> Stefan
>>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
And another one:
#0 0x00007f7d81ba1f23 in apr_brigade_length () from
/usr/lib/x86_64-linux-gnu/libaprutil-1.so.0
(gdb) bt
#0 0x00007f7d81ba1f23 in apr_brigade_length () from
/usr/lib/x86_64-linux-gnu/libaprutil-1.so.0
#1 0x000000000052a8bc in h2_util_bb_avail ()
#2 0x000000007112ca10 in ?? ()
#3 0x00007f7d7112ca8c in ?? ()
#4 0x00007f7d7112ca90 in ?? ()
#5 0x00007f7d60a4bad0 in ?? ()
#6 0x0000000000000000 in ?? ()
Am 19.01.2017 um 09:20 schrieb Stefan Priebe - Profihost AG:
> Hi,
> Am 19.01.2017 um 09:11 schrieb Yann Ylavic:
>> On Thu, Jan 19, 2017 at 9:05 AM, Stefan Priebe - Profihost AG
>> <s....@profihost.ag> wrote:
>>> With a vanilla apache 2.4.25 i got this one:
>>>
>>> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>> #0 0x00007fe2bb7b8add in read () from /lib/x86_64-linux-gnu/libpthread.so.0
>>> (gdb) bt
>>> #0 0x00007fe2bb7b8add in read () from /lib/x86_64-linux-gnu/libpthread.so.0
>>> #1 0x000000000048043a in ap_mpm_podx_check ()
>>> #2 <signal handler called>
>>> #3 0x000000004b351bd0 in ?? ()
>>> Cannot access memory at address 0x0
>>
>> This is probably not the faulting thread, could you find it with
>> "thread apply all bt" or paste the whole output please?
>
> ah i found it via thread apply all bt:
>
> (gdb) bt
> #0 0x0000000000521d98 in h2_stream_out_prepare ()
> #1 0x00007f5cbdfeaa80 in ?? ()
> #2 0x00007f5cbdfeaa8c in ?? ()
> #3 0x00007f5cbdfeaa90 in ?? ()
> #4 0x00007f5cb97ec0a0 in ?? ()
> #5 0x00007f5cb97ec988 in ?? ()
> #6 0x00007f5cbdfeaac0 in ?? ()
> #7 0x0000000000000000 in ?? ()
>
> this is with a vanilla 2.4.25.
>
> Greets,
> Stefan
>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Hi,
Am 19.01.2017 um 09:11 schrieb Yann Ylavic:
> On Thu, Jan 19, 2017 at 9:05 AM, Stefan Priebe - Profihost AG
> <s....@profihost.ag> wrote:
>> With a vanilla apache 2.4.25 i got this one:
>>
>> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
>> Program terminated with signal SIGSEGV, Segmentation fault.
>> #0 0x00007fe2bb7b8add in read () from /lib/x86_64-linux-gnu/libpthread.so.0
>> (gdb) bt
>> #0 0x00007fe2bb7b8add in read () from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1 0x000000000048043a in ap_mpm_podx_check ()
>> #2 <signal handler called>
>> #3 0x000000004b351bd0 in ?? ()
>> Cannot access memory at address 0x0
>
> This is probably not the faulting thread, could you find it with
> "thread apply all bt" or paste the whole output please?
ah i found it via thread apply all bt:
(gdb) bt
#0 0x0000000000521d98 in h2_stream_out_prepare ()
#1 0x00007f5cbdfeaa80 in ?? ()
#2 0x00007f5cbdfeaa8c in ?? ()
#3 0x00007f5cbdfeaa90 in ?? ()
#4 0x00007f5cb97ec0a0 in ?? ()
#5 0x00007f5cb97ec988 in ?? ()
#6 0x00007f5cbdfeaac0 in ?? ()
#7 0x0000000000000000 in ?? ()
this is with a vanilla 2.4.25.
Greets,
Stefan
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
On Thu, Jan 19, 2017 at 9:05 AM, Stefan Priebe - Profihost AG
<s....@profihost.ag> wrote:
> With a vanilla apache 2.4.25 i got this one:
>
> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0 0x00007fe2bb7b8add in read () from /lib/x86_64-linux-gnu/libpthread.so.0
> (gdb) bt
> #0 0x00007fe2bb7b8add in read () from /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x000000000048043a in ap_mpm_podx_check ()
> #2 <signal handler called>
> #3 0x000000004b351bd0 in ?? ()
> Cannot access memory at address 0x0
This is probably not the faulting thread, could you find it with
"thread apply all bt" or paste the whole output please?
Thanks,
Yann.
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
With a vanilla apache 2.4.25 i got this one:
Core was generated by `/usr/local/apache2/bin/httpd -k start'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007fe2bb7b8add in read () from /lib/x86_64-linux-gnu/libpthread.so.0
(gdb) bt
#0 0x00007fe2bb7b8add in read () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000048043a in ap_mpm_podx_check ()
#2 <signal handler called>
#3 0x000000004b351bd0 in ?? ()
Cannot access memory at address 0x0
I'm compiling apache 2.4.25 against:
libapr1: 1.5.1-3
libaprutil1: 1.5.4-1
from debian jessie and
libnghttp2-14: 1.13.0-1
build from upstream sources. May this be relevant?
Still not oberving any crashes running apache 2.4.23.
Greets,
Stefan
Am 19.01.2017 um 08:22 schrieb Stefan Priebe - Profihost AG:
> Hi Stefan,
>
> Apache 2.4.25 + mpm_event V7 and core mod_http2:
> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0 0x0000000000521d98 in h2_stream_out_prepare ()
> (gdb) bt
> #0 0x0000000000521d98 in h2_stream_out_prepare ()
> #1 0x00007f54f27eba80 in ?? ()
> #2 0x00007f54f27eba8c in ?? ()
> #3 0x00007f54f27eba90 in ?? ()
> #4 0x00007f54ef00e0a0 in ?? ()
> #5 0x00007f54ef00e9a8 in ?? ()
> #6 0x00007f54f27ebac0 in ?? ()
> #7 0x0000000000000000 in ?? ()
>
> so it's crashing with mpm_event and without mpm_event. And it does with
> core mod_http2 and mod_http2 v1.8.8.
>
> Is a regression in mod_http2 possible?
>
> Greets,
> Stefan
> Am 19.01.2017 um 07:55 schrieb Stefan Priebe - Profihost AG:
>> Dear Stefan,
>> dear Yann,
>>
>> a longer test run shows it's also crashing without any mpm_event patch
>> at all. So I'm really sorry. It just starts crashing faster with the
>> mpm_event patches.
>>
>> I'll now retest regarding mod_http2 core, plain apache 2.4.25 and
>> mod_http v1.8.8
>>
>> may be it's a regression in 2.4.25 itself or in mod_http2.
>>
>> Stefan
>>
>> Am 19.01.2017 um 00:52 schrieb Yann Ylavic:
>>> On Wed, Jan 18, 2017 at 10:49 PM, Stefan Priebe - Profihost AG
>>> <s....@profihost.ag> wrote:
>>>>
>>>> v5 does not apply to 2.4.25. If you can send me a v5 version that
>>>> applies to 2.4.25 i'll test.
>>>
>>> Here it is (slightly modified to fix possible deadlocks, per r1762742
>>> and r1774538).
>>>
>>> Thanks!
>>>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
On Wed, Jan 18, 2017 at 10:42 PM, Stefan Priebe - Profihost AG
<s....@profihost.ag> wrote:
> I also tried to remove rev 1762701 from the v6 patchset. But it does not
> match.
That's more or less what v7 does..
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
I also tried to remove rev 1762701 from the v6 patchset. But it does not
match.
Stefan
Am 18.01.2017 um 22:23 schrieb Stefan Priebe - Profihost AG:
> Hi Yann,
>
> sadly it does not solve the issue.
>
> Trace looks still the same:
> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0 0x00007fe0228a9014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> (gdb) bt
> #0 0x00007fe0228a9014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> #1 0x00007fe022d3c036 in ?? () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> #2 0x00007fe022d3c46f in apr_hash_set () from
> /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> #3 0x000000000052a238 in h2_ihash_remove ()
> #4 0x0000004500000000 in ?? ()
> #5 0x00007fe0022c5328 in ?? ()
> #6 0x00007fe002fec300 in ?? ()
> #7 0x0000000000506b24 in purge_stream ()
> #8 0x00007fe0014690a0 in ?? ()
> #9 0x00007fe000fad2d8 in ?? ()
> #10 0x00007fe00194e0a0 in ?? ()
> #11 0x0000004500000000 in ?? ()
> #12 0x00007fe0014690a0 in ?? ()
> #13 0x00007fe000fad2d8 in ?? ()
> #14 0x00007fe002fec340 in ?? ()
> #15 0x000000000052a18f in ihash_iter ()
> #16 0x00007fe0014690a0 in ?? ()
> #17 0x0000000000000004 in ?? ()
> #18 0x00007fe0014690a0 in ?? ()
> #19 0x00007fe002fec3c0 in ?? ()
> #20 0x0000000000000000 in ?? ()
>
> Stefan
>
> Am 18.01.2017 um 17:49 schrieb Yann Ylavic:
>> On Wed, Jan 18, 2017 at 4:38 PM, Stefan Priebe - Profihost AG
>> <s....@profihost.ag> wrote:
>>> Ok will wait whether you provide a fix or I should revert that part.
>>
>> Patch updated in PR 57399.
>>
>> Thanks,
>> Yann.
>>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Am 19.01.2017 um 08:26 schrieb Stefan Eissing:
> Will look into this. With which patch do you have it most frequently?
It happens pretty fast while using:
Apache 2.4.25 + mpm_event V6 + mod_http2 V1.8.8
Greets,
Stefan
> Cheers, Stefan
>
>> Am 19.01.2017 um 08:22 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>>
>> Hi Stefan,
>>
>> Apache 2.4.25 + mpm_event V7 and core mod_http2:
>> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
>> Program terminated with signal SIGSEGV, Segmentation fault.
>> #0 0x0000000000521d98 in h2_stream_out_prepare ()
>> (gdb) bt
>> #0 0x0000000000521d98 in h2_stream_out_prepare ()
>> #1 0x00007f54f27eba80 in ?? ()
>> #2 0x00007f54f27eba8c in ?? ()
>> #3 0x00007f54f27eba90 in ?? ()
>> #4 0x00007f54ef00e0a0 in ?? ()
>> #5 0x00007f54ef00e9a8 in ?? ()
>> #6 0x00007f54f27ebac0 in ?? ()
>> #7 0x0000000000000000 in ?? ()
>>
>> so it's crashing with mpm_event and without mpm_event. And it does with
>> core mod_http2 and mod_http2 v1.8.8.
>>
>> Is a regression in mod_http2 possible?
>>
>> Greets,
>> Stefan
>>> Am 19.01.2017 um 07:55 schrieb Stefan Priebe - Profihost AG:
>>> Dear Stefan,
>>> dear Yann,
>>>
>>> a longer test run shows it's also crashing without any mpm_event patch
>>> at all. So I'm really sorry. It just starts crashing faster with the
>>> mpm_event patches.
>>>
>>> I'll now retest regarding mod_http2 core, plain apache 2.4.25 and
>>> mod_http v1.8.8
>>>
>>> may be it's a regression in 2.4.25 itself or in mod_http2.
>>>
>>> Stefan
>>>
>>>> Am 19.01.2017 um 00:52 schrieb Yann Ylavic:
>>>> On Wed, Jan 18, 2017 at 10:49 PM, Stefan Priebe - Profihost AG
>>>> <s....@profihost.ag> wrote:
>>>>>
>>>>> v5 does not apply to 2.4.25. If you can send me a v5 version that
>>>>> applies to 2.4.25 i'll test.
>>>>
>>>> Here it is (slightly modified to fix possible deadlocks, per r1762742
>>>> and r1774538).
>>>>
>>>> Thanks!
>>>>
>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Eissing <st...@greenbytes.de>.
Will look into this. With which patch do you have it most frequently?
Cheers, Stefan
> Am 19.01.2017 um 08:22 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>
> Hi Stefan,
>
> Apache 2.4.25 + mpm_event V7 and core mod_http2:
> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0 0x0000000000521d98 in h2_stream_out_prepare ()
> (gdb) bt
> #0 0x0000000000521d98 in h2_stream_out_prepare ()
> #1 0x00007f54f27eba80 in ?? ()
> #2 0x00007f54f27eba8c in ?? ()
> #3 0x00007f54f27eba90 in ?? ()
> #4 0x00007f54ef00e0a0 in ?? ()
> #5 0x00007f54ef00e9a8 in ?? ()
> #6 0x00007f54f27ebac0 in ?? ()
> #7 0x0000000000000000 in ?? ()
>
> so it's crashing with mpm_event and without mpm_event. And it does with
> core mod_http2 and mod_http2 v1.8.8.
>
> Is a regression in mod_http2 possible?
>
> Greets,
> Stefan
>> Am 19.01.2017 um 07:55 schrieb Stefan Priebe - Profihost AG:
>> Dear Stefan,
>> dear Yann,
>>
>> a longer test run shows it's also crashing without any mpm_event patch
>> at all. So I'm really sorry. It just starts crashing faster with the
>> mpm_event patches.
>>
>> I'll now retest regarding mod_http2 core, plain apache 2.4.25 and
>> mod_http v1.8.8
>>
>> may be it's a regression in 2.4.25 itself or in mod_http2.
>>
>> Stefan
>>
>>> Am 19.01.2017 um 00:52 schrieb Yann Ylavic:
>>> On Wed, Jan 18, 2017 at 10:49 PM, Stefan Priebe - Profihost AG
>>> <s....@profihost.ag> wrote:
>>>>
>>>> v5 does not apply to 2.4.25. If you can send me a v5 version that
>>>> applies to 2.4.25 i'll test.
>>>
>>> Here it is (slightly modified to fix possible deadlocks, per r1762742
>>> and r1774538).
>>>
>>> Thanks!
>>>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Hi Stefan,
Apache 2.4.25 + mpm_event V7 and core mod_http2:
Core was generated by `/usr/local/apache2/bin/httpd -k start'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000000000521d98 in h2_stream_out_prepare ()
(gdb) bt
#0 0x0000000000521d98 in h2_stream_out_prepare ()
#1 0x00007f54f27eba80 in ?? ()
#2 0x00007f54f27eba8c in ?? ()
#3 0x00007f54f27eba90 in ?? ()
#4 0x00007f54ef00e0a0 in ?? ()
#5 0x00007f54ef00e9a8 in ?? ()
#6 0x00007f54f27ebac0 in ?? ()
#7 0x0000000000000000 in ?? ()
so it's crashing with mpm_event and without mpm_event. And it does with
core mod_http2 and mod_http2 v1.8.8.
Is a regression in mod_http2 possible?
Greets,
Stefan
Am 19.01.2017 um 07:55 schrieb Stefan Priebe - Profihost AG:
> Dear Stefan,
> dear Yann,
>
> a longer test run shows it's also crashing without any mpm_event patch
> at all. So I'm really sorry. It just starts crashing faster with the
> mpm_event patches.
>
> I'll now retest regarding mod_http2 core, plain apache 2.4.25 and
> mod_http v1.8.8
>
> may be it's a regression in 2.4.25 itself or in mod_http2.
>
> Stefan
>
> Am 19.01.2017 um 00:52 schrieb Yann Ylavic:
>> On Wed, Jan 18, 2017 at 10:49 PM, Stefan Priebe - Profihost AG
>> <s....@profihost.ag> wrote:
>>>
>>> v5 does not apply to 2.4.25. If you can send me a v5 version that
>>> applies to 2.4.25 i'll test.
>>
>> Here it is (slightly modified to fix possible deadlocks, per r1762742
>> and r1774538).
>>
>> Thanks!
>>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Dear Stefan,
dear Yann,
a longer test run shows it's also crashing without any mpm_event patch
at all. So I'm really sorry. It just starts crashing faster with the
mpm_event patches.
I'll now retest regarding mod_http2 core, plain apache 2.4.25 and
mod_http v1.8.8
may be it's a regression in 2.4.25 itself or in mod_http2.
Stefan
Am 19.01.2017 um 00:52 schrieb Yann Ylavic:
> On Wed, Jan 18, 2017 at 10:49 PM, Stefan Priebe - Profihost AG
> <s....@profihost.ag> wrote:
>>
>> v5 does not apply to 2.4.25. If you can send me a v5 version that
>> applies to 2.4.25 i'll test.
>
> Here it is (slightly modified to fix possible deadlocks, per r1762742
> and r1774538).
>
> Thanks!
>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
On Wed, Jan 18, 2017 at 10:49 PM, Stefan Priebe - Profihost AG
<s....@profihost.ag> wrote:
>
> v5 does not apply to 2.4.25. If you can send me a v5 version that
> applies to 2.4.25 i'll test.
Here it is (slightly modified to fix possible deadlocks, per r1762742
and r1774538).
Thanks!
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Am 18.01.2017 um 22:44 schrieb Yann Ylavic:
> On Wed, Jan 18, 2017 at 10:23 PM, Stefan Priebe - Profihost AG
> <s....@profihost.ag> wrote:
>>
>> sadly it does not solve the issue.
>
> OK, thanks, back to the paper.
>
> Any [error] log maybe?
What kind of? server-error.log just says
[Wed Jan 18 22:45:34.050450 2017] [core:notice] [pid 16119:tid
140600654485376] AH00051: child pid 16372 exit signal Segmentation fault
(11), possible coredump in /tmp/
> Also, do you confirm that:
> 1. it didn't crash with v5 + original (2.4.25's) mod_http2?
v5 does not apply to 2.4.25. If you can send me a v5 version that
applies to 2.4.25 i'll test.
> 2. it started crashing with v6 + original mod_http2 (before update to v1.8.8)?
Yes.
Stefan
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Am 18.01.2017 um 22:50 schrieb Yann Ylavic:
> On Wed, Jan 18, 2017 at 10:44 PM, Yann Ylavic <yl...@gmail.com> wrote:
>> On Wed, Jan 18, 2017 at 10:23 PM, Stefan Priebe - Profihost AG
>> <s....@profihost.ag> wrote:
>>>
>>
>> Also, do you confirm that:
>> 1. it didn't crash with v5 + original (2.4.25's) mod_http2?
>
> Hmm, v5 was against 2.4.23, but still no crash happened with it right?
yes
>
>> 2. it started crashing with v6 + original mod_http2 (before update to v1.8.8)?
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
On Wed, Jan 18, 2017 at 10:50 PM, Yann Ylavic <yl...@gmail.com> wrote:
> On Wed, Jan 18, 2017 at 10:44 PM, Yann Ylavic <yl...@gmail.com> wrote:
>> On Wed, Jan 18, 2017 at 10:23 PM, Stefan Priebe - Profihost AG
>> <s....@profihost.ag> wrote:
>>>
>>
>> Also, do you confirm that:
>> 1. it didn't crash with v5 + original (2.4.25's) mod_http2?
>
> Hmm, v5 was against 2.4.23, but still no crash happened with it right?
>
>> 2. it started crashing with v6 + original mod_http2 (before update to v1.8.8)?
And of course, no crash in 2.4.25 with no patch at all?
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
On Wed, Jan 18, 2017 at 10:44 PM, Yann Ylavic <yl...@gmail.com> wrote:
> On Wed, Jan 18, 2017 at 10:23 PM, Stefan Priebe - Profihost AG
> <s....@profihost.ag> wrote:
>>
>
> Also, do you confirm that:
> 1. it didn't crash with v5 + original (2.4.25's) mod_http2?
Hmm, v5 was against 2.4.23, but still no crash happened with it right?
> 2. it started crashing with v6 + original mod_http2 (before update to v1.8.8)?
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
On Wed, Jan 18, 2017 at 10:23 PM, Stefan Priebe - Profihost AG
<s....@profihost.ag> wrote:
>
> sadly it does not solve the issue.
OK, thanks, back to the paper.
Any [error] log maybe?
Also, do you confirm that:
1. it didn't crash with v5 + original (2.4.25's) mod_http2?
2. it started crashing with v6 + original mod_http2 (before update to v1.8.8)?
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Hi Yann,
sadly it does not solve the issue.
Trace looks still the same:
Core was generated by `/usr/local/apache2/bin/httpd -k start'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007fe0228a9014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0 0x00007fe0228a9014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fe022d3c036 in ?? () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
#2 0x00007fe022d3c46f in apr_hash_set () from
/usr/lib/x86_64-linux-gnu/libapr-1.so.0
#3 0x000000000052a238 in h2_ihash_remove ()
#4 0x0000004500000000 in ?? ()
#5 0x00007fe0022c5328 in ?? ()
#6 0x00007fe002fec300 in ?? ()
#7 0x0000000000506b24 in purge_stream ()
#8 0x00007fe0014690a0 in ?? ()
#9 0x00007fe000fad2d8 in ?? ()
#10 0x00007fe00194e0a0 in ?? ()
#11 0x0000004500000000 in ?? ()
#12 0x00007fe0014690a0 in ?? ()
#13 0x00007fe000fad2d8 in ?? ()
#14 0x00007fe002fec340 in ?? ()
#15 0x000000000052a18f in ihash_iter ()
#16 0x00007fe0014690a0 in ?? ()
#17 0x0000000000000004 in ?? ()
#18 0x00007fe0014690a0 in ?? ()
#19 0x00007fe002fec3c0 in ?? ()
#20 0x0000000000000000 in ?? ()
Stefan
Am 18.01.2017 um 17:49 schrieb Yann Ylavic:
> On Wed, Jan 18, 2017 at 4:38 PM, Stefan Priebe - Profihost AG
> <s....@profihost.ag> wrote:
>> Ok will wait whether you provide a fix or I should revert that part.
>
> Patch updated in PR 57399.
>
> Thanks,
> Yann.
>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Eissing <st...@greenbytes.de>.
> Am 18.01.2017 um 13:39 schrieb Stefan Eissing <st...@greenbytes.de>:
>
> There is currently no cleanup register for shutting down that way. If the main connection pool gets cleaned before the special meta is destroyed, these problem might happen. Not sure.
I am babbling. Of course there is. Question is: does it work correctly when the order changes...
Stefan Eissing
<green/>bytes GmbH
Hafenstrasse 16
48155 Münster
www.greenbytes.de
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Eissing <st...@greenbytes.de>.
Yann,
all these traces point to the connection shutdown case for mod_http2 which is triggered by writing a special meta bucket and cleaning up on its destruction. That's the trick to make sure that all data has been sent and similar to the EOR bucket.
There is currently no cleanup register for shutting down that way. If the main connection pool gets cleaned before the special meta is destroyed, these problem might happen. Not sure.
-Stefan
> Am 18.01.2017 um 13:32 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>
> Hi,
>
> and a new one just some seconds ago:
> (gdb) bt
> #0 0x00007f32a3747014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> #1 0x00007f32a3bda036 in ?? () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> #2 0x00007f32a3bda46f in apr_hash_set () from
> /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> #3 0x000000000052a238 in h2_ihash_remove ()
> #4 0x000000b700000000 in ?? ()
> #5 0x00007f329410e328 in ?? ()
> #6 0x00007f3283fe6300 in ?? ()
> #7 0x0000000000506b24 in purge_stream ()
> #8 0x00007f3282edf0a0 in ?? ()
> #9 0x00007f3282ecf2d8 in ?? ()
> #10 0x00007f32940e60a0 in ?? ()
> #11 0x000000b700000000 in ?? ()
> #12 0x00007f3282edf0a0 in ?? ()
> #13 0x00007f3282ecf2d8 in ?? ()
> #14 0x00007f3283fe6340 in ?? ()
> #15 0x000000000052a18f in ihash_iter ()
> #16 0x00007f3282edf0a0 in ?? ()
> #17 0x0000000000000004 in ?? ()
> #18 0x00007f3282edf0a0 in ?? ()
> #19 0x00007f3283fe63c0 in ?? ()
> #20 0x0000000000000000 in ?? ()
>
> Stefan
>
> Am 18.01.2017 um 12:49 schrieb Stefan Priebe - Profihost AG:
>> Hi,
>>
>> site is mostly used with http2. So it may be totally unrelated to
>> mod_http2. Sorry for the noise than. I just thought it by digging
>> through the traces.
>>
>> Stefan
>>
>> Am 18.01.2017 um 12:34 schrieb Yann Ylavic:
>>> Hi Stefan,
>>>
>>> On Wed, Jan 18, 2017 at 11:33 AM, Stefan Priebe - Profihost AG
>>> <s....@profihost.ag> wrote:
>>>>
>>>> after applying the event patch to 2.4.25 from
>>>> https://bz.apache.org/bugzilla/show_bug.cgi?id=57399.
>>>>
>>>> I'm seeing segfaults in the mod_http2 code. I already bumped mod_http2
>>>> to v1.8.8. But the segfaults are still happening.
>>>
>>> Does it happen only with http2, or maybe your site is mostly h2-used only?
>>>
Stefan Eissing
<green/>bytes GmbH
Hafenstrasse 16
48155 Münster
www.greenbytes.de
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Hi,
and a new one just some seconds ago:
(gdb) bt
#0 0x00007f32a3747014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f32a3bda036 in ?? () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
#2 0x00007f32a3bda46f in apr_hash_set () from
/usr/lib/x86_64-linux-gnu/libapr-1.so.0
#3 0x000000000052a238 in h2_ihash_remove ()
#4 0x000000b700000000 in ?? ()
#5 0x00007f329410e328 in ?? ()
#6 0x00007f3283fe6300 in ?? ()
#7 0x0000000000506b24 in purge_stream ()
#8 0x00007f3282edf0a0 in ?? ()
#9 0x00007f3282ecf2d8 in ?? ()
#10 0x00007f32940e60a0 in ?? ()
#11 0x000000b700000000 in ?? ()
#12 0x00007f3282edf0a0 in ?? ()
#13 0x00007f3282ecf2d8 in ?? ()
#14 0x00007f3283fe6340 in ?? ()
#15 0x000000000052a18f in ihash_iter ()
#16 0x00007f3282edf0a0 in ?? ()
#17 0x0000000000000004 in ?? ()
#18 0x00007f3282edf0a0 in ?? ()
#19 0x00007f3283fe63c0 in ?? ()
#20 0x0000000000000000 in ?? ()
Stefan
Am 18.01.2017 um 12:49 schrieb Stefan Priebe - Profihost AG:
> Hi,
>
> site is mostly used with http2. So it may be totally unrelated to
> mod_http2. Sorry for the noise than. I just thought it by digging
> through the traces.
>
> Stefan
>
> Am 18.01.2017 um 12:34 schrieb Yann Ylavic:
>> Hi Stefan,
>>
>> On Wed, Jan 18, 2017 at 11:33 AM, Stefan Priebe - Profihost AG
>> <s....@profihost.ag> wrote:
>>>
>>> after applying the event patch to 2.4.25 from
>>> https://bz.apache.org/bugzilla/show_bug.cgi?id=57399.
>>>
>>> I'm seeing segfaults in the mod_http2 code. I already bumped mod_http2
>>> to v1.8.8. But the segfaults are still happening.
>>
>> Does it happen only with http2, or maybe your site is mostly h2-used only?
>>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
On Wed, Jan 18, 2017 at 4:38 PM, Stefan Priebe - Profihost AG
<s....@profihost.ag> wrote:
> Ok will wait whether you provide a fix or I should revert that part.
Patch updated in PR 57399.
Thanks,
Yann.
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Ok will wait whether you provide a fix or I should revert that part.
Stefan
Excuse my typo sent from my mobile phone.
> Am 18.01.2017 um 16:32 schrieb Yann Ylavic <yl...@gmail.com>:
>
> On Wed, Jan 18, 2017 at 4:17 PM, Stefan Priebe - Profihost AG
> <s....@profihost.ag> wrote:
>>
>> is it enought to revert that commit?
>
> I think so (worth a try at least).
> I'm about to commit sort of this revert to trunk, with a possible explanation...
>
> Thanks Stefan for all these follow ups.
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
On Wed, Jan 18, 2017 at 4:17 PM, Stefan Priebe - Profihost AG
<s....@profihost.ag> wrote:
>
> is it enought to revert that commit?
I think so (worth a try at least).
I'm about to commit sort of this revert to trunk, with a possible explanation...
Thanks Stefan for all these follow ups.
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Hi Yann,
is it enought to revert that commit?
http://svn.apache.org/r1762701
Stefan
Am 18.01.2017 um 12:49 schrieb Stefan Priebe - Profihost AG:
> Hi,
>
> site is mostly used with http2. So it may be totally unrelated to
> mod_http2. Sorry for the noise than. I just thought it by digging
> through the traces.
>
> Stefan
>
> Am 18.01.2017 um 12:34 schrieb Yann Ylavic:
>> Hi Stefan,
>>
>> On Wed, Jan 18, 2017 at 11:33 AM, Stefan Priebe - Profihost AG
>> <s....@profihost.ag> wrote:
>>>
>>> after applying the event patch to 2.4.25 from
>>> https://bz.apache.org/bugzilla/show_bug.cgi?id=57399.
>>>
>>> I'm seeing segfaults in the mod_http2 code. I already bumped mod_http2
>>> to v1.8.8. But the segfaults are still happening.
>>
>> Does it happen only with http2, or maybe your site is mostly h2-used only?
>>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Hi,
site is mostly used with http2. So it may be totally unrelated to
mod_http2. Sorry for the noise than. I just thought it by digging
through the traces.
Stefan
Am 18.01.2017 um 12:34 schrieb Yann Ylavic:
> Hi Stefan,
>
> On Wed, Jan 18, 2017 at 11:33 AM, Stefan Priebe - Profihost AG
> <s....@profihost.ag> wrote:
>>
>> after applying the event patch to 2.4.25 from
>> https://bz.apache.org/bugzilla/show_bug.cgi?id=57399.
>>
>> I'm seeing segfaults in the mod_http2 code. I already bumped mod_http2
>> to v1.8.8. But the segfaults are still happening.
>
> Does it happen only with http2, or maybe your site is mostly h2-used only?
>
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
Hi Stefan,
On Wed, Jan 18, 2017 at 11:33 AM, Stefan Priebe - Profihost AG
<s....@profihost.ag> wrote:
>
> after applying the event patch to 2.4.25 from
> https://bz.apache.org/bugzilla/show_bug.cgi?id=57399.
>
> I'm seeing segfaults in the mod_http2 code. I already bumped mod_http2
> to v1.8.8. But the segfaults are still happening.
Does it happen only with http2, or maybe your site is mostly h2-used only?
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
On Wed, Jan 18, 2017 at 12:09 PM, Stefan Eissing
<st...@greenbytes.de> wrote:
> Can't have it all, Stefan!
:)
>
> Seriously, will have a look at this. Thanks for the traces.
I possibly messed up with event's locking in latest patch from PR 57399.
Actually it removes locks around pollset_add/del ([1]) which may be
the cause, though I don't see why for now...
Will take a look too.
[1] http://svn.apache.org/r1762701
Re: mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Eissing <st...@greenbytes.de>.
Can't have it all, Stefan!
Seriously, will have a look at this. Thanks for the traces.
-Stefan
> Am 18.01.2017 um 11:33 schrieb Stefan Priebe - Profihost AG <s....@profihost.ag>:
>
> Hi Stefan,
> Hi Yann,
>
> after applying the event patch to 2.4.25 from
> https://bz.apache.org/bugzilla/show_bug.cgi?id=57399.
>
> I'm seeing segfaults in the mod_http2 code. I already bumped mod_http2
> to v1.8.8. But the segfaults are still happening.
>
> gdb shows this:
> Core was generated by `/usr/local/apache2/bin/httpd -k start'.
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0 0x00007fb75242cd45 in apr_pool_cleanup_kill () from
> /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> (gdb) bt
> #0 0x00007fb75242cd45 in apr_pool_cleanup_kill () from
> /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> #1 0x00007fb75242ce21 in apr_pool_cleanup_run () from
> /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> #2 0x000000000051f268 in stream_pool_cleanup ()
> #3 0x00007fb7327eb270 in ?? ()
> #4 0x00007fb730c180a0 in ?? ()
> #5 0x00007fb7327eb294 in ?? ()
> #6 0x00007fb7309046c0 in ?? ()
> #7 0x00007fb730c180a0 in ?? ()
> #8 0x0000000030c18028 in ?? ()
> #9 0x00007fb730c18028 in ?? ()
> #10 0x00007fb75242b9be in apr_pool_destroy () from
> /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> #11 0x000000000051fe61 in h2_stream_destroy ()
> #12 0x0000002d317482d8 in ?? ()
> #13 0x00007fb730c180a0 in ?? ()
> #14 0x00007fb7327eb300 in ?? ()
> #15 0x0000000000507ab7 in stream_done ()
> #16 0x0000000000000000 in ?? ()
>
>
> and or this:
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0 0x00007fd2e9034014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> (gdb) bt
> #0 0x00007fd2e9034014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> #1 0x00007fd2e94c7036 in ?? () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> #2 0x00007fd2e94c746f in apr_hash_set () from
> /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> #3 0x000000000052a238 in h2_ihash_remove ()
> #4 0x0000004b00000000 in ?? ()
> #5 0x00007fd2c8d60328 in ?? ()
> #6 0x00007fd2c97e9300 in ?? ()
> #7 0x0000000000506b24 in purge_stream ()
> #8 0x00007fd2d82140a0 in ?? ()
> #9 0x00007fd2c831e2d8 in ?? ()
> #10 0x00007fd20352c0a0 in ?? ()
> #11 0x0000004b00000000 in ?? ()
> #12 0x00007fd2d82140a0 in ?? ()
> #13 0x00007fd2c831e2d8 in ?? ()
> #14 0x00007fd2c97e9340 in ?? ()
> #15 0x000000000052a18f in ihash_iter ()
> #16 0x00007fd2d82140a0 in ?? ()
> #17 0x0000000000000004 in ?? ()
> #18 0x00007fd2d82140a0 in ?? ()
> #19 0x00007fd2c97e93c0 in ?? ()
> #20 0x0000000000000000 in ?? ()
>
>
> Greets,
> Stefan
>
> Am 17.01.2017 um 21:53 schrieb Stefan Priebe:
>> Hi Yann,
>>
>> while testing V6 i'm experiencing segfaults.
>>
>> exit signal Segmentation
>>
>> server-error.log:
>> AH00052: child pid 14110 exit signal Segmentation fault (11)
>>
>> currently i'm trying to grab a core dump.
>>
>> Greets,
>> Stefan
>>
>> Am 26.12.2016 um 21:18 schrieb Stefan Priebe - Profihost AG:
>>> Am 23.12.2016 um 01:41 schrieb Yann Ylavic:
>>>> Hi Stefan,
>>>>
>>>> On Tue, Dec 20, 2016 at 1:52 PM, Stefan Priebe - Profihost AG
>>>> <s....@profihost.ag> wrote:
>>>>>
>>>>> Today i had another server giving no answers to any requests. apache
>>>>> fullstatus did not respond.
>>>>
>>>> Since v5 of the patch, I committed another related change in trunk,
>>>> namely: http://svn.apache.org/r1774538
>>>> It's about lingering keepalive connections on graceful restart which
>>>> may not cause a wakeup.
>>>> Does it help?
>>>
>>> I'll try that but wanted to rebuild based on http 2.4.25. But your
>>> mpm_event_listener_wakeup_bug57399_V5 patch does no longer apply to http
>>> 2.2.25. Can you rebase it?
>>>
>>>>> gdb bt shows this for all httpd childs:
>>>>
>>>> These backtraces are the ones of the main thread, probably not the
>>>> culprit.
>>>> What does "thread apply all bt" say?
>>>
>>> Will redo / save that output next time.
>>>
>>> Thanks!
>>>
>>> Greets,
>>> Stefan
>>>
>>>> Regards,
>>>> Yann.
>>>>
Stefan Eissing
<green/>bytes GmbH
Hafenstrasse 16
48155 Münster
www.greenbytes.de
mod_http2 and Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Hi Stefan,
Hi Yann,
after applying the event patch to 2.4.25 from
https://bz.apache.org/bugzilla/show_bug.cgi?id=57399.
I'm seeing segfaults in the mod_http2 code. I already bumped mod_http2
to v1.8.8. But the segfaults are still happening.
gdb shows this:
Core was generated by `/usr/local/apache2/bin/httpd -k start'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007fb75242cd45 in apr_pool_cleanup_kill () from
/usr/lib/x86_64-linux-gnu/libapr-1.so.0
(gdb) bt
#0 0x00007fb75242cd45 in apr_pool_cleanup_kill () from
/usr/lib/x86_64-linux-gnu/libapr-1.so.0
#1 0x00007fb75242ce21 in apr_pool_cleanup_run () from
/usr/lib/x86_64-linux-gnu/libapr-1.so.0
#2 0x000000000051f268 in stream_pool_cleanup ()
#3 0x00007fb7327eb270 in ?? ()
#4 0x00007fb730c180a0 in ?? ()
#5 0x00007fb7327eb294 in ?? ()
#6 0x00007fb7309046c0 in ?? ()
#7 0x00007fb730c180a0 in ?? ()
#8 0x0000000030c18028 in ?? ()
#9 0x00007fb730c18028 in ?? ()
#10 0x00007fb75242b9be in apr_pool_destroy () from
/usr/lib/x86_64-linux-gnu/libapr-1.so.0
#11 0x000000000051fe61 in h2_stream_destroy ()
#12 0x0000002d317482d8 in ?? ()
#13 0x00007fb730c180a0 in ?? ()
#14 0x00007fb7327eb300 in ?? ()
#15 0x0000000000507ab7 in stream_done ()
#16 0x0000000000000000 in ?? ()
and or this:
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007fd2e9034014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0 0x00007fd2e9034014 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fd2e94c7036 in ?? () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
#2 0x00007fd2e94c746f in apr_hash_set () from
/usr/lib/x86_64-linux-gnu/libapr-1.so.0
#3 0x000000000052a238 in h2_ihash_remove ()
#4 0x0000004b00000000 in ?? ()
#5 0x00007fd2c8d60328 in ?? ()
#6 0x00007fd2c97e9300 in ?? ()
#7 0x0000000000506b24 in purge_stream ()
#8 0x00007fd2d82140a0 in ?? ()
#9 0x00007fd2c831e2d8 in ?? ()
#10 0x00007fd20352c0a0 in ?? ()
#11 0x0000004b00000000 in ?? ()
#12 0x00007fd2d82140a0 in ?? ()
#13 0x00007fd2c831e2d8 in ?? ()
#14 0x00007fd2c97e9340 in ?? ()
#15 0x000000000052a18f in ihash_iter ()
#16 0x00007fd2d82140a0 in ?? ()
#17 0x0000000000000004 in ?? ()
#18 0x00007fd2d82140a0 in ?? ()
#19 0x00007fd2c97e93c0 in ?? ()
#20 0x0000000000000000 in ?? ()
Greets,
Stefan
Am 17.01.2017 um 21:53 schrieb Stefan Priebe:
> Hi Yann,
>
> while testing V6 i'm experiencing segfaults.
>
> exit signal Segmentation
>
> server-error.log:
> AH00052: child pid 14110 exit signal Segmentation fault (11)
>
> currently i'm trying to grab a core dump.
>
> Greets,
> Stefan
>
> Am 26.12.2016 um 21:18 schrieb Stefan Priebe - Profihost AG:
>> Am 23.12.2016 um 01:41 schrieb Yann Ylavic:
>>> Hi Stefan,
>>>
>>> On Tue, Dec 20, 2016 at 1:52 PM, Stefan Priebe - Profihost AG
>>> <s....@profihost.ag> wrote:
>>>>
>>>> Today i had another server giving no answers to any requests. apache
>>>> fullstatus did not respond.
>>>
>>> Since v5 of the patch, I committed another related change in trunk,
>>> namely: http://svn.apache.org/r1774538
>>> It's about lingering keepalive connections on graceful restart which
>>> may not cause a wakeup.
>>> Does it help?
>>
>> I'll try that but wanted to rebuild based on http 2.4.25. But your
>> mpm_event_listener_wakeup_bug57399_V5 patch does no longer apply to http
>> 2.2.25. Can you rebase it?
>>
>>>> gdb bt shows this for all httpd childs:
>>>
>>> These backtraces are the ones of the main thread, probably not the
>>> culprit.
>>> What does "thread apply all bt" say?
>>
>> Will redo / save that output next time.
>>
>> Thanks!
>>
>> Greets,
>> Stefan
>>
>>> Regards,
>>> Yann.
>>>
Re: Frequent wake-ups for mpm_event
Posted by Stefan Priebe <s....@profihost.ag>.
Hi Yann,
while testing V6 i'm experiencing segfaults.
exit signal Segmentation
server-error.log:
AH00052: child pid 14110 exit signal Segmentation fault (11)
currently i'm trying to grab a core dump.
Greets,
Stefan
Am 26.12.2016 um 21:18 schrieb Stefan Priebe - Profihost AG:
> Am 23.12.2016 um 01:41 schrieb Yann Ylavic:
>> Hi Stefan,
>>
>> On Tue, Dec 20, 2016 at 1:52 PM, Stefan Priebe - Profihost AG
>> <s....@profihost.ag> wrote:
>>>
>>> Today i had another server giving no answers to any requests. apache
>>> fullstatus did not respond.
>>
>> Since v5 of the patch, I committed another related change in trunk,
>> namely: http://svn.apache.org/r1774538
>> It's about lingering keepalive connections on graceful restart which
>> may not cause a wakeup.
>> Does it help?
>
> I'll try that but wanted to rebuild based on http 2.4.25. But your
> mpm_event_listener_wakeup_bug57399_V5 patch does no longer apply to http
> 2.2.25. Can you rebase it?
>
>>> gdb bt shows this for all httpd childs:
>>
>> These backtraces are the ones of the main thread, probably not the culprit.
>> What does "thread apply all bt" say?
>
> Will redo / save that output next time.
>
> Thanks!
>
> Greets,
> Stefan
>
>> Regards,
>> Yann.
>>
Re: Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Am 23.12.2016 um 01:41 schrieb Yann Ylavic:
> Hi Stefan,
>
> On Tue, Dec 20, 2016 at 1:52 PM, Stefan Priebe - Profihost AG
> <s....@profihost.ag> wrote:
>>
>> Today i had another server giving no answers to any requests. apache
>> fullstatus did not respond.
>
> Since v5 of the patch, I committed another related change in trunk,
> namely: http://svn.apache.org/r1774538
> It's about lingering keepalive connections on graceful restart which
> may not cause a wakeup.
> Does it help?
I'll try that but wanted to rebuild based on http 2.4.25. But your
mpm_event_listener_wakeup_bug57399_V5 patch does no longer apply to http
2.2.25. Can you rebase it?
>> gdb bt shows this for all httpd childs:
>
> These backtraces are the ones of the main thread, probably not the culprit.
> What does "thread apply all bt" say?
Will redo / save that output next time.
Thanks!
Greets,
Stefan
> Regards,
> Yann.
>
Re: Frequent wake-ups for mpm_event
Posted by Luca Toscano <to...@gmail.com>.
2016-10-10 22:01 GMT+02:00 Stefan Priebe - Profihost AG <
s.priebe@profihost.ag>:
> Hi Luca,
> Am 10.10.2016 um 16:48 schrieb Luca Toscano:
> > Hi Stefan,
> >
> > 2016-10-07 10:11 GMT+02:00 Stefan Priebe - Profihost AG
> > <s.priebe@profihost.ag <ma...@profihost.ag>>:
> >
> > Am 05.10.2016 <tel:05.10.2016> um 12:50 schrieb Luca Toscano:
> > > Hi Stefan,
> > >
> > > 2016-09-26 14:26 GMT+02:00 Stefan Priebe - Profihost AG
> > > <s.priebe@profihost.ag <ma...@profihost.ag>
> > <mailto:s.priebe@profihost.ag <ma...@profihost.ag>>>:
> > >
> > > currently no deadlocks since V5 also no old httpd processes
> anymore.
> > >
> > >
> > > if you still have patience and time, I'd like to ask you a
> question:
> > > have you noticed any performance improvement after applying the
> patch?
> > > For example (Yann correct me if I am wrong) I'd expect some
> reduction in
> > > system CPU utilization since the number of wake ups / context
> switches
> > > has been reduced dramatically. Moreover, it would be great to know
> some
> > > info about the status of the worker threads over time from the
> > > scoreboard, since it would be great not to introduce weird
> regressions
> > > with this patch :)
> >
> > Dear Luca,
> >
> > sure what can i exactly provide?
> >
> >
> > I would be interested in scoreboard related metrics over time, namely if
> > the number of busy/idle/graceful/etc.. worker statuses change from a
> > "regular" 2.4.23 httpd. I don't expect a bit difference, but I'd really
> > like to make sure that no big regression is introduced (like let's say,
> > an increase over time of worker threads stuck in a certain status like G
> > or more busy workers at request peak load time).
> >
> > Related to the above: have you noticed any improvement in
> > latency/throughput metrics? For example, an improvement in total amount
> > of requests handled in the same period of time o more generally in the
> > responsiveness of the websites served?
>
> It seems i cannot provide any of the requested data ;-(
>
You have already provided an awesome support to this patch Stefan, thanks a
lot!
Luca
Re: Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Hi Luca,
Am 10.10.2016 um 16:48 schrieb Luca Toscano:
> Hi Stefan,
>
> 2016-10-07 10:11 GMT+02:00 Stefan Priebe - Profihost AG
> <s.priebe@profihost.ag <ma...@profihost.ag>>:
>
> Am 05.10.2016 <tel:05.10.2016> um 12:50 schrieb Luca Toscano:
> > Hi Stefan,
> >
> > 2016-09-26 14:26 GMT+02:00 Stefan Priebe - Profihost AG
> > <s.priebe@profihost.ag <ma...@profihost.ag>
> <mailto:s.priebe@profihost.ag <ma...@profihost.ag>>>:
> >
> > currently no deadlocks since V5 also no old httpd processes anymore.
> >
> >
> > if you still have patience and time, I'd like to ask you a question:
> > have you noticed any performance improvement after applying the patch?
> > For example (Yann correct me if I am wrong) I'd expect some reduction in
> > system CPU utilization since the number of wake ups / context switches
> > has been reduced dramatically. Moreover, it would be great to know some
> > info about the status of the worker threads over time from the
> > scoreboard, since it would be great not to introduce weird regressions
> > with this patch :)
>
> Dear Luca,
>
> sure what can i exactly provide?
>
>
> I would be interested in scoreboard related metrics over time, namely if
> the number of busy/idle/graceful/etc.. worker statuses change from a
> "regular" 2.4.23 httpd. I don't expect a bit difference, but I'd really
> like to make sure that no big regression is introduced (like let's say,
> an increase over time of worker threads stuck in a certain status like G
> or more busy workers at request peak load time).
>
> Related to the above: have you noticed any improvement in
> latency/throughput metrics? For example, an improvement in total amount
> of requests handled in the same period of time o more generally in the
> responsiveness of the websites served?
It seems i cannot provide any of the requested data ;-(
This is an example output of a server running with the patch:
Current Time: Monday, 10-Oct-2016 21:57:49 CEST
Restart Time: Wednesday, 05-Oct-2016 06:30:31 CEST
Parent Server Config. Generation: 12
Parent Server MPM Generation: 11
Server uptime: 5 days 15 hours 27 minutes 17 seconds
Server load: 4.01 4.22 4.10
Total accesses: 6485371 - Total Traffic: 139.0 GB
CPU Usage: u2686.2 s1651.82 cu0 cs0 - .89% CPU load
13.3 requests/sec - 298.9 kB/second - 22.5 kB/request
10 requests currently being processed, 115 idle workers
Slot PID Stopping Connections Threads Async connections
total accepting busy idle writing keep-alive closing
0 6044 no 8 yes 2 23 0 1 4
1 15254 no 3 yes 2 23 0 0 2
2 15189 no 5 yes 3 22 0 0 2
3 15081 no 5 yes 0 25 0 0 4
4 15082 no 1 yes 3 22 0 0 0
Sum 5 0 22 10 115 0 1 12
L__W__________________________W__________________W_W__W_________
__W_____________________________________W______________L_W___...
................................................................
................................................................
................................................................
................................................................
................
Stefan
Re: Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
Hi Stefan,
On Tue, Dec 20, 2016 at 1:52 PM, Stefan Priebe - Profihost AG
<s....@profihost.ag> wrote:
>
> Today i had another server giving no answers to any requests. apache
> fullstatus did not respond.
Since v5 of the patch, I committed another related change in trunk,
namely: http://svn.apache.org/r1774538
It's about lingering keepalive connections on graceful restart which
may not cause a wakeup.
Does it help?
>
> gdb bt shows this for all httpd childs:
These backtraces are the ones of the main thread, probably not the culprit.
What does "thread apply all bt" say?
Regards,
Yann.
Re: Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Hi Yann,
here is a follow up on this old thread with some problems again. See below.
Am 23.09.2016 um 21:26 schrieb Stefan Priebe - Profihost AG:
> Hi Yann,
> Hi Luca,
>
> right now i had another server which shows:
> AH00485: scoreboard is full, not at MaxRequestWorkers msg.
>
> I never saw that before applying the event patch.
>
> A fullstatus view is not answered - so the whole apache hangs. Also it
> has 84 running processes instead of normally just 4-8.
> # ps aux|grep httpd | wc -l
> 84
>
> There are also not many connections:
> # netstat -na | egrep ":80|:443"|wc -l
> 110
>
> Even after a restart all and even more httpd processes are there:
> # ps aux|grep httpd | wc -l
> 89
>
> ps aux also tell me that there a lot of pretty old processes:
> # ps aux|grep httpd | grep Sep22 | wc -l
> 40
>
> and even a lot of processes are from Sep 23 many hours ago. So i don't
> think this is bug #53555.
>
> It just seems a lot of threads are hanging / deadlocked.
>
> gdb says:
> (gdb) bt
> #0 0x00007f9235b634db in pthread_join () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00007f9235d9e9bb in apr_thread_join () from
> /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> #2 0x000000000055b283 in join_workers ()
> #3 0x000000000055b8cc in child_main ()
> #4 0x000000000055ba3f in make_child ()
> #5 0x000000000055c21e in perform_idle_server_maintenance ()
> #6 0x000000000055c619 in server_main_loop ()
> #7 0x000000000055c959 in event_run ()
> #8 0x0000000000445f91 in ap_run_mpm ()
> #9 0x000000000043dd19 in main ()
>
>
> bt from all threads of an old httpd process:
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> 0x00007f9235b634db in pthread_join () from
> /lib/x86_64-linux-gnu/libpthread.so.0
>
> Thread 52 (Thread 0x7f92325ee700 (LWP 13704)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00000000005286f9 in get_mplx_next ()
> #2 0x000000000052fb34 in execute ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 51 (Thread 0x7f9231ded700 (LWP 13705)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00000000005286f9 in get_mplx_next ()
> #2 0x000000000052fb34 in execute ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 50 (Thread 0x7f92315ec700 (LWP 13706)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00000000005286f9 in get_mplx_next ()
> #2 0x000000000052fb34 in execute ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 49 (Thread 0x7f9230deb700 (LWP 13707)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00000000005286f9 in get_mplx_next ()
> #2 0x000000000052fb34 in execute ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 48 (Thread 0x7f92305ea700 (LWP 13708)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00000000005286f9 in get_mplx_next ()
> #2 0x000000000052fb34 in execute ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 47 (Thread 0x7f922fde9700 (LWP 13709)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00000000005286f9 in get_mplx_next ()
> #2 0x000000000052fb34 in execute ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 46 (Thread 0x7f922f5e8700 (LWP 13710)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00000000005286f9 in get_mplx_next ()
> #2 0x000000000052fb34 in execute ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 45 (Thread 0x7f922ede7700 (LWP 13711)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00000000005286f9 in get_mplx_next ()
> #2 0x000000000052fb34 in execute ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 44 (Thread 0x7f922e5e6700 (LWP 13712)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00000000005286f9 in get_mplx_next ()
> #2 0x000000000052fb34 in execute ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 43 (Thread 0x7f922dde5700 (LWP 13713)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00000000005286f9 in get_mplx_next ()
> #2 0x000000000052fb34 in execute ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 42 (Thread 0x7f922d5e4700 (LWP 13714)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00000000005286f9 in get_mplx_next ()
> #2 0x000000000052fb34 in execute ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 41 (Thread 0x7f922cde3700 (LWP 13715)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00000000005286f9 in get_mplx_next ()
> #2 0x000000000052fb34 in execute ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 40 (Thread 0x7f922c5e2700 (LWP 13716)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00000000005286f9 in get_mplx_next ()
> #2 0x000000000052fb34 in execute ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 39 (Thread 0x7f922bde1700 (LWP 13717)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00000000005286f9 in get_mplx_next ()
> #2 0x000000000052fb34 in execute ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 38 (Thread 0x7f922b5e0700 (LWP 13718)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00000000005286f9 in get_mplx_next ()
> #2 0x000000000052fb34 in execute ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 37 (Thread 0x7f922addf700 (LWP 13719)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00000000005286f9 in get_mplx_next ()
> #2 0x000000000052fb34 in execute ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 36 (Thread 0x7f922a5de700 (LWP 13720)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00000000005286f9 in get_mplx_next ()
> #2 0x000000000052fb34 in execute ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 35 (Thread 0x7f9229ddd700 (LWP 13721)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00000000005286f9 in get_mplx_next ()
> #2 0x000000000052fb34 in execute ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 34 (Thread 0x7f92295dc700 (LWP 13722)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00000000005286f9 in get_mplx_next ()
> #2 0x000000000052fb34 in execute ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 33 (Thread 0x7f9228ddb700 (LWP 13723)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00000000005286f9 in get_mplx_next ()
> #2 0x000000000052fb34 in execute ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 32 (Thread 0x7f92285da700 (LWP 13724)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00000000005286f9 in get_mplx_next ()
> #2 0x000000000052fb34 in execute ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 31 (Thread 0x7f9227dd9700 (LWP 13725)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00000000005286f9 in get_mplx_next ()
> #2 0x000000000052fb34 in execute ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 30 (Thread 0x7f92275d8700 (LWP 13726)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00000000005286f9 in get_mplx_next ()
> #2 0x000000000052fb34 in execute ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 29 (Thread 0x7f9226dd7700 (LWP 13727)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00000000005286f9 in get_mplx_next ()
> #2 0x000000000052fb34 in execute ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 28 (Thread 0x7f92265d6700 (LWP 13728)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00000000005286f9 in get_mplx_next ()
> #2 0x000000000052fb34 in execute ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 27 (Thread 0x7f92255d4700 (LWP 13783)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x000000000055fa26 in ap_queue_pop_something ()
> #2 0x000000000055a633 in worker_thread ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 26 (Thread 0x7f9224dd3700 (LWP 13784)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x000000000055fa26 in ap_queue_pop_something ()
> #2 0x000000000055a633 in worker_thread ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 25 (Thread 0x7f921ffff700 (LWP 13785)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x000000000055fa26 in ap_queue_pop_something ()
> #2 0x000000000055a633 in worker_thread ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 24 (Thread 0x7f921f7fe700 (LWP 13786)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x000000000055fa26 in ap_queue_pop_something ()
> #2 0x000000000055a633 in worker_thread ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 23 (Thread 0x7f921effd700 (LWP 13787)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x000000000055fa26 in ap_queue_pop_something ()
> #2 0x000000000055a633 in worker_thread ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 22 (Thread 0x7f921e7fc700 (LWP 13788)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x000000000055fa26 in ap_queue_pop_something ()
> #2 0x000000000055a633 in worker_thread ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 21 (Thread 0x7f921dffb700 (LWP 13789)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x000000000055fa26 in ap_queue_pop_something ()
> #2 0x000000000055a633 in worker_thread ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 20 (Thread 0x7f921d7fa700 (LWP 13790)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x000000000055fa26 in ap_queue_pop_something ()
> #2 0x000000000055a633 in worker_thread ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 19 (Thread 0x7f921cff9700 (LWP 13791)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x000000000055fa26 in ap_queue_pop_something ()
> #2 0x000000000055a633 in worker_thread ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 18 (Thread 0x7f921c7f8700 (LWP 13792)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x000000000055fa26 in ap_queue_pop_something ()
> #2 0x000000000055a633 in worker_thread ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 17 (Thread 0x7f921bff7700 (LWP 13793)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x000000000055fa26 in ap_queue_pop_something ()
> #2 0x000000000055a633 in worker_thread ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 16 (Thread 0x7f921b7f6700 (LWP 13794)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x000000000055fa26 in ap_queue_pop_something ()
> #2 0x000000000055a633 in worker_thread ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 15 (Thread 0x7f921aff5700 (LWP 13795)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x000000000055fa26 in ap_queue_pop_something ()
> #2 0x000000000055a633 in worker_thread ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 14 (Thread 0x7f921a7f4700 (LWP 13796)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x000000000055fa26 in ap_queue_pop_something ()
> #2 0x000000000055a633 in worker_thread ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 13 (Thread 0x7f9219ff3700 (LWP 13797)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x000000000055fa26 in ap_queue_pop_something ()
> #2 0x000000000055a633 in worker_thread ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 12 (Thread 0x7f92197f2700 (LWP 13798)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x000000000055fa26 in ap_queue_pop_something ()
> #2 0x000000000055a633 in worker_thread ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 11 (Thread 0x7f9218ff1700 (LWP 13799)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x000000000055fa26 in ap_queue_pop_something ()
> #2 0x000000000055a633 in worker_thread ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 10 (Thread 0x7f92187f0700 (LWP 13800)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x000000000055fa26 in ap_queue_pop_something ()
> #2 0x000000000055a633 in worker_thread ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 9 (Thread 0x7f9217fef700 (LWP 13801)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x000000000055fa26 in ap_queue_pop_something ()
> #2 0x000000000055a633 in worker_thread ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 8 (Thread 0x7f92177ee700 (LWP 13802)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x000000000055fa26 in ap_queue_pop_something ()
> #2 0x000000000055a633 in worker_thread ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 7 (Thread 0x7f9216fed700 (LWP 13803)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x000000000055fa26 in ap_queue_pop_something ()
> #2 0x000000000055a633 in worker_thread ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 6 (Thread 0x7f92167ec700 (LWP 13804)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x000000000055fa26 in ap_queue_pop_something ()
> #2 0x000000000055a633 in worker_thread ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 5 (Thread 0x7f9215feb700 (LWP 13805)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x000000000055fa26 in ap_queue_pop_something ()
> #2 0x000000000055a633 in worker_thread ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 4 (Thread 0x7f92157ea700 (LWP 13806)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x000000000055fa26 in ap_queue_pop_something ()
> #2 0x000000000055a633 in worker_thread ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 3 (Thread 0x7f9214fe9700 (LWP 13807)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x000000000055fa26 in ap_queue_pop_something ()
> #2 0x000000000055a633 in worker_thread ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 2 (Thread 0x7f92147e8700 (LWP 13808)):
> #0 0x00007f9235897bb3 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
> #1 0x00007f9235d99fc3 in ?? () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> #2 0x00000000005595bd in listener_thread ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 1 (Thread 0x7f9237159780 (LWP 13703)):
> #0 0x00007f9235b634db in pthread_join () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00007f9235d9e9bb in apr_thread_join () from
> /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> #2 0x000000000055b283 in join_workers ()
> #3 0x000000000055b8cc in child_main ()
> #4 0x000000000055ba3f in make_child ()
> #5 0x000000000055c21e in perform_idle_server_maintenance ()
> #6 0x000000000055c619 in server_main_loop ()
> #7 0x000000000055c959 in event_run ()
> #8 0x0000000000445f91 in ap_run_mpm ()
> #9 0x000000000043dd19 in main ()
>
>
> Stefan
>
> Am 23.09.2016 um 16:53 schrieb Yann Ylavic:
>> With the patch this time...
>>
>> On Fri, Sep 23, 2016 at 4:53 PM, Yann Ylavic <yl...@gmail.com> wrote:
>>> On Fri, Sep 23, 2016 at 3:13 PM, Yann Ylavic <yl...@gmail.com> wrote:
>>>>
>>>> If you can try another (additive) patch on the faulty server, it would
>>>> be awesome to test the one from [1], not sure it applies correctly
>>>> with my 2.4.x patch though, possibly better with the trunk version
>>>> over [1] (let me know otherwise).
>>>
>>> Not a big deal actually, the attached patch includes both [1] and [2] combined.
>>>
>>> Regards,
>>> Yann.
>>>
>>> [1] https://bz.apache.org/bugzilla/show_bug.cgi?id=53555#c55
>>> [2] https://bz.apache.org/bugzilla/show_bug.cgi?id=57399#c9
Today i had another server giving no answers to any requests. apache
fullstatus did not respond.
[ ~]# netstat -na | grep ":80" | wc -l
29
[ ~]# netstat -na | grep ":443" | wc -l
57
ps aux|grep -i httpd
root 575 0.0 0.2 161060 40092 ? Ss Dec07 1:47
/usr/local/apache2/bin/httpd -k start
wwwrun 17894 3.1 1.6 4891824 278748 ? Sl 11:52 3:42
/usr/local/apache2/bin/httpd -k start
wwwrun 25574 0.0 0.0 108316 6684 ? S 06:00 0:00
/usr/local/apache2/bin/httpd -k start
wwwrun 25575 0.0 0.0 108788 6504 ? S 06:00 0:00
/usr/local/apache2/bin/httpd -k start
wwwrun 25741 2.1 0.6 4778760 110512 ? Sl 06:00 10:02
/usr/local/apache2/bin/httpd -k start
wwwrun 28094 3.1 1.5 4916324 252428 ? Sl 06:31 13:59
/usr/local/apache2/bin/httpd -k start
wwwrun 29575 5.2 2.6 5081956 443192 ? Sl 06:53 22:00
/usr/local/apache2/bin/httpd -k start
an strace only shows:
[pid 28100] futex(0x7fc9f40301f4, FUTEX_WAIT_PRIVATE, 6214, NULL
<unfinished ...>
[pid 28099] futex(0x7fc9f40301f4, FUTEX_WAIT_PRIVATE, 6192, NULL
<unfinished ...>
[pid 28101] futex(0x7fc9f40301f4, FUTEX_WAIT_PRIVATE, 6214, NULL
<unfinished ...>
[pid 28099] <... futex resumed> ) = -1 EAGAIN (Resource
temporarily unavailable)
[pid 28098] futex(0x7fc9f40301f4, FUTEX_WAIT_PRIVATE, 6187, NULL
<unfinished ...>
[pid 28099] futex(0x7fc9f40301f4, FUTEX_WAIT_PRIVATE, 6214, NULL
<unfinished ...>
[pid 28098] <... futex resumed> ) = -1 EAGAIN (Resource
temporarily unavailable)
[pid 28097] futex(0x7fc9f40301f4, FUTEX_WAIT_PRIVATE, 6206, NULL
<unfinished ...>
[pid 28098] futex(0x7fc9f40301f4, FUTEX_WAIT_PRIVATE, 6214, NULL
<unfinished ...>
[pid 28097] <... futex resumed> ) = -1 EAGAIN (Resource
temporarily unavailable)
[pid 28096] futex(0x7fc9f40301f4, FUTEX_WAIT_PRIVATE, 6200, NULL
<unfinished ...>
[pid 28097] futex(0x7fc9f40301f4, FUTEX_WAIT_PRIVATE, 6214, NULL
<unfinished ...>
[pid 28096] <... futex resumed> ) = -1 EAGAIN (Resource
temporarily unavailable)
[pid 28095] futex(0x7fc9f40301f4, FUTEX_WAIT_PRIVATE, 6168, NULL
<unfinished ...>
[pid 28096] futex(0x7fc9f40301f4, FUTEX_WAIT_PRIVATE, 6214, NULL
<unfinished ...>
[pid 28095] <... futex resumed> ) = -1 EAGAIN (Resource
temporarily unavailable)
[pid 28094] read(8, <unfinished ...>
[pid 28095] futex(0x7fc9f40301f4, FUTEX_WAIT_PRIVATE, 6214, NULL
<unfinished ...>
[pid 28542] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection
timed out)
[pid 28542] futex(0x173db88, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 28542] futex(0x173d344,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 51873, {1482238254,
882451000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 28542] futex(0x173db88, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 28542] futex(0x173d344,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 51875, {1482238255,
883132000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 28542] futex(0x173db88, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 28542] futex(0x173d344,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 51877, {1482238256,
883880000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 28542] futex(0x173db88, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 28542] futex(0x173d344,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 51879, {1482238257,
884511000}, ffffffff <unfinished ...>
[pid 28146] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection
timed out)
[pid 28146] futex(0x7fc9ed2540a8, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 28146] futex(0x7fc9ca61a28c,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 4671, {1482238262,
784057000}, ffffffff <unfinished ...>
[pid 28542] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed
out)
[pid 28542] futex(0x173db88, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 28542] futex(0x173d344,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 51881, {1482238258,
885329000}, ffffffffq) = -1 ETIMEDOUT (Connection timed out)
[pid 28542] futex(0x173db88, FUTEX_WAKE_PRIVATE, 1) = 0
gdb bt shows this for all httpd childs:
(gdb) bt
#0 0x00007fc9f2b7dadd in read () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000047cd89 in ap_mpm_podx_check ()
#2 0x000000000055b98f in child_main ()
#3 0x000000000055bc2e in make_child ()
#4 0x000000000055c5a0 in perform_idle_server_maintenance ()
#5 0x000000000055c940 in server_main_loop ()
#6 0x000000000055cc80 in event_run ()
#7 0x0000000000445ff1 in ap_run_mpm ()
#8 0x000000000043dd79 in main ()
greets,
Stefan
Re: Frequent wake-ups for mpm_event
Posted by Luca Toscano <to...@gmail.com>.
Hi Stefan,
2016-10-07 10:11 GMT+02:00 Stefan Priebe - Profihost AG <
s.priebe@profihost.ag>:
> Am 05.10.2016 um 12:50 schrieb Luca Toscano:
> > Hi Stefan,
> >
> > 2016-09-26 14:26 GMT+02:00 Stefan Priebe - Profihost AG
> > <s.priebe@profihost.ag <ma...@profihost.ag>>:
> >
> > currently no deadlocks since V5 also no old httpd processes anymore.
> >
> >
> > if you still have patience and time, I'd like to ask you a question:
> > have you noticed any performance improvement after applying the patch?
> > For example (Yann correct me if I am wrong) I'd expect some reduction in
> > system CPU utilization since the number of wake ups / context switches
> > has been reduced dramatically. Moreover, it would be great to know some
> > info about the status of the worker threads over time from the
> > scoreboard, since it would be great not to introduce weird regressions
> > with this patch :)
>
> Dear Luca,
>
> sure what can i exactly provide?
>
I would be interested in scoreboard related metrics over time, namely if
the number of busy/idle/graceful/etc.. worker statuses change from a
"regular" 2.4.23 httpd. I don't expect a bit difference, but I'd really
like to make sure that no big regression is introduced (like let's say, an
increase over time of worker threads stuck in a certain status like G or
more busy workers at request peak load time).
Related to the above: have you noticed any improvement in
latency/throughput metrics? For example, an improvement in total amount of
requests handled in the same period of time o more generally in the
responsiveness of the websites served?
I don't expect big changes but I am curious :)
Thanks again!
Regards,
Luca
Re: Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Am 05.10.2016 um 12:50 schrieb Luca Toscano:
> Hi Stefan,
>
> 2016-09-26 14:26 GMT+02:00 Stefan Priebe - Profihost AG
> <s.priebe@profihost.ag <ma...@profihost.ag>>:
>
> currently no deadlocks since V5 also no old httpd processes anymore.
>
>
> if you still have patience and time, I'd like to ask you a question:
> have you noticed any performance improvement after applying the patch?
> For example (Yann correct me if I am wrong) I'd expect some reduction in
> system CPU utilization since the number of wake ups / context switches
> has been reduced dramatically. Moreover, it would be great to know some
> info about the status of the worker threads over time from the
> scoreboard, since it would be great not to introduce weird regressions
> with this patch :)
Dear Luca,
sure what can i exactly provide?
I don't see a difference in CPU usage but the machines are pretty big
(Dual Xeon E5 2667) so i'm not sure if it is noticable.
Greets,
Stefan
Re: Frequent wake-ups for mpm_event
Posted by Luca Toscano <to...@gmail.com>.
Hi Stefan,
2016-09-26 14:26 GMT+02:00 Stefan Priebe - Profihost AG <
s.priebe@profihost.ag>:
> currently no deadlocks since V5 also no old httpd processes anymore.
>
if you still have patience and time, I'd like to ask you a question: have
you noticed any performance improvement after applying the patch? For
example (Yann correct me if I am wrong) I'd expect some reduction in system
CPU utilization since the number of wake ups / context switches has been
reduced dramatically. Moreover, it would be great to know some info about
the status of the worker threads over time from the scoreboard, since it
would be great not to introduce weird regressions with this patch :)
Thanks again!
Luca
Re: Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
currently no deadlocks since V5 also no old httpd processes anymore.
Stefan
Am 24.09.2016 um 17:32 schrieb Yann Ylavic:
> On Sat, Sep 24, 2016 at 5:13 PM, Stefan Priebe - Profihost AG
> <s....@profihost.ag> wrote:
>>
>> saw v5 attached to bugzilla. Which one should I take?
>
> Either, v5 is the same as v4 plus the fix (hopefully) for when no
> wakeable poll method exists on the system, which shouldn't be the case
> for you (although it was triggered in your previous testing with v3
> where linux' epoll was not detected as wakeable (this is addressed in
> both v4 et v5).
>
> Regards,
> Yann.
>
Re: Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
On Sat, Sep 24, 2016 at 5:13 PM, Stefan Priebe - Profihost AG
<s....@profihost.ag> wrote:
>
> saw v5 attached to bugzilla. Which one should I take?
Either, v5 is the same as v4 plus the fix (hopefully) for when no
wakeable poll method exists on the system, which shouldn't be the case
for you (although it was triggered in your previous testing with v3
where linux' epoll was not detected as wakeable (this is addressed in
both v4 et v5).
Regards,
Yann.
Re: Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Hi,
saw v5 attached to bugzilla. Which one should I take?
Greets,
Stefan
Excuse my typo sent from my mobile phone.
> Am 24.09.2016 um 15:05 schrieb Yann Ylavic <yl...@gmail.com>:
>
> Hi Stefan,
>
> thanks again for the tests and very useful feedbacks!
>
> On Fri, Sep 23, 2016 at 9:26 PM, Stefan Priebe - Profihost AG
> <s....@profihost.ag> wrote:
>>
>> It just seems a lot of threads are hanging / deadlocked.
>
> Clearly...
>
>>
>> gdb says:
> [...]
>>
>> Thread 27 (Thread 0x7f92255d4700 (LWP 13783)):
>> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib/x86_64-linux-gnu/libpthread.so.0
>> #1 0x000000000055fa26 in ap_queue_pop_something ()
>> #2 0x000000000055a633 in worker_thread ()
>> #3 0x00007f9235b620a4 in start_thread () from
>> /lib/x86_64-linux-gnu/libpthread.so.0
>> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> [...]
>>
>> Thread 2 (Thread 0x7f92147e8700 (LWP 13808)):
>> #0 0x00007f9235897bb3 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
>> #1 0x00007f9235d99fc3 in ?? () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>> #2 0x00000000005595bd in listener_thread ()
>> #3 0x00007f9235b620a4 in start_thread () from
>> /lib/x86_64-linux-gnu/libpthread.so.0
>> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>>
>> Thread 1 (Thread 0x7f9237159780 (LWP 13703)):
>> #0 0x00007f9235b634db in pthread_join () from
>> /lib/x86_64-linux-gnu/libpthread.so.0
>> #1 0x00007f9235d9e9bb in apr_thread_join () from
>> /usr/lib/x86_64-linux-gnu/libapr-1.so.0
>> #2 0x000000000055b283 in join_workers ()
>> #3 0x000000000055b8cc in child_main ()
>> #4 0x000000000055ba3f in make_child ()
>> #5 0x000000000055c21e in perform_idle_server_maintenance ()
>> #6 0x000000000055c619 in server_main_loop ()
>> #7 0x000000000055c959 in event_run ()
>> #8 0x0000000000445f91 in ap_run_mpm ()
>> #9 0x000000000043dd19 in main ()
>
> So the listener is not woken up while the child process is being
> gracefuly stopped, and no worker seems to be working.
>
> It seems that 2.4.x is missing an important fix for my patch to work
> correctly (namely http://svn.apache.org/r1732228, which was never
> backported to 2.4).
> Without it, the listener/poll()ing wake-up thing is not enabled on linux...
>
> This says that my patch is broken when no "good" polling method is
> available on the system (I can probably fix that later), but this
> shouldn't the case for linux' epoll.
>
> Could you please give a(nother) try to this new v4 patch (attached)?
> This is simply v3 (from PR 57399) plus trunk's r1732228 mentioned above.
>
> Regards,
> Yann.
> <httpd-2.4.x-mpm_event-wakeup-v4.patch>
Re: Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
Hi Stefan,
thanks again for the tests and very useful feedbacks!
On Fri, Sep 23, 2016 at 9:26 PM, Stefan Priebe - Profihost AG
<s....@profihost.ag> wrote:
>
> It just seems a lot of threads are hanging / deadlocked.
Clearly...
>
> gdb says:
[...]
>
> Thread 27 (Thread 0x7f92255d4700 (LWP 13783)):
> #0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x000000000055fa26 in ap_queue_pop_something ()
> #2 0x000000000055a633 in worker_thread ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
[...]
>
> Thread 2 (Thread 0x7f92147e8700 (LWP 13808)):
> #0 0x00007f9235897bb3 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
> #1 0x00007f9235d99fc3 in ?? () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> #2 0x00000000005595bd in listener_thread ()
> #3 0x00007f9235b620a4 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 1 (Thread 0x7f9237159780 (LWP 13703)):
> #0 0x00007f9235b634db in pthread_join () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00007f9235d9e9bb in apr_thread_join () from
> /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> #2 0x000000000055b283 in join_workers ()
> #3 0x000000000055b8cc in child_main ()
> #4 0x000000000055ba3f in make_child ()
> #5 0x000000000055c21e in perform_idle_server_maintenance ()
> #6 0x000000000055c619 in server_main_loop ()
> #7 0x000000000055c959 in event_run ()
> #8 0x0000000000445f91 in ap_run_mpm ()
> #9 0x000000000043dd19 in main ()
So the listener is not woken up while the child process is being
gracefuly stopped, and no worker seems to be working.
It seems that 2.4.x is missing an important fix for my patch to work
correctly (namely http://svn.apache.org/r1732228, which was never
backported to 2.4).
Without it, the listener/poll()ing wake-up thing is not enabled on linux...
This says that my patch is broken when no "good" polling method is
available on the system (I can probably fix that later), but this
shouldn't the case for linux' epoll.
Could you please give a(nother) try to this new v4 patch (attached)?
This is simply v3 (from PR 57399) plus trunk's r1732228 mentioned above.
Regards,
Yann.
Re: Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Hi Yann,
Hi Luca,
right now i had another server which shows:
AH00485: scoreboard is full, not at MaxRequestWorkers msg.
I never saw that before applying the event patch.
A fullstatus view is not answered - so the whole apache hangs. Also it
has 84 running processes instead of normally just 4-8.
# ps aux|grep httpd | wc -l
84
There are also not many connections:
# netstat -na | egrep ":80|:443"|wc -l
110
Even after a restart all and even more httpd processes are there:
# ps aux|grep httpd | wc -l
89
ps aux also tell me that there a lot of pretty old processes:
# ps aux|grep httpd | grep Sep22 | wc -l
40
and even a lot of processes are from Sep 23 many hours ago. So i don't
think this is bug #53555.
It just seems a lot of threads are hanging / deadlocked.
gdb says:
(gdb) bt
#0 0x00007f9235b634db in pthread_join () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f9235d9e9bb in apr_thread_join () from
/usr/lib/x86_64-linux-gnu/libapr-1.so.0
#2 0x000000000055b283 in join_workers ()
#3 0x000000000055b8cc in child_main ()
#4 0x000000000055ba3f in make_child ()
#5 0x000000000055c21e in perform_idle_server_maintenance ()
#6 0x000000000055c619 in server_main_loop ()
#7 0x000000000055c959 in event_run ()
#8 0x0000000000445f91 in ap_run_mpm ()
#9 0x000000000043dd19 in main ()
bt from all threads of an old httpd process:
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f9235b634db in pthread_join () from
/lib/x86_64-linux-gnu/libpthread.so.0
Thread 52 (Thread 0x7f92325ee700 (LWP 13704)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000005286f9 in get_mplx_next ()
#2 0x000000000052fb34 in execute ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 51 (Thread 0x7f9231ded700 (LWP 13705)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000005286f9 in get_mplx_next ()
#2 0x000000000052fb34 in execute ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 50 (Thread 0x7f92315ec700 (LWP 13706)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000005286f9 in get_mplx_next ()
#2 0x000000000052fb34 in execute ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 49 (Thread 0x7f9230deb700 (LWP 13707)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000005286f9 in get_mplx_next ()
#2 0x000000000052fb34 in execute ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 48 (Thread 0x7f92305ea700 (LWP 13708)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000005286f9 in get_mplx_next ()
#2 0x000000000052fb34 in execute ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 47 (Thread 0x7f922fde9700 (LWP 13709)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000005286f9 in get_mplx_next ()
#2 0x000000000052fb34 in execute ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 46 (Thread 0x7f922f5e8700 (LWP 13710)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000005286f9 in get_mplx_next ()
#2 0x000000000052fb34 in execute ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 45 (Thread 0x7f922ede7700 (LWP 13711)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000005286f9 in get_mplx_next ()
#2 0x000000000052fb34 in execute ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 44 (Thread 0x7f922e5e6700 (LWP 13712)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000005286f9 in get_mplx_next ()
#2 0x000000000052fb34 in execute ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 43 (Thread 0x7f922dde5700 (LWP 13713)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000005286f9 in get_mplx_next ()
#2 0x000000000052fb34 in execute ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 42 (Thread 0x7f922d5e4700 (LWP 13714)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000005286f9 in get_mplx_next ()
#2 0x000000000052fb34 in execute ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 41 (Thread 0x7f922cde3700 (LWP 13715)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000005286f9 in get_mplx_next ()
#2 0x000000000052fb34 in execute ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 40 (Thread 0x7f922c5e2700 (LWP 13716)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000005286f9 in get_mplx_next ()
#2 0x000000000052fb34 in execute ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 39 (Thread 0x7f922bde1700 (LWP 13717)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000005286f9 in get_mplx_next ()
#2 0x000000000052fb34 in execute ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 38 (Thread 0x7f922b5e0700 (LWP 13718)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000005286f9 in get_mplx_next ()
#2 0x000000000052fb34 in execute ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 37 (Thread 0x7f922addf700 (LWP 13719)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000005286f9 in get_mplx_next ()
#2 0x000000000052fb34 in execute ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 36 (Thread 0x7f922a5de700 (LWP 13720)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000005286f9 in get_mplx_next ()
#2 0x000000000052fb34 in execute ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 35 (Thread 0x7f9229ddd700 (LWP 13721)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000005286f9 in get_mplx_next ()
#2 0x000000000052fb34 in execute ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 34 (Thread 0x7f92295dc700 (LWP 13722)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000005286f9 in get_mplx_next ()
#2 0x000000000052fb34 in execute ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 33 (Thread 0x7f9228ddb700 (LWP 13723)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000005286f9 in get_mplx_next ()
#2 0x000000000052fb34 in execute ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 32 (Thread 0x7f92285da700 (LWP 13724)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000005286f9 in get_mplx_next ()
#2 0x000000000052fb34 in execute ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 31 (Thread 0x7f9227dd9700 (LWP 13725)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000005286f9 in get_mplx_next ()
#2 0x000000000052fb34 in execute ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 30 (Thread 0x7f92275d8700 (LWP 13726)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000005286f9 in get_mplx_next ()
#2 0x000000000052fb34 in execute ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 29 (Thread 0x7f9226dd7700 (LWP 13727)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000005286f9 in get_mplx_next ()
#2 0x000000000052fb34 in execute ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 28 (Thread 0x7f92265d6700 (LWP 13728)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000005286f9 in get_mplx_next ()
#2 0x000000000052fb34 in execute ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 27 (Thread 0x7f92255d4700 (LWP 13783)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000055fa26 in ap_queue_pop_something ()
#2 0x000000000055a633 in worker_thread ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 26 (Thread 0x7f9224dd3700 (LWP 13784)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000055fa26 in ap_queue_pop_something ()
#2 0x000000000055a633 in worker_thread ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 25 (Thread 0x7f921ffff700 (LWP 13785)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000055fa26 in ap_queue_pop_something ()
#2 0x000000000055a633 in worker_thread ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 24 (Thread 0x7f921f7fe700 (LWP 13786)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000055fa26 in ap_queue_pop_something ()
#2 0x000000000055a633 in worker_thread ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 23 (Thread 0x7f921effd700 (LWP 13787)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000055fa26 in ap_queue_pop_something ()
#2 0x000000000055a633 in worker_thread ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 22 (Thread 0x7f921e7fc700 (LWP 13788)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000055fa26 in ap_queue_pop_something ()
#2 0x000000000055a633 in worker_thread ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 21 (Thread 0x7f921dffb700 (LWP 13789)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000055fa26 in ap_queue_pop_something ()
#2 0x000000000055a633 in worker_thread ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 20 (Thread 0x7f921d7fa700 (LWP 13790)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000055fa26 in ap_queue_pop_something ()
#2 0x000000000055a633 in worker_thread ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 19 (Thread 0x7f921cff9700 (LWP 13791)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000055fa26 in ap_queue_pop_something ()
#2 0x000000000055a633 in worker_thread ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 18 (Thread 0x7f921c7f8700 (LWP 13792)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000055fa26 in ap_queue_pop_something ()
#2 0x000000000055a633 in worker_thread ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 17 (Thread 0x7f921bff7700 (LWP 13793)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000055fa26 in ap_queue_pop_something ()
#2 0x000000000055a633 in worker_thread ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 16 (Thread 0x7f921b7f6700 (LWP 13794)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000055fa26 in ap_queue_pop_something ()
#2 0x000000000055a633 in worker_thread ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 15 (Thread 0x7f921aff5700 (LWP 13795)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000055fa26 in ap_queue_pop_something ()
#2 0x000000000055a633 in worker_thread ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 14 (Thread 0x7f921a7f4700 (LWP 13796)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000055fa26 in ap_queue_pop_something ()
#2 0x000000000055a633 in worker_thread ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 13 (Thread 0x7f9219ff3700 (LWP 13797)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000055fa26 in ap_queue_pop_something ()
#2 0x000000000055a633 in worker_thread ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 12 (Thread 0x7f92197f2700 (LWP 13798)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000055fa26 in ap_queue_pop_something ()
#2 0x000000000055a633 in worker_thread ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 11 (Thread 0x7f9218ff1700 (LWP 13799)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000055fa26 in ap_queue_pop_something ()
#2 0x000000000055a633 in worker_thread ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 10 (Thread 0x7f92187f0700 (LWP 13800)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000055fa26 in ap_queue_pop_something ()
#2 0x000000000055a633 in worker_thread ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 9 (Thread 0x7f9217fef700 (LWP 13801)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000055fa26 in ap_queue_pop_something ()
#2 0x000000000055a633 in worker_thread ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 8 (Thread 0x7f92177ee700 (LWP 13802)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000055fa26 in ap_queue_pop_something ()
#2 0x000000000055a633 in worker_thread ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 7 (Thread 0x7f9216fed700 (LWP 13803)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000055fa26 in ap_queue_pop_something ()
#2 0x000000000055a633 in worker_thread ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 6 (Thread 0x7f92167ec700 (LWP 13804)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000055fa26 in ap_queue_pop_something ()
#2 0x000000000055a633 in worker_thread ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 5 (Thread 0x7f9215feb700 (LWP 13805)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000055fa26 in ap_queue_pop_something ()
#2 0x000000000055a633 in worker_thread ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 4 (Thread 0x7f92157ea700 (LWP 13806)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000055fa26 in ap_queue_pop_something ()
#2 0x000000000055a633 in worker_thread ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 3 (Thread 0x7f9214fe9700 (LWP 13807)):
#0 0x00007f9235b6608f in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000055fa26 in ap_queue_pop_something ()
#2 0x000000000055a633 in worker_thread ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 2 (Thread 0x7f92147e8700 (LWP 13808)):
#0 0x00007f9235897bb3 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f9235d99fc3 in ?? () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
#2 0x00000000005595bd in listener_thread ()
#3 0x00007f9235b620a4 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f92358975dd in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 1 (Thread 0x7f9237159780 (LWP 13703)):
#0 0x00007f9235b634db in pthread_join () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f9235d9e9bb in apr_thread_join () from
/usr/lib/x86_64-linux-gnu/libapr-1.so.0
#2 0x000000000055b283 in join_workers ()
#3 0x000000000055b8cc in child_main ()
#4 0x000000000055ba3f in make_child ()
#5 0x000000000055c21e in perform_idle_server_maintenance ()
#6 0x000000000055c619 in server_main_loop ()
#7 0x000000000055c959 in event_run ()
#8 0x0000000000445f91 in ap_run_mpm ()
#9 0x000000000043dd19 in main ()
Stefan
Am 23.09.2016 um 16:53 schrieb Yann Ylavic:
> With the patch this time...
>
> On Fri, Sep 23, 2016 at 4:53 PM, Yann Ylavic <yl...@gmail.com> wrote:
>> On Fri, Sep 23, 2016 at 3:13 PM, Yann Ylavic <yl...@gmail.com> wrote:
>>>
>>> If you can try another (additive) patch on the faulty server, it would
>>> be awesome to test the one from [1], not sure it applies correctly
>>> with my 2.4.x patch though, possibly better with the trunk version
>>> over [1] (let me know otherwise).
>>
>> Not a big deal actually, the attached patch includes both [1] and [2] combined.
>>
>> Regards,
>> Yann.
>>
>> [1] https://bz.apache.org/bugzilla/show_bug.cgi?id=53555#c55
>> [2] https://bz.apache.org/bugzilla/show_bug.cgi?id=57399#c9
Re: Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
With the patch this time...
On Fri, Sep 23, 2016 at 4:53 PM, Yann Ylavic <yl...@gmail.com> wrote:
> On Fri, Sep 23, 2016 at 3:13 PM, Yann Ylavic <yl...@gmail.com> wrote:
>>
>> If you can try another (additive) patch on the faulty server, it would
>> be awesome to test the one from [1], not sure it applies correctly
>> with my 2.4.x patch though, possibly better with the trunk version
>> over [1] (let me know otherwise).
>
> Not a big deal actually, the attached patch includes both [1] and [2] combined.
>
> Regards,
> Yann.
>
> [1] https://bz.apache.org/bugzilla/show_bug.cgi?id=53555#c55
> [2] https://bz.apache.org/bugzilla/show_bug.cgi?id=57399#c9
Re: Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
On Fri, Sep 23, 2016 at 3:13 PM, Yann Ylavic <yl...@gmail.com> wrote:
>
> If you can try another (additive) patch on the faulty server, it would
> be awesome to test the one from [1], not sure it applies correctly
> with my 2.4.x patch though, possibly better with the trunk version
> over [1] (let me know otherwise).
Not a big deal actually, the attached patch includes both [1] and [2] combined.
Regards,
Yann.
[1] https://bz.apache.org/bugzilla/show_bug.cgi?id=53555#c55
[2] https://bz.apache.org/bugzilla/show_bug.cgi?id=57399#c9
Re: Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
Hi Stefan,
On Fri, Sep 23, 2016 at 1:15 PM, Stefan Priebe - Profihost AG
<s....@profihost.ag> wrote:
>
> a applied the latest patch (incl. fudge factor) to 1500 real life servers.
Thank you very much for testing in real conditions.
>
> Today i had the situation that no websites were delivered. The error.log
> says:
>
> [Fri Sep 23 09:02:41.974245 2016] [mpm_event:error] [pid 6495:tid
> 139689321596800] AH00485: scoreboard is full, not at MaxRequestWorkers
> [Fri Sep 23 09:02:42.975267 2016] [mpm_event:error] [pid 6495:tid
> 139689321596800] AH00485: scoreboard is full, not at MaxRequestWorkers
> [Fri Sep 23 09:02:43.976217 2016] [mpm_event:error] [pid 6495:tid
> 139689321596800] AH00485: scoreboard is full, not at MaxRequestWorkers
> [Fri Sep 23 09:02:44.977276 2016] [mpm_event:error] [pid 6495:tid
> 139689321596800] AH00485: scoreboard is full, not at MaxRequestWorkers
> [Fri Sep 23 09:02:45.978314 2016] [mpm_event:error] [pid 6495:tid
> 139689321596800] AH00485: scoreboard is full, not at MaxRequestWorkers
> [Fri Sep 23 09:02:46.979222 2016] [mpm_event:error] [pid 6495:tid
> 139689321596800] AH00485: scoreboard is full, not at MaxRequestWorkers
> [Fri Sep 23 09:02:47.980304 2016] [mpm_event:error] [pid 6495:tid
> 139689321596800] AH00485: scoreboard is full, not at MaxRequestWorkers
> [Fri Sep 23 09:02:48.981213 2016] [mpm_event:error] [pid 6495:tid
> 139689321596800] AH00485: scoreboard is full, not at MaxRequestWorkers
>
> Not sure if this is related to your patch.
Hm, this never happened before you applied the patch?
Did you observe a difference in the overall traffic (req/s, latency,
...) and/or CPU usage before it stopped responding?
What's your MaxRequestsPerChild setting, and if it's zero, do you
issue frequent graceful restarts?
It could of course be caused by (a bug in )the patch, but I don't see
for now how it could change the number of active/idle worker threads
on its own, but with a different traffic.
If it never happened before it may well be the case, though...
It would help to know the (mod_)status of these workers.
> I also found this one:
> https://bz.apache.org/bugzilla/show_bug.cgi?id=53555
If you can try another (additive) patch on the faulty server, it would
be awesome to test the one from [1], not sure it applies correctly
with my 2.4.x patch though, possibly better with the trunk version
over [1] (let me know otherwise).
Regards,
Yann.
[1] https://bz.apache.org/bugzilla/show_bug.cgi?id=53555#c55
Re: Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Hi Luca,
Am 23.09.2016 um 13:47 schrieb Luca Toscano:
> Hi Stefan,
>
> 2016-09-23 13:15 GMT+02:00 Stefan Priebe - Profihost AG
> <s.priebe@profihost.ag <ma...@profihost.ag>>:
>
> Hi Yann,
>
> a applied the latest patch (incl. fudge factor) to 1500 real life
> servers.
>
> Today i had the situation that no websites were delivered. The error.log
> says:
>
> [Fri Sep 23 09:02:41.974245 2016] [mpm_event:error] [pid 6495:tid
> 139689321596800] AH00485: scoreboard is full, not at MaxRequestWorkers
> [Fri Sep 23 09:02:42.975267 2016] [mpm_event:error] [pid 6495:tid
> [..]
> [Fri Sep 23 09:02:48.981213 2016] [mpm_event:error] [pid 6495:tid
> 139689321596800] AH00485: scoreboard is full, not at MaxRequestWorkers
>
> Not sure if this is related to your patch. I also found this one:
> https://bz.apache.org/bugzilla/show_bug.cgi?id=53555
> <https://bz.apache.org/bugzilla/show_bug.cgi?id=53555>
>
>
> Thanks a lot for the test and really sorry for the trouble. I have a
> couple of followup questions for you:
No problem - was just one server.
> 1) Did it happen to all the patched servers or just to one of them?
Just to one.
> Moreover, did it happen right after the deploy/upgrade or after a long
> time?
After 24hours.
> I am asking because I don't know your settings and it would help
> to know the scale of the impact to the 1500 servers.
>
> 2) Have you checked the scoreboard in mod_status when the issue happened
> by any chance? It would be useful to know the status of the workers when
> they reached AH00485.
Ah no i did not.
Regards,
Stefan
> It could be an occurrence of bug 53555 or maybe a new behavior of
> mpm_event after this patch..
>
> Regards,
>
> Luca
>
>
Re: Frequent wake-ups for mpm_event
Posted by Luca Toscano <to...@gmail.com>.
Hi Stefan,
2016-09-23 13:15 GMT+02:00 Stefan Priebe - Profihost AG <
s.priebe@profihost.ag>:
> Hi Yann,
>
> a applied the latest patch (incl. fudge factor) to 1500 real life servers.
>
> Today i had the situation that no websites were delivered. The error.log
> says:
>
> [Fri Sep 23 09:02:41.974245 2016] [mpm_event:error] [pid 6495:tid
> 139689321596800] AH00485: scoreboard is full, not at MaxRequestWorkers
> [Fri Sep 23 09:02:42.975267 2016] [mpm_event:error] [pid 6495:tid
> [..]
> [Fri Sep 23 09:02:48.981213 2016] [mpm_event:error] [pid 6495:tid
> 139689321596800] AH00485: scoreboard is full, not at MaxRequestWorkers
>
> Not sure if this is related to your patch. I also found this one:
> https://bz.apache.org/bugzilla/show_bug.cgi?id=53555
>
>
Thanks a lot for the test and really sorry for the trouble. I have a couple
of followup questions for you:
1) Did it happen to all the patched servers or just to one of them?
Moreover, did it happen right after the deploy/upgrade or after a long
time? I am asking because I don't know your settings and it would help to
know the scale of the impact to the 1500 servers.
2) Have you checked the scoreboard in mod_status when the issue happened by
any chance? It would be useful to know the status of the workers when they
reached AH00485.
It could be an occurrence of bug 53555 or maybe a new behavior of mpm_event
after this patch..
Regards,
Luca
Re: Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Hi Yann,
a applied the latest patch (incl. fudge factor) to 1500 real life servers.
Today i had the situation that no websites were delivered. The error.log
says:
[Fri Sep 23 09:02:41.974245 2016] [mpm_event:error] [pid 6495:tid
139689321596800] AH00485: scoreboard is full, not at MaxRequestWorkers
[Fri Sep 23 09:02:42.975267 2016] [mpm_event:error] [pid 6495:tid
139689321596800] AH00485: scoreboard is full, not at MaxRequestWorkers
[Fri Sep 23 09:02:43.976217 2016] [mpm_event:error] [pid 6495:tid
139689321596800] AH00485: scoreboard is full, not at MaxRequestWorkers
[Fri Sep 23 09:02:44.977276 2016] [mpm_event:error] [pid 6495:tid
139689321596800] AH00485: scoreboard is full, not at MaxRequestWorkers
[Fri Sep 23 09:02:45.978314 2016] [mpm_event:error] [pid 6495:tid
139689321596800] AH00485: scoreboard is full, not at MaxRequestWorkers
[Fri Sep 23 09:02:46.979222 2016] [mpm_event:error] [pid 6495:tid
139689321596800] AH00485: scoreboard is full, not at MaxRequestWorkers
[Fri Sep 23 09:02:47.980304 2016] [mpm_event:error] [pid 6495:tid
139689321596800] AH00485: scoreboard is full, not at MaxRequestWorkers
[Fri Sep 23 09:02:48.981213 2016] [mpm_event:error] [pid 6495:tid
139689321596800] AH00485: scoreboard is full, not at MaxRequestWorkers
Not sure if this is related to your patch. I also found this one:
https://bz.apache.org/bugzilla/show_bug.cgi?id=53555
Greets,
Stefan
Am 18.09.2016 um 14:16 schrieb Yann Ylavic:
> On Sun, Sep 18, 2016 at 12:16 PM, Luca Toscano <to...@gmail.com> wrote:
>>
>> 2016-09-15 11:41 GMT+02:00 Stefan Priebe - Profihost AG
>> <s....@profihost.ag>:
>>>
>>> that sounds great. Is there a version available that applies to 2.4 ?
>>
>> I tried to study/port Yann's patch to 2.4.x. but it is not that
>> straightforward since there are a lot of differences between the two
>> branches, most of them afaics are related to the skiplist's handling and
>> usage.
>
> Patch against 2.4.x attached.
>
> Regards,
> Yann.
>
Re: Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Hi Luca,
Am 18.09.2016 um 12:16 schrieb Luca Toscano:
> Hi Stefan,
>
> 2016-09-15 11:41 GMT+02:00 Stefan Priebe - Profihost AG
> <s.priebe@profihost.ag <ma...@profihost.ag>>:
>
> Hi,
>
> Am 15.09.2016 um 11:20 schrieb Stefan Eissing:
> > The patch works nicely on my OS X workhorse. Without it, I see 5-15 reactivations of the child processes every second, with the patch it stays at 0 most of the time.
> >
> > Very nice work, Yann!
>
> that sounds great. Is there a version available that applies to 2.4 ?
>
>
> I tried to study/port Yann's patch to 2.4.x. but it is not that
> straightforward since there are a lot of differences between the two
> branches, most of them afaics are related to the skiplist's handling and
> usage. In my opinion it would be great to use this patch to also make
> future backports easier, but I'll wait other opinions :)
>
> If you have a way to test the 2.4.x patch (when it will be ready) in a
> "real" environment and give us feedback it would be really awesome.
Yes will do so. Already installed it on 3 production machines.
Greets,
Stefan
>
>
>
Re: Frequent wake-ups for mpm_event
Posted by Yann Ylavic <yl...@gmail.com>.
On Sun, Sep 18, 2016 at 12:16 PM, Luca Toscano <to...@gmail.com> wrote:
>
> 2016-09-15 11:41 GMT+02:00 Stefan Priebe - Profihost AG
> <s....@profihost.ag>:
>>
>> that sounds great. Is there a version available that applies to 2.4 ?
>
> I tried to study/port Yann's patch to 2.4.x. but it is not that
> straightforward since there are a lot of differences between the two
> branches, most of them afaics are related to the skiplist's handling and
> usage.
Patch against 2.4.x attached.
Regards,
Yann.
Re: Frequent wake-ups for mpm_event
Posted by Luca Toscano <to...@gmail.com>.
Hi Stefan,
2016-09-15 11:41 GMT+02:00 Stefan Priebe - Profihost AG <
s.priebe@profihost.ag>:
> Hi,
>
> Am 15.09.2016 um 11:20 schrieb Stefan Eissing:
> > The patch works nicely on my OS X workhorse. Without it, I see 5-15
> reactivations of the child processes every second, with the patch it stays
> at 0 most of the time.
> >
> > Very nice work, Yann!
>
> that sounds great. Is there a version available that applies to 2.4 ?
I tried to study/port Yann's patch to 2.4.x. but it is not that
straightforward since there are a lot of differences between the two
branches, most of them afaics are related to the skiplist's handling and
usage. In my opinion it would be great to use this patch to also make
future backports easier, but I'll wait other opinions :)
If you have a way to test the 2.4.x patch (when it will be ready) in a
"real" environment and give us feedback it would be really awesome.
Thanks!
Regards,
Luca
Re: Frequent wake-ups for mpm_event
Posted by Stefan Priebe - Profihost AG <s....@profihost.ag>.
Hi,
Am 15.09.2016 um 11:20 schrieb Stefan Eissing:
> The patch works nicely on my OS X workhorse. Without it, I see 5-15 reactivations of the child processes every second, with the patch it stays at 0 most of the time.
>
> Very nice work, Yann!
that sounds great. Is there a version available that applies to 2.4 ?
Greets,
Stefan
>> Am 14.09.2016 um 18:47 schrieb Luca Toscano <to...@gmail.com>:
>>
>>
>>
>> 2016-08-25 16:45 GMT+02:00 Yann Ylavic <yl...@gmail.com>:
>>
>> Possibly others will test it too, and report...
>>
>>
>>
>> Ping to everybody interested in the list to check Yann's patch in https://bz.apache.org/bugzilla/show_bug.cgi?id=57399 and provide feedback :)
>>
>> The end goal would be to reduce as much as possible the amount of wake-ups done by the mpm-event's listener thread.
>>
>> Thanks!
>>
>> Regards,
>>
>> Luca
>>
>
Re: Frequent wake-ups for mpm_event
Posted by Stefan Eissing <st...@greenbytes.de>.
The patch works nicely on my OS X workhorse. Without it, I see 5-15 reactivations of the child processes every second, with the patch it stays at 0 most of the time.
Very nice work, Yann!
> Am 14.09.2016 um 18:47 schrieb Luca Toscano <to...@gmail.com>:
>
>
>
> 2016-08-25 16:45 GMT+02:00 Yann Ylavic <yl...@gmail.com>:
>
> Possibly others will test it too, and report...
>
>
>
> Ping to everybody interested in the list to check Yann's patch in https://bz.apache.org/bugzilla/show_bug.cgi?id=57399 and provide feedback :)
>
> The end goal would be to reduce as much as possible the amount of wake-ups done by the mpm-event's listener thread.
>
> Thanks!
>
> Regards,
>
> Luca
>