You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Eric Covener <co...@gmail.com> on 2015/06/21 17:07:53 UTC
Re: buckets across threads - question
re: https://github.com/icing/mod_h2/issues/28 has this design further evolved?
Is there a way to make the deep copy un-necessary by injecting your
own bucket_alloc created with a pool with a threadsafe allocator very
early? It seems like all of the buckets are created in the h2 thread
after you've had a chance to manipulate stuff.
On Tue, Mar 31, 2015 at 2:57 PM, Stefan Eissing
<st...@greenbytes.de> wrote:
> Thanks, Jim!
>
>
>
>> Am 31.03.2015 um 19:29 schrieb Jim Jagielski <ji...@jaguNET.com>:
>>
>> What I did is used the alpn patch as a guide and updated trunk
>> to add the functionality:
>>
>> http://svn.apache.org/r1670397
>>
>> I'll give it a few days to work out and then propose for a
>> 2.4 backport.
>>
>>> On Mar 31, 2015, at 1:13 PM, Stefan Eissing <st...@greenbytes.de> wrote:
>>>
>>> I think the old NPN patch is in trunk. The ALPN not AFAIK. I undertstand there is a trunk first policy, but it'd be good to get it at least going there... thanks.
>>>
>>> Stefan
>>>
>>>
>>>
>>> Am 31.03.2015 um 18:22 schrieb Jim Jagielski <ji...@jaguNET.com>:
>>>
>>>>>
>>>>> PS. As a small quid-pro-quo and because you were asking about a possibly 2.4.13: there has so far not been a volunteer to integrate the ALPN patch I adapted from mod_spdy. It would be great if someone could take a look at it. It is the biggest obstacle so far for experimenters (early sufferers) of mod_h2 and it would be very nice to have it out of the way.
>>>>
>>>> I'll take a look. It's not currently in trunk, is that right?
>>
--
Eric Covener
covener@gmail.com
Re: buckets across threads - question
Posted by Eric Covener <co...@gmail.com>.
On Sun, Jun 21, 2015 at 11:07 AM, Eric Covener <co...@gmail.com> wrote:
> re: https://github.com/icing/mod_h2/issues/28 has this design further evolved?
>
> Is there a way to make the deep copy un-necessary by injecting your
> own bucket_alloc created with a pool with a threadsafe allocator very
> early? It seems like all of the buckets are created in the h2 thread
> after you've had a chance to manipulate stuff.
Or a bucket_alloc per (faux) request_rec somehow.
Re: buckets across threads - question
Posted by Graham Leggett <mi...@sharp.fm>.
On 22 Jun 2015, at 10:07 AM, Stefan Eissing <st...@greenbytes.de> wrote:
> Sort answer: no.
> Re issue 28: I will look into this more today. I was able to generate a segfault on connection shutdown on Friday and work on a fix.
>
> Longer version re buckets:
> - even though buckets for a request are created inside a single worker thread, they are mutated/split/placed on free list upon processing down the main connection. Which then all affects their bucket_alloc_t from several threads. Especially on longer response bodies,
> - since there are no function pointers for bucket_alloc_t, it is not obvious to me how one would replace them with another implementation.
> - what can be added are, of course, new bucket implementations. Maybe there is an approach to move the data itself without copying more than once. With a clever handling of file buckets, maybe no copying at all.
>
> Otherwise, I am open to ideas.
The secret to all of this is the pool hierarchy - if a request is to be served by any thread in a pool of threads, then that request mustn’t ever contain memory allocated from a pool tied to a specific thread.
When mod_proxy learned how to do connection pooling there were subtle pool lifetime issues uncovered caused by the backend connections allocated from a connection pool being interfaced to the frontend connections per request. You have to get the pools 100% right.
Regards,
Graham
—
Re: module configs across (pseudo) connections
Posted by Eric Covener <co...@gmail.com>.
Should probably note somewhere that mod_logio won't log meaningful
results for h2 connections (based on how it tracks at connection
level)
On Mon, Jun 22, 2015 at 8:48 AM, Stefan Eissing
<st...@greenbytes.de> wrote:
> Eric, thanks for the help! When enabling mod_logio it became immediately clear that mod_h2 wrongly prevented some pre_connection hooks to run. mod_logio however expects its allocated module config to be there when a request gets cleaned up... So, with v0.7.2 all pre_conn hooks are run again and it is part of my test setup now.
>
> Which adds the issue about proper handling of module configurations in pseudo connections. There seem to be two approaches:
> a) treat pseudo connections like real ones -> run all connection hooks
> b) treat them as "shadows" of the real connection -> copy module configs
>
> While a) is the least dangerous, it misses gives a false impression about the properties of a connection. For example, mod_h2 currently copies over the mod_ssl config, so that SSL variables are available during request processing on pseudo connections. On the other hand, code is not really prepared for b) since this means that many threads may operate on the same module config.
>
> So, mod_h2 now follow a) for now (with the exception of mod_ssl). A future proposal for pseudo connections will need to reevaluate this.
>
> Cheers,
> Stefan
>
>> Am 22.06.2015 um 14:23 schrieb Eric Covener <co...@gmail.com>:
>>
>> On Mon, Jun 22, 2015 at 7:38 AM, Stefan Eissing
>> <st...@greenbytes.de> wrote:
>>> Thanks, now I see what you mean. What I do not understand:
>>> - why is this EOR processed too early?
>>
>> Usually it is at the end of a brigade and doesn't get cleaned up until
>> all of the data is written. But the copy and delete causes the cleanup
>> to run while you're iterating over the brigade to copy it in advance
>> of writing.
>>
>>> - what is causing the SegFault in the ap_run_log_transaction()
>>
>> I don't know. I would have guessed running it early would only impact
>> something later.
>>
>>> - and why am I seeing no errors on my system. Is this a configuration issue with logging?
>>
>> Looks like you figured this out -- must have mod_logio plus its %B or
>> whatever in your LogFormat.
>
> <green/>bytes GmbH
> Hafenweg 16, 48155 Münster, Germany
> Phone: +49 251 2807760. Amtsgericht Münster: HRB5782
>
>
>
--
Eric Covener
covener@gmail.com
Re: module configs across (pseudo) connections
Posted by William A Rowe Jr <wr...@rowe-clan.net>.
On Jun 24, 2015 8:39 AM, "Eric Covener" <co...@gmail.com> wrote:
>
> On Wed, Jun 24, 2015 at 9:26 AM, Graham Leggett <mi...@sharp.fm> wrote:
> > I believe we should be treating the “pseudo” connections as real
connections, and perhaps by linking a “subconnection” to a “connection”
(c->main) in the same way we currently link a subrequest to a request
(r->main).
>
> There are some basics for this in trunk from jim. I think if the slave
> connections had their own bucket_alloc it might remove some of the
> copying in h2.
This will be the best-case solution. Even 'connection-level' filters could
be applied before 'network transport' filters are invoked.
This would have simplified mod_ftp data channel connection mechanics as
well.
Re: module configs across (pseudo) connections
Posted by Stefan Eissing <st...@greenbytes.de>.
> Am 24.06.2015 um 16:14 schrieb Eric Covener <co...@gmail.com>:
>
> On Wed, Jun 24, 2015 at 10:07 AM, Stefan Eissing
> <st...@greenbytes.de> wrote:
>> Hmm, yes, well. It's the thought that counts... ;-)
>>
>> I think this will not be enough, though, if I understood the failures of my various attempts correctly. But it will certainly be good if more heads than one have a go at this.
>>
>> Let Tm :-= main thread, Tw := worker thread, TmB() the main connection bucket brigade, TwB() the worker (request) brigade.
>>
>> When a response from a worker starts:
>> TmB( , , , , , , , ) TwB(b1,b2,b3, , , , )
>> so we move buckets across threads
>> TmB(b1,b2, , , , , , ) TwB(b3, , , , , , )
>> and send b1 out and new response data arrives
>> TmB(b2, , , , , , , ) TwB(b3,b4, , , , , )
>> we have the destruction of b1 and the creation of b4 that go against the same bucket_alloc_t instance from two threads.
>>
>> Similar operations happen when b1 needs to be split or is a file bucket that gets read. So refraining from destroying buckets in the main thread is not enough.
>>
>> Have I missed something here?
>
>
> One thing I missed was that the httpd thread did the writing vs the h2
> thread. I thought the workers wrote but were serialized by the httpd
> thread.
I see. I was afraid of too many thread switches/waits/sync during main connection writes. For streams of same priority, h2 wants round-robin frames of streams to come out, preferably.
Ideally, I thought, for static file resources at least, the httpd thread would have all the file buckets in its buffers and read from them directly. With that goal in mind, I thought about moving things from h2 thread to httpd.
> --
> Eric Covener
> covener@gmail.com
<green/>bytes GmbH
Hafenweg 16, 48155 Münster, Germany
Phone: +49 251 2807760. Amtsgericht Münster: HRB5782
Re: module configs across (pseudo) connections
Posted by Eric Covener <co...@gmail.com>.
On Wed, Jun 24, 2015 at 10:07 AM, Stefan Eissing
<st...@greenbytes.de> wrote:
> Hmm, yes, well. It's the thought that counts... ;-)
>
> I think this will not be enough, though, if I understood the failures of my various attempts correctly. But it will certainly be good if more heads than one have a go at this.
>
> Let Tm :-= main thread, Tw := worker thread, TmB() the main connection bucket brigade, TwB() the worker (request) brigade.
>
> When a response from a worker starts:
> TmB( , , , , , , , ) TwB(b1,b2,b3, , , , )
> so we move buckets across threads
> TmB(b1,b2, , , , , , ) TwB(b3, , , , , , )
> and send b1 out and new response data arrives
> TmB(b2, , , , , , , ) TwB(b3,b4, , , , , )
> we have the destruction of b1 and the creation of b4 that go against the same bucket_alloc_t instance from two threads.
>
> Similar operations happen when b1 needs to be split or is a file bucket that gets read. So refraining from destroying buckets in the main thread is not enough.
>
> Have I missed something here?
One thing I missed was that the httpd thread did the writing vs the h2
thread. I thought the workers wrote but were serialized by the httpd
thread.
--
Eric Covener
covener@gmail.com
Re: module configs across (pseudo) connections
Posted by Graham Leggett <mi...@sharp.fm>.
On 24 Jun 2015, at 4:07 PM, Stefan Eissing <st...@greenbytes.de> wrote:
> Hmm, yes, well. It's the thought that counts... ;-)
>
> I think this will not be enough, though, if I understood the failures of my various attempts correctly. But it will certainly be good if more heads than one have a go at this.
>
> Let Tm :-= main thread, Tw := worker thread, TmB() the main connection bucket brigade, TwB() the worker (request) brigade.
>
> When a response from a worker starts:
> TmB( , , , , , , , ) TwB(b1,b2,b3, , , , )
> so we move buckets across threads
> TmB(b1,b2, , , , , , ) TwB(b3, , , , , , )
> and send b1 out and new response data arrives
> TmB(b2, , , , , , , ) TwB(b3,b4, , , , , )
> we have the destruction of b1 and the creation of b4 that go against the same bucket_alloc_t instance from two threads.
>
> Similar operations happen when b1 needs to be split or is a file bucket that gets read. So refraining from destroying buckets in the main thread is not enough.
>
> Have I missed something here?
In theory, a bucket should only be considered by one thread at a time.
A brigade containing buckets may be worked on by a different thread, but this should never happen concurrently with another thread.
Am I right in understanding that the problem comes about when there is a transition of a bucket from the “slave” connection to the “master” connection?
In theory, if the “slave” connection was terminated by a filter that collected buckets and set them aside in a brigade, and the “master” connection started with a filter that collected buckets from various brigades and then sent them down the rest of the filter chain we would solve any threading issues.
We would probably need some kind of mechanism for flow control, so that we could temporarily suspend a “slave” connection until the master is ready to receive more data from that connection, but this should be straightforward.
I was working on a patch last year that allowed any filter to set aside buckets and return early before all data was processed. This may help you to implement the “slave to master” filters:
http://mail-archives.apache.org/mod_mbox/httpd-dev/201409.mbox/%3C04776F50-0A53-4C1D-AA1C-9F57B60BC0AC@sharp.fm%3E
Regards,
Graham
—
Re: module configs across (pseudo) connections
Posted by Stefan Eissing <st...@greenbytes.de>.
> Am 24.06.2015 um 15:50 schrieb Jim Jagielski <ji...@jaguNET.com>:
>
>
>> On Jun 24, 2015, at 9:39 AM, Eric Covener <co...@gmail.com> wrote:
>>
>> On Wed, Jun 24, 2015 at 9:26 AM, Graham Leggett <mi...@sharp.fm> wrote:
>>> I believe we should be treating the “pseudo” connections as real connections, and perhaps by linking a “subconnection” to a “connection” (c->main) in the same way we currently link a subrequest to a request (r->main).
>>
>> There are some basics for this in trunk from jim. I think if the slave
>> connections had their own bucket_alloc it might remove some of the
>> copying in h2.
>
> +1... that's the thought at least ;)
Hmm, yes, well. It's the thought that counts... ;-)
I think this will not be enough, though, if I understood the failures of my various attempts correctly. But it will certainly be good if more heads than one have a go at this.
Let Tm :-= main thread, Tw := worker thread, TmB() the main connection bucket brigade, TwB() the worker (request) brigade.
When a response from a worker starts:
TmB( , , , , , , , ) TwB(b1,b2,b3, , , , )
so we move buckets across threads
TmB(b1,b2, , , , , , ) TwB(b3, , , , , , )
and send b1 out and new response data arrives
TmB(b2, , , , , , , ) TwB(b3,b4, , , , , )
we have the destruction of b1 and the creation of b4 that go against the same bucket_alloc_t instance from two threads.
Similar operations happen when b1 needs to be split or is a file bucket that gets read. So refraining from destroying buckets in the main thread is not enough.
Have I missed something here?
<green/>bytes GmbH
Hafenweg 16, 48155 Münster, Germany
Phone: +49 251 2807760. Amtsgericht Münster: HRB5782
Re: module configs across (pseudo) connections
Posted by Jim Jagielski <ji...@jaguNET.com>.
> On Jun 24, 2015, at 9:39 AM, Eric Covener <co...@gmail.com> wrote:
>
> On Wed, Jun 24, 2015 at 9:26 AM, Graham Leggett <mi...@sharp.fm> wrote:
>> I believe we should be treating the “pseudo” connections as real connections, and perhaps by linking a “subconnection” to a “connection” (c->main) in the same way we currently link a subrequest to a request (r->main).
>
> There are some basics for this in trunk from jim. I think if the slave
> connections had their own bucket_alloc it might remove some of the
> copying in h2.
+1... that's the thought at least ;)
Re: module configs across (pseudo) connections
Posted by Eric Covener <co...@gmail.com>.
On Wed, Jun 24, 2015 at 9:26 AM, Graham Leggett <mi...@sharp.fm> wrote:
> I believe we should be treating the “pseudo” connections as real connections, and perhaps by linking a “subconnection” to a “connection” (c->main) in the same way we currently link a subrequest to a request (r->main).
There are some basics for this in trunk from jim. I think if the slave
connections had their own bucket_alloc it might remove some of the
copying in h2.
--
Eric Covener
covener@gmail.com
Re: module configs across (pseudo) connections
Posted by Graham Leggett <mi...@sharp.fm>.
On 24 Jun 2015, at 3:58 PM, Stefan Eissing <st...@greenbytes.de> wrote:
> Totally agree. That is why it is not implemented like that. With the side effect that mod_logio, for example, does not aggregate data for the main connection.
>
> The only exception in the current implementation is mod_ssl. mod_h2 copies that one to the slave connection, so that ssl_var_lookup() works.
This may break if a change was made to mod_ssl and structure sizes changed.
If there was something like c->main, we could teach mod_ssl how to find the “root” connection so that ssl_var_lookup() “just works” without copying.
Regards,
Graham
—
Re: module configs across (pseudo) connections
Posted by Stefan Eissing <st...@greenbytes.de>.
> Am 24.06.2015 um 15:26 schrieb Graham Leggett <mi...@sharp.fm>:
>
> I would argue that the copying option above is most dangerous, as this has side effects that may not be catered for by pool cleanups.
Totally agree. That is why it is not implemented like that. With the side effect that mod_logio, for example, does not aggregate data for the main connection.
The only exception in the current implementation is mod_ssl. mod_h2 copies that one to the slave connection, so that ssl_var_lookup() works.
> I believe we should be treating the “pseudo” connections as real connections, and perhaps by linking a “subconnection” to a “connection” (c->main) in the same way we currently link a subrequest to a request (r->main).
Sounds good. Then mod_ssl/mod_logio can treat these as they seem fit.
> If you run into any blockers while doing this I would argue those blockers would be bugs and we want to fix them.
Cheers,
Stefan
<green/>bytes GmbH
Hafenweg 16, 48155 Münster, Germany
Phone: +49 251 2807760. Amtsgericht Münster: HRB5782
Re: module configs across (pseudo) connections
Posted by Graham Leggett <mi...@sharp.fm>.
On 22 Jun 2015, at 2:48 PM, Stefan Eissing <st...@greenbytes.de> wrote:
> Eric, thanks for the help! When enabling mod_logio it became immediately clear that mod_h2 wrongly prevented some pre_connection hooks to run. mod_logio however expects its allocated module config to be there when a request gets cleaned up... So, with v0.7.2 all pre_conn hooks are run again and it is part of my test setup now.
>
> Which adds the issue about proper handling of module configurations in pseudo connections. There seem to be two approaches:
> a) treat pseudo connections like real ones -> run all connection hooks
> b) treat them as "shadows" of the real connection -> copy module configs
>
> While a) is the least dangerous, it misses gives a false impression about the properties of a connection. For example, mod_h2 currently copies over the mod_ssl config, so that SSL variables are available during request processing on pseudo connections. On the other hand, code is not really prepared for b) since this means that many threads may operate on the same module config.
>
> So, mod_h2 now follow a) for now (with the exception of mod_ssl). A future proposal for pseudo connections will need to reevaluate this.
I would argue that the copying option above is most dangerous, as this has side effects that may not be catered for by pool cleanups.
I believe we should be treating the “pseudo” connections as real connections, and perhaps by linking a “subconnection” to a “connection” (c->main) in the same way we currently link a subrequest to a request (r->main).
If you run into any blockers while doing this I would argue those blockers would be bugs and we want to fix them.
Regards,
Graham
—
module configs across (pseudo) connections
Posted by Stefan Eissing <st...@greenbytes.de>.
Eric, thanks for the help! When enabling mod_logio it became immediately clear that mod_h2 wrongly prevented some pre_connection hooks to run. mod_logio however expects its allocated module config to be there when a request gets cleaned up... So, with v0.7.2 all pre_conn hooks are run again and it is part of my test setup now.
Which adds the issue about proper handling of module configurations in pseudo connections. There seem to be two approaches:
a) treat pseudo connections like real ones -> run all connection hooks
b) treat them as "shadows" of the real connection -> copy module configs
While a) is the least dangerous, it misses gives a false impression about the properties of a connection. For example, mod_h2 currently copies over the mod_ssl config, so that SSL variables are available during request processing on pseudo connections. On the other hand, code is not really prepared for b) since this means that many threads may operate on the same module config.
So, mod_h2 now follow a) for now (with the exception of mod_ssl). A future proposal for pseudo connections will need to reevaluate this.
Cheers,
Stefan
> Am 22.06.2015 um 14:23 schrieb Eric Covener <co...@gmail.com>:
>
> On Mon, Jun 22, 2015 at 7:38 AM, Stefan Eissing
> <st...@greenbytes.de> wrote:
>> Thanks, now I see what you mean. What I do not understand:
>> - why is this EOR processed too early?
>
> Usually it is at the end of a brigade and doesn't get cleaned up until
> all of the data is written. But the copy and delete causes the cleanup
> to run while you're iterating over the brigade to copy it in advance
> of writing.
>
>> - what is causing the SegFault in the ap_run_log_transaction()
>
> I don't know. I would have guessed running it early would only impact
> something later.
>
>> - and why am I seeing no errors on my system. Is this a configuration issue with logging?
>
> Looks like you figured this out -- must have mod_logio plus its %B or
> whatever in your LogFormat.
<green/>bytes GmbH
Hafenweg 16, 48155 Münster, Germany
Phone: +49 251 2807760. Amtsgericht Münster: HRB5782
Re: buckets across threads - question
Posted by Eric Covener <co...@gmail.com>.
On Mon, Jun 22, 2015 at 4:07 AM, Stefan Eissing
<st...@greenbytes.de> wrote:
> Sort answer: no.
> Re issue 28: I will look into this more today. I was able to generate a segfault on connection shutdown on Friday and work on a fix.
>
> Longer version re buckets:
> - even though buckets for a request are created inside a single worker thread, they are mutated/split/placed on free list upon processing down the main connection. Which then all affects their bucket_alloc_t from several threads. Especially on longer response bodies,
> - since there are no function pointers for bucket_alloc_t, it is not obvious to me how one would replace them with another implementation.
> - what can be added are, of course, new bucket implementations. Maybe there is an approach to move the data itself without copying more than once. With a clever handling of file buckets, maybe no copying at all.
>
> Otherwise, I am open to ideas.
I am afraid the change will not fix #28. In the backtrace for #28 an
EOR bucket is being deleted while the buckets in a brigade are being
copied to a new pool. That triggers all of the end of request
processing way too early.
I was naively thinking as a POC it would be possible to pmemdup the
conn_rec and create a new bucket allocator (when you create the faux
request_rec) If it worked, we'd want to find some way to get the same
result in the core. If we continue to share one at the real
connection level, I think it needs an option of a mutex under h2.
--
Eric Covener
covener@gmail.com
Re: buckets across threads - question
Posted by Stefan Eissing <st...@greenbytes.de>.
Sort answer: no.
Re issue 28: I will look into this more today. I was able to generate a segfault on connection shutdown on Friday and work on a fix.
Longer version re buckets:
- even though buckets for a request are created inside a single worker thread, they are mutated/split/placed on free list upon processing down the main connection. Which then all affects their bucket_alloc_t from several threads. Especially on longer response bodies,
- since there are no function pointers for bucket_alloc_t, it is not obvious to me how one would replace them with another implementation.
- what can be added are, of course, new bucket implementations. Maybe there is an approach to move the data itself without copying more than once. With a clever handling of file buckets, maybe no copying at all.
Otherwise, I am open to ideas.
//Stefan
> Am 21.06.2015 um 17:07 schrieb Eric Covener <co...@gmail.com>:
>
> re: https://github.com/icing/mod_h2/issues/28 has this design further evolved?
>
> Is there a way to make the deep copy un-necessary by injecting your
> own bucket_alloc created with a pool with a threadsafe allocator very
> early? It seems like all of the buckets are created in the h2 thread
> after you've had a chance to manipulate stuff.
>
>
>
> On Tue, Mar 31, 2015 at 2:57 PM, Stefan Eissing
> <st...@greenbytes.de> wrote:
>> Thanks, Jim!
>>
>>
>>
>>> Am 31.03.2015 um 19:29 schrieb Jim Jagielski <ji...@jaguNET.com>:
>>>
>>> What I did is used the alpn patch as a guide and updated trunk
>>> to add the functionality:
>>>
>>> http://svn.apache.org/r1670397
>>>
>>> I'll give it a few days to work out and then propose for a
>>> 2.4 backport.
>>>
>>>> On Mar 31, 2015, at 1:13 PM, Stefan Eissing <st...@greenbytes.de> wrote:
>>>>
>>>> I think the old NPN patch is in trunk. The ALPN not AFAIK. I undertstand there is a trunk first policy, but it'd be good to get it at least going there... thanks.
>>>>
>>>> Stefan
>>>>
>>>>
>>>>
>>>> Am 31.03.2015 um 18:22 schrieb Jim Jagielski <ji...@jaguNET.com>:
>>>>
>>>>>>
>>>>>> PS. As a small quid-pro-quo and because you were asking about a possibly 2.4.13: there has so far not been a volunteer to integrate the ALPN patch I adapted from mod_spdy. It would be great if someone could take a look at it. It is the biggest obstacle so far for experimenters (early sufferers) of mod_h2 and it would be very nice to have it out of the way.
>>>>>
>>>>> I'll take a look. It's not currently in trunk, is that right?
>>>
>
>
>
> --
> Eric Covener
> covener@gmail.com
<green/>bytes GmbH
Hafenweg 16, 48155 Münster, Germany
Phone: +49 251 2807760. Amtsgericht Münster: HRB5782