You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@trafficserver.apache.org by Susan Hinrichs <sh...@network-geographics.com> on 2015/07/24 03:16:01 UTC
Another cross thread race condition and thoughts on server VC migration
Hello,
Another latent cross-thread race condition has become very active in our
environment (TS-3797). Given that we just spent time within the last
month squashing another cross thread race condition (TS-3486) that was
active in several environments, Alan and I would like to step back and
try to reduce the cross thread impact of the global session pools.
I wrote up our thoughts and plan for implementation. Given that
threading and race conditions are always tricky, I'd appreciate more
eyes looking for flaws in our approach or suggestions for alternatives.
https://cwiki.apache.org/confluence/display/TS/Threading+Issues+And+NetVC+Migration
Thanks,
Susan
Re: Another cross thread race condition and thoughts on server VC
migration
Posted by Leif Hedstrom <zw...@apache.org>.
> On Jul 24, 2015, at 9:17 AM, Leif Hedstrom <zw...@apache.org> wrote:
>
>>
>> On Jul 24, 2015, at 9:11 AM, Leif Hedstrom <zw...@apache.org> wrote:
>>
>>>
>>> On Jul 24, 2015, at 3:16 AM, Susan Hinrichs <sh...@network-geographics.com> wrote:
>>>
>>> Hello,
>>>
>>> Another latent cross-thread race condition has become very active in our environment (TS-3797). Given that we just spent time within the last month squashing another cross thread race condition (TS-3486) that was active in several environments, Alan and I would like to step back and try to reduce the cross thread impact of the global session pools.
>>>
>>> I wrote up our thoughts and plan for implementation. Given that threading and race conditions are always tricky, I'd appreciate more eyes looking for flaws in our approach or suggestions for alternatives.
>>>
>>> https://cwiki.apache.org/confluence/display/TS/Threading+Issues+And+NetVC+Migration
>>
>>
>>
>> My gut reaction to this is that this makes our efforts for NUMA / thread affinity very, very difficult to achieve. The goal is to avoid memory migrating cross NUMA sockets, to avoid QPI traffic. This would encourage the opposite unless I misread it ? It also obviously violates the original design goals, where VCs do *not* migrate.
>
>
> Also, William Bardwell made an attempt to do these VC migrations long ago, and it did not work well. That was in fact the reason why the per-thread session pools where implemented.
>
> See the patches / discussions on
>
> https://issues.apache.org/jira/browse/TS-880
And
https://issues.apache.org/jira/browse/TS-924
— Leif
Re: Another cross thread race condition and thoughts on server VC
migration
Posted by Leif Hedstrom <zw...@apache.org>.
> On Jul 24, 2015, at 9:11 AM, Leif Hedstrom <zw...@apache.org> wrote:
>
>>
>> On Jul 24, 2015, at 3:16 AM, Susan Hinrichs <sh...@network-geographics.com> wrote:
>>
>> Hello,
>>
>> Another latent cross-thread race condition has become very active in our environment (TS-3797). Given that we just spent time within the last month squashing another cross thread race condition (TS-3486) that was active in several environments, Alan and I would like to step back and try to reduce the cross thread impact of the global session pools.
>>
>> I wrote up our thoughts and plan for implementation. Given that threading and race conditions are always tricky, I'd appreciate more eyes looking for flaws in our approach or suggestions for alternatives.
>>
>> https://cwiki.apache.org/confluence/display/TS/Threading+Issues+And+NetVC+Migration
>
>
>
> My gut reaction to this is that this makes our efforts for NUMA / thread affinity very, very difficult to achieve. The goal is to avoid memory migrating cross NUMA sockets, to avoid QPI traffic. This would encourage the opposite unless I misread it ? It also obviously violates the original design goals, where VCs do *not* migrate.
Also, William Bardwell made an attempt to do these VC migrations long ago, and it did not work well. That was in fact the reason why the per-thread session pools where implemented.
See the patches / discussions on
https://issues.apache.org/jira/browse/TS-880
The Subject of that Jira is somewhat confusing, but it’s the same issue: A KA connection to origin is used by client VCs on different threads that the origin connections.
— Leif
Re: Another cross thread race condition and thoughts on server VC
migration
Posted by Alan Carroll <so...@yahoo-inc.com.INVALID>.
It's unclear to me that you wouldn't get better NUMA affinity with the migration. The logic spends much more time messing with the IOBuffers than with the socket buffers and if the socket is migrated the IOBuffers will be thread local whereas currently they are not. That is, IOBuffers for server data are in a different thread than the HttpSM so actions such as servicing READ_READY events from the user agent are crossing thread boundaries.
On Friday, July 24, 2015 8:27 AM, Susan Hinrichs <sh...@network-geographics.com> wrote:
On 7/24/2015 2:11 AM, Leif Hedstrom wrote:
>> On Jul 24, 2015, at 3:16 AM, Susan Hinrichs <sh...@network-geographics.com> wrote:
>>
>> Hello,
>>
>> Another latent cross-thread race condition has become very active in our environment (TS-3797). Given that we just spent time within the last month squashing another cross thread race condition (TS-3486) that was active in several environments, Alan and I would like to step back and try to reduce the cross thread impact of the global session pools.
>>
>> I wrote up our thoughts and plan for implementation. Given that threading and race conditions are always tricky, I'd appreciate more eyes looking for flaws in our approach or suggestions for alternatives.
>>
>> https://cwiki.apache.org/confluence/display/TS/Threading+Issues+And+NetVC+Migration
>
>
> My gut reaction to this is that this makes our efforts for NUMA / thread affinity very, very difficult to achieve. The goal is to avoid memory migrating cross NUMA sockets, to avoid QPI traffic. This would encourage the opposite unless I misread it ? It also obviously violates the original design goals, where VCs do *not* migrate.
>
> It’d be very interesting to hear from John Plevyak and what their initial design had considered for these issues?
>
> Cheers,
>
> — Leif
>
Leif,
Thanks for the pointers to the historical precedence. I'll look over
them this morning.
I will think more into NUMA issues as well. I had been focusing on
reducing thread cross-talk and associated possibility for errors. I
thought this solution was all good since a new net VC is created in the
target thread and the socket and SSL object are copied over. But upon
reflection, I realize that there are buffers associated with the socket
and the SSL object.
Susan
Re: Another cross thread race condition and thoughts on server VC migration
Posted by Leif Hedstrom <zw...@apache.org>.
> On Jul 29, 2015, at 4:39 PM, Alan Carroll <so...@yahoo-inc.com.INVALID> wrote:
>
> We're testing this in production now, although cautiously. I think it important to note the VC migration is conceptual, not actual. As noted in the original proposal on the wiki the important bits are removed from the old VC and put in the new VC, the latter then being thread local.
Cool. amc explained to me that we’re actually not migrating VC (allocated data), but releasing the original VC, allocating a new one, and transferring the origin server socket. That makes a lot more sense, and falls nicely within the concept of NUMA awareness.
Thanks,
— leif
Re: Another cross thread race condition and thoughts on server VC
migration
Posted by Alan Carroll <so...@yahoo-inc.com.INVALID>.
We're testing this in production now, although cautiously. I think it important to note the VC migration is conceptual, not actual. As noted in the original proposal on the wiki the important bits are removed from the old VC and put in the new VC, the latter then being thread local.
On Friday, July 24, 2015 9:31 AM, Sudheer Vinukonda <su...@yahoo-inc.com.INVALID> wrote:
I agree with Leif as well - the "cost" in constantly reshuffling VC's across threads may outweigh the extra latency in using per-thread pool. If we do not want cross-thread communication, it seems like using per-thread pool is a more cleaner solution. Would it makes sense to spend time to investigate why the per-thread pool makes things worse (in terms of latency), perhaps? Other users of per-thread pool (Brian Geffon) claimed significant gains with that, so, may be, it's probably important/useful to actually understand why the behavior is not similar?
Also, without changing the design too much, I wonder if we could actually, enhance the global pool slightly, by adding the thread_id to the key and basically try to "pick" on the same net thread as the client VC first. If there's no match found on the same thread, then simply fall back to the current design of picking from the rest of the threads. My initial thoughts are that there might more "misses" for the client VC's thread initially, but, after a few hours (min?) of reaching a steady state, there's a good chance of reducing the cross-thread "picks". Unless I'm missing something and there are serious flaws in this hybrid approach, I can even try a POC patch for this and come back with results.
Thanks,
Sudheer
On Friday, July 24, 2015 6:27 AM, Susan Hinrichs <sh...@network-geographics.com> wrote:
On 7/24/2015 2:11 AM, Leif Hedstrom wrote:
>> On Jul 24, 2015, at 3:16 AM, Susan Hinrichs <sh...@network-geographics.com> wrote:
>>
>> Hello,
>>
>> Another latent cross-thread race condition has become very active in our environment (TS-3797). Given that we just spent time within the last month squashing another cross thread race condition (TS-3486) that was active in several environments, Alan and I would like to step back and try to reduce the cross thread impact of the global session pools.
>>
>> I wrote up our thoughts and plan for implementation. Given that threading and race conditions are always tricky, I'd appreciate more eyes looking for flaws in our approach or suggestions for alternatives.
>>
>> https://cwiki.apache.org/confluence/display/TS/Threading+Issues+And+NetVC+Migration
>
>
> My gut reaction to this is that this makes our efforts for NUMA / thread affinity very, very difficult to achieve. The goal is to avoid memory migrating cross NUMA sockets, to avoid QPI traffic. This would encourage the opposite unless I misread it ? It also obviously violates the original design goals, where VCs do *not* migrate.
>
> It’d be very interesting to hear from John Plevyak and what their initial design had considered for these issues?
>
> Cheers,
>
> — Leif
>
Leif,
Thanks for the pointers to the historical precedence. I'll look over
them this morning.
I will think more into NUMA issues as well. I had been focusing on
reducing thread cross-talk and associated possibility for errors. I
thought this solution was all good since a new net VC is created in the
target thread and the socket and SSL object are copied over. But upon
reflection, I realize that there are buffers associated with the socket
and the SSL object.
Susan
Re: Another cross thread race condition and thoughts on server VC
migration
Posted by Sudheer Vinukonda <su...@yahoo-inc.com.INVALID>.
I agree with Leif as well - the "cost" in constantly reshuffling VC's across threads may outweigh the extra latency in using per-thread pool. If we do not want cross-thread communication, it seems like using per-thread pool is a more cleaner solution. Would it makes sense to spend time to investigate why the per-thread pool makes things worse (in terms of latency), perhaps? Other users of per-thread pool (Brian Geffon) claimed significant gains with that, so, may be, it's probably important/useful to actually understand why the behavior is not similar?
Also, without changing the design too much, I wonder if we could actually, enhance the global pool slightly, by adding the thread_id to the key and basically try to "pick" on the same net thread as the client VC first. If there's no match found on the same thread, then simply fall back to the current design of picking from the rest of the threads. My initial thoughts are that there might more "misses" for the client VC's thread initially, but, after a few hours (min?) of reaching a steady state, there's a good chance of reducing the cross-thread "picks". Unless I'm missing something and there are serious flaws in this hybrid approach, I can even try a POC patch for this and come back with results.
Thanks,
Sudheer
On Friday, July 24, 2015 6:27 AM, Susan Hinrichs <sh...@network-geographics.com> wrote:
On 7/24/2015 2:11 AM, Leif Hedstrom wrote:
>> On Jul 24, 2015, at 3:16 AM, Susan Hinrichs <sh...@network-geographics.com> wrote:
>>
>> Hello,
>>
>> Another latent cross-thread race condition has become very active in our environment (TS-3797). Given that we just spent time within the last month squashing another cross thread race condition (TS-3486) that was active in several environments, Alan and I would like to step back and try to reduce the cross thread impact of the global session pools.
>>
>> I wrote up our thoughts and plan for implementation. Given that threading and race conditions are always tricky, I'd appreciate more eyes looking for flaws in our approach or suggestions for alternatives.
>>
>> https://cwiki.apache.org/confluence/display/TS/Threading+Issues+And+NetVC+Migration
>
>
> My gut reaction to this is that this makes our efforts for NUMA / thread affinity very, very difficult to achieve. The goal is to avoid memory migrating cross NUMA sockets, to avoid QPI traffic. This would encourage the opposite unless I misread it ? It also obviously violates the original design goals, where VCs do *not* migrate.
>
> It’d be very interesting to hear from John Plevyak and what their initial design had considered for these issues?
>
> Cheers,
>
> — Leif
>
Leif,
Thanks for the pointers to the historical precedence. I'll look over
them this morning.
I will think more into NUMA issues as well. I had been focusing on
reducing thread cross-talk and associated possibility for errors. I
thought this solution was all good since a new net VC is created in the
target thread and the socket and SSL object are copied over. But upon
reflection, I realize that there are buffers associated with the socket
and the SSL object.
Susan
Re: Another cross thread race condition and thoughts on server VC
migration
Posted by Susan Hinrichs <sh...@network-geographics.com>.
On 7/24/2015 2:11 AM, Leif Hedstrom wrote:
>> On Jul 24, 2015, at 3:16 AM, Susan Hinrichs <sh...@network-geographics.com> wrote:
>>
>> Hello,
>>
>> Another latent cross-thread race condition has become very active in our environment (TS-3797). Given that we just spent time within the last month squashing another cross thread race condition (TS-3486) that was active in several environments, Alan and I would like to step back and try to reduce the cross thread impact of the global session pools.
>>
>> I wrote up our thoughts and plan for implementation. Given that threading and race conditions are always tricky, I'd appreciate more eyes looking for flaws in our approach or suggestions for alternatives.
>>
>> https://cwiki.apache.org/confluence/display/TS/Threading+Issues+And+NetVC+Migration
>
>
> My gut reaction to this is that this makes our efforts for NUMA / thread affinity very, very difficult to achieve. The goal is to avoid memory migrating cross NUMA sockets, to avoid QPI traffic. This would encourage the opposite unless I misread it ? It also obviously violates the original design goals, where VCs do *not* migrate.
>
> It’d be very interesting to hear from John Plevyak and what their initial design had considered for these issues?
>
> Cheers,
>
> — Leif
>
Leif,
Thanks for the pointers to the historical precedence. I'll look over
them this morning.
I will think more into NUMA issues as well. I had been focusing on
reducing thread cross-talk and associated possibility for errors. I
thought this solution was all good since a new net VC is created in the
target thread and the socket and SSL object are copied over. But upon
reflection, I realize that there are buffers associated with the socket
and the SSL object.
Susan
Re: Another cross thread race condition and thoughts on server VC
migration
Posted by Leif Hedstrom <zw...@apache.org>.
> On Jul 24, 2015, at 3:16 AM, Susan Hinrichs <sh...@network-geographics.com> wrote:
>
> Hello,
>
> Another latent cross-thread race condition has become very active in our environment (TS-3797). Given that we just spent time within the last month squashing another cross thread race condition (TS-3486) that was active in several environments, Alan and I would like to step back and try to reduce the cross thread impact of the global session pools.
>
> I wrote up our thoughts and plan for implementation. Given that threading and race conditions are always tricky, I'd appreciate more eyes looking for flaws in our approach or suggestions for alternatives.
>
> https://cwiki.apache.org/confluence/display/TS/Threading+Issues+And+NetVC+Migration
My gut reaction to this is that this makes our efforts for NUMA / thread affinity very, very difficult to achieve. The goal is to avoid memory migrating cross NUMA sockets, to avoid QPI traffic. This would encourage the opposite unless I misread it ? It also obviously violates the original design goals, where VCs do *not* migrate.
It’d be very interesting to hear from John Plevyak and what their initial design had considered for these issues?
Cheers,
— Leif