You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@trafficserver.apache.org by zzz <zz...@apache.org> on 2018/08/15 00:58:52 UTC
[New API Proposal] TSContScheduleToCurrentThread and TSContScheduleToCurrentThreadEvery
Hi, team,
The background here is we'd like to clean up a thread local cache after
some timeout. Unfortunately, we're lacking an API to schedule a
continuation to the current thread.
There are two ways I can think of to achieve this.
One is a pair of new APIs *TSContScheduleToCurrentThread(TSCont contp,
ink_hrtime timeout) *and* TSContScheduleToCurrentThreadEvery(TSCont contp,
ink_hrtime every).*
Or we can append the TSThreadPool enum with a new entry called
*TS_THREAD_CURRENT
*and add the logic into *TSContSchedule() *and *TSContScheduleEvery(). *
The issue with the latter solution is *TS_THREAD_CURRENT* doesn't quite fit
in TSThreadPool. If we change TSThreadPool to some more proper name, the
change will not be backward compatible.
Please feel free to share your insights. Any ideas would be much
appreciated!
Thanks,
Zizhong
Re: [New API Proposal] TSContScheduleToCurrentThread and TSContScheduleToCurrentThreadEvery
Posted by Otto van der Schaaf <os...@gmail.com>.
A TSEventThread API makes sense to me; maybe Kees has something similar in
the
implementation for XXXOnThread() he mentioned.
Regardless, it sounds like this would allow fine-grained control of where
code associated to a
continuation is executed, which would be great as far as I am concerned.
Otto
On Thu, Aug 16, 2018 at 12:19 AM Alan Carroll
<so...@oath.com.invalid> wrote:
> In the core, when a continuation is scheduled on a specific thread that is
> not the current thread, it's because there is a locked object with the lock
> held on another thread, and the goal is to have the continuation run on
> that thread. This can help gather interacting objects on the same thread
> where they won't have lock contention (because if I try to get a mutex
> that's locked on *this* thread, it is successful).
>
Re: [New API Proposal] TSContScheduleToCurrentThread and TSContScheduleToCurrentThreadEvery
Posted by Fei Deng <fe...@oath.com.INVALID>.
This is the current state of the APIs. I have this tracked on my private
branch on github, please provide feedback for it. Thanks!
https://github.com/duke8253/trafficserver/pull/5
On Tue, Aug 21, 2018 at 3:36 PM zzz <zz...@apache.org> wrote:
> Thanks, Fei Deng, glad to hear the good news! Looking forward to the new
> PR.
>
> 2018-08-21 7:50 GMT-07:00 Alan Carroll <so...@oath.com.invalid>:
>
> > I wait, with bright eyed anticipation, for the documentation for this!
> >
> > On Tue, Aug 21, 2018 at 9:47 AM Fei Deng <fe...@oath.com.invalid> wrote:
> >
> > > I'm looking into this right now. Already have the TSContScheduleOn
> > > and TSContScheduleOnEvery APIs done. Still working on getting the
> thread
> > > affinity.
> > >
> > > On Thu, Aug 16, 2018 at 12:59 PM Alan Carroll
> > > <so...@oath.com.invalid> wrote:
> > >
> > > > As a quick response, it sounds like most of your issues would be
> > > addressed
> > > > by thread affinity for continuations. I'll see if I can get some
> write
> > up
> > > > done for the summit. There are other issues beyond yours that I think
> > > would
> > > > also benefit from that.
> > > >
> > > > On Thu, Aug 16, 2018 at 12:56 PM Kees Spoelstra <
> kspoelstra@we-amp.com
> > >
> > > > wrote:
> > > >
> > > > > On Thu, Aug 16, 2018, 20:39 Alan Carroll <solidwallofcode@oath.com
> > > > > .invalid>
> > > > > wrote:
> > > > >
> > > > > > "getting the TSEventThread from a vconn, txn, ssn instead of
> > > > implicitly
> > > > > > trusting that the current thread is relevant?"
> > > > > >
> > > > > > I'm not sure what this means. Those objects don't have threads
> > > > explicitly
> > > > > > associated, it's more implicit. If the code is in a callback on a
> > > hook
> > > > > for
> > > > > > the object, then the current thread is relevant, effectively by
> > > > > definition.
> > > > > >
> > > > >
> > > > > When coordinating between txns and reenabling or accessing object a
> > > from
> > > > > object b's hook you might not be so lucky.A hook for A might be
> > called
> > > on
> > > > > an unexpected thread and fun stuff will happen. Often you will be
> > lucky
> > > > > because core has a lot of explicit alignment checks and reenabling
> A
> > > will
> > > > > be done on it its original thread.
> > > > >
> > > > >
> > > > > But as soon as I read this I realized you were going to bring up
> the
> > > > > > continuation thread affinity issue.
> > > > > >
> > > > > > "moving a client vconn to a certain thread" - that would be
> > > > challenging.
> > > > > > I'd rather move the plugins to the NetVC thread. This is one
> aspect
> > > of
> > > > > > getting the mutex locking thread - that's the thread to which to
> > > move.
> > > > > >
> > > > >
> > > > > I meant outgoing netvc connections, my bad, but we have the
> > technology
> > > to
> > > > > make it better, :), in the global http server connectionpool we do
> > this
> > > > > with the netvcs to align with the txn/sm threads. I don't know what
> > has
> > > > an
> > > > > higher impact thread jumping or moving the netvc.
> > > > > Moving netvcs to other threads might have some impact on data in
> > > flight,
> > > > I
> > > > > think Susan knows about the limitations of this.
> > > > >
> > > > > Wouldn't it be nice if we had connection pool constructs which we
> > could
> > > > > reuse :) instead of reinventing.
> > > > > Somehow when pulling threads the whole world starts to unravel.
> > > > >
> > > > >
> > > > > > "check the callback continuations in the current API?" - check
> them
> > > for
> > > > > > what? I don't understand this question.
> > > > > >
> > > > >
> > > > >
> > > > > The callback cont for some async tsapi calls might not be called
> on
> > > the
> > > > > same thread. It would be nice to know the guaranteed behavior of
> > calls
> > > > with
> > > > > callback continuations without checking core code (for example DNS)
> > > > >
> > > >
> > > >
> > > > --
> > > > *Beware the fisherman who's casting out his line in to a dried up
> > > > riverbed.*
> > > > *Oh don't try to tell him 'cause he won't believe. Throw some bread
> to
> > > the
> > > > ducks instead.*
> > > > *It's easier that way. *- Genesis : Duke : VI 25-28
> > > >
> > >
> > >
> > > --
> > > *Fei Deng*
> > >
> > > E: feid@oath.com
> > >
> > > 1908 S First Street
> > > Champaign, IL, 61820
> > >
> >
> >
> > --
> > *Beware the fisherman who's casting out his line in to a dried up
> > riverbed.*
> > *Oh don't try to tell him 'cause he won't believe. Throw some bread to
> the
> > ducks instead.*
> > *It's easier that way. *- Genesis : Duke : VI 25-28
> >
>
--
*Fei Deng*
E: feid@oath.com
1908 S First Street
Champaign, IL, 61820
Re: [New API Proposal] TSContScheduleToCurrentThread and TSContScheduleToCurrentThreadEvery
Posted by zzz <zz...@apache.org>.
Thanks, Fei Deng, glad to hear the good news! Looking forward to the new PR.
2018-08-21 7:50 GMT-07:00 Alan Carroll <so...@oath.com.invalid>:
> I wait, with bright eyed anticipation, for the documentation for this!
>
> On Tue, Aug 21, 2018 at 9:47 AM Fei Deng <fe...@oath.com.invalid> wrote:
>
> > I'm looking into this right now. Already have the TSContScheduleOn
> > and TSContScheduleOnEvery APIs done. Still working on getting the thread
> > affinity.
> >
> > On Thu, Aug 16, 2018 at 12:59 PM Alan Carroll
> > <so...@oath.com.invalid> wrote:
> >
> > > As a quick response, it sounds like most of your issues would be
> > addressed
> > > by thread affinity for continuations. I'll see if I can get some write
> up
> > > done for the summit. There are other issues beyond yours that I think
> > would
> > > also benefit from that.
> > >
> > > On Thu, Aug 16, 2018 at 12:56 PM Kees Spoelstra <kspoelstra@we-amp.com
> >
> > > wrote:
> > >
> > > > On Thu, Aug 16, 2018, 20:39 Alan Carroll <solidwallofcode@oath.com
> > > > .invalid>
> > > > wrote:
> > > >
> > > > > "getting the TSEventThread from a vconn, txn, ssn instead of
> > > implicitly
> > > > > trusting that the current thread is relevant?"
> > > > >
> > > > > I'm not sure what this means. Those objects don't have threads
> > > explicitly
> > > > > associated, it's more implicit. If the code is in a callback on a
> > hook
> > > > for
> > > > > the object, then the current thread is relevant, effectively by
> > > > definition.
> > > > >
> > > >
> > > > When coordinating between txns and reenabling or accessing object a
> > from
> > > > object b's hook you might not be so lucky.A hook for A might be
> called
> > on
> > > > an unexpected thread and fun stuff will happen. Often you will be
> lucky
> > > > because core has a lot of explicit alignment checks and reenabling A
> > will
> > > > be done on it its original thread.
> > > >
> > > >
> > > > But as soon as I read this I realized you were going to bring up the
> > > > > continuation thread affinity issue.
> > > > >
> > > > > "moving a client vconn to a certain thread" - that would be
> > > challenging.
> > > > > I'd rather move the plugins to the NetVC thread. This is one aspect
> > of
> > > > > getting the mutex locking thread - that's the thread to which to
> > move.
> > > > >
> > > >
> > > > I meant outgoing netvc connections, my bad, but we have the
> technology
> > to
> > > > make it better, :), in the global http server connectionpool we do
> this
> > > > with the netvcs to align with the txn/sm threads. I don't know what
> has
> > > an
> > > > higher impact thread jumping or moving the netvc.
> > > > Moving netvcs to other threads might have some impact on data in
> > flight,
> > > I
> > > > think Susan knows about the limitations of this.
> > > >
> > > > Wouldn't it be nice if we had connection pool constructs which we
> could
> > > > reuse :) instead of reinventing.
> > > > Somehow when pulling threads the whole world starts to unravel.
> > > >
> > > >
> > > > > "check the callback continuations in the current API?" - check them
> > for
> > > > > what? I don't understand this question.
> > > > >
> > > >
> > > >
> > > > The callback cont for some async tsapi calls might not be called on
> > the
> > > > same thread. It would be nice to know the guaranteed behavior of
> calls
> > > with
> > > > callback continuations without checking core code (for example DNS)
> > > >
> > >
> > >
> > > --
> > > *Beware the fisherman who's casting out his line in to a dried up
> > > riverbed.*
> > > *Oh don't try to tell him 'cause he won't believe. Throw some bread to
> > the
> > > ducks instead.*
> > > *It's easier that way. *- Genesis : Duke : VI 25-28
> > >
> >
> >
> > --
> > *Fei Deng*
> >
> > E: feid@oath.com
> >
> > 1908 S First Street
> > Champaign, IL, 61820
> >
>
>
> --
> *Beware the fisherman who's casting out his line in to a dried up
> riverbed.*
> *Oh don't try to tell him 'cause he won't believe. Throw some bread to the
> ducks instead.*
> *It's easier that way. *- Genesis : Duke : VI 25-28
>
Re: [New API Proposal] TSContScheduleToCurrentThread and TSContScheduleToCurrentThreadEvery
Posted by Alan Carroll <so...@oath.com.INVALID>.
I wait, with bright eyed anticipation, for the documentation for this!
On Tue, Aug 21, 2018 at 9:47 AM Fei Deng <fe...@oath.com.invalid> wrote:
> I'm looking into this right now. Already have the TSContScheduleOn
> and TSContScheduleOnEvery APIs done. Still working on getting the thread
> affinity.
>
> On Thu, Aug 16, 2018 at 12:59 PM Alan Carroll
> <so...@oath.com.invalid> wrote:
>
> > As a quick response, it sounds like most of your issues would be
> addressed
> > by thread affinity for continuations. I'll see if I can get some write up
> > done for the summit. There are other issues beyond yours that I think
> would
> > also benefit from that.
> >
> > On Thu, Aug 16, 2018 at 12:56 PM Kees Spoelstra <ks...@we-amp.com>
> > wrote:
> >
> > > On Thu, Aug 16, 2018, 20:39 Alan Carroll <solidwallofcode@oath.com
> > > .invalid>
> > > wrote:
> > >
> > > > "getting the TSEventThread from a vconn, txn, ssn instead of
> > implicitly
> > > > trusting that the current thread is relevant?"
> > > >
> > > > I'm not sure what this means. Those objects don't have threads
> > explicitly
> > > > associated, it's more implicit. If the code is in a callback on a
> hook
> > > for
> > > > the object, then the current thread is relevant, effectively by
> > > definition.
> > > >
> > >
> > > When coordinating between txns and reenabling or accessing object a
> from
> > > object b's hook you might not be so lucky.A hook for A might be called
> on
> > > an unexpected thread and fun stuff will happen. Often you will be lucky
> > > because core has a lot of explicit alignment checks and reenabling A
> will
> > > be done on it its original thread.
> > >
> > >
> > > But as soon as I read this I realized you were going to bring up the
> > > > continuation thread affinity issue.
> > > >
> > > > "moving a client vconn to a certain thread" - that would be
> > challenging.
> > > > I'd rather move the plugins to the NetVC thread. This is one aspect
> of
> > > > getting the mutex locking thread - that's the thread to which to
> move.
> > > >
> > >
> > > I meant outgoing netvc connections, my bad, but we have the technology
> to
> > > make it better, :), in the global http server connectionpool we do this
> > > with the netvcs to align with the txn/sm threads. I don't know what has
> > an
> > > higher impact thread jumping or moving the netvc.
> > > Moving netvcs to other threads might have some impact on data in
> flight,
> > I
> > > think Susan knows about the limitations of this.
> > >
> > > Wouldn't it be nice if we had connection pool constructs which we could
> > > reuse :) instead of reinventing.
> > > Somehow when pulling threads the whole world starts to unravel.
> > >
> > >
> > > > "check the callback continuations in the current API?" - check them
> for
> > > > what? I don't understand this question.
> > > >
> > >
> > >
> > > The callback cont for some async tsapi calls might not be called on
> the
> > > same thread. It would be nice to know the guaranteed behavior of calls
> > with
> > > callback continuations without checking core code (for example DNS)
> > >
> >
> >
> > --
> > *Beware the fisherman who's casting out his line in to a dried up
> > riverbed.*
> > *Oh don't try to tell him 'cause he won't believe. Throw some bread to
> the
> > ducks instead.*
> > *It's easier that way. *- Genesis : Duke : VI 25-28
> >
>
>
> --
> *Fei Deng*
>
> E: feid@oath.com
>
> 1908 S First Street
> Champaign, IL, 61820
>
--
*Beware the fisherman who's casting out his line in to a dried up riverbed.*
*Oh don't try to tell him 'cause he won't believe. Throw some bread to the
ducks instead.*
*It's easier that way. *- Genesis : Duke : VI 25-28
Re: [New API Proposal] TSContScheduleToCurrentThread and TSContScheduleToCurrentThreadEvery
Posted by Fei Deng <fe...@oath.com.INVALID>.
I'm looking into this right now. Already have the TSContScheduleOn
and TSContScheduleOnEvery APIs done. Still working on getting the thread
affinity.
On Thu, Aug 16, 2018 at 12:59 PM Alan Carroll
<so...@oath.com.invalid> wrote:
> As a quick response, it sounds like most of your issues would be addressed
> by thread affinity for continuations. I'll see if I can get some write up
> done for the summit. There are other issues beyond yours that I think would
> also benefit from that.
>
> On Thu, Aug 16, 2018 at 12:56 PM Kees Spoelstra <ks...@we-amp.com>
> wrote:
>
> > On Thu, Aug 16, 2018, 20:39 Alan Carroll <solidwallofcode@oath.com
> > .invalid>
> > wrote:
> >
> > > "getting the TSEventThread from a vconn, txn, ssn instead of
> implicitly
> > > trusting that the current thread is relevant?"
> > >
> > > I'm not sure what this means. Those objects don't have threads
> explicitly
> > > associated, it's more implicit. If the code is in a callback on a hook
> > for
> > > the object, then the current thread is relevant, effectively by
> > definition.
> > >
> >
> > When coordinating between txns and reenabling or accessing object a from
> > object b's hook you might not be so lucky.A hook for A might be called on
> > an unexpected thread and fun stuff will happen. Often you will be lucky
> > because core has a lot of explicit alignment checks and reenabling A will
> > be done on it its original thread.
> >
> >
> > But as soon as I read this I realized you were going to bring up the
> > > continuation thread affinity issue.
> > >
> > > "moving a client vconn to a certain thread" - that would be
> challenging.
> > > I'd rather move the plugins to the NetVC thread. This is one aspect of
> > > getting the mutex locking thread - that's the thread to which to move.
> > >
> >
> > I meant outgoing netvc connections, my bad, but we have the technology to
> > make it better, :), in the global http server connectionpool we do this
> > with the netvcs to align with the txn/sm threads. I don't know what has
> an
> > higher impact thread jumping or moving the netvc.
> > Moving netvcs to other threads might have some impact on data in flight,
> I
> > think Susan knows about the limitations of this.
> >
> > Wouldn't it be nice if we had connection pool constructs which we could
> > reuse :) instead of reinventing.
> > Somehow when pulling threads the whole world starts to unravel.
> >
> >
> > > "check the callback continuations in the current API?" - check them for
> > > what? I don't understand this question.
> > >
> >
> >
> > The callback cont for some async tsapi calls might not be called on the
> > same thread. It would be nice to know the guaranteed behavior of calls
> with
> > callback continuations without checking core code (for example DNS)
> >
>
>
> --
> *Beware the fisherman who's casting out his line in to a dried up
> riverbed.*
> *Oh don't try to tell him 'cause he won't believe. Throw some bread to the
> ducks instead.*
> *It's easier that way. *- Genesis : Duke : VI 25-28
>
--
*Fei Deng*
E: feid@oath.com
1908 S First Street
Champaign, IL, 61820
Re: [New API Proposal] TSContScheduleToCurrentThread and TSContScheduleToCurrentThreadEvery
Posted by Alan Carroll <so...@oath.com.INVALID>.
As a quick response, it sounds like most of your issues would be addressed
by thread affinity for continuations. I'll see if I can get some write up
done for the summit. There are other issues beyond yours that I think would
also benefit from that.
On Thu, Aug 16, 2018 at 12:56 PM Kees Spoelstra <ks...@we-amp.com>
wrote:
> On Thu, Aug 16, 2018, 20:39 Alan Carroll <solidwallofcode@oath.com
> .invalid>
> wrote:
>
> > "getting the TSEventThread from a vconn, txn, ssn instead of implicitly
> > trusting that the current thread is relevant?"
> >
> > I'm not sure what this means. Those objects don't have threads explicitly
> > associated, it's more implicit. If the code is in a callback on a hook
> for
> > the object, then the current thread is relevant, effectively by
> definition.
> >
>
> When coordinating between txns and reenabling or accessing object a from
> object b's hook you might not be so lucky.A hook for A might be called on
> an unexpected thread and fun stuff will happen. Often you will be lucky
> because core has a lot of explicit alignment checks and reenabling A will
> be done on it its original thread.
>
>
> But as soon as I read this I realized you were going to bring up the
> > continuation thread affinity issue.
> >
> > "moving a client vconn to a certain thread" - that would be challenging.
> > I'd rather move the plugins to the NetVC thread. This is one aspect of
> > getting the mutex locking thread - that's the thread to which to move.
> >
>
> I meant outgoing netvc connections, my bad, but we have the technology to
> make it better, :), in the global http server connectionpool we do this
> with the netvcs to align with the txn/sm threads. I don't know what has an
> higher impact thread jumping or moving the netvc.
> Moving netvcs to other threads might have some impact on data in flight, I
> think Susan knows about the limitations of this.
>
> Wouldn't it be nice if we had connection pool constructs which we could
> reuse :) instead of reinventing.
> Somehow when pulling threads the whole world starts to unravel.
>
>
> > "check the callback continuations in the current API?" - check them for
> > what? I don't understand this question.
> >
>
>
> The callback cont for some async tsapi calls might not be called on the
> same thread. It would be nice to know the guaranteed behavior of calls with
> callback continuations without checking core code (for example DNS)
>
--
*Beware the fisherman who's casting out his line in to a dried up riverbed.*
*Oh don't try to tell him 'cause he won't believe. Throw some bread to the
ducks instead.*
*It's easier that way. *- Genesis : Duke : VI 25-28
Re: [New API Proposal] TSContScheduleToCurrentThread and TSContScheduleToCurrentThreadEvery
Posted by Kees Spoelstra <ks...@we-amp.com>.
On Thu, Aug 16, 2018, 20:39 Alan Carroll <so...@oath.com.invalid>
wrote:
> "getting the TSEventThread from a vconn, txn, ssn instead of implicitly
> trusting that the current thread is relevant?"
>
> I'm not sure what this means. Those objects don't have threads explicitly
> associated, it's more implicit. If the code is in a callback on a hook for
> the object, then the current thread is relevant, effectively by definition.
>
When coordinating between txns and reenabling or accessing object a from
object b's hook you might not be so lucky.A hook for A might be called on
an unexpected thread and fun stuff will happen. Often you will be lucky
because core has a lot of explicit alignment checks and reenabling A will
be done on it its original thread.
But as soon as I read this I realized you were going to bring up the
> continuation thread affinity issue.
>
> "moving a client vconn to a certain thread" - that would be challenging.
> I'd rather move the plugins to the NetVC thread. This is one aspect of
> getting the mutex locking thread - that's the thread to which to move.
>
I meant outgoing netvc connections, my bad, but we have the technology to
make it better, :), in the global http server connectionpool we do this
with the netvcs to align with the txn/sm threads. I don't know what has an
higher impact thread jumping or moving the netvc.
Moving netvcs to other threads might have some impact on data in flight, I
think Susan knows about the limitations of this.
Wouldn't it be nice if we had connection pool constructs which we could
reuse :) instead of reinventing.
Somehow when pulling threads the whole world starts to unravel.
> "check the callback continuations in the current API?" - check them for
> what? I don't understand this question.
>
The callback cont for some async tsapi calls might not be called on the
same thread. It would be nice to know the guaranteed behavior of calls with
callback continuations without checking core code (for example DNS)
Re: [New API Proposal] TSContScheduleToCurrentThread and TSContScheduleToCurrentThreadEvery
Posted by Alan Carroll <so...@oath.com.INVALID>.
"getting the TSEventThread from a vconn, txn, ssn instead of implicitly
trusting that the current thread is relevant?"
I'm not sure what this means. Those objects don't have threads explicitly
associated, it's more implicit. If the code is in a callback on a hook for
the object, then the current thread is relevant, effectively by definition.
But as soon as I read this I realized you were going to bring up the
continuation thread affinity issue.
"moving a client vconn to a certain thread" - that would be challenging.
I'd rather move the plugins to the NetVC thread. This is one aspect of
getting the mutex locking thread - that's the thread to which to move.
"check the callback continuations in the current API?" - check them for
what? I don't understand this question.
"getting the TSEventThread of a TSThread" - that's fine, with the
understanding that not every TSThread is a TSEventThread so this may yield
a null result.
Unwinding the call stack to deal with conflicting locks is something that's
done in the core quite a bit, so yes it would be nice if plugins could do
that as well. It's effectively a "yield" with regard to the ATS event
dispatching logic. I will try to remember what we talked about with thread
aligned scheduling, there have been multiple schemes I've discussed with
Jason.
Re: [New API Proposal] TSContScheduleToCurrentThread and TSContScheduleToCurrentThreadEvery
Posted by Kees Spoelstra <ks...@we-amp.com>.
Adding to that, we might need have to look into this:
- getting the TSEventThread from a vconn, txn, ssn instead of implicitly
trusting that the current thread is relevant?
- moving a client vconn to a certain thread (we've seen that the thread
jumping in plugins can really kill performance with pooled client-server
vconns), we ended up pooling vconns per thread for the fastcgi plugin,
which could be a waste and not even possible for some fastcgi servers (more
threads running in TS than connections allowed to the fastcgi server)
- check the callback continuations in the current API? do we want to have
guaranteed callbacks on the same thread or do we want the plugin to handle
this?
- getting the TSEventThread of a TSThread
- some documentation for plugins regarding thread alignment
Naming: TSEventThreadSelf() in line with the current API?
Another advantage of scheduling a unit of work on the same thread is that
we can escape some locks which are on the stack due to the recursive nature
of some handling in ATS, only requesting relevant locks. This reduces the
chances of deadlock or retrying. I know people doing coordination between
txns will love this.
A problem with this approach is that it is not always clear to developers
what the relevant locks are, resulting in very interesting situations (TM)
:)
I remember talking with Alan and Jason about the thread aligned scheduling,
and there were some concerns about possible starvation, but if core already
goes out of its way to align per thread it makes sense to follow that in
plugin land. What are the thoughts about this now?
@Otto: We used the TSThread, but using a TSEventThread type will enhance
readability and protect the developer, the implementation will be the same.
On 16 August 2018 at 06:18, Alan Carroll <so...@oath.com.invalid>
wrote:
> In the core, when a continuation is scheduled on a specific thread that is
> not the current thread, it's because there is a locked object with the lock
> held on another thread, and the goal is to have the continuation run on
> that thread. This can help gather interacting objects on the same thread
> where they won't have lock contention (because if I try to get a mutex
> that's locked on *this* thread, it is successful).
>
Re: [New API Proposal] TSContScheduleToCurrentThread and TSContScheduleToCurrentThreadEvery
Posted by Alan Carroll <so...@oath.com.INVALID>.
In the core, when a continuation is scheduled on a specific thread that is
not the current thread, it's because there is a locked object with the lock
held on another thread, and the goal is to have the continuation run on
that thread. This can help gather interacting objects on the same thread
where they won't have lock contention (because if I try to get a mutex
that's locked on *this* thread, it is successful).
Re: [New API Proposal] TSContScheduleToCurrentThread and TSContScheduleToCurrentThreadEvery
Posted by CrazyCow <zh...@gmail.com>.
Thanks, Kees, Otto, and Alan.
TSScheduleOnThread() seems perfect and much more powerful. Any use cases
you guys can figure for scheduling a cont on a specific different thread?
For scheduling on the current thread case, we don't even need a mutex. If
we'd schedule the cont with mutex on the mutex->thread_holding, we only
need the mutex from the user and don't need to have the new API
TSMutexThreadHoldingGet().
2018-08-15 7:29 GMT-07:00 Alan Carroll <so...@oath.com.invalid>:
> I am fine with extending "TSThreadPool" to include a "this thread" value..
> I don't think it breaks the name, it's just specifying a pool of size 1
> :-).
>
> For Otto's suggestion, I'd go with adding a new API type, "TSEventThread".
> This, in contrast to TSThread, would represent a thread that has an event
> loop and which therefore can handle event scheduling. A TSThread does not
> have this capability in general. Additional API could be built around this,
> such as "TSEventThread TSThisEventThread()" which returns the current
> thread. Then you can easily implement what Otto requested,
> "TSScheduleOnThread(TSCont, ink_hrtime, TSEventThread)". The original
> request would then be "TSScheduleOnThread(contp, 1, TSThisEventThread());".
> We might want to add "TSMutexThreadHoldingGet" which gets the thread
> holding a mutex. For the kind of scheduling you guys are discussing, this
> has proven quite handy in the core. I can work on a write up if you think
> this is a reasonable direction to go.
>
> On Wed, Aug 15, 2018 at 1:57 AM Otto van der Schaaf <os...@gmail.com>
> wrote:
>
> > +1 for APIs that allow scheduling on an arbitrary thread
> >
> > On Wed, 15 Aug 2018 at 03:37, Kees Spoelstra <ks...@we-amp.com>
> > wrote:
> >
> > > +1, we currently have it implemented but did not come around to make a
> > pull
> > > request and a proposal.
> > >
> > > Some remarks, instead of current thread we should be able to schedule
> on
> > an
> > > arbitrary thread (*OnThread)
> > >
> > > This can also improve performance by preventing thread jumping and
> > provide
> > > simpler concurrency because there is no concurrency :)
> > >
> > > Not having access to source code, but I recall the change not being
> too
> > > big, it is mostly adding the thread to some calls.
> > >
> > > If people are interested I'll try to provide a pr in the next two days
> > >
> > >
> > > On Wed, Aug 15, 2018, 08:58 zzz <zz...@apache.org> wrote:
> > >
> > > > Hi, team,
> > > >
> > > > The background here is we'd like to clean up a thread local cache
> after
> > > > some timeout. Unfortunately, we're lacking an API to schedule a
> > > > continuation to the current thread.
> > > >
> > > > There are two ways I can think of to achieve this.
> > > > One is a pair of new APIs *TSContScheduleToCurrentThread(TSCont
> contp,
> > > > ink_hrtime timeout) *and* TSContScheduleToCurrentThreadEvery(TSCont
> > > contp,
> > > > ink_hrtime every).*
> > > > Or we can append the TSThreadPool enum with a new entry called
> > > > *TS_THREAD_CURRENT
> > > > *and add the logic into *TSContSchedule() *and
> *TSContScheduleEvery().
> > *
> > > > The issue with the latter solution is *TS_THREAD_CURRENT* doesn't
> quite
> > > fit
> > > > in TSThreadPool. If we change TSThreadPool to some more proper name,
> > the
> > > > change will not be backward compatible.
> > > >
> > > > Please feel free to share your insights. Any ideas would be much
> > > > appreciated!
> > > >
> > > >
> > > > Thanks,
> > > > Zizhong
> > > >
> > >
> >
>
>
> --
> *Beware the fisherman who's casting out his line in to a dried up
> riverbed.*
> *Oh don't try to tell him 'cause he won't believe. Throw some bread to the
> ducks instead.*
> *It's easier that way. *- Genesis : Duke : VI 25-28
>
Re: [New API Proposal] TSContScheduleToCurrentThread and TSContScheduleToCurrentThreadEvery
Posted by Alan Carroll <so...@oath.com.INVALID>.
I am fine with extending "TSThreadPool" to include a "this thread" value..
I don't think it breaks the name, it's just specifying a pool of size 1 :-).
For Otto's suggestion, I'd go with adding a new API type, "TSEventThread".
This, in contrast to TSThread, would represent a thread that has an event
loop and which therefore can handle event scheduling. A TSThread does not
have this capability in general. Additional API could be built around this,
such as "TSEventThread TSThisEventThread()" which returns the current
thread. Then you can easily implement what Otto requested,
"TSScheduleOnThread(TSCont, ink_hrtime, TSEventThread)". The original
request would then be "TSScheduleOnThread(contp, 1, TSThisEventThread());".
We might want to add "TSMutexThreadHoldingGet" which gets the thread
holding a mutex. For the kind of scheduling you guys are discussing, this
has proven quite handy in the core. I can work on a write up if you think
this is a reasonable direction to go.
On Wed, Aug 15, 2018 at 1:57 AM Otto van der Schaaf <os...@gmail.com>
wrote:
> +1 for APIs that allow scheduling on an arbitrary thread
>
> On Wed, 15 Aug 2018 at 03:37, Kees Spoelstra <ks...@we-amp.com>
> wrote:
>
> > +1, we currently have it implemented but did not come around to make a
> pull
> > request and a proposal.
> >
> > Some remarks, instead of current thread we should be able to schedule on
> an
> > arbitrary thread (*OnThread)
> >
> > This can also improve performance by preventing thread jumping and
> provide
> > simpler concurrency because there is no concurrency :)
> >
> > Not having access to source code, but I recall the change not being too
> > big, it is mostly adding the thread to some calls.
> >
> > If people are interested I'll try to provide a pr in the next two days
> >
> >
> > On Wed, Aug 15, 2018, 08:58 zzz <zz...@apache.org> wrote:
> >
> > > Hi, team,
> > >
> > > The background here is we'd like to clean up a thread local cache after
> > > some timeout. Unfortunately, we're lacking an API to schedule a
> > > continuation to the current thread.
> > >
> > > There are two ways I can think of to achieve this.
> > > One is a pair of new APIs *TSContScheduleToCurrentThread(TSCont contp,
> > > ink_hrtime timeout) *and* TSContScheduleToCurrentThreadEvery(TSCont
> > contp,
> > > ink_hrtime every).*
> > > Or we can append the TSThreadPool enum with a new entry called
> > > *TS_THREAD_CURRENT
> > > *and add the logic into *TSContSchedule() *and *TSContScheduleEvery().
> *
> > > The issue with the latter solution is *TS_THREAD_CURRENT* doesn't quite
> > fit
> > > in TSThreadPool. If we change TSThreadPool to some more proper name,
> the
> > > change will not be backward compatible.
> > >
> > > Please feel free to share your insights. Any ideas would be much
> > > appreciated!
> > >
> > >
> > > Thanks,
> > > Zizhong
> > >
> >
>
--
*Beware the fisherman who's casting out his line in to a dried up riverbed.*
*Oh don't try to tell him 'cause he won't believe. Throw some bread to the
ducks instead.*
*It's easier that way. *- Genesis : Duke : VI 25-28
Re: [New API Proposal] TSContScheduleToCurrentThread and TSContScheduleToCurrentThreadEvery
Posted by Otto van der Schaaf <os...@gmail.com>.
+1 for APIs that allow scheduling on an arbitrary thread
On Wed, 15 Aug 2018 at 03:37, Kees Spoelstra <ks...@we-amp.com> wrote:
> +1, we currently have it implemented but did not come around to make a pull
> request and a proposal.
>
> Some remarks, instead of current thread we should be able to schedule on an
> arbitrary thread (*OnThread)
>
> This can also improve performance by preventing thread jumping and provide
> simpler concurrency because there is no concurrency :)
>
> Not having access to source code, but I recall the change not being too
> big, it is mostly adding the thread to some calls.
>
> If people are interested I'll try to provide a pr in the next two days
>
>
> On Wed, Aug 15, 2018, 08:58 zzz <zz...@apache.org> wrote:
>
> > Hi, team,
> >
> > The background here is we'd like to clean up a thread local cache after
> > some timeout. Unfortunately, we're lacking an API to schedule a
> > continuation to the current thread.
> >
> > There are two ways I can think of to achieve this.
> > One is a pair of new APIs *TSContScheduleToCurrentThread(TSCont contp,
> > ink_hrtime timeout) *and* TSContScheduleToCurrentThreadEvery(TSCont
> contp,
> > ink_hrtime every).*
> > Or we can append the TSThreadPool enum with a new entry called
> > *TS_THREAD_CURRENT
> > *and add the logic into *TSContSchedule() *and *TSContScheduleEvery(). *
> > The issue with the latter solution is *TS_THREAD_CURRENT* doesn't quite
> fit
> > in TSThreadPool. If we change TSThreadPool to some more proper name, the
> > change will not be backward compatible.
> >
> > Please feel free to share your insights. Any ideas would be much
> > appreciated!
> >
> >
> > Thanks,
> > Zizhong
> >
>
Re: [New API Proposal] TSContScheduleToCurrentThread and TSContScheduleToCurrentThreadEvery
Posted by Kees Spoelstra <ks...@we-amp.com>.
+1, we currently have it implemented but did not come around to make a pull
request and a proposal.
Some remarks, instead of current thread we should be able to schedule on an
arbitrary thread (*OnThread)
This can also improve performance by preventing thread jumping and provide
simpler concurrency because there is no concurrency :)
Not having access to source code, but I recall the change not being too
big, it is mostly adding the thread to some calls.
If people are interested I'll try to provide a pr in the next two days
On Wed, Aug 15, 2018, 08:58 zzz <zz...@apache.org> wrote:
> Hi, team,
>
> The background here is we'd like to clean up a thread local cache after
> some timeout. Unfortunately, we're lacking an API to schedule a
> continuation to the current thread.
>
> There are two ways I can think of to achieve this.
> One is a pair of new APIs *TSContScheduleToCurrentThread(TSCont contp,
> ink_hrtime timeout) *and* TSContScheduleToCurrentThreadEvery(TSCont contp,
> ink_hrtime every).*
> Or we can append the TSThreadPool enum with a new entry called
> *TS_THREAD_CURRENT
> *and add the logic into *TSContSchedule() *and *TSContScheduleEvery(). *
> The issue with the latter solution is *TS_THREAD_CURRENT* doesn't quite fit
> in TSThreadPool. If we change TSThreadPool to some more proper name, the
> change will not be backward compatible.
>
> Please feel free to share your insights. Any ideas would be much
> appreciated!
>
>
> Thanks,
> Zizhong
>