You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@trafficserver.apache.org by James Peach <jp...@apache.org> on 2013/04/11 17:28:19 UTC

VolInit change for linux AIO

Hi Weijin,

I'm looking at the Linux AOI changes, and there is one place where you lift the Vol::init out into a VolInit continuation. This change doesn't seem specific to Linux AIO ... can you explain why it's being done? Is it safe to do it when Linux AIO is not enabled?

thanks,
James

Re: VolInit change for linux AIO

Posted by Leif Hedstrom <zw...@apache.org>.
On 4/11/13 8:25 PM, weijin wrote:
> after changed to Linux Native AIO, the AIO callback happened in the same 
> thread, that means all subsequent operations of vol init will be also 
> happened in only one thread.
> But the previous aio can callback in any ET_CALL thread, so in most case 
> the vol init operations can spread to many thread.

Hmmm, so why is that a bad ? The callback has to happen on one single thread?

-- Leif

Re: VolInit change for linux AIO

Posted by Leif Hedstrom <zw...@apache.org>.
On 4/13/13 10:42 AM, weijin wrote:
> On 04/13/2013 07:27 AM, James Peach wrote:
>> On Apr 11, 2013, at 7:25 PM, weijin <we...@126.com> wrote:
>>
>>> after changed to Linux Native AIO, the AIO callback happened in the same 
>>> thread, that means all subsequent operations of vol init will be also 
>>> happened in only one thread.
>> Why is this? Is it due to the way your change works or the way Linux AIO 
>> works?
> yes, If enable_native_aio, which thread submit, which handle callbacks

So this implies that a system with only one or a couple of disks, all AIO 
happens on one (or two) threads? That seems like a pretty common use case, 
and performance could potentially suffer.

Maybe we should have a way to combine this with "AIO callback" threads, such 
that I can have these callbacks on a thread that's not doubling up as 
Net-thread (ET_TASK) ? As example, it'd be a pretty normal setup to have one 
cache disk, and 2 net-threads, running on the same box as the "origin". In 
this setup, having a third thread for native AIO seems like a better solution?

>>
>>> But the previous aio can callback in any ET_CALL thread, so in most case 
>>> the vol init operations can spread to many thread.
>> So does that mean that you only have 1 thread handling AIO for each disk? 
>> What is the performance profile of that?
>>
> the vol::init will first read the dirs,  then do the recover logics. the 
> recover logics is complicate and may read data more than one time. We may 
> have many vols and I will Vol::init (recover logics) can be done paralleled.

Hmmm, on my prod box, there's only one volume...

Cheers,

-- Leif

Re: VolInit change for linux AIO

Posted by weijin <we...@126.com>.
On 04/13/2013 07:27 AM, James Peach wrote:
> On Apr 11, 2013, at 7:25 PM, weijin <we...@126.com> wrote:
>
>> after changed to Linux Native AIO, the AIO callback happened in the same thread, that means all subsequent operations of vol init will be also happened in only one thread.
> Why is this? Is it due to the way your change works or the way Linux AIO works?
yes, If enable_native_aio, which thread submit, which handle callbacks
>
>> But the previous aio can callback in any ET_CALL thread, so in most case the vol init operations can spread to many thread.
> So does that mean that you only have 1 thread handling AIO for each disk? What is the performance profile of that?
>
the vol::init will first read the dirs,  then do the recover logics. the 
recover logics is complicate and may read data more than one time. We 
may have many vols and I will Vol::init (recover logics) can be done 
paralleled.
>> On 04/11/2013 11:28 PM, James Peach wrote:
>>> Hi Weijin,
>>>
>>> I'm looking at the Linux AOI changes, and there is one place where you lift the Vol::init out into a VolInit continuation. This change doesn't seem specific to Linux AIO ... can you explain why it's being done? Is it safe to do it when Linux AIO is not enabled?
>>>
>>> thanks,
>>> James
>>>
>>
>

Re: VolInit change for linux AIO

Posted by James Peach <jp...@apache.org>.
On Apr 11, 2013, at 7:25 PM, weijin <we...@126.com> wrote:

> after changed to Linux Native AIO, the AIO callback happened in the same thread, that means all subsequent operations of vol init will be also happened in only one thread.

Why is this? Is it due to the way your change works or the way Linux AIO works?

> But the previous aio can callback in any ET_CALL thread, so in most case the vol init operations can spread to many thread.

So does that mean that you only have 1 thread handling AIO for each disk? What is the performance profile of that?

> 
> On 04/11/2013 11:28 PM, James Peach wrote:
>> Hi Weijin,
>> 
>> I'm looking at the Linux AOI changes, and there is one place where you lift the Vol::init out into a VolInit continuation. This change doesn't seem specific to Linux AIO ... can you explain why it's being done? Is it safe to do it when Linux AIO is not enabled?
>> 
>> thanks,
>> James
>> 
> 
>

Re: VolInit change for linux AIO

Posted by weijin <we...@126.com>.
after changed to Linux Native AIO, the AIO callback happened in the same 
thread, that means all subsequent operations of vol init will be also 
happened in only one thread.
But the previous aio can callback in any ET_CALL thread, so in most case 
the vol init operations can spread to many thread.

On 04/11/2013 11:28 PM, James Peach wrote:
> Hi Weijin,
>
> I'm looking at the Linux AOI changes, and there is one place where you lift the Vol::init out into a VolInit continuation. This change doesn't seem specific to Linux AIO ... can you explain why it's being done? Is it safe to do it when Linux AIO is not enabled?
>
> thanks,
> James
>