You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@asterixdb.apache.org by Wail Alkowaileet <wa...@gmail.com> on 2017/10/24 02:14:27 UTC
PrimaryIndexOperationTracker
Dear devs,
I have a question regarding the opTracker. Currently, we initialize one
opTracker per dataset in every NC.
My question is why it's per dataset not per partition ? Is there a
transactional constraints for that ?
From what I can see that the opTracker can create a lot of contention in
case there're many IO devices. For instance, each insert will call
*LSMHarness.getAndEnterComponents()* [1], which *synchronize(opTracker). *That
means (correct me if I'm wrong), insert is going to serialize the
*enterComponent()* part among partitions.
[1]
https://github.com/apache/asterixdb/blob/master/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/LSMHarness.java#L86
--
*Regards,*
Wail Alkowaileet
Re: PrimaryIndexOperationTracker
Posted by Sattam Alsubaiee <sa...@gmail.com>.
And of course I meant partitions of a dataset in the same machine...
Btw it is mentioned in our storage paper. "Currently in AsterixDB, all
partitions of any one dataset and its
indexes in a machine share the same memory budget for in-memory
components (divided equally across the indexes). This implies that
when the memory component of index in a partition is declared to
be full, all memory components of the indexes for the same dataset
in that machine are also declared to be full. Therefore, multiple
consecutive flush requests are sent to the I/O scheduler for flushing
the memory components of that dataset and its indexes for all the
partitions located on the machine.“
Sattam
On Oct 24, 2017 6:12 AM, "Sattam Alsubaiee" <sa...@gmail.com> wrote:
> There is a fundamental reason behind that (unless that has changed). All
> partitions of a dataset (primary and secondary indexes) share the memory
> budget for their in-memory components. So they use the same optracker to
> synchronize their operations.
>
>
> Sattam
>
> On Oct 24, 2017 5:59 AM, "Wail Alkowaileet" <wa...@gmail.com> wrote:
>
>> Thanks Abdullah!
>>
>> On Mon, Oct 23, 2017 at 7:15 PM, abdullah alamoudi <ba...@gmail.com>
>> wrote:
>>
>> > Hi Wail,
>> > There is no fundamental reason why it is one. In fact, it has been on
>> our
>> > todo for a long time to make it one per partition.
>> >
>> > Cheers,
>> > Abdullah.
>> >
>> > > On Oct 23, 2017, at 7:14 PM, Wail Alkowaileet <wa...@gmail.com>
>> > wrote:
>> > >
>> > > Dear devs,
>> > >
>> > > I have a question regarding the opTracker. Currently, we initialize
>> one
>> > > opTracker per dataset in every NC.
>> > >
>> > > My question is why it's per dataset not per partition ? Is there a
>> > > transactional constraints for that ?
>> > >
>> > > From what I can see that the opTracker can create a lot of contention
>> in
>> > > case there're many IO devices. For instance, each insert will call
>> > > *LSMHarness.getAndEnterComponents()* [1], which
>> > *synchronize(opTracker). *That
>> > > means (correct me if I'm wrong), insert is going to serialize the
>> > > *enterComponent()* part among partitions.
>> > >
>> > > [1]
>> > > https://github.com/apache/asterixdb/blob/master/hyracks-
>> > fullstack/hyracks/hyracks-storage-am-lsm-common/src/
>> > main/java/org/apache/hyracks/storage/am/lsm/common/impls/
>> > LSMHarness.java#L86
>> > >
>> > > --
>> > >
>> > > *Regards,*
>> > > Wail Alkowaileet
>> >
>> >
>>
>>
>> --
>>
>> *Regards,*
>> Wail Alkowaileet
>>
>
Re: PrimaryIndexOperationTracker
Posted by abdullah alamoudi <ba...@gmail.com>.
+1
> On Oct 23, 2017, at 9:24 PM, Till Westmann <ti...@apache.org> wrote:
>
> Hi,
>
> I think that it’s actually quite common to have multiple iodevices.
> Even if there is only one disk, configuring multiple iodevices can
> improve performance - especially if a significant part of the data fits
> into memory (and disk IO is not the limiting factor). And the reason
> for that seems to be that the degree of parallelism for the "lower"
> part of a query plan is limited by the number of available iodevices.
> Also, I think that multiple disks are not uncommon for heavier data
> management workloads. The d2 instances [1] on AWS give some example
> configurations.
>
> All this is just to say that we should not discount
> a) the need to support multiple disks and
> b) the limitations to linear scalability introduced by "1-per-NC" (vs.
> "1-per-partition") objects.
> On machines with a few more cores (10+) we've used multiple NCs (as
> opposed to multiple partitions per NC) to achieve better scale-out and
> better query performance.
>
> Cheers,
> Till
>
> [1] https://aws.amazon.com/about-aws/whats-new/2015/03/now-available-d2-instances-the-latest-generation-of-amazon-ec2-dense-storage-instances/
>
> On 23 Oct 2017, at 20:18, Chen Luo wrote:
>
>> I was confusing about this problem last quarter, and I remembered (maybe
>> Ian) told me the same reason that it's mainly used for memory budgeting
>> (primary and secondary indexes of dataset in the same node). Moreover, it's
>> not very common to have too many I/O devices on a node, unless it has
>> multiple disks.
>>
>> Best regards,
>> Chen Luo
>>
>> On Mon, Oct 23, 2017 at 8:12 PM, Sattam Alsubaiee <sa...@gmail.com>
>> wrote:
>>
>>> There is a fundamental reason behind that (unless that has changed). All
>>> partitions of a dataset (primary and secondary indexes) share the memory
>>> budget for their in-memory components. So they use the same optracker to
>>> synchronize their operations.
>>>
>>>
>>> Sattam
>>>
>>> On Oct 24, 2017 5:59 AM, "Wail Alkowaileet" <wa...@gmail.com> wrote:
>>>
>>>> Thanks Abdullah!
>>>>
>>>> On Mon, Oct 23, 2017 at 7:15 PM, abdullah alamoudi <ba...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Wail,
>>>>> There is no fundamental reason why it is one. In fact, it has been on
>>> our
>>>>> todo for a long time to make it one per partition.
>>>>>
>>>>> Cheers,
>>>>> Abdullah.
>>>>>
>>>>>> On Oct 23, 2017, at 7:14 PM, Wail Alkowaileet <wa...@gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> Dear devs,
>>>>>>
>>>>>> I have a question regarding the opTracker. Currently, we initialize
>>> one
>>>>>> opTracker per dataset in every NC.
>>>>>>
>>>>>> My question is why it's per dataset not per partition ? Is there a
>>>>>> transactional constraints for that ?
>>>>>>
>>>>>> From what I can see that the opTracker can create a lot of contention
>>>> in
>>>>>> case there're many IO devices. For instance, each insert will call
>>>>>> *LSMHarness.getAndEnterComponents()* [1], which
>>>>> *synchronize(opTracker). *That
>>>>>> means (correct me if I'm wrong), insert is going to serialize the
>>>>>> *enterComponent()* part among partitions.
>>>>>>
>>>>>> [1]
>>>>>> https://github.com/apache/asterixdb/blob/master/hyracks-
>>>>> fullstack/hyracks/hyracks-storage-am-lsm-common/src/
>>>>> main/java/org/apache/hyracks/storage/am/lsm/common/impls/
>>>>> LSMHarness.java#L86
>>>>>>
>>>>>> --
>>>>>>
>>>>>> *Regards,*
>>>>>> Wail Alkowaileet
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> *Regards,*
>>>> Wail Alkowaileet
>>>>
>>>
Re: PrimaryIndexOperationTracker
Posted by Till Westmann <ti...@apache.org>.
Hi,
I think that it’s actually quite common to have multiple iodevices.
Even if there is only one disk, configuring multiple iodevices can
improve performance - especially if a significant part of the data fits
into memory (and disk IO is not the limiting factor). And the reason
for that seems to be that the degree of parallelism for the "lower"
part of a query plan is limited by the number of available iodevices.
Also, I think that multiple disks are not uncommon for heavier data
management workloads. The d2 instances [1] on AWS give some example
configurations.
All this is just to say that we should not discount
a) the need to support multiple disks and
b) the limitations to linear scalability introduced by "1-per-NC" (vs.
"1-per-partition") objects.
On machines with a few more cores (10+) we've used multiple NCs (as
opposed to multiple partitions per NC) to achieve better scale-out and
better query performance.
Cheers,
Till
[1]
https://aws.amazon.com/about-aws/whats-new/2015/03/now-available-d2-instances-the-latest-generation-of-amazon-ec2-dense-storage-instances/
On 23 Oct 2017, at 20:18, Chen Luo wrote:
> I was confusing about this problem last quarter, and I remembered
> (maybe
> Ian) told me the same reason that it's mainly used for memory
> budgeting
> (primary and secondary indexes of dataset in the same node). Moreover,
> it's
> not very common to have too many I/O devices on a node, unless it has
> multiple disks.
>
> Best regards,
> Chen Luo
>
> On Mon, Oct 23, 2017 at 8:12 PM, Sattam Alsubaiee
> <sa...@gmail.com>
> wrote:
>
>> There is a fundamental reason behind that (unless that has changed).
>> All
>> partitions of a dataset (primary and secondary indexes) share the
>> memory
>> budget for their in-memory components. So they use the same optracker
>> to
>> synchronize their operations.
>>
>>
>> Sattam
>>
>> On Oct 24, 2017 5:59 AM, "Wail Alkowaileet" <wa...@gmail.com>
>> wrote:
>>
>>> Thanks Abdullah!
>>>
>>> On Mon, Oct 23, 2017 at 7:15 PM, abdullah alamoudi
>>> <ba...@gmail.com>
>>> wrote:
>>>
>>>> Hi Wail,
>>>> There is no fundamental reason why it is one. In fact, it has been
>>>> on
>> our
>>>> todo for a long time to make it one per partition.
>>>>
>>>> Cheers,
>>>> Abdullah.
>>>>
>>>>> On Oct 23, 2017, at 7:14 PM, Wail Alkowaileet <wa...@gmail.com>
>>>> wrote:
>>>>>
>>>>> Dear devs,
>>>>>
>>>>> I have a question regarding the opTracker. Currently, we
>>>>> initialize
>> one
>>>>> opTracker per dataset in every NC.
>>>>>
>>>>> My question is why it's per dataset not per partition ? Is there a
>>>>> transactional constraints for that ?
>>>>>
>>>>> From what I can see that the opTracker can create a lot of
>>>>> contention
>>> in
>>>>> case there're many IO devices. For instance, each insert will call
>>>>> *LSMHarness.getAndEnterComponents()* [1], which
>>>> *synchronize(opTracker). *That
>>>>> means (correct me if I'm wrong), insert is going to serialize the
>>>>> *enterComponent()* part among partitions.
>>>>>
>>>>> [1]
>>>>> https://github.com/apache/asterixdb/blob/master/hyracks-
>>>> fullstack/hyracks/hyracks-storage-am-lsm-common/src/
>>>> main/java/org/apache/hyracks/storage/am/lsm/common/impls/
>>>> LSMHarness.java#L86
>>>>>
>>>>> --
>>>>>
>>>>> *Regards,*
>>>>> Wail Alkowaileet
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> *Regards,*
>>> Wail Alkowaileet
>>>
>>
Re: PrimaryIndexOperationTracker
Posted by Chen Luo <cl...@uci.edu>.
I was confusing about this problem last quarter, and I remembered (maybe
Ian) told me the same reason that it's mainly used for memory budgeting
(primary and secondary indexes of dataset in the same node). Moreover, it's
not very common to have too many I/O devices on a node, unless it has
multiple disks.
Best regards,
Chen Luo
On Mon, Oct 23, 2017 at 8:12 PM, Sattam Alsubaiee <sa...@gmail.com>
wrote:
> There is a fundamental reason behind that (unless that has changed). All
> partitions of a dataset (primary and secondary indexes) share the memory
> budget for their in-memory components. So they use the same optracker to
> synchronize their operations.
>
>
> Sattam
>
> On Oct 24, 2017 5:59 AM, "Wail Alkowaileet" <wa...@gmail.com> wrote:
>
> > Thanks Abdullah!
> >
> > On Mon, Oct 23, 2017 at 7:15 PM, abdullah alamoudi <ba...@gmail.com>
> > wrote:
> >
> > > Hi Wail,
> > > There is no fundamental reason why it is one. In fact, it has been on
> our
> > > todo for a long time to make it one per partition.
> > >
> > > Cheers,
> > > Abdullah.
> > >
> > > > On Oct 23, 2017, at 7:14 PM, Wail Alkowaileet <wa...@gmail.com>
> > > wrote:
> > > >
> > > > Dear devs,
> > > >
> > > > I have a question regarding the opTracker. Currently, we initialize
> one
> > > > opTracker per dataset in every NC.
> > > >
> > > > My question is why it's per dataset not per partition ? Is there a
> > > > transactional constraints for that ?
> > > >
> > > > From what I can see that the opTracker can create a lot of contention
> > in
> > > > case there're many IO devices. For instance, each insert will call
> > > > *LSMHarness.getAndEnterComponents()* [1], which
> > > *synchronize(opTracker). *That
> > > > means (correct me if I'm wrong), insert is going to serialize the
> > > > *enterComponent()* part among partitions.
> > > >
> > > > [1]
> > > > https://github.com/apache/asterixdb/blob/master/hyracks-
> > > fullstack/hyracks/hyracks-storage-am-lsm-common/src/
> > > main/java/org/apache/hyracks/storage/am/lsm/common/impls/
> > > LSMHarness.java#L86
> > > >
> > > > --
> > > >
> > > > *Regards,*
> > > > Wail Alkowaileet
> > >
> > >
> >
> >
> > --
> >
> > *Regards,*
> > Wail Alkowaileet
> >
>
Re: PrimaryIndexOperationTracker
Posted by Sattam Alsubaiee <sa...@gmail.com>.
There is a fundamental reason behind that (unless that has changed). All
partitions of a dataset (primary and secondary indexes) share the memory
budget for their in-memory components. So they use the same optracker to
synchronize their operations.
Sattam
On Oct 24, 2017 5:59 AM, "Wail Alkowaileet" <wa...@gmail.com> wrote:
> Thanks Abdullah!
>
> On Mon, Oct 23, 2017 at 7:15 PM, abdullah alamoudi <ba...@gmail.com>
> wrote:
>
> > Hi Wail,
> > There is no fundamental reason why it is one. In fact, it has been on our
> > todo for a long time to make it one per partition.
> >
> > Cheers,
> > Abdullah.
> >
> > > On Oct 23, 2017, at 7:14 PM, Wail Alkowaileet <wa...@gmail.com>
> > wrote:
> > >
> > > Dear devs,
> > >
> > > I have a question regarding the opTracker. Currently, we initialize one
> > > opTracker per dataset in every NC.
> > >
> > > My question is why it's per dataset not per partition ? Is there a
> > > transactional constraints for that ?
> > >
> > > From what I can see that the opTracker can create a lot of contention
> in
> > > case there're many IO devices. For instance, each insert will call
> > > *LSMHarness.getAndEnterComponents()* [1], which
> > *synchronize(opTracker). *That
> > > means (correct me if I'm wrong), insert is going to serialize the
> > > *enterComponent()* part among partitions.
> > >
> > > [1]
> > > https://github.com/apache/asterixdb/blob/master/hyracks-
> > fullstack/hyracks/hyracks-storage-am-lsm-common/src/
> > main/java/org/apache/hyracks/storage/am/lsm/common/impls/
> > LSMHarness.java#L86
> > >
> > > --
> > >
> > > *Regards,*
> > > Wail Alkowaileet
> >
> >
>
>
> --
>
> *Regards,*
> Wail Alkowaileet
>
Re: PrimaryIndexOperationTracker
Posted by Wail Alkowaileet <wa...@gmail.com>.
Thanks Abdullah!
On Mon, Oct 23, 2017 at 7:15 PM, abdullah alamoudi <ba...@gmail.com>
wrote:
> Hi Wail,
> There is no fundamental reason why it is one. In fact, it has been on our
> todo for a long time to make it one per partition.
>
> Cheers,
> Abdullah.
>
> > On Oct 23, 2017, at 7:14 PM, Wail Alkowaileet <wa...@gmail.com>
> wrote:
> >
> > Dear devs,
> >
> > I have a question regarding the opTracker. Currently, we initialize one
> > opTracker per dataset in every NC.
> >
> > My question is why it's per dataset not per partition ? Is there a
> > transactional constraints for that ?
> >
> > From what I can see that the opTracker can create a lot of contention in
> > case there're many IO devices. For instance, each insert will call
> > *LSMHarness.getAndEnterComponents()* [1], which
> *synchronize(opTracker). *That
> > means (correct me if I'm wrong), insert is going to serialize the
> > *enterComponent()* part among partitions.
> >
> > [1]
> > https://github.com/apache/asterixdb/blob/master/hyracks-
> fullstack/hyracks/hyracks-storage-am-lsm-common/src/
> main/java/org/apache/hyracks/storage/am/lsm/common/impls/
> LSMHarness.java#L86
> >
> > --
> >
> > *Regards,*
> > Wail Alkowaileet
>
>
--
*Regards,*
Wail Alkowaileet
Re: PrimaryIndexOperationTracker
Posted by abdullah alamoudi <ba...@gmail.com>.
Hi Wail,
There is no fundamental reason why it is one. In fact, it has been on our todo for a long time to make it one per partition.
Cheers,
Abdullah.
> On Oct 23, 2017, at 7:14 PM, Wail Alkowaileet <wa...@gmail.com> wrote:
>
> Dear devs,
>
> I have a question regarding the opTracker. Currently, we initialize one
> opTracker per dataset in every NC.
>
> My question is why it's per dataset not per partition ? Is there a
> transactional constraints for that ?
>
> From what I can see that the opTracker can create a lot of contention in
> case there're many IO devices. For instance, each insert will call
> *LSMHarness.getAndEnterComponents()* [1], which *synchronize(opTracker). *That
> means (correct me if I'm wrong), insert is going to serialize the
> *enterComponent()* part among partitions.
>
> [1]
> https://github.com/apache/asterixdb/blob/master/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/LSMHarness.java#L86
>
> --
>
> *Regards,*
> Wail Alkowaileet