You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@asterixdb.apache.org by abdullah alamoudi <am...@apache.org> on 2015/09/22 08:11:12 UTC
Question about open indexes
Hi Dev, @Ildar,
In the insert pipeline for datasets with open indexes, we introduce a cast
function before the insert and so one would expect the records to look like
the casted record type which I assume has {{the closed fields + a nullable
field}}.
The question is, what happens to the previously existing records?, since
now the index has both, records of the original type and records of the
casted type.
Thanks,
Abdullah.
Re: Question about open indexes
Posted by Mike Carey <dt...@gmail.com>.
Hmmm - I agree about the logical equivalence of altering the type - but
- I would think that the physical format would still be the original
(and that the cast would really be a check and a value extraction for
the index). I will keep reading this thread....
On 9/22/15 12:18 AM, abdullah alamoudi wrote:
> I see.
> Thanks Ildar,
>
> Abdullah.
>
> Amoudi, Abdullah.
>
> On Tue, Sep 22, 2015 at 10:04 AM, Ildar Absalyamov <
> ildar.absalyamov@gmail.com> wrote:
>
>> Abdullah,
>>
>> If I remember correctly whenever a secondary open index is created all
>> existing records would be casted to a proper type to ensure that the index
>> creation is valid.
>> As for the overall correctness of casting operation, semantically creating
>> an open index is the same thing as altering the dataset type. The current
>> implementation allows only one open index of particular type created on a
>> single field. If we would have had “alter datatype” functionality the open
>> indexing would not be required at all.
>>
>>> On Sep 21, 2015, at 23:25, abdullah alamoudi <am...@apache.org> wrote:
>>>
>>> More thoughts:
>>> I assume the intention of the cast was just to make sure if the open
>> field
>>> exists, it is of the specified type. Moreover, the un-casted record
>> should
>>> be inserted into the index.
>>> If my assumptions are not correct, please, let me know ASAP.
>>>
>>> I have two thoughts on this:
>>> 1. Actually, insert plans show that the records being inserted into the
>>> primary index is actually the casted record creating the issue described
>>> above.
>>>
>>> 2. I don't believe this is the right way to ensure that the open field if
>>> exists is of the right type. why not extract the field using field access
>>> by name function and then verify the type using the field tag?
>>>
>>>
>>>
>>> On Tue, Sep 22, 2015 at 9:11 AM, abdullah alamoudi <am...@apache.org>
>>> wrote:
>>>
>>>> Hi Dev, @Ildar,
>>>>
>>>> In the insert pipeline for datasets with open indexes, we introduce a
>> cast
>>>> function before the insert and so one would expect the records to look
>> like
>>>> the casted record type which I assume has {{the closed fields + a
>> nullable
>>>> field}}.
>>>>
>>>> The question is, what happens to the previously existing records?, since
>>>> now the index has both, records of the original type and records of the
>>>> casted type.
>>>>
>>>> Thanks,
>>>> Abdullah.
>>>>
>> Best regards,
>> Ildar
>>
>>
Re: Question about open indexes
Posted by abdullah alamoudi <ba...@gmail.com>.
P.S
The cast function should definitely have better output in the plan that
includes input and output types.
Amoudi, Abdullah.
On Tue, Sep 22, 2015 at 1:43 PM, abdullah alamoudi <ba...@gmail.com>
wrote:
> Never mind,
> I figured it out.
>
> The cast in red actually changes the record in primary index into the
> casted record. The cast before the insert operator into the primary index
> actually casts from the input to the open type since they are compatible.
>
> Regards,
> Abdullah.
>
>
> Amoudi, Abdullah.
>
> On Tue, Sep 22, 2015 at 11:19 AM, abdullah alamoudi <ba...@gmail.com>
> wrote:
>
>> @Ildar,
>> If that is the case, then why do we cast as well after the primary index
>> insert operator. If all the records are casted already, then why is the
>> second cast needed?
>>
>> For example, look at the following plan:
>> Statement:
>> insert into dataset OrdersOpen (
>> for $x in dataset Orders
>> return $x
>> );
>> Plan:
>> commit
>> -- COMMIT |PARTITIONED|
>> project ([$$3])
>> -- STREAM_PROJECT |PARTITIONED|
>> exchange
>> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
>> insert into idx_Orders_Custkey on tpch:OrdersOpen from [%0->$$7]
>> -- INDEX_INSERT_DELETE |PARTITIONED|
>> exchange
>> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
>> project ([$$3, $$7])
>> -- STREAM_PROJECT |PARTITIONED|
>> assign [$$7] <- [function-call:
>> asterix:field-access-by-index, Args:[function-call: asterix:cast-record,
>> Args:[%0->$$4], AInt32: {8}]]
>> -- ASSIGN |PARTITIONED|
>> exchange
>> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
>> insert into tpch:OrdersOpen from %0->$$4 partitioned by
>> [%0->$$3]
>> -- INSERT_DELETE |PARTITIONED|
>> exchange
>> -- HASH_PARTITION_EXCHANGE [$$3] |PARTITIONED|
>> assign [$$3] <- [function-call:
>> asterix:field-access-by-index, Args:[%0->$$4, AInt32: {0}]]
>> -- ASSIGN |PARTITIONED|
>> project ([$$4])
>> -- STREAM_PROJECT |PARTITIONED|
>> assign [$$4] <- [function-call:
>> asterix:cast-record, Args:[%0->$$0]]
>> -- ASSIGN |PARTITIONED|
>> project ([$$0])
>> -- STREAM_PROJECT |PARTITIONED|
>> exchange
>> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
>> data-scan []<-[$$5, $$0] <- tpch:Orders
>> -- DATASOURCE_SCAN |PARTITIONED|
>> exchange
>> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
>> empty-tuple-source
>> -- EMPTY_TUPLE_SOURCE |PARTITIONED|
>>
>> what is the point of the cast in red?
>>
>>
>> Amoudi, Abdullah.
>>
>> On Tue, Sep 22, 2015 at 10:18 AM, abdullah alamoudi <ba...@gmail.com>
>> wrote:
>>
>>> I see.
>>> Thanks Ildar,
>>>
>>> Abdullah.
>>>
>>> Amoudi, Abdullah.
>>>
>>> On Tue, Sep 22, 2015 at 10:04 AM, Ildar Absalyamov <
>>> ildar.absalyamov@gmail.com> wrote:
>>>
>>>> Abdullah,
>>>>
>>>> If I remember correctly whenever a secondary open index is created all
>>>> existing records would be casted to a proper type to ensure that the index
>>>> creation is valid.
>>>> As for the overall correctness of casting operation, semantically
>>>> creating an open index is the same thing as altering the dataset type. The
>>>> current implementation allows only one open index of particular type
>>>> created on a single field. If we would have had “alter datatype”
>>>> functionality the open indexing would not be required at all.
>>>>
>>>> > On Sep 21, 2015, at 23:25, abdullah alamoudi <am...@apache.org>
>>>> wrote:
>>>> >
>>>> > More thoughts:
>>>> > I assume the intention of the cast was just to make sure if the open
>>>> field
>>>> > exists, it is of the specified type. Moreover, the un-casted record
>>>> should
>>>> > be inserted into the index.
>>>> > If my assumptions are not correct, please, let me know ASAP.
>>>> >
>>>> > I have two thoughts on this:
>>>> > 1. Actually, insert plans show that the records being inserted into
>>>> the
>>>> > primary index is actually the casted record creating the issue
>>>> described
>>>> > above.
>>>> >
>>>> > 2. I don't believe this is the right way to ensure that the open
>>>> field if
>>>> > exists is of the right type. why not extract the field using field
>>>> access
>>>> > by name function and then verify the type using the field tag?
>>>> >
>>>> >
>>>> >
>>>> > On Tue, Sep 22, 2015 at 9:11 AM, abdullah alamoudi <amoudi@apache.org
>>>> >
>>>> > wrote:
>>>> >
>>>> >> Hi Dev, @Ildar,
>>>> >>
>>>> >> In the insert pipeline for datasets with open indexes, we introduce
>>>> a cast
>>>> >> function before the insert and so one would expect the records to
>>>> look like
>>>> >> the casted record type which I assume has {{the closed fields + a
>>>> nullable
>>>> >> field}}.
>>>> >>
>>>> >> The question is, what happens to the previously existing records?,
>>>> since
>>>> >> now the index has both, records of the original type and records of
>>>> the
>>>> >> casted type.
>>>> >>
>>>> >> Thanks,
>>>> >> Abdullah.
>>>> >>
>>>>
>>>> Best regards,
>>>> Ildar
>>>>
>>>>
>>>
>>
>
Re: Question about open indexes
Posted by abdullah alamoudi <ba...@gmail.com>.
Never mind,
I figured it out.
The cast in red actually changes the record in primary index into the
casted record. The cast before the insert operator into the primary index
actually casts from the input to the open type since they are compatible.
Regards,
Abdullah.
Amoudi, Abdullah.
On Tue, Sep 22, 2015 at 11:19 AM, abdullah alamoudi <ba...@gmail.com>
wrote:
> @Ildar,
> If that is the case, then why do we cast as well after the primary index
> insert operator. If all the records are casted already, then why is the
> second cast needed?
>
> For example, look at the following plan:
> Statement:
> insert into dataset OrdersOpen (
> for $x in dataset Orders
> return $x
> );
> Plan:
> commit
> -- COMMIT |PARTITIONED|
> project ([$$3])
> -- STREAM_PROJECT |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> insert into idx_Orders_Custkey on tpch:OrdersOpen from [%0->$$7]
> -- INDEX_INSERT_DELETE |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> project ([$$3, $$7])
> -- STREAM_PROJECT |PARTITIONED|
> assign [$$7] <- [function-call: asterix:field-access-by-index,
> Args:[function-call: asterix:cast-record, Args:[%0->$$4], AInt32: {8}]]
> -- ASSIGN |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> insert into tpch:OrdersOpen from %0->$$4 partitioned by
> [%0->$$3]
> -- INSERT_DELETE |PARTITIONED|
> exchange
> -- HASH_PARTITION_EXCHANGE [$$3] |PARTITIONED|
> assign [$$3] <- [function-call:
> asterix:field-access-by-index, Args:[%0->$$4, AInt32: {0}]]
> -- ASSIGN |PARTITIONED|
> project ([$$4])
> -- STREAM_PROJECT |PARTITIONED|
> assign [$$4] <- [function-call:
> asterix:cast-record, Args:[%0->$$0]]
> -- ASSIGN |PARTITIONED|
> project ([$$0])
> -- STREAM_PROJECT |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> data-scan []<-[$$5, $$0] <- tpch:Orders
> -- DATASOURCE_SCAN |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> empty-tuple-source
> -- EMPTY_TUPLE_SOURCE |PARTITIONED|
>
> what is the point of the cast in red?
>
>
> Amoudi, Abdullah.
>
> On Tue, Sep 22, 2015 at 10:18 AM, abdullah alamoudi <ba...@gmail.com>
> wrote:
>
>> I see.
>> Thanks Ildar,
>>
>> Abdullah.
>>
>> Amoudi, Abdullah.
>>
>> On Tue, Sep 22, 2015 at 10:04 AM, Ildar Absalyamov <
>> ildar.absalyamov@gmail.com> wrote:
>>
>>> Abdullah,
>>>
>>> If I remember correctly whenever a secondary open index is created all
>>> existing records would be casted to a proper type to ensure that the index
>>> creation is valid.
>>> As for the overall correctness of casting operation, semantically
>>> creating an open index is the same thing as altering the dataset type. The
>>> current implementation allows only one open index of particular type
>>> created on a single field. If we would have had “alter datatype”
>>> functionality the open indexing would not be required at all.
>>>
>>> > On Sep 21, 2015, at 23:25, abdullah alamoudi <am...@apache.org>
>>> wrote:
>>> >
>>> > More thoughts:
>>> > I assume the intention of the cast was just to make sure if the open
>>> field
>>> > exists, it is of the specified type. Moreover, the un-casted record
>>> should
>>> > be inserted into the index.
>>> > If my assumptions are not correct, please, let me know ASAP.
>>> >
>>> > I have two thoughts on this:
>>> > 1. Actually, insert plans show that the records being inserted into the
>>> > primary index is actually the casted record creating the issue
>>> described
>>> > above.
>>> >
>>> > 2. I don't believe this is the right way to ensure that the open field
>>> if
>>> > exists is of the right type. why not extract the field using field
>>> access
>>> > by name function and then verify the type using the field tag?
>>> >
>>> >
>>> >
>>> > On Tue, Sep 22, 2015 at 9:11 AM, abdullah alamoudi <am...@apache.org>
>>> > wrote:
>>> >
>>> >> Hi Dev, @Ildar,
>>> >>
>>> >> In the insert pipeline for datasets with open indexes, we introduce a
>>> cast
>>> >> function before the insert and so one would expect the records to
>>> look like
>>> >> the casted record type which I assume has {{the closed fields + a
>>> nullable
>>> >> field}}.
>>> >>
>>> >> The question is, what happens to the previously existing records?,
>>> since
>>> >> now the index has both, records of the original type and records of
>>> the
>>> >> casted type.
>>> >>
>>> >> Thanks,
>>> >> Abdullah.
>>> >>
>>>
>>> Best regards,
>>> Ildar
>>>
>>>
>>
>
Re: Question about open indexes
Posted by abdullah alamoudi <ba...@gmail.com>.
@Ildar,
If that is the case, then why do we cast as well after the primary index
insert operator. If all the records are casted already, then why is the
second cast needed?
For example, look at the following plan:
Statement:
insert into dataset OrdersOpen (
for $x in dataset Orders
return $x
);
Plan:
commit
-- COMMIT |PARTITIONED|
project ([$$3])
-- STREAM_PROJECT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
insert into idx_Orders_Custkey on tpch:OrdersOpen from [%0->$$7]
-- INDEX_INSERT_DELETE |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
project ([$$3, $$7])
-- STREAM_PROJECT |PARTITIONED|
assign [$$7] <- [function-call: asterix:field-access-by-index,
Args:[function-call: asterix:cast-record, Args:[%0->$$4], AInt32: {8}]]
-- ASSIGN |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
insert into tpch:OrdersOpen from %0->$$4 partitioned by
[%0->$$3]
-- INSERT_DELETE |PARTITIONED|
exchange
-- HASH_PARTITION_EXCHANGE [$$3] |PARTITIONED|
assign [$$3] <- [function-call:
asterix:field-access-by-index, Args:[%0->$$4, AInt32: {0}]]
-- ASSIGN |PARTITIONED|
project ([$$4])
-- STREAM_PROJECT |PARTITIONED|
assign [$$4] <- [function-call:
asterix:cast-record, Args:[%0->$$0]]
-- ASSIGN |PARTITIONED|
project ([$$0])
-- STREAM_PROJECT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
data-scan []<-[$$5, $$0] <- tpch:Orders
-- DATASOURCE_SCAN |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
empty-tuple-source
-- EMPTY_TUPLE_SOURCE |PARTITIONED|
what is the point of the cast in red?
Amoudi, Abdullah.
On Tue, Sep 22, 2015 at 10:18 AM, abdullah alamoudi <ba...@gmail.com>
wrote:
> I see.
> Thanks Ildar,
>
> Abdullah.
>
> Amoudi, Abdullah.
>
> On Tue, Sep 22, 2015 at 10:04 AM, Ildar Absalyamov <
> ildar.absalyamov@gmail.com> wrote:
>
>> Abdullah,
>>
>> If I remember correctly whenever a secondary open index is created all
>> existing records would be casted to a proper type to ensure that the index
>> creation is valid.
>> As for the overall correctness of casting operation, semantically
>> creating an open index is the same thing as altering the dataset type. The
>> current implementation allows only one open index of particular type
>> created on a single field. If we would have had “alter datatype”
>> functionality the open indexing would not be required at all.
>>
>> > On Sep 21, 2015, at 23:25, abdullah alamoudi <am...@apache.org> wrote:
>> >
>> > More thoughts:
>> > I assume the intention of the cast was just to make sure if the open
>> field
>> > exists, it is of the specified type. Moreover, the un-casted record
>> should
>> > be inserted into the index.
>> > If my assumptions are not correct, please, let me know ASAP.
>> >
>> > I have two thoughts on this:
>> > 1. Actually, insert plans show that the records being inserted into the
>> > primary index is actually the casted record creating the issue described
>> > above.
>> >
>> > 2. I don't believe this is the right way to ensure that the open field
>> if
>> > exists is of the right type. why not extract the field using field
>> access
>> > by name function and then verify the type using the field tag?
>> >
>> >
>> >
>> > On Tue, Sep 22, 2015 at 9:11 AM, abdullah alamoudi <am...@apache.org>
>> > wrote:
>> >
>> >> Hi Dev, @Ildar,
>> >>
>> >> In the insert pipeline for datasets with open indexes, we introduce a
>> cast
>> >> function before the insert and so one would expect the records to look
>> like
>> >> the casted record type which I assume has {{the closed fields + a
>> nullable
>> >> field}}.
>> >>
>> >> The question is, what happens to the previously existing records?,
>> since
>> >> now the index has both, records of the original type and records of the
>> >> casted type.
>> >>
>> >> Thanks,
>> >> Abdullah.
>> >>
>>
>> Best regards,
>> Ildar
>>
>>
>
Re: Question about open indexes
Posted by abdullah alamoudi <ba...@gmail.com>.
I see.
Thanks Ildar,
Abdullah.
Amoudi, Abdullah.
On Tue, Sep 22, 2015 at 10:04 AM, Ildar Absalyamov <
ildar.absalyamov@gmail.com> wrote:
> Abdullah,
>
> If I remember correctly whenever a secondary open index is created all
> existing records would be casted to a proper type to ensure that the index
> creation is valid.
> As for the overall correctness of casting operation, semantically creating
> an open index is the same thing as altering the dataset type. The current
> implementation allows only one open index of particular type created on a
> single field. If we would have had “alter datatype” functionality the open
> indexing would not be required at all.
>
> > On Sep 21, 2015, at 23:25, abdullah alamoudi <am...@apache.org> wrote:
> >
> > More thoughts:
> > I assume the intention of the cast was just to make sure if the open
> field
> > exists, it is of the specified type. Moreover, the un-casted record
> should
> > be inserted into the index.
> > If my assumptions are not correct, please, let me know ASAP.
> >
> > I have two thoughts on this:
> > 1. Actually, insert plans show that the records being inserted into the
> > primary index is actually the casted record creating the issue described
> > above.
> >
> > 2. I don't believe this is the right way to ensure that the open field if
> > exists is of the right type. why not extract the field using field access
> > by name function and then verify the type using the field tag?
> >
> >
> >
> > On Tue, Sep 22, 2015 at 9:11 AM, abdullah alamoudi <am...@apache.org>
> > wrote:
> >
> >> Hi Dev, @Ildar,
> >>
> >> In the insert pipeline for datasets with open indexes, we introduce a
> cast
> >> function before the insert and so one would expect the records to look
> like
> >> the casted record type which I assume has {{the closed fields + a
> nullable
> >> field}}.
> >>
> >> The question is, what happens to the previously existing records?, since
> >> now the index has both, records of the original type and records of the
> >> casted type.
> >>
> >> Thanks,
> >> Abdullah.
> >>
>
> Best regards,
> Ildar
>
>
Re: Question about open indexes
Posted by Till Westmann <ti...@apache.org>.
Sounds good. My main intent was to find out if I had missed it :)
Cheers,
Till
On 28 Sep 2015, at 13:14, Ildar Absalyamov wrote:
> Not yet. Was planning to have one in few days though.
>
>> On Sep 28, 2015, at 13:00, Till Westmann <ti...@apache.org> wrote:
>>
>> Ok, makes sense. We also don’t have a fix for it, right?
>>
>> Cheers,
>> Till
>>
>> On 28 Sep 2015, at 12:15, Ian Maxon wrote:
>>
>>> I think the general thought that is unfortunately the fix for this
>>> can't go into 0.8.7 as it is right now, because it needs changes to
>>> Hyracks first. We should probably do a bugfix release right after.
>>>
>>> -Ian
>>>
>>> On Mon, Sep 28, 2015 at 11:51 AM, Till Westmann <ti...@apache.org>
>>> wrote:
>>>> I’ve added the “0.8.7-blocker” to
>>>> https://issues.apache.org/jira/browse/ASTERIXDB-1109 (which I
>>>> believe covers
>>>> this issue).
>>>> Is this what we agree on right now?
>>>>
>>>> Also, do we already have a review for this?
>>>>
>>>> Thanks,
>>>> Till
>>>>
>>>>
>>>> On 26 Sep 2015, at 1:37, abdullah alamoudi wrote:
>>>>
>>>>> I agree with Chen especially with the system not yet production
>>>>> ready.
>>>>> It seems that going through with the release is more important.
>>>>>
>>>>> Cheers,
>>>>>
>>>>>
>>>>> Amoudi, Abdullah.
>>>>>
>>>>> On Sat, Sep 26, 2015 at 3:33 AM, Chen Li <ch...@gmail.com> wrote:
>>>>>
>>>>>> I vote for including this fix in the next Asterxi/Hyracks
>>>>>> release, not
>>>>>> this
>>>>>> one.
>>>>>>
>>>>>> Chen
>>>>>>
>>>>>> On Fri, Sep 25, 2015 at 4:23 PM, Ildar Absalyamov <
>>>>>> ildar.absalyamov@gmail.com> wrote:
>>>>>>
>>>>>>> It did not really occur to me during today during the meeting,
>>>>>>> but
>>>>>>
>>>>>> Preston
>>>>>>>
>>>>>>> pointed out that the secondary index delete fix, that I
>>>>>>> proposed, spans
>>>>>>> both Hyracks & Asterix codebase. Thus we will either have to
>>>>>>> release
>>>>>>> Hyracks once again, or bite the bullet, sign the RC without this
>>>>>>> fixing
>>>>>>> this issue and create bug-fix releases for both Hyracks&Asterix
>>>>>>> right
>>>>>>
>>>>>> after.
>>>>>>>
>>>>>>>
>>>>>>>> On Sep 22, 2015, at 22:27, Mike Carey <dt...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Ah - that makes sense now. Thx. (And welcome back. :-))
>>>>>>>>
>>>>>>>> On 9/22/15 10:02 PM, Ildar Absalyamov wrote:
>>>>>>>>>
>>>>>>>>> Sorry for confusion, my initial answer was not correct enough,
>>>>>>
>>>>>> probably
>>>>>>>
>>>>>>> should have waited sometime after I drove 1500 miles form
>>>>>>> Seattle :)
>>>>>>>>>
>>>>>>>>> The casting in the insert pipeline, which Abdullah mentioned,
>>>>>>>>> is
>>>>>>
>>>>>> needed
>>>>>>>
>>>>>>> only for secondary index insert. The reasoning behind this
>>>>>>> casting is to
>>>>>>> ensure that the record is equivalent, thus it is safe to create
>>>>>>> an open
>>>>>>> index. It is true that we can get <Pk, Sk> pairs out of original
>>>>>>> record
>>>>>>> using get-field-by-name\index, but the cast operator is
>>>>>>> introduced
>>>>>>> merely
>>>>>>> to kill the pipeline if the dataset input is not correct.
>>>>>>>>>
>>>>>>>>> Thus the records in primary are never touched of modified, not
>>>>>>>>> matter
>>>>>>>
>>>>>>> what indexes were created.
>>>>>>>>>
>>>>>>>>> I am not sure however what is the second cast in Abdullah’s
>>>>>>>>> plan, and
>>>>>>>
>>>>>>> where is comes from.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> @Taewoo, so scan-delete-btree-secondary-index-open test does
>>>>>>>>> not
>>>>>>>
>>>>>>> actually delete data from the secondary index? I have checked
>>>>>>> the plan
>>>>>>
>>>>>> and
>>>>>>>
>>>>>>> it has the delete operator. Maybe it is initialized with wrong
>>>>>>
>>>>>> parameters,
>>>>>>>
>>>>>>> I’ll have a close look.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Sep 22, 2015, at 18:33, Mike Carey <dt...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Sounds kinda bad! Also, I wonder what happens when the
>>>>>>>>>> compiler
>>>>>>>
>>>>>>> encounters records in the dataset - whose type in the catalog
>>>>>>> doesn't
>>>>>>
>>>>>> claim
>>>>>>>
>>>>>>> to have a given (but now indexed) open field - e.g., during a
>>>>>>> data scan
>>>>>>
>>>>>> or
>>>>>>>
>>>>>>> an access via some other path? Can Bad Things Happen due to the
>>>>>>> compiler
>>>>>>> not properly anticipating the casted form of the records?
>>>>>>> (Maybe I am
>>>>>>> misunderstanding something, but we should probably take a
>>>>>>> careful look
>>>>>>> at
>>>>>>> the test cases - and make sure we do things like add a bunch of
>>>>>>> records,
>>>>>>> then add such an index, then add some more records, then
>>>>>>> stress-test
>>>>>>> type-related things that come at the dataset (i) thru the index,
>>>>>>> (ii)
>>>>>>
>>>>>> thru
>>>>>>>
>>>>>>> a primary dataset scan, and (iii) thru some other index.)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 9/22/15 4:06 PM, Taewoo Kim wrote:
>>>>>>>>>>>
>>>>>>>>>>> I think this issue:
>>>>>>>
>>>>>>> https://issues.apache.org/jira/browse/ASTERIXDB-1109 is
>>>>>>>>>>>
>>>>>>>>>>> related. Currently, index entries (SK, PK) are not deleted
>>>>>>>>>>> on an
>>>>>>>
>>>>>>> open-type
>>>>>>>>>>>
>>>>>>>>>>> secondary index during a deletion. This issue was not
>>>>>>>>>>> surfaced due
>>>>>>
>>>>>> to
>>>>>>>
>>>>>>> the
>>>>>>>>>>>
>>>>>>>>>>> fact that every search after a secondary index search had to
>>>>>>>>>>> go
>>>>>>>
>>>>>>> through the
>>>>>>>>>>>
>>>>>>>>>>> primary index lookup.
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> Taewoo
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Sep 22, 2015 at 12:04 AM, Ildar Absalyamov <
>>>>>>>>>>> ildar.absalyamov@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Abdullah,
>>>>>>>>>>>>
>>>>>>>>>>>> If I remember correctly whenever a secondary open index is
>>>>>>>>>>>> created
>>>>>>>
>>>>>>> all
>>>>>>>>>>>>
>>>>>>>>>>>> existing records would be casted to a proper type to ensure
>>>>>>>>>>>> that
>>>>>>
>>>>>> the
>>>>>>>
>>>>>>> index
>>>>>>>>>>>>
>>>>>>>>>>>> creation is valid.
>>>>>>>>>>>> As for the overall correctness of casting operation,
>>>>>>>>>>>> semantically
>>>>>>>
>>>>>>> creating
>>>>>>>>>>>>
>>>>>>>>>>>> an open index is the same thing as altering the dataset
>>>>>>>>>>>> type. The
>>>>>>>
>>>>>>> current
>>>>>>>>>>>>
>>>>>>>>>>>> implementation allows only one open index of particular
>>>>>>>>>>>> type
>>>>>>
>>>>>> created
>>>>>>>
>>>>>>> on a
>>>>>>>>>>>>
>>>>>>>>>>>> single field. If we would have had “alter datatype”
>>>>>>>>>>>> functionality
>>>>>>>
>>>>>>> the open
>>>>>>>>>>>>
>>>>>>>>>>>> indexing would not be required at all.
>>>>>>>>>>>>
>>>>>>>>>>>>> On Sep 21, 2015, at 23:25, abdullah alamoudi
>>>>>>>>>>>>> <am...@apache.org>
>>>>>>>
>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> More thoughts:
>>>>>>>>>>>>> I assume the intention of the cast was just to make sure
>>>>>>>>>>>>> if the
>>>>>>
>>>>>> open
>>>>>>>>>>>>
>>>>>>>>>>>> field
>>>>>>>>>>>>>
>>>>>>>>>>>>> exists, it is of the specified type. Moreover, the
>>>>>>>>>>>>> un-casted
>>>>>>
>>>>>> record
>>>>>>>>>>>>
>>>>>>>>>>>> should
>>>>>>>>>>>>>
>>>>>>>>>>>>> be inserted into the index.
>>>>>>>>>>>>> If my assumptions are not correct, please, let me know
>>>>>>>>>>>>> ASAP.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have two thoughts on this:
>>>>>>>>>>>>> 1. Actually, insert plans show that the records being
>>>>>>>>>>>>> inserted
>>>>>>
>>>>>> into
>>>>>>>
>>>>>>> the
>>>>>>>>>>>>>
>>>>>>>>>>>>> primary index is actually the casted record creating the
>>>>>>>>>>>>> issue
>>>>>>>
>>>>>>> described
>>>>>>>>>>>>>
>>>>>>>>>>>>> above.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2. I don't believe this is the right way to ensure that
>>>>>>>>>>>>> the open
>>>>>>>
>>>>>>> field if
>>>>>>>>>>>>>
>>>>>>>>>>>>> exists is of the right type. why not extract the field
>>>>>>>>>>>>> using field
>>>>>>>
>>>>>>> access
>>>>>>>>>>>>>
>>>>>>>>>>>>> by name function and then verify the type using the field
>>>>>>>>>>>>> tag?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Sep 22, 2015 at 9:11 AM, abdullah alamoudi <
>>>>>>>
>>>>>>> amoudi@apache.org>
>>>>>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Dev, @Ildar,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> In the insert pipeline for datasets with open indexes, we
>>>>>>>
>>>>>>> introduce a
>>>>>>>>>>>>
>>>>>>>>>>>> cast
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> function before the insert and so one would expect the
>>>>>>>>>>>>>> records to
>>>>>>>
>>>>>>> look
>>>>>>>>>>>>
>>>>>>>>>>>> like
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> the casted record type which I assume has {{the closed
>>>>>>>>>>>>>> fields + a
>>>>>>>>>>>>
>>>>>>>>>>>> nullable
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> field}}.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The question is, what happens to the previously existing
>>>>>>
>>>>>> records?,
>>>>>>>
>>>>>>> since
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> now the index has both, records of the original type and
>>>>>>>>>>>>>> records
>>>>>>>
>>>>>>> of the
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> casted type.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Abdullah.
>>>>>>>>>>>>>>
>>>>>>>>>>>> Best regards,
>>>>>>>>>>>> Ildar
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>> Best regards,
>>>>>>>>> Ildar
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> Best regards,
>>>>>>> Ildar
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>
> Best regards,
> Ildar
Re: Question about open indexes
Posted by Ildar Absalyamov <il...@gmail.com>.
Not yet. Was planning to have one in few days though.
> On Sep 28, 2015, at 13:00, Till Westmann <ti...@apache.org> wrote:
>
> Ok, makes sense. We also don’t have a fix for it, right?
>
> Cheers,
> Till
>
> On 28 Sep 2015, at 12:15, Ian Maxon wrote:
>
>> I think the general thought that is unfortunately the fix for this
>> can't go into 0.8.7 as it is right now, because it needs changes to
>> Hyracks first. We should probably do a bugfix release right after.
>>
>> -Ian
>>
>> On Mon, Sep 28, 2015 at 11:51 AM, Till Westmann <ti...@apache.org> wrote:
>>> I’ve added the “0.8.7-blocker” to
>>> https://issues.apache.org/jira/browse/ASTERIXDB-1109 (which I believe covers
>>> this issue).
>>> Is this what we agree on right now?
>>>
>>> Also, do we already have a review for this?
>>>
>>> Thanks,
>>> Till
>>>
>>>
>>> On 26 Sep 2015, at 1:37, abdullah alamoudi wrote:
>>>
>>>> I agree with Chen especially with the system not yet production ready.
>>>> It seems that going through with the release is more important.
>>>>
>>>> Cheers,
>>>>
>>>>
>>>> Amoudi, Abdullah.
>>>>
>>>> On Sat, Sep 26, 2015 at 3:33 AM, Chen Li <ch...@gmail.com> wrote:
>>>>
>>>>> I vote for including this fix in the next Asterxi/Hyracks release, not
>>>>> this
>>>>> one.
>>>>>
>>>>> Chen
>>>>>
>>>>> On Fri, Sep 25, 2015 at 4:23 PM, Ildar Absalyamov <
>>>>> ildar.absalyamov@gmail.com> wrote:
>>>>>
>>>>>> It did not really occur to me during today during the meeting, but
>>>>>
>>>>> Preston
>>>>>>
>>>>>> pointed out that the secondary index delete fix, that I proposed, spans
>>>>>> both Hyracks & Asterix codebase. Thus we will either have to release
>>>>>> Hyracks once again, or bite the bullet, sign the RC without this fixing
>>>>>> this issue and create bug-fix releases for both Hyracks&Asterix right
>>>>>
>>>>> after.
>>>>>>
>>>>>>
>>>>>>> On Sep 22, 2015, at 22:27, Mike Carey <dt...@gmail.com> wrote:
>>>>>>>
>>>>>>> Ah - that makes sense now. Thx. (And welcome back. :-))
>>>>>>>
>>>>>>> On 9/22/15 10:02 PM, Ildar Absalyamov wrote:
>>>>>>>>
>>>>>>>> Sorry for confusion, my initial answer was not correct enough,
>>>>>
>>>>> probably
>>>>>>
>>>>>> should have waited sometime after I drove 1500 miles form Seattle :)
>>>>>>>>
>>>>>>>> The casting in the insert pipeline, which Abdullah mentioned, is
>>>>>
>>>>> needed
>>>>>>
>>>>>> only for secondary index insert. The reasoning behind this casting is to
>>>>>> ensure that the record is equivalent, thus it is safe to create an open
>>>>>> index. It is true that we can get <Pk, Sk> pairs out of original record
>>>>>> using get-field-by-name\index, but the cast operator is introduced
>>>>>> merely
>>>>>> to kill the pipeline if the dataset input is not correct.
>>>>>>>>
>>>>>>>> Thus the records in primary are never touched of modified, not matter
>>>>>>
>>>>>> what indexes were created.
>>>>>>>>
>>>>>>>> I am not sure however what is the second cast in Abdullah’s plan, and
>>>>>>
>>>>>> where is comes from.
>>>>>>>>
>>>>>>>>
>>>>>>>> @Taewoo, so scan-delete-btree-secondary-index-open test does not
>>>>>>
>>>>>> actually delete data from the secondary index? I have checked the plan
>>>>>
>>>>> and
>>>>>>
>>>>>> it has the delete operator. Maybe it is initialized with wrong
>>>>>
>>>>> parameters,
>>>>>>
>>>>>> I’ll have a close look.
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Sep 22, 2015, at 18:33, Mike Carey <dt...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> Sounds kinda bad! Also, I wonder what happens when the compiler
>>>>>>
>>>>>> encounters records in the dataset - whose type in the catalog doesn't
>>>>>
>>>>> claim
>>>>>>
>>>>>> to have a given (but now indexed) open field - e.g., during a data scan
>>>>>
>>>>> or
>>>>>>
>>>>>> an access via some other path? Can Bad Things Happen due to the
>>>>>> compiler
>>>>>> not properly anticipating the casted form of the records? (Maybe I am
>>>>>> misunderstanding something, but we should probably take a careful look
>>>>>> at
>>>>>> the test cases - and make sure we do things like add a bunch of records,
>>>>>> then add such an index, then add some more records, then stress-test
>>>>>> type-related things that come at the dataset (i) thru the index, (ii)
>>>>>
>>>>> thru
>>>>>>
>>>>>> a primary dataset scan, and (iii) thru some other index.)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 9/22/15 4:06 PM, Taewoo Kim wrote:
>>>>>>>>>>
>>>>>>>>>> I think this issue:
>>>>>>
>>>>>> https://issues.apache.org/jira/browse/ASTERIXDB-1109 is
>>>>>>>>>>
>>>>>>>>>> related. Currently, index entries (SK, PK) are not deleted on an
>>>>>>
>>>>>> open-type
>>>>>>>>>>
>>>>>>>>>> secondary index during a deletion. This issue was not surfaced due
>>>>>
>>>>> to
>>>>>>
>>>>>> the
>>>>>>>>>>
>>>>>>>>>> fact that every search after a secondary index search had to go
>>>>>>
>>>>>> through the
>>>>>>>>>>
>>>>>>>>>> primary index lookup.
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> Taewoo
>>>>>>>>>>
>>>>>>>>>> On Tue, Sep 22, 2015 at 12:04 AM, Ildar Absalyamov <
>>>>>>>>>> ildar.absalyamov@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Abdullah,
>>>>>>>>>>>
>>>>>>>>>>> If I remember correctly whenever a secondary open index is created
>>>>>>
>>>>>> all
>>>>>>>>>>>
>>>>>>>>>>> existing records would be casted to a proper type to ensure that
>>>>>
>>>>> the
>>>>>>
>>>>>> index
>>>>>>>>>>>
>>>>>>>>>>> creation is valid.
>>>>>>>>>>> As for the overall correctness of casting operation, semantically
>>>>>>
>>>>>> creating
>>>>>>>>>>>
>>>>>>>>>>> an open index is the same thing as altering the dataset type. The
>>>>>>
>>>>>> current
>>>>>>>>>>>
>>>>>>>>>>> implementation allows only one open index of particular type
>>>>>
>>>>> created
>>>>>>
>>>>>> on a
>>>>>>>>>>>
>>>>>>>>>>> single field. If we would have had “alter datatype” functionality
>>>>>>
>>>>>> the open
>>>>>>>>>>>
>>>>>>>>>>> indexing would not be required at all.
>>>>>>>>>>>
>>>>>>>>>>>> On Sep 21, 2015, at 23:25, abdullah alamoudi <am...@apache.org>
>>>>>>
>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> More thoughts:
>>>>>>>>>>>> I assume the intention of the cast was just to make sure if the
>>>>>
>>>>> open
>>>>>>>>>>>
>>>>>>>>>>> field
>>>>>>>>>>>>
>>>>>>>>>>>> exists, it is of the specified type. Moreover, the un-casted
>>>>>
>>>>> record
>>>>>>>>>>>
>>>>>>>>>>> should
>>>>>>>>>>>>
>>>>>>>>>>>> be inserted into the index.
>>>>>>>>>>>> If my assumptions are not correct, please, let me know ASAP.
>>>>>>>>>>>>
>>>>>>>>>>>> I have two thoughts on this:
>>>>>>>>>>>> 1. Actually, insert plans show that the records being inserted
>>>>>
>>>>> into
>>>>>>
>>>>>> the
>>>>>>>>>>>>
>>>>>>>>>>>> primary index is actually the casted record creating the issue
>>>>>>
>>>>>> described
>>>>>>>>>>>>
>>>>>>>>>>>> above.
>>>>>>>>>>>>
>>>>>>>>>>>> 2. I don't believe this is the right way to ensure that the open
>>>>>>
>>>>>> field if
>>>>>>>>>>>>
>>>>>>>>>>>> exists is of the right type. why not extract the field using field
>>>>>>
>>>>>> access
>>>>>>>>>>>>
>>>>>>>>>>>> by name function and then verify the type using the field tag?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Sep 22, 2015 at 9:11 AM, abdullah alamoudi <
>>>>>>
>>>>>> amoudi@apache.org>
>>>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Dev, @Ildar,
>>>>>>>>>>>>>
>>>>>>>>>>>>> In the insert pipeline for datasets with open indexes, we
>>>>>>
>>>>>> introduce a
>>>>>>>>>>>
>>>>>>>>>>> cast
>>>>>>>>>>>>>
>>>>>>>>>>>>> function before the insert and so one would expect the records to
>>>>>>
>>>>>> look
>>>>>>>>>>>
>>>>>>>>>>> like
>>>>>>>>>>>>>
>>>>>>>>>>>>> the casted record type which I assume has {{the closed fields + a
>>>>>>>>>>>
>>>>>>>>>>> nullable
>>>>>>>>>>>>>
>>>>>>>>>>>>> field}}.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The question is, what happens to the previously existing
>>>>>
>>>>> records?,
>>>>>>
>>>>>> since
>>>>>>>>>>>>>
>>>>>>>>>>>>> now the index has both, records of the original type and records
>>>>>>
>>>>>> of the
>>>>>>>>>>>>>
>>>>>>>>>>>>> casted type.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Abdullah.
>>>>>>>>>>>>>
>>>>>>>>>>> Best regards,
>>>>>>>>>>> Ildar
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>> Ildar
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> Best regards,
>>>>>> Ildar
>>>>>>
>>>>>>
>>>>>
>>>
Best regards,
Ildar
Re: Question about open indexes
Posted by Till Westmann <ti...@apache.org>.
Ok, makes sense. We also don’t have a fix for it, right?
Cheers,
Till
On 28 Sep 2015, at 12:15, Ian Maxon wrote:
> I think the general thought that is unfortunately the fix for this
> can't go into 0.8.7 as it is right now, because it needs changes to
> Hyracks first. We should probably do a bugfix release right after.
>
> -Ian
>
> On Mon, Sep 28, 2015 at 11:51 AM, Till Westmann <ti...@apache.org>
> wrote:
>> I’ve added the “0.8.7-blocker” to
>> https://issues.apache.org/jira/browse/ASTERIXDB-1109 (which I believe
>> covers
>> this issue).
>> Is this what we agree on right now?
>>
>> Also, do we already have a review for this?
>>
>> Thanks,
>> Till
>>
>>
>> On 26 Sep 2015, at 1:37, abdullah alamoudi wrote:
>>
>>> I agree with Chen especially with the system not yet production
>>> ready.
>>> It seems that going through with the release is more important.
>>>
>>> Cheers,
>>>
>>>
>>> Amoudi, Abdullah.
>>>
>>> On Sat, Sep 26, 2015 at 3:33 AM, Chen Li <ch...@gmail.com> wrote:
>>>
>>>> I vote for including this fix in the next Asterxi/Hyracks release,
>>>> not
>>>> this
>>>> one.
>>>>
>>>> Chen
>>>>
>>>> On Fri, Sep 25, 2015 at 4:23 PM, Ildar Absalyamov <
>>>> ildar.absalyamov@gmail.com> wrote:
>>>>
>>>>> It did not really occur to me during today during the meeting, but
>>>>
>>>> Preston
>>>>>
>>>>> pointed out that the secondary index delete fix, that I proposed,
>>>>> spans
>>>>> both Hyracks & Asterix codebase. Thus we will either have to
>>>>> release
>>>>> Hyracks once again, or bite the bullet, sign the RC without this
>>>>> fixing
>>>>> this issue and create bug-fix releases for both Hyracks&Asterix
>>>>> right
>>>>
>>>> after.
>>>>>
>>>>>
>>>>>> On Sep 22, 2015, at 22:27, Mike Carey <dt...@gmail.com> wrote:
>>>>>>
>>>>>> Ah - that makes sense now. Thx. (And welcome back. :-))
>>>>>>
>>>>>> On 9/22/15 10:02 PM, Ildar Absalyamov wrote:
>>>>>>>
>>>>>>> Sorry for confusion, my initial answer was not correct enough,
>>>>
>>>> probably
>>>>>
>>>>> should have waited sometime after I drove 1500 miles form Seattle
>>>>> :)
>>>>>>>
>>>>>>> The casting in the insert pipeline, which Abdullah mentioned, is
>>>>
>>>> needed
>>>>>
>>>>> only for secondary index insert. The reasoning behind this casting
>>>>> is to
>>>>> ensure that the record is equivalent, thus it is safe to create an
>>>>> open
>>>>> index. It is true that we can get <Pk, Sk> pairs out of original
>>>>> record
>>>>> using get-field-by-name\index, but the cast operator is introduced
>>>>> merely
>>>>> to kill the pipeline if the dataset input is not correct.
>>>>>>>
>>>>>>> Thus the records in primary are never touched of modified, not
>>>>>>> matter
>>>>>
>>>>> what indexes were created.
>>>>>>>
>>>>>>> I am not sure however what is the second cast in Abdullah’s
>>>>>>> plan, and
>>>>>
>>>>> where is comes from.
>>>>>>>
>>>>>>>
>>>>>>> @Taewoo, so scan-delete-btree-secondary-index-open test does not
>>>>>
>>>>> actually delete data from the secondary index? I have checked the
>>>>> plan
>>>>
>>>> and
>>>>>
>>>>> it has the delete operator. Maybe it is initialized with wrong
>>>>
>>>> parameters,
>>>>>
>>>>> I’ll have a close look.
>>>>>>>
>>>>>>>
>>>>>>>> On Sep 22, 2015, at 18:33, Mike Carey <dt...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Sounds kinda bad! Also, I wonder what happens when the
>>>>>>>> compiler
>>>>>
>>>>> encounters records in the dataset - whose type in the catalog
>>>>> doesn't
>>>>
>>>> claim
>>>>>
>>>>> to have a given (but now indexed) open field - e.g., during a data
>>>>> scan
>>>>
>>>> or
>>>>>
>>>>> an access via some other path? Can Bad Things Happen due to the
>>>>> compiler
>>>>> not properly anticipating the casted form of the records? (Maybe
>>>>> I am
>>>>> misunderstanding something, but we should probably take a careful
>>>>> look
>>>>> at
>>>>> the test cases - and make sure we do things like add a bunch of
>>>>> records,
>>>>> then add such an index, then add some more records, then
>>>>> stress-test
>>>>> type-related things that come at the dataset (i) thru the index,
>>>>> (ii)
>>>>
>>>> thru
>>>>>
>>>>> a primary dataset scan, and (iii) thru some other index.)
>>>>>>>>
>>>>>>>>
>>>>>>>> On 9/22/15 4:06 PM, Taewoo Kim wrote:
>>>>>>>>>
>>>>>>>>> I think this issue:
>>>>>
>>>>> https://issues.apache.org/jira/browse/ASTERIXDB-1109 is
>>>>>>>>>
>>>>>>>>> related. Currently, index entries (SK, PK) are not deleted on
>>>>>>>>> an
>>>>>
>>>>> open-type
>>>>>>>>>
>>>>>>>>> secondary index during a deletion. This issue was not surfaced
>>>>>>>>> due
>>>>
>>>> to
>>>>>
>>>>> the
>>>>>>>>>
>>>>>>>>> fact that every search after a secondary index search had to
>>>>>>>>> go
>>>>>
>>>>> through the
>>>>>>>>>
>>>>>>>>> primary index lookup.
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Taewoo
>>>>>>>>>
>>>>>>>>> On Tue, Sep 22, 2015 at 12:04 AM, Ildar Absalyamov <
>>>>>>>>> ildar.absalyamov@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Abdullah,
>>>>>>>>>>
>>>>>>>>>> If I remember correctly whenever a secondary open index is
>>>>>>>>>> created
>>>>>
>>>>> all
>>>>>>>>>>
>>>>>>>>>> existing records would be casted to a proper type to ensure
>>>>>>>>>> that
>>>>
>>>> the
>>>>>
>>>>> index
>>>>>>>>>>
>>>>>>>>>> creation is valid.
>>>>>>>>>> As for the overall correctness of casting operation,
>>>>>>>>>> semantically
>>>>>
>>>>> creating
>>>>>>>>>>
>>>>>>>>>> an open index is the same thing as altering the dataset type.
>>>>>>>>>> The
>>>>>
>>>>> current
>>>>>>>>>>
>>>>>>>>>> implementation allows only one open index of particular type
>>>>
>>>> created
>>>>>
>>>>> on a
>>>>>>>>>>
>>>>>>>>>> single field. If we would have had “alter datatype”
>>>>>>>>>> functionality
>>>>>
>>>>> the open
>>>>>>>>>>
>>>>>>>>>> indexing would not be required at all.
>>>>>>>>>>
>>>>>>>>>>> On Sep 21, 2015, at 23:25, abdullah alamoudi
>>>>>>>>>>> <am...@apache.org>
>>>>>
>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> More thoughts:
>>>>>>>>>>> I assume the intention of the cast was just to make sure if
>>>>>>>>>>> the
>>>>
>>>> open
>>>>>>>>>>
>>>>>>>>>> field
>>>>>>>>>>>
>>>>>>>>>>> exists, it is of the specified type. Moreover, the un-casted
>>>>
>>>> record
>>>>>>>>>>
>>>>>>>>>> should
>>>>>>>>>>>
>>>>>>>>>>> be inserted into the index.
>>>>>>>>>>> If my assumptions are not correct, please, let me know ASAP.
>>>>>>>>>>>
>>>>>>>>>>> I have two thoughts on this:
>>>>>>>>>>> 1. Actually, insert plans show that the records being
>>>>>>>>>>> inserted
>>>>
>>>> into
>>>>>
>>>>> the
>>>>>>>>>>>
>>>>>>>>>>> primary index is actually the casted record creating the
>>>>>>>>>>> issue
>>>>>
>>>>> described
>>>>>>>>>>>
>>>>>>>>>>> above.
>>>>>>>>>>>
>>>>>>>>>>> 2. I don't believe this is the right way to ensure that the
>>>>>>>>>>> open
>>>>>
>>>>> field if
>>>>>>>>>>>
>>>>>>>>>>> exists is of the right type. why not extract the field using
>>>>>>>>>>> field
>>>>>
>>>>> access
>>>>>>>>>>>
>>>>>>>>>>> by name function and then verify the type using the field
>>>>>>>>>>> tag?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Sep 22, 2015 at 9:11 AM, abdullah alamoudi <
>>>>>
>>>>> amoudi@apache.org>
>>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Dev, @Ildar,
>>>>>>>>>>>>
>>>>>>>>>>>> In the insert pipeline for datasets with open indexes, we
>>>>>
>>>>> introduce a
>>>>>>>>>>
>>>>>>>>>> cast
>>>>>>>>>>>>
>>>>>>>>>>>> function before the insert and so one would expect the
>>>>>>>>>>>> records to
>>>>>
>>>>> look
>>>>>>>>>>
>>>>>>>>>> like
>>>>>>>>>>>>
>>>>>>>>>>>> the casted record type which I assume has {{the closed
>>>>>>>>>>>> fields + a
>>>>>>>>>>
>>>>>>>>>> nullable
>>>>>>>>>>>>
>>>>>>>>>>>> field}}.
>>>>>>>>>>>>
>>>>>>>>>>>> The question is, what happens to the previously existing
>>>>
>>>> records?,
>>>>>
>>>>> since
>>>>>>>>>>>>
>>>>>>>>>>>> now the index has both, records of the original type and
>>>>>>>>>>>> records
>>>>>
>>>>> of the
>>>>>>>>>>>>
>>>>>>>>>>>> casted type.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Abdullah.
>>>>>>>>>>>>
>>>>>>>>>> Best regards,
>>>>>>>>>> Ildar
>>>>>>>>>>
>>>>>>>>>>
>>>>>>> Best regards,
>>>>>>> Ildar
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> Best regards,
>>>>> Ildar
>>>>>
>>>>>
>>>>
>>
Re: Question about open indexes
Posted by Ian Maxon <im...@uci.edu>.
I think the general thought that is unfortunately the fix for this
can't go into 0.8.7 as it is right now, because it needs changes to
Hyracks first. We should probably do a bugfix release right after.
-Ian
On Mon, Sep 28, 2015 at 11:51 AM, Till Westmann <ti...@apache.org> wrote:
> I’ve added the “0.8.7-blocker” to
> https://issues.apache.org/jira/browse/ASTERIXDB-1109 (which I believe covers
> this issue).
> Is this what we agree on right now?
>
> Also, do we already have a review for this?
>
> Thanks,
> Till
>
>
> On 26 Sep 2015, at 1:37, abdullah alamoudi wrote:
>
>> I agree with Chen especially with the system not yet production ready.
>> It seems that going through with the release is more important.
>>
>> Cheers,
>>
>>
>> Amoudi, Abdullah.
>>
>> On Sat, Sep 26, 2015 at 3:33 AM, Chen Li <ch...@gmail.com> wrote:
>>
>>> I vote for including this fix in the next Asterxi/Hyracks release, not
>>> this
>>> one.
>>>
>>> Chen
>>>
>>> On Fri, Sep 25, 2015 at 4:23 PM, Ildar Absalyamov <
>>> ildar.absalyamov@gmail.com> wrote:
>>>
>>>> It did not really occur to me during today during the meeting, but
>>>
>>> Preston
>>>>
>>>> pointed out that the secondary index delete fix, that I proposed, spans
>>>> both Hyracks & Asterix codebase. Thus we will either have to release
>>>> Hyracks once again, or bite the bullet, sign the RC without this fixing
>>>> this issue and create bug-fix releases for both Hyracks&Asterix right
>>>
>>> after.
>>>>
>>>>
>>>>> On Sep 22, 2015, at 22:27, Mike Carey <dt...@gmail.com> wrote:
>>>>>
>>>>> Ah - that makes sense now. Thx. (And welcome back. :-))
>>>>>
>>>>> On 9/22/15 10:02 PM, Ildar Absalyamov wrote:
>>>>>>
>>>>>> Sorry for confusion, my initial answer was not correct enough,
>>>
>>> probably
>>>>
>>>> should have waited sometime after I drove 1500 miles form Seattle :)
>>>>>>
>>>>>> The casting in the insert pipeline, which Abdullah mentioned, is
>>>
>>> needed
>>>>
>>>> only for secondary index insert. The reasoning behind this casting is to
>>>> ensure that the record is equivalent, thus it is safe to create an open
>>>> index. It is true that we can get <Pk, Sk> pairs out of original record
>>>> using get-field-by-name\index, but the cast operator is introduced
>>>> merely
>>>> to kill the pipeline if the dataset input is not correct.
>>>>>>
>>>>>> Thus the records in primary are never touched of modified, not matter
>>>>
>>>> what indexes were created.
>>>>>>
>>>>>> I am not sure however what is the second cast in Abdullah’s plan, and
>>>>
>>>> where is comes from.
>>>>>>
>>>>>>
>>>>>> @Taewoo, so scan-delete-btree-secondary-index-open test does not
>>>>
>>>> actually delete data from the secondary index? I have checked the plan
>>>
>>> and
>>>>
>>>> it has the delete operator. Maybe it is initialized with wrong
>>>
>>> parameters,
>>>>
>>>> I’ll have a close look.
>>>>>>
>>>>>>
>>>>>>> On Sep 22, 2015, at 18:33, Mike Carey <dt...@gmail.com> wrote:
>>>>>>>
>>>>>>> Sounds kinda bad! Also, I wonder what happens when the compiler
>>>>
>>>> encounters records in the dataset - whose type in the catalog doesn't
>>>
>>> claim
>>>>
>>>> to have a given (but now indexed) open field - e.g., during a data scan
>>>
>>> or
>>>>
>>>> an access via some other path? Can Bad Things Happen due to the
>>>> compiler
>>>> not properly anticipating the casted form of the records? (Maybe I am
>>>> misunderstanding something, but we should probably take a careful look
>>>> at
>>>> the test cases - and make sure we do things like add a bunch of records,
>>>> then add such an index, then add some more records, then stress-test
>>>> type-related things that come at the dataset (i) thru the index, (ii)
>>>
>>> thru
>>>>
>>>> a primary dataset scan, and (iii) thru some other index.)
>>>>>>>
>>>>>>>
>>>>>>> On 9/22/15 4:06 PM, Taewoo Kim wrote:
>>>>>>>>
>>>>>>>> I think this issue:
>>>>
>>>> https://issues.apache.org/jira/browse/ASTERIXDB-1109 is
>>>>>>>>
>>>>>>>> related. Currently, index entries (SK, PK) are not deleted on an
>>>>
>>>> open-type
>>>>>>>>
>>>>>>>> secondary index during a deletion. This issue was not surfaced due
>>>
>>> to
>>>>
>>>> the
>>>>>>>>
>>>>>>>> fact that every search after a secondary index search had to go
>>>>
>>>> through the
>>>>>>>>
>>>>>>>> primary index lookup.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Taewoo
>>>>>>>>
>>>>>>>> On Tue, Sep 22, 2015 at 12:04 AM, Ildar Absalyamov <
>>>>>>>> ildar.absalyamov@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Abdullah,
>>>>>>>>>
>>>>>>>>> If I remember correctly whenever a secondary open index is created
>>>>
>>>> all
>>>>>>>>>
>>>>>>>>> existing records would be casted to a proper type to ensure that
>>>
>>> the
>>>>
>>>> index
>>>>>>>>>
>>>>>>>>> creation is valid.
>>>>>>>>> As for the overall correctness of casting operation, semantically
>>>>
>>>> creating
>>>>>>>>>
>>>>>>>>> an open index is the same thing as altering the dataset type. The
>>>>
>>>> current
>>>>>>>>>
>>>>>>>>> implementation allows only one open index of particular type
>>>
>>> created
>>>>
>>>> on a
>>>>>>>>>
>>>>>>>>> single field. If we would have had “alter datatype” functionality
>>>>
>>>> the open
>>>>>>>>>
>>>>>>>>> indexing would not be required at all.
>>>>>>>>>
>>>>>>>>>> On Sep 21, 2015, at 23:25, abdullah alamoudi <am...@apache.org>
>>>>
>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> More thoughts:
>>>>>>>>>> I assume the intention of the cast was just to make sure if the
>>>
>>> open
>>>>>>>>>
>>>>>>>>> field
>>>>>>>>>>
>>>>>>>>>> exists, it is of the specified type. Moreover, the un-casted
>>>
>>> record
>>>>>>>>>
>>>>>>>>> should
>>>>>>>>>>
>>>>>>>>>> be inserted into the index.
>>>>>>>>>> If my assumptions are not correct, please, let me know ASAP.
>>>>>>>>>>
>>>>>>>>>> I have two thoughts on this:
>>>>>>>>>> 1. Actually, insert plans show that the records being inserted
>>>
>>> into
>>>>
>>>> the
>>>>>>>>>>
>>>>>>>>>> primary index is actually the casted record creating the issue
>>>>
>>>> described
>>>>>>>>>>
>>>>>>>>>> above.
>>>>>>>>>>
>>>>>>>>>> 2. I don't believe this is the right way to ensure that the open
>>>>
>>>> field if
>>>>>>>>>>
>>>>>>>>>> exists is of the right type. why not extract the field using field
>>>>
>>>> access
>>>>>>>>>>
>>>>>>>>>> by name function and then verify the type using the field tag?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Sep 22, 2015 at 9:11 AM, abdullah alamoudi <
>>>>
>>>> amoudi@apache.org>
>>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Dev, @Ildar,
>>>>>>>>>>>
>>>>>>>>>>> In the insert pipeline for datasets with open indexes, we
>>>>
>>>> introduce a
>>>>>>>>>
>>>>>>>>> cast
>>>>>>>>>>>
>>>>>>>>>>> function before the insert and so one would expect the records to
>>>>
>>>> look
>>>>>>>>>
>>>>>>>>> like
>>>>>>>>>>>
>>>>>>>>>>> the casted record type which I assume has {{the closed fields + a
>>>>>>>>>
>>>>>>>>> nullable
>>>>>>>>>>>
>>>>>>>>>>> field}}.
>>>>>>>>>>>
>>>>>>>>>>> The question is, what happens to the previously existing
>>>
>>> records?,
>>>>
>>>> since
>>>>>>>>>>>
>>>>>>>>>>> now the index has both, records of the original type and records
>>>>
>>>> of the
>>>>>>>>>>>
>>>>>>>>>>> casted type.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Abdullah.
>>>>>>>>>>>
>>>>>>>>> Best regards,
>>>>>>>>> Ildar
>>>>>>>>>
>>>>>>>>>
>>>>>> Best regards,
>>>>>> Ildar
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> Best regards,
>>>> Ildar
>>>>
>>>>
>>>
>
Re: Question about open indexes
Posted by Till Westmann <ti...@apache.org>.
I’ve added the “0.8.7-blocker” to
https://issues.apache.org/jira/browse/ASTERIXDB-1109 (which I believe
covers this issue).
Is this what we agree on right now?
Also, do we already have a review for this?
Thanks,
Till
On 26 Sep 2015, at 1:37, abdullah alamoudi wrote:
> I agree with Chen especially with the system not yet production ready.
> It seems that going through with the release is more important.
>
> Cheers,
>
>
> Amoudi, Abdullah.
>
> On Sat, Sep 26, 2015 at 3:33 AM, Chen Li <ch...@gmail.com> wrote:
>
>> I vote for including this fix in the next Asterxi/Hyracks release,
>> not this
>> one.
>>
>> Chen
>>
>> On Fri, Sep 25, 2015 at 4:23 PM, Ildar Absalyamov <
>> ildar.absalyamov@gmail.com> wrote:
>>
>>> It did not really occur to me during today during the meeting, but
>> Preston
>>> pointed out that the secondary index delete fix, that I proposed,
>>> spans
>>> both Hyracks & Asterix codebase. Thus we will either have to release
>>> Hyracks once again, or bite the bullet, sign the RC without this
>>> fixing
>>> this issue and create bug-fix releases for both Hyracks&Asterix
>>> right
>> after.
>>>
>>>> On Sep 22, 2015, at 22:27, Mike Carey <dt...@gmail.com> wrote:
>>>>
>>>> Ah - that makes sense now. Thx. (And welcome back. :-))
>>>>
>>>> On 9/22/15 10:02 PM, Ildar Absalyamov wrote:
>>>>> Sorry for confusion, my initial answer was not correct enough,
>> probably
>>> should have waited sometime after I drove 1500 miles form Seattle :)
>>>>> The casting in the insert pipeline, which Abdullah mentioned, is
>> needed
>>> only for secondary index insert. The reasoning behind this casting
>>> is to
>>> ensure that the record is equivalent, thus it is safe to create an
>>> open
>>> index. It is true that we can get <Pk, Sk> pairs out of original
>>> record
>>> using get-field-by-name\index, but the cast operator is introduced
>>> merely
>>> to kill the pipeline if the dataset input is not correct.
>>>>> Thus the records in primary are never touched of modified, not
>>>>> matter
>>> what indexes were created.
>>>>> I am not sure however what is the second cast in Abdullah’s
>>>>> plan, and
>>> where is comes from.
>>>>>
>>>>> @Taewoo, so scan-delete-btree-secondary-index-open test does not
>>> actually delete data from the secondary index? I have checked the
>>> plan
>> and
>>> it has the delete operator. Maybe it is initialized with wrong
>> parameters,
>>> I’ll have a close look.
>>>>>
>>>>>> On Sep 22, 2015, at 18:33, Mike Carey <dt...@gmail.com> wrote:
>>>>>>
>>>>>> Sounds kinda bad! Also, I wonder what happens when the compiler
>>> encounters records in the dataset - whose type in the catalog
>>> doesn't
>> claim
>>> to have a given (but now indexed) open field - e.g., during a data
>>> scan
>> or
>>> an access via some other path? Can Bad Things Happen due to the
>>> compiler
>>> not properly anticipating the casted form of the records? (Maybe I
>>> am
>>> misunderstanding something, but we should probably take a careful
>>> look at
>>> the test cases - and make sure we do things like add a bunch of
>>> records,
>>> then add such an index, then add some more records, then stress-test
>>> type-related things that come at the dataset (i) thru the index,
>>> (ii)
>> thru
>>> a primary dataset scan, and (iii) thru some other index.)
>>>>>>
>>>>>> On 9/22/15 4:06 PM, Taewoo Kim wrote:
>>>>>>> I think this issue:
>>> https://issues.apache.org/jira/browse/ASTERIXDB-1109 is
>>>>>>> related. Currently, index entries (SK, PK) are not deleted on an
>>> open-type
>>>>>>> secondary index during a deletion. This issue was not surfaced
>>>>>>> due
>> to
>>> the
>>>>>>> fact that every search after a secondary index search had to go
>>> through the
>>>>>>> primary index lookup.
>>>>>>>
>>>>>>> Best,
>>>>>>> Taewoo
>>>>>>>
>>>>>>> On Tue, Sep 22, 2015 at 12:04 AM, Ildar Absalyamov <
>>>>>>> ildar.absalyamov@gmail.com> wrote:
>>>>>>>
>>>>>>>> Abdullah,
>>>>>>>>
>>>>>>>> If I remember correctly whenever a secondary open index is
>>>>>>>> created
>>> all
>>>>>>>> existing records would be casted to a proper type to ensure
>>>>>>>> that
>> the
>>> index
>>>>>>>> creation is valid.
>>>>>>>> As for the overall correctness of casting operation,
>>>>>>>> semantically
>>> creating
>>>>>>>> an open index is the same thing as altering the dataset type.
>>>>>>>> The
>>> current
>>>>>>>> implementation allows only one open index of particular type
>> created
>>> on a
>>>>>>>> single field. If we would have had “alter datatype”
>>>>>>>> functionality
>>> the open
>>>>>>>> indexing would not be required at all.
>>>>>>>>
>>>>>>>>> On Sep 21, 2015, at 23:25, abdullah alamoudi
>>>>>>>>> <am...@apache.org>
>>> wrote:
>>>>>>>>>
>>>>>>>>> More thoughts:
>>>>>>>>> I assume the intention of the cast was just to make sure if
>>>>>>>>> the
>> open
>>>>>>>> field
>>>>>>>>> exists, it is of the specified type. Moreover, the un-casted
>> record
>>>>>>>> should
>>>>>>>>> be inserted into the index.
>>>>>>>>> If my assumptions are not correct, please, let me know ASAP.
>>>>>>>>>
>>>>>>>>> I have two thoughts on this:
>>>>>>>>> 1. Actually, insert plans show that the records being inserted
>> into
>>> the
>>>>>>>>> primary index is actually the casted record creating the issue
>>> described
>>>>>>>>> above.
>>>>>>>>>
>>>>>>>>> 2. I don't believe this is the right way to ensure that the
>>>>>>>>> open
>>> field if
>>>>>>>>> exists is of the right type. why not extract the field using
>>>>>>>>> field
>>> access
>>>>>>>>> by name function and then verify the type using the field tag?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Sep 22, 2015 at 9:11 AM, abdullah alamoudi <
>>> amoudi@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Dev, @Ildar,
>>>>>>>>>>
>>>>>>>>>> In the insert pipeline for datasets with open indexes, we
>>> introduce a
>>>>>>>> cast
>>>>>>>>>> function before the insert and so one would expect the
>>>>>>>>>> records to
>>> look
>>>>>>>> like
>>>>>>>>>> the casted record type which I assume has {{the closed fields
>>>>>>>>>> + a
>>>>>>>> nullable
>>>>>>>>>> field}}.
>>>>>>>>>>
>>>>>>>>>> The question is, what happens to the previously existing
>> records?,
>>> since
>>>>>>>>>> now the index has both, records of the original type and
>>>>>>>>>> records
>>> of the
>>>>>>>>>> casted type.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Abdullah.
>>>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>> Ildar
>>>>>>>>
>>>>>>>>
>>>>> Best regards,
>>>>> Ildar
>>>>>
>>>>>
>>>>
>>>
>>> Best regards,
>>> Ildar
>>>
>>>
>>
Re: Question about open indexes
Posted by abdullah alamoudi <ba...@gmail.com>.
I agree with Chen especially with the system not yet production ready.
It seems that going through with the release is more important.
Cheers,
Amoudi, Abdullah.
On Sat, Sep 26, 2015 at 3:33 AM, Chen Li <ch...@gmail.com> wrote:
> I vote for including this fix in the next Asterxi/Hyracks release, not this
> one.
>
> Chen
>
> On Fri, Sep 25, 2015 at 4:23 PM, Ildar Absalyamov <
> ildar.absalyamov@gmail.com> wrote:
>
> > It did not really occur to me during today during the meeting, but
> Preston
> > pointed out that the secondary index delete fix, that I proposed, spans
> > both Hyracks & Asterix codebase. Thus we will either have to release
> > Hyracks once again, or bite the bullet, sign the RC without this fixing
> > this issue and create bug-fix releases for both Hyracks&Asterix right
> after.
> >
> > > On Sep 22, 2015, at 22:27, Mike Carey <dt...@gmail.com> wrote:
> > >
> > > Ah - that makes sense now. Thx. (And welcome back. :-))
> > >
> > > On 9/22/15 10:02 PM, Ildar Absalyamov wrote:
> > >> Sorry for confusion, my initial answer was not correct enough,
> probably
> > should have waited sometime after I drove 1500 miles form Seattle :)
> > >> The casting in the insert pipeline, which Abdullah mentioned, is
> needed
> > only for secondary index insert. The reasoning behind this casting is to
> > ensure that the record is equivalent, thus it is safe to create an open
> > index. It is true that we can get <Pk, Sk> pairs out of original record
> > using get-field-by-name\index, but the cast operator is introduced merely
> > to kill the pipeline if the dataset input is not correct.
> > >> Thus the records in primary are never touched of modified, not matter
> > what indexes were created.
> > >> I am not sure however what is the second cast in Abdullah’s plan, and
> > where is comes from.
> > >>
> > >> @Taewoo, so scan-delete-btree-secondary-index-open test does not
> > actually delete data from the secondary index? I have checked the plan
> and
> > it has the delete operator. Maybe it is initialized with wrong
> parameters,
> > I’ll have a close look.
> > >>
> > >>> On Sep 22, 2015, at 18:33, Mike Carey <dt...@gmail.com> wrote:
> > >>>
> > >>> Sounds kinda bad! Also, I wonder what happens when the compiler
> > encounters records in the dataset - whose type in the catalog doesn't
> claim
> > to have a given (but now indexed) open field - e.g., during a data scan
> or
> > an access via some other path? Can Bad Things Happen due to the compiler
> > not properly anticipating the casted form of the records? (Maybe I am
> > misunderstanding something, but we should probably take a careful look at
> > the test cases - and make sure we do things like add a bunch of records,
> > then add such an index, then add some more records, then stress-test
> > type-related things that come at the dataset (i) thru the index, (ii)
> thru
> > a primary dataset scan, and (iii) thru some other index.)
> > >>>
> > >>> On 9/22/15 4:06 PM, Taewoo Kim wrote:
> > >>>> I think this issue:
> > https://issues.apache.org/jira/browse/ASTERIXDB-1109 is
> > >>>> related. Currently, index entries (SK, PK) are not deleted on an
> > open-type
> > >>>> secondary index during a deletion. This issue was not surfaced due
> to
> > the
> > >>>> fact that every search after a secondary index search had to go
> > through the
> > >>>> primary index lookup.
> > >>>>
> > >>>> Best,
> > >>>> Taewoo
> > >>>>
> > >>>> On Tue, Sep 22, 2015 at 12:04 AM, Ildar Absalyamov <
> > >>>> ildar.absalyamov@gmail.com> wrote:
> > >>>>
> > >>>>> Abdullah,
> > >>>>>
> > >>>>> If I remember correctly whenever a secondary open index is created
> > all
> > >>>>> existing records would be casted to a proper type to ensure that
> the
> > index
> > >>>>> creation is valid.
> > >>>>> As for the overall correctness of casting operation, semantically
> > creating
> > >>>>> an open index is the same thing as altering the dataset type. The
> > current
> > >>>>> implementation allows only one open index of particular type
> created
> > on a
> > >>>>> single field. If we would have had “alter datatype” functionality
> > the open
> > >>>>> indexing would not be required at all.
> > >>>>>
> > >>>>>> On Sep 21, 2015, at 23:25, abdullah alamoudi <am...@apache.org>
> > wrote:
> > >>>>>>
> > >>>>>> More thoughts:
> > >>>>>> I assume the intention of the cast was just to make sure if the
> open
> > >>>>> field
> > >>>>>> exists, it is of the specified type. Moreover, the un-casted
> record
> > >>>>> should
> > >>>>>> be inserted into the index.
> > >>>>>> If my assumptions are not correct, please, let me know ASAP.
> > >>>>>>
> > >>>>>> I have two thoughts on this:
> > >>>>>> 1. Actually, insert plans show that the records being inserted
> into
> > the
> > >>>>>> primary index is actually the casted record creating the issue
> > described
> > >>>>>> above.
> > >>>>>>
> > >>>>>> 2. I don't believe this is the right way to ensure that the open
> > field if
> > >>>>>> exists is of the right type. why not extract the field using field
> > access
> > >>>>>> by name function and then verify the type using the field tag?
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> On Tue, Sep 22, 2015 at 9:11 AM, abdullah alamoudi <
> > amoudi@apache.org>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> Hi Dev, @Ildar,
> > >>>>>>>
> > >>>>>>> In the insert pipeline for datasets with open indexes, we
> > introduce a
> > >>>>> cast
> > >>>>>>> function before the insert and so one would expect the records to
> > look
> > >>>>> like
> > >>>>>>> the casted record type which I assume has {{the closed fields + a
> > >>>>> nullable
> > >>>>>>> field}}.
> > >>>>>>>
> > >>>>>>> The question is, what happens to the previously existing
> records?,
> > since
> > >>>>>>> now the index has both, records of the original type and records
> > of the
> > >>>>>>> casted type.
> > >>>>>>>
> > >>>>>>> Thanks,
> > >>>>>>> Abdullah.
> > >>>>>>>
> > >>>>> Best regards,
> > >>>>> Ildar
> > >>>>>
> > >>>>>
> > >> Best regards,
> > >> Ildar
> > >>
> > >>
> > >
> >
> > Best regards,
> > Ildar
> >
> >
>
Re: Question about open indexes
Posted by Mike Carey <dt...@gmail.com>.
Seems like less work, I'd think, to fix Hyracks first?
On Sep 25, 2015 5:34 PM, "Chen Li" <ch...@gmail.com> wrote:
> I vote for including this fix in the next Asterxi/Hyracks release, not this
> one.
>
> Chen
>
> On Fri, Sep 25, 2015 at 4:23 PM, Ildar Absalyamov <
> ildar.absalyamov@gmail.com> wrote:
>
> > It did not really occur to me during today during the meeting, but
> Preston
> > pointed out that the secondary index delete fix, that I proposed, spans
> > both Hyracks & Asterix codebase. Thus we will either have to release
> > Hyracks once again, or bite the bullet, sign the RC without this fixing
> > this issue and create bug-fix releases for both Hyracks&Asterix right
> after.
> >
> > > On Sep 22, 2015, at 22:27, Mike Carey <dt...@gmail.com> wrote:
> > >
> > > Ah - that makes sense now. Thx. (And welcome back. :-))
> > >
> > > On 9/22/15 10:02 PM, Ildar Absalyamov wrote:
> > >> Sorry for confusion, my initial answer was not correct enough,
> probably
> > should have waited sometime after I drove 1500 miles form Seattle :)
> > >> The casting in the insert pipeline, which Abdullah mentioned, is
> needed
> > only for secondary index insert. The reasoning behind this casting is to
> > ensure that the record is equivalent, thus it is safe to create an open
> > index. It is true that we can get <Pk, Sk> pairs out of original record
> > using get-field-by-name\index, but the cast operator is introduced merely
> > to kill the pipeline if the dataset input is not correct.
> > >> Thus the records in primary are never touched of modified, not matter
> > what indexes were created.
> > >> I am not sure however what is the second cast in Abdullah’s plan, and
> > where is comes from.
> > >>
> > >> @Taewoo, so scan-delete-btree-secondary-index-open test does not
> > actually delete data from the secondary index? I have checked the plan
> and
> > it has the delete operator. Maybe it is initialized with wrong
> parameters,
> > I’ll have a close look.
> > >>
> > >>> On Sep 22, 2015, at 18:33, Mike Carey <dt...@gmail.com> wrote:
> > >>>
> > >>> Sounds kinda bad! Also, I wonder what happens when the compiler
> > encounters records in the dataset - whose type in the catalog doesn't
> claim
> > to have a given (but now indexed) open field - e.g., during a data scan
> or
> > an access via some other path? Can Bad Things Happen due to the compiler
> > not properly anticipating the casted form of the records? (Maybe I am
> > misunderstanding something, but we should probably take a careful look at
> > the test cases - and make sure we do things like add a bunch of records,
> > then add such an index, then add some more records, then stress-test
> > type-related things that come at the dataset (i) thru the index, (ii)
> thru
> > a primary dataset scan, and (iii) thru some other index.)
> > >>>
> > >>> On 9/22/15 4:06 PM, Taewoo Kim wrote:
> > >>>> I think this issue:
> > https://issues.apache.org/jira/browse/ASTERIXDB-1109 is
> > >>>> related. Currently, index entries (SK, PK) are not deleted on an
> > open-type
> > >>>> secondary index during a deletion. This issue was not surfaced due
> to
> > the
> > >>>> fact that every search after a secondary index search had to go
> > through the
> > >>>> primary index lookup.
> > >>>>
> > >>>> Best,
> > >>>> Taewoo
> > >>>>
> > >>>> On Tue, Sep 22, 2015 at 12:04 AM, Ildar Absalyamov <
> > >>>> ildar.absalyamov@gmail.com> wrote:
> > >>>>
> > >>>>> Abdullah,
> > >>>>>
> > >>>>> If I remember correctly whenever a secondary open index is created
> > all
> > >>>>> existing records would be casted to a proper type to ensure that
> the
> > index
> > >>>>> creation is valid.
> > >>>>> As for the overall correctness of casting operation, semantically
> > creating
> > >>>>> an open index is the same thing as altering the dataset type. The
> > current
> > >>>>> implementation allows only one open index of particular type
> created
> > on a
> > >>>>> single field. If we would have had “alter datatype” functionality
> > the open
> > >>>>> indexing would not be required at all.
> > >>>>>
> > >>>>>> On Sep 21, 2015, at 23:25, abdullah alamoudi <am...@apache.org>
> > wrote:
> > >>>>>>
> > >>>>>> More thoughts:
> > >>>>>> I assume the intention of the cast was just to make sure if the
> open
> > >>>>> field
> > >>>>>> exists, it is of the specified type. Moreover, the un-casted
> record
> > >>>>> should
> > >>>>>> be inserted into the index.
> > >>>>>> If my assumptions are not correct, please, let me know ASAP.
> > >>>>>>
> > >>>>>> I have two thoughts on this:
> > >>>>>> 1. Actually, insert plans show that the records being inserted
> into
> > the
> > >>>>>> primary index is actually the casted record creating the issue
> > described
> > >>>>>> above.
> > >>>>>>
> > >>>>>> 2. I don't believe this is the right way to ensure that the open
> > field if
> > >>>>>> exists is of the right type. why not extract the field using field
> > access
> > >>>>>> by name function and then verify the type using the field tag?
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> On Tue, Sep 22, 2015 at 9:11 AM, abdullah alamoudi <
> > amoudi@apache.org>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> Hi Dev, @Ildar,
> > >>>>>>>
> > >>>>>>> In the insert pipeline for datasets with open indexes, we
> > introduce a
> > >>>>> cast
> > >>>>>>> function before the insert and so one would expect the records to
> > look
> > >>>>> like
> > >>>>>>> the casted record type which I assume has {{the closed fields + a
> > >>>>> nullable
> > >>>>>>> field}}.
> > >>>>>>>
> > >>>>>>> The question is, what happens to the previously existing
> records?,
> > since
> > >>>>>>> now the index has both, records of the original type and records
> > of the
> > >>>>>>> casted type.
> > >>>>>>>
> > >>>>>>> Thanks,
> > >>>>>>> Abdullah.
> > >>>>>>>
> > >>>>> Best regards,
> > >>>>> Ildar
> > >>>>>
> > >>>>>
> > >> Best regards,
> > >> Ildar
> > >>
> > >>
> > >
> >
> > Best regards,
> > Ildar
> >
> >
>
Re: Question about open indexes
Posted by Chen Li <ch...@gmail.com>.
I vote for including this fix in the next Asterxi/Hyracks release, not this
one.
Chen
On Fri, Sep 25, 2015 at 4:23 PM, Ildar Absalyamov <
ildar.absalyamov@gmail.com> wrote:
> It did not really occur to me during today during the meeting, but Preston
> pointed out that the secondary index delete fix, that I proposed, spans
> both Hyracks & Asterix codebase. Thus we will either have to release
> Hyracks once again, or bite the bullet, sign the RC without this fixing
> this issue and create bug-fix releases for both Hyracks&Asterix right after.
>
> > On Sep 22, 2015, at 22:27, Mike Carey <dt...@gmail.com> wrote:
> >
> > Ah - that makes sense now. Thx. (And welcome back. :-))
> >
> > On 9/22/15 10:02 PM, Ildar Absalyamov wrote:
> >> Sorry for confusion, my initial answer was not correct enough, probably
> should have waited sometime after I drove 1500 miles form Seattle :)
> >> The casting in the insert pipeline, which Abdullah mentioned, is needed
> only for secondary index insert. The reasoning behind this casting is to
> ensure that the record is equivalent, thus it is safe to create an open
> index. It is true that we can get <Pk, Sk> pairs out of original record
> using get-field-by-name\index, but the cast operator is introduced merely
> to kill the pipeline if the dataset input is not correct.
> >> Thus the records in primary are never touched of modified, not matter
> what indexes were created.
> >> I am not sure however what is the second cast in Abdullah’s plan, and
> where is comes from.
> >>
> >> @Taewoo, so scan-delete-btree-secondary-index-open test does not
> actually delete data from the secondary index? I have checked the plan and
> it has the delete operator. Maybe it is initialized with wrong parameters,
> I’ll have a close look.
> >>
> >>> On Sep 22, 2015, at 18:33, Mike Carey <dt...@gmail.com> wrote:
> >>>
> >>> Sounds kinda bad! Also, I wonder what happens when the compiler
> encounters records in the dataset - whose type in the catalog doesn't claim
> to have a given (but now indexed) open field - e.g., during a data scan or
> an access via some other path? Can Bad Things Happen due to the compiler
> not properly anticipating the casted form of the records? (Maybe I am
> misunderstanding something, but we should probably take a careful look at
> the test cases - and make sure we do things like add a bunch of records,
> then add such an index, then add some more records, then stress-test
> type-related things that come at the dataset (i) thru the index, (ii) thru
> a primary dataset scan, and (iii) thru some other index.)
> >>>
> >>> On 9/22/15 4:06 PM, Taewoo Kim wrote:
> >>>> I think this issue:
> https://issues.apache.org/jira/browse/ASTERIXDB-1109 is
> >>>> related. Currently, index entries (SK, PK) are not deleted on an
> open-type
> >>>> secondary index during a deletion. This issue was not surfaced due to
> the
> >>>> fact that every search after a secondary index search had to go
> through the
> >>>> primary index lookup.
> >>>>
> >>>> Best,
> >>>> Taewoo
> >>>>
> >>>> On Tue, Sep 22, 2015 at 12:04 AM, Ildar Absalyamov <
> >>>> ildar.absalyamov@gmail.com> wrote:
> >>>>
> >>>>> Abdullah,
> >>>>>
> >>>>> If I remember correctly whenever a secondary open index is created
> all
> >>>>> existing records would be casted to a proper type to ensure that the
> index
> >>>>> creation is valid.
> >>>>> As for the overall correctness of casting operation, semantically
> creating
> >>>>> an open index is the same thing as altering the dataset type. The
> current
> >>>>> implementation allows only one open index of particular type created
> on a
> >>>>> single field. If we would have had “alter datatype” functionality
> the open
> >>>>> indexing would not be required at all.
> >>>>>
> >>>>>> On Sep 21, 2015, at 23:25, abdullah alamoudi <am...@apache.org>
> wrote:
> >>>>>>
> >>>>>> More thoughts:
> >>>>>> I assume the intention of the cast was just to make sure if the open
> >>>>> field
> >>>>>> exists, it is of the specified type. Moreover, the un-casted record
> >>>>> should
> >>>>>> be inserted into the index.
> >>>>>> If my assumptions are not correct, please, let me know ASAP.
> >>>>>>
> >>>>>> I have two thoughts on this:
> >>>>>> 1. Actually, insert plans show that the records being inserted into
> the
> >>>>>> primary index is actually the casted record creating the issue
> described
> >>>>>> above.
> >>>>>>
> >>>>>> 2. I don't believe this is the right way to ensure that the open
> field if
> >>>>>> exists is of the right type. why not extract the field using field
> access
> >>>>>> by name function and then verify the type using the field tag?
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Tue, Sep 22, 2015 at 9:11 AM, abdullah alamoudi <
> amoudi@apache.org>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Hi Dev, @Ildar,
> >>>>>>>
> >>>>>>> In the insert pipeline for datasets with open indexes, we
> introduce a
> >>>>> cast
> >>>>>>> function before the insert and so one would expect the records to
> look
> >>>>> like
> >>>>>>> the casted record type which I assume has {{the closed fields + a
> >>>>> nullable
> >>>>>>> field}}.
> >>>>>>>
> >>>>>>> The question is, what happens to the previously existing records?,
> since
> >>>>>>> now the index has both, records of the original type and records
> of the
> >>>>>>> casted type.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Abdullah.
> >>>>>>>
> >>>>> Best regards,
> >>>>> Ildar
> >>>>>
> >>>>>
> >> Best regards,
> >> Ildar
> >>
> >>
> >
>
> Best regards,
> Ildar
>
>
Re: Question about open indexes
Posted by Ildar Absalyamov <il...@gmail.com>.
It did not really occur to me during today during the meeting, but Preston pointed out that the secondary index delete fix, that I proposed, spans both Hyracks & Asterix codebase. Thus we will either have to release Hyracks once again, or bite the bullet, sign the RC without this fixing this issue and create bug-fix releases for both Hyracks&Asterix right after.
> On Sep 22, 2015, at 22:27, Mike Carey <dt...@gmail.com> wrote:
>
> Ah - that makes sense now. Thx. (And welcome back. :-))
>
> On 9/22/15 10:02 PM, Ildar Absalyamov wrote:
>> Sorry for confusion, my initial answer was not correct enough, probably should have waited sometime after I drove 1500 miles form Seattle :)
>> The casting in the insert pipeline, which Abdullah mentioned, is needed only for secondary index insert. The reasoning behind this casting is to ensure that the record is equivalent, thus it is safe to create an open index. It is true that we can get <Pk, Sk> pairs out of original record using get-field-by-name\index, but the cast operator is introduced merely to kill the pipeline if the dataset input is not correct.
>> Thus the records in primary are never touched of modified, not matter what indexes were created.
>> I am not sure however what is the second cast in Abdullah’s plan, and where is comes from.
>>
>> @Taewoo, so scan-delete-btree-secondary-index-open test does not actually delete data from the secondary index? I have checked the plan and it has the delete operator. Maybe it is initialized with wrong parameters, I’ll have a close look.
>>
>>> On Sep 22, 2015, at 18:33, Mike Carey <dt...@gmail.com> wrote:
>>>
>>> Sounds kinda bad! Also, I wonder what happens when the compiler encounters records in the dataset - whose type in the catalog doesn't claim to have a given (but now indexed) open field - e.g., during a data scan or an access via some other path? Can Bad Things Happen due to the compiler not properly anticipating the casted form of the records? (Maybe I am misunderstanding something, but we should probably take a careful look at the test cases - and make sure we do things like add a bunch of records, then add such an index, then add some more records, then stress-test type-related things that come at the dataset (i) thru the index, (ii) thru a primary dataset scan, and (iii) thru some other index.)
>>>
>>> On 9/22/15 4:06 PM, Taewoo Kim wrote:
>>>> I think this issue:https://issues.apache.org/jira/browse/ASTERIXDB-1109 is
>>>> related. Currently, index entries (SK, PK) are not deleted on an open-type
>>>> secondary index during a deletion. This issue was not surfaced due to the
>>>> fact that every search after a secondary index search had to go through the
>>>> primary index lookup.
>>>>
>>>> Best,
>>>> Taewoo
>>>>
>>>> On Tue, Sep 22, 2015 at 12:04 AM, Ildar Absalyamov <
>>>> ildar.absalyamov@gmail.com> wrote:
>>>>
>>>>> Abdullah,
>>>>>
>>>>> If I remember correctly whenever a secondary open index is created all
>>>>> existing records would be casted to a proper type to ensure that the index
>>>>> creation is valid.
>>>>> As for the overall correctness of casting operation, semantically creating
>>>>> an open index is the same thing as altering the dataset type. The current
>>>>> implementation allows only one open index of particular type created on a
>>>>> single field. If we would have had “alter datatype” functionality the open
>>>>> indexing would not be required at all.
>>>>>
>>>>>> On Sep 21, 2015, at 23:25, abdullah alamoudi <am...@apache.org> wrote:
>>>>>>
>>>>>> More thoughts:
>>>>>> I assume the intention of the cast was just to make sure if the open
>>>>> field
>>>>>> exists, it is of the specified type. Moreover, the un-casted record
>>>>> should
>>>>>> be inserted into the index.
>>>>>> If my assumptions are not correct, please, let me know ASAP.
>>>>>>
>>>>>> I have two thoughts on this:
>>>>>> 1. Actually, insert plans show that the records being inserted into the
>>>>>> primary index is actually the casted record creating the issue described
>>>>>> above.
>>>>>>
>>>>>> 2. I don't believe this is the right way to ensure that the open field if
>>>>>> exists is of the right type. why not extract the field using field access
>>>>>> by name function and then verify the type using the field tag?
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Sep 22, 2015 at 9:11 AM, abdullah alamoudi <am...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Dev, @Ildar,
>>>>>>>
>>>>>>> In the insert pipeline for datasets with open indexes, we introduce a
>>>>> cast
>>>>>>> function before the insert and so one would expect the records to look
>>>>> like
>>>>>>> the casted record type which I assume has {{the closed fields + a
>>>>> nullable
>>>>>>> field}}.
>>>>>>>
>>>>>>> The question is, what happens to the previously existing records?, since
>>>>>>> now the index has both, records of the original type and records of the
>>>>>>> casted type.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Abdullah.
>>>>>>>
>>>>> Best regards,
>>>>> Ildar
>>>>>
>>>>>
>> Best regards,
>> Ildar
>>
>>
>
Best regards,
Ildar
Re: Question about open indexes
Posted by Mike Carey <dt...@gmail.com>.
Ah - that makes sense now. Thx. (And welcome back. :-))
On 9/22/15 10:02 PM, Ildar Absalyamov wrote:
> Sorry for confusion, my initial answer was not correct enough, probably should have waited sometime after I drove 1500 miles form Seattle :)
> The casting in the insert pipeline, which Abdullah mentioned, is needed only for secondary index insert. The reasoning behind this casting is to ensure that the record is equivalent, thus it is safe to create an open index. It is true that we can get <Pk, Sk> pairs out of original record using get-field-by-name\index, but the cast operator is introduced merely to kill the pipeline if the dataset input is not correct.
> Thus the records in primary are never touched of modified, not matter what indexes were created.
> I am not sure however what is the second cast in Abdullah’s plan, and where is comes from.
>
> @Taewoo, so scan-delete-btree-secondary-index-open test does not actually delete data from the secondary index? I have checked the plan and it has the delete operator. Maybe it is initialized with wrong parameters, I’ll have a close look.
>
>> On Sep 22, 2015, at 18:33, Mike Carey <dt...@gmail.com> wrote:
>>
>> Sounds kinda bad! Also, I wonder what happens when the compiler encounters records in the dataset - whose type in the catalog doesn't claim to have a given (but now indexed) open field - e.g., during a data scan or an access via some other path? Can Bad Things Happen due to the compiler not properly anticipating the casted form of the records? (Maybe I am misunderstanding something, but we should probably take a careful look at the test cases - and make sure we do things like add a bunch of records, then add such an index, then add some more records, then stress-test type-related things that come at the dataset (i) thru the index, (ii) thru a primary dataset scan, and (iii) thru some other index.)
>>
>> On 9/22/15 4:06 PM, Taewoo Kim wrote:
>>> I think this issue:https://issues.apache.org/jira/browse/ASTERIXDB-1109 is
>>> related. Currently, index entries (SK, PK) are not deleted on an open-type
>>> secondary index during a deletion. This issue was not surfaced due to the
>>> fact that every search after a secondary index search had to go through the
>>> primary index lookup.
>>>
>>> Best,
>>> Taewoo
>>>
>>> On Tue, Sep 22, 2015 at 12:04 AM, Ildar Absalyamov <
>>> ildar.absalyamov@gmail.com> wrote:
>>>
>>>> Abdullah,
>>>>
>>>> If I remember correctly whenever a secondary open index is created all
>>>> existing records would be casted to a proper type to ensure that the index
>>>> creation is valid.
>>>> As for the overall correctness of casting operation, semantically creating
>>>> an open index is the same thing as altering the dataset type. The current
>>>> implementation allows only one open index of particular type created on a
>>>> single field. If we would have had “alter datatype” functionality the open
>>>> indexing would not be required at all.
>>>>
>>>>> On Sep 21, 2015, at 23:25, abdullah alamoudi <am...@apache.org> wrote:
>>>>>
>>>>> More thoughts:
>>>>> I assume the intention of the cast was just to make sure if the open
>>>> field
>>>>> exists, it is of the specified type. Moreover, the un-casted record
>>>> should
>>>>> be inserted into the index.
>>>>> If my assumptions are not correct, please, let me know ASAP.
>>>>>
>>>>> I have two thoughts on this:
>>>>> 1. Actually, insert plans show that the records being inserted into the
>>>>> primary index is actually the casted record creating the issue described
>>>>> above.
>>>>>
>>>>> 2. I don't believe this is the right way to ensure that the open field if
>>>>> exists is of the right type. why not extract the field using field access
>>>>> by name function and then verify the type using the field tag?
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Sep 22, 2015 at 9:11 AM, abdullah alamoudi <am...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Hi Dev, @Ildar,
>>>>>>
>>>>>> In the insert pipeline for datasets with open indexes, we introduce a
>>>> cast
>>>>>> function before the insert and so one would expect the records to look
>>>> like
>>>>>> the casted record type which I assume has {{the closed fields + a
>>>> nullable
>>>>>> field}}.
>>>>>>
>>>>>> The question is, what happens to the previously existing records?, since
>>>>>> now the index has both, records of the original type and records of the
>>>>>> casted type.
>>>>>>
>>>>>> Thanks,
>>>>>> Abdullah.
>>>>>>
>>>> Best regards,
>>>> Ildar
>>>>
>>>>
> Best regards,
> Ildar
>
>
Re: Question about open indexes
Posted by Young-Seok Kim <ki...@gmail.com>.
The plan has the delete operator, but the operator doesn't actually delete
entries from the secondary index due to the following tupleFilter.accept()
code in AsterixLSMInsertDeleteOperatorNodePushable.nextFrame()
The accept() returns false because the secondary key fields fed from the
underlying operator delivers null even if those field values shouldn't be
null. (We also tested that if the same operation is applied to the closed
type dataset and the non-open index, the delete operator actually deletes
entries.)
if (tupleFilter != null) {
frameTuple.reset(accessor, i);
if (!tupleFilter.accept(frameTuple)) {
tmpCnt++;
continue;
}
}
Young-Seok
On Tue, Sep 22, 2015 at 10:02 PM, Ildar Absalyamov <
ildar.absalyamov@gmail.com> wrote:
> Sorry for confusion, my initial answer was not correct enough, probably
> should have waited sometime after I drove 1500 miles form Seattle :)
> The casting in the insert pipeline, which Abdullah mentioned, is needed
> only for secondary index insert. The reasoning behind this casting is to
> ensure that the record is equivalent, thus it is safe to create an open
> index. It is true that we can get <Pk, Sk> pairs out of original record
> using get-field-by-name\index, but the cast operator is introduced merely
> to kill the pipeline if the dataset input is not correct.
> Thus the records in primary are never touched of modified, not matter what
> indexes were created.
> I am not sure however what is the second cast in Abdullah’s plan, and
> where is comes from.
>
> @Taewoo, so scan-delete-btree-secondary-index-open test does not actually
> delete data from the secondary index? I have checked the plan and it has
> the delete operator. Maybe it is initialized with wrong parameters, I’ll
> have a close look.
>
> > On Sep 22, 2015, at 18:33, Mike Carey <dt...@gmail.com> wrote:
> >
> > Sounds kinda bad! Also, I wonder what happens when the compiler
> encounters records in the dataset - whose type in the catalog doesn't claim
> to have a given (but now indexed) open field - e.g., during a data scan or
> an access via some other path? Can Bad Things Happen due to the compiler
> not properly anticipating the casted form of the records? (Maybe I am
> misunderstanding something, but we should probably take a careful look at
> the test cases - and make sure we do things like add a bunch of records,
> then add such an index, then add some more records, then stress-test
> type-related things that come at the dataset (i) thru the index, (ii) thru
> a primary dataset scan, and (iii) thru some other index.)
> >
> > On 9/22/15 4:06 PM, Taewoo Kim wrote:
> >> I think this issue:https://issues.apache.org/jira/browse/ASTERIXDB-1109
> is
> >> related. Currently, index entries (SK, PK) are not deleted on an
> open-type
> >> secondary index during a deletion. This issue was not surfaced due to
> the
> >> fact that every search after a secondary index search had to go through
> the
> >> primary index lookup.
> >>
> >> Best,
> >> Taewoo
> >>
> >> On Tue, Sep 22, 2015 at 12:04 AM, Ildar Absalyamov <
> >> ildar.absalyamov@gmail.com> wrote:
> >>
> >>> Abdullah,
> >>>
> >>> If I remember correctly whenever a secondary open index is created all
> >>> existing records would be casted to a proper type to ensure that the
> index
> >>> creation is valid.
> >>> As for the overall correctness of casting operation, semantically
> creating
> >>> an open index is the same thing as altering the dataset type. The
> current
> >>> implementation allows only one open index of particular type created
> on a
> >>> single field. If we would have had “alter datatype” functionality the
> open
> >>> indexing would not be required at all.
> >>>
> >>>> On Sep 21, 2015, at 23:25, abdullah alamoudi <am...@apache.org>
> wrote:
> >>>>
> >>>> More thoughts:
> >>>> I assume the intention of the cast was just to make sure if the open
> >>> field
> >>>> exists, it is of the specified type. Moreover, the un-casted record
> >>> should
> >>>> be inserted into the index.
> >>>> If my assumptions are not correct, please, let me know ASAP.
> >>>>
> >>>> I have two thoughts on this:
> >>>> 1. Actually, insert plans show that the records being inserted into
> the
> >>>> primary index is actually the casted record creating the issue
> described
> >>>> above.
> >>>>
> >>>> 2. I don't believe this is the right way to ensure that the open
> field if
> >>>> exists is of the right type. why not extract the field using field
> access
> >>>> by name function and then verify the type using the field tag?
> >>>>
> >>>>
> >>>>
> >>>> On Tue, Sep 22, 2015 at 9:11 AM, abdullah alamoudi <amoudi@apache.org
> >
> >>>> wrote:
> >>>>
> >>>>> Hi Dev, @Ildar,
> >>>>>
> >>>>> In the insert pipeline for datasets with open indexes, we introduce a
> >>> cast
> >>>>> function before the insert and so one would expect the records to
> look
> >>> like
> >>>>> the casted record type which I assume has {{the closed fields + a
> >>> nullable
> >>>>> field}}.
> >>>>>
> >>>>> The question is, what happens to the previously existing records?,
> since
> >>>>> now the index has both, records of the original type and records of
> the
> >>>>> casted type.
> >>>>>
> >>>>> Thanks,
> >>>>> Abdullah.
> >>>>>
> >>> Best regards,
> >>> Ildar
> >>>
> >>>
> >
>
> Best regards,
> Ildar
>
>
Re: Question about open indexes
Posted by Ildar Absalyamov <il...@gmail.com>.
Sorry for confusion, my initial answer was not correct enough, probably should have waited sometime after I drove 1500 miles form Seattle :)
The casting in the insert pipeline, which Abdullah mentioned, is needed only for secondary index insert. The reasoning behind this casting is to ensure that the record is equivalent, thus it is safe to create an open index. It is true that we can get <Pk, Sk> pairs out of original record using get-field-by-name\index, but the cast operator is introduced merely to kill the pipeline if the dataset input is not correct.
Thus the records in primary are never touched of modified, not matter what indexes were created.
I am not sure however what is the second cast in Abdullah’s plan, and where is comes from.
@Taewoo, so scan-delete-btree-secondary-index-open test does not actually delete data from the secondary index? I have checked the plan and it has the delete operator. Maybe it is initialized with wrong parameters, I’ll have a close look.
> On Sep 22, 2015, at 18:33, Mike Carey <dt...@gmail.com> wrote:
>
> Sounds kinda bad! Also, I wonder what happens when the compiler encounters records in the dataset - whose type in the catalog doesn't claim to have a given (but now indexed) open field - e.g., during a data scan or an access via some other path? Can Bad Things Happen due to the compiler not properly anticipating the casted form of the records? (Maybe I am misunderstanding something, but we should probably take a careful look at the test cases - and make sure we do things like add a bunch of records, then add such an index, then add some more records, then stress-test type-related things that come at the dataset (i) thru the index, (ii) thru a primary dataset scan, and (iii) thru some other index.)
>
> On 9/22/15 4:06 PM, Taewoo Kim wrote:
>> I think this issue:https://issues.apache.org/jira/browse/ASTERIXDB-1109 is
>> related. Currently, index entries (SK, PK) are not deleted on an open-type
>> secondary index during a deletion. This issue was not surfaced due to the
>> fact that every search after a secondary index search had to go through the
>> primary index lookup.
>>
>> Best,
>> Taewoo
>>
>> On Tue, Sep 22, 2015 at 12:04 AM, Ildar Absalyamov <
>> ildar.absalyamov@gmail.com> wrote:
>>
>>> Abdullah,
>>>
>>> If I remember correctly whenever a secondary open index is created all
>>> existing records would be casted to a proper type to ensure that the index
>>> creation is valid.
>>> As for the overall correctness of casting operation, semantically creating
>>> an open index is the same thing as altering the dataset type. The current
>>> implementation allows only one open index of particular type created on a
>>> single field. If we would have had “alter datatype” functionality the open
>>> indexing would not be required at all.
>>>
>>>> On Sep 21, 2015, at 23:25, abdullah alamoudi <am...@apache.org> wrote:
>>>>
>>>> More thoughts:
>>>> I assume the intention of the cast was just to make sure if the open
>>> field
>>>> exists, it is of the specified type. Moreover, the un-casted record
>>> should
>>>> be inserted into the index.
>>>> If my assumptions are not correct, please, let me know ASAP.
>>>>
>>>> I have two thoughts on this:
>>>> 1. Actually, insert plans show that the records being inserted into the
>>>> primary index is actually the casted record creating the issue described
>>>> above.
>>>>
>>>> 2. I don't believe this is the right way to ensure that the open field if
>>>> exists is of the right type. why not extract the field using field access
>>>> by name function and then verify the type using the field tag?
>>>>
>>>>
>>>>
>>>> On Tue, Sep 22, 2015 at 9:11 AM, abdullah alamoudi <am...@apache.org>
>>>> wrote:
>>>>
>>>>> Hi Dev, @Ildar,
>>>>>
>>>>> In the insert pipeline for datasets with open indexes, we introduce a
>>> cast
>>>>> function before the insert and so one would expect the records to look
>>> like
>>>>> the casted record type which I assume has {{the closed fields + a
>>> nullable
>>>>> field}}.
>>>>>
>>>>> The question is, what happens to the previously existing records?, since
>>>>> now the index has both, records of the original type and records of the
>>>>> casted type.
>>>>>
>>>>> Thanks,
>>>>> Abdullah.
>>>>>
>>> Best regards,
>>> Ildar
>>>
>>>
>
Best regards,
Ildar
Re: Question about open indexes
Posted by Mike Carey <dt...@gmail.com>.
Sounds kinda bad! Also, I wonder what happens when the compiler
encounters records in the dataset - whose type in the catalog doesn't
claim to have a given (but now indexed) open field - e.g., during a data
scan or an access via some other path? Can Bad Things Happen due to the
compiler not properly anticipating the casted form of the records?
(Maybe I am misunderstanding something, but we should probably take a
careful look at the test cases - and make sure we do things like add a
bunch of records, then add such an index, then add some more records,
then stress-test type-related things that come at the dataset (i) thru
the index, (ii) thru a primary dataset scan, and (iii) thru some other
index.)
On 9/22/15 4:06 PM, Taewoo Kim wrote:
> I think this issue:https://issues.apache.org/jira/browse/ASTERIXDB-1109 is
> related. Currently, index entries (SK, PK) are not deleted on an open-type
> secondary index during a deletion. This issue was not surfaced due to the
> fact that every search after a secondary index search had to go through the
> primary index lookup.
>
> Best,
> Taewoo
>
> On Tue, Sep 22, 2015 at 12:04 AM, Ildar Absalyamov <
> ildar.absalyamov@gmail.com> wrote:
>
>> Abdullah,
>>
>> If I remember correctly whenever a secondary open index is created all
>> existing records would be casted to a proper type to ensure that the index
>> creation is valid.
>> As for the overall correctness of casting operation, semantically creating
>> an open index is the same thing as altering the dataset type. The current
>> implementation allows only one open index of particular type created on a
>> single field. If we would have had “alter datatype” functionality the open
>> indexing would not be required at all.
>>
>>> On Sep 21, 2015, at 23:25, abdullah alamoudi <am...@apache.org> wrote:
>>>
>>> More thoughts:
>>> I assume the intention of the cast was just to make sure if the open
>> field
>>> exists, it is of the specified type. Moreover, the un-casted record
>> should
>>> be inserted into the index.
>>> If my assumptions are not correct, please, let me know ASAP.
>>>
>>> I have two thoughts on this:
>>> 1. Actually, insert plans show that the records being inserted into the
>>> primary index is actually the casted record creating the issue described
>>> above.
>>>
>>> 2. I don't believe this is the right way to ensure that the open field if
>>> exists is of the right type. why not extract the field using field access
>>> by name function and then verify the type using the field tag?
>>>
>>>
>>>
>>> On Tue, Sep 22, 2015 at 9:11 AM, abdullah alamoudi <am...@apache.org>
>>> wrote:
>>>
>>>> Hi Dev, @Ildar,
>>>>
>>>> In the insert pipeline for datasets with open indexes, we introduce a
>> cast
>>>> function before the insert and so one would expect the records to look
>> like
>>>> the casted record type which I assume has {{the closed fields + a
>> nullable
>>>> field}}.
>>>>
>>>> The question is, what happens to the previously existing records?, since
>>>> now the index has both, records of the original type and records of the
>>>> casted type.
>>>>
>>>> Thanks,
>>>> Abdullah.
>>>>
>> Best regards,
>> Ildar
>>
>>
Re: Question about open indexes
Posted by Taewoo Kim <wa...@gmail.com>.
I think this issue:https://issues.apache.org/jira/browse/ASTERIXDB-1109 is
related. Currently, index entries (SK, PK) are not deleted on an open-type
secondary index during a deletion. This issue was not surfaced due to the
fact that every search after a secondary index search had to go through the
primary index lookup.
Best,
Taewoo
On Tue, Sep 22, 2015 at 12:04 AM, Ildar Absalyamov <
ildar.absalyamov@gmail.com> wrote:
> Abdullah,
>
> If I remember correctly whenever a secondary open index is created all
> existing records would be casted to a proper type to ensure that the index
> creation is valid.
> As for the overall correctness of casting operation, semantically creating
> an open index is the same thing as altering the dataset type. The current
> implementation allows only one open index of particular type created on a
> single field. If we would have had “alter datatype” functionality the open
> indexing would not be required at all.
>
> > On Sep 21, 2015, at 23:25, abdullah alamoudi <am...@apache.org> wrote:
> >
> > More thoughts:
> > I assume the intention of the cast was just to make sure if the open
> field
> > exists, it is of the specified type. Moreover, the un-casted record
> should
> > be inserted into the index.
> > If my assumptions are not correct, please, let me know ASAP.
> >
> > I have two thoughts on this:
> > 1. Actually, insert plans show that the records being inserted into the
> > primary index is actually the casted record creating the issue described
> > above.
> >
> > 2. I don't believe this is the right way to ensure that the open field if
> > exists is of the right type. why not extract the field using field access
> > by name function and then verify the type using the field tag?
> >
> >
> >
> > On Tue, Sep 22, 2015 at 9:11 AM, abdullah alamoudi <am...@apache.org>
> > wrote:
> >
> >> Hi Dev, @Ildar,
> >>
> >> In the insert pipeline for datasets with open indexes, we introduce a
> cast
> >> function before the insert and so one would expect the records to look
> like
> >> the casted record type which I assume has {{the closed fields + a
> nullable
> >> field}}.
> >>
> >> The question is, what happens to the previously existing records?, since
> >> now the index has both, records of the original type and records of the
> >> casted type.
> >>
> >> Thanks,
> >> Abdullah.
> >>
>
> Best regards,
> Ildar
>
>
Re: Question about open indexes
Posted by Ildar Absalyamov <il...@gmail.com>.
Abdullah,
If I remember correctly whenever a secondary open index is created all existing records would be casted to a proper type to ensure that the index creation is valid.
As for the overall correctness of casting operation, semantically creating an open index is the same thing as altering the dataset type. The current implementation allows only one open index of particular type created on a single field. If we would have had “alter datatype” functionality the open indexing would not be required at all.
> On Sep 21, 2015, at 23:25, abdullah alamoudi <am...@apache.org> wrote:
>
> More thoughts:
> I assume the intention of the cast was just to make sure if the open field
> exists, it is of the specified type. Moreover, the un-casted record should
> be inserted into the index.
> If my assumptions are not correct, please, let me know ASAP.
>
> I have two thoughts on this:
> 1. Actually, insert plans show that the records being inserted into the
> primary index is actually the casted record creating the issue described
> above.
>
> 2. I don't believe this is the right way to ensure that the open field if
> exists is of the right type. why not extract the field using field access
> by name function and then verify the type using the field tag?
>
>
>
> On Tue, Sep 22, 2015 at 9:11 AM, abdullah alamoudi <am...@apache.org>
> wrote:
>
>> Hi Dev, @Ildar,
>>
>> In the insert pipeline for datasets with open indexes, we introduce a cast
>> function before the insert and so one would expect the records to look like
>> the casted record type which I assume has {{the closed fields + a nullable
>> field}}.
>>
>> The question is, what happens to the previously existing records?, since
>> now the index has both, records of the original type and records of the
>> casted type.
>>
>> Thanks,
>> Abdullah.
>>
Best regards,
Ildar
Re: Question about open indexes
Posted by abdullah alamoudi <am...@apache.org>.
More thoughts:
I assume the intention of the cast was just to make sure if the open field
exists, it is of the specified type. Moreover, the un-casted record should
be inserted into the index.
If my assumptions are not correct, please, let me know ASAP.
I have two thoughts on this:
1. Actually, insert plans show that the records being inserted into the
primary index is actually the casted record creating the issue described
above.
2. I don't believe this is the right way to ensure that the open field if
exists is of the right type. why not extract the field using field access
by name function and then verify the type using the field tag?
On Tue, Sep 22, 2015 at 9:11 AM, abdullah alamoudi <am...@apache.org>
wrote:
> Hi Dev, @Ildar,
>
> In the insert pipeline for datasets with open indexes, we introduce a cast
> function before the insert and so one would expect the records to look like
> the casted record type which I assume has {{the closed fields + a nullable
> field}}.
>
> The question is, what happens to the previously existing records?, since
> now the index has both, records of the original type and records of the
> casted type.
>
> Thanks,
> Abdullah.
>