You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@asterixdb.apache.org by Mike Carey <dt...@gmail.com> on 2015/08/13 22:08:07 UTC
First Apache incubating release - blockers?
So where are we on feeds and metadata - the release blockers?
(@Raman? Feed tutorial and API key? @Ildar? Your changes?)
Re: First Apache incubating release - blockers?
Posted by Ted Dunning <te...@gmail.com>.
The actions should be essentially the same. Those items on the incubator
page is just the check-list that the actions with RAT should be used to
verify.
On Thu, Aug 13, 2015 at 1:55 PM, Ian Maxon <im...@uci.edu> wrote:
> We kind of did this but I wasn't aware of this section on the incubator
> page. It was part of the change that introduced Apache RAT.
>
> -Ian
>
> On Thu, Aug 13, 2015 at 1:11 PM, Jochen Wiedmann <
> jochen.wiedmann@gmail.com>
> wrote:
>
> > On Thu, Aug 13, 2015 at 10:08 PM, Mike Carey <dt...@gmail.com> wrote:
> > > So where are we on feeds and metadata - the release blockers?
> > > (@Raman? Feed tutorial and API key? @Ildar? Your changes?)
> >
> > I'd suggest to complete the "Verify distribution rights" section on
> > http://incubator.apache.org/projects/asterixdb.html, before cutting a
> > release.
> >
> > Jochen
> >
> >
> > --
> > Any world that can produce the Taj Mahal, William Shakespeare,
> > and Stripe toothpaste can't be all bad. (C.R. MacNamara, One Two Three)
> >
>
Re: First Apache incubating release - blockers?
Posted by Ian Maxon <im...@uci.edu>.
We kind of did this but I wasn't aware of this section on the incubator
page. It was part of the change that introduced Apache RAT.
-Ian
On Thu, Aug 13, 2015 at 1:11 PM, Jochen Wiedmann <jo...@gmail.com>
wrote:
> On Thu, Aug 13, 2015 at 10:08 PM, Mike Carey <dt...@gmail.com> wrote:
> > So where are we on feeds and metadata - the release blockers?
> > (@Raman? Feed tutorial and API key? @Ildar? Your changes?)
>
> I'd suggest to complete the "Verify distribution rights" section on
> http://incubator.apache.org/projects/asterixdb.html, before cutting a
> release.
>
> Jochen
>
>
> --
> Any world that can produce the Taj Mahal, William Shakespeare,
> and Stripe toothpaste can't be all bad. (C.R. MacNamara, One Two Three)
>
Re: First Apache incubating release - blockers?
Posted by Ildar Absalyamov <il...@gmail.com>.
> On Aug 13, 2015, at 23:07, abdullah alamoudi <ba...@gmail.com> wrote:
>
> I assume you meant “Dataset" record type definition and not "Datatype" and
> you're right. they are the same and can be elevated one level higher :-).
OK, I will move it as a part of this change in this case.
>
> For everyone's information, here is a description of the problem:
> In our regular secondary indexes, the structure of the tuple consists of
> <SecondaryKeys,PrimaryKeys> and the Hyracks jobs get constructed based on
> this assumption. However, our Metadata secondary indexes don't follow this
> convention and so we end up with runtime errors whenever they are picked
> for access.
Correct, instead these indexes construct tuple in a form <SecondaryKey, StuffThatReallyNeeded>, which allows them to avoid probing the primary for the needed information, returning via index-only lookup.
>
> Cheers,
> Abdullah.
>
> On Fri, Aug 14, 2015 at 8:59 AM, Ildar Absalyamov <
> ildar.absalyamov@gmail.com> wrote:
>
>> @Abdullah,
>> OK, now I see that change. IMHO this is a hack, but I will adopt that
>> approach that as well to submit the code review. However this issue really
>> needs to be addressed.
>>
>> Side question, when you introduced external indexes you have added a field
>> “ExternalDetails” (which in turn is a nested record) into “Datatype" record
>> type definition in addition to existing “InternalDetails” field (BTW I
>> believe either of the fields exists at a particular point in time, maybe be
>> can use UNION for them?). However both of them have nested field
>> “GroupName”, which is used by one of the secondary metadata indexes. Is
>> that the same GroupName in both cases? If yes, can we move this field 1
>> level up from {External\Internal}Details to avoid duplication?
>>
>>> On Aug 13, 2015, at 20:48, abdullah alamoudi <ba...@gmail.com> wrote:
>>>
>>> Ildar,
>>> I remember that I had an issue with the secondary indexes of Metadata
>> when
>>> I enabled the prefix search for our indexes. I believe with the current
>>> structure of the secondary indexes, they are unusable in a typical query
>>> execution job and so I disabled secondary index access for Metadata
>> indexes.
>>>
>>> I think we should stick to that at the moment at least. I also believe
>> that
>>> the point of the secondary indexes for Metadata is to validate ddl
>>> operations and not to be used for dataset access within Hyracks.
>>>
>>> Cheers,
>>> Abdullah.
>>>
>>> On Fri, Aug 14, 2015 at 12:10 AM, Ildar Absalyamov <
>>> ildar.absalyamov@gmail.com> wrote:
>>>
>>>> End of this\early next week would be the estimate.
>>>>
>>>> 2015-08-13 13:53 GMT-07:00 Mike Carey <dt...@gmail.com>:
>>>>
>>>>> Okay - cool - future improvements needed, clearly.
>>>>> Any ETA for the interim solution?
>>>>> Thx!
>>>>> Mike
>>>>>
>>>>>
>>>>> On 8/13/15 1:42 PM, Ildar Absalyamov wrote:
>>>>>
>>>>>> With respect to metadata change I am fixing the issue with
>> bootstrapped
>>>>>> secondary metadata indexes.
>>>>>> Strictly speaking they are not indexes per se, more similar to
>>>>>> materialized
>>>>>> views.
>>>>>> Writing\reading to\from those indexes is done manually bypassing the
>>>>>> regular index insert\delete codepath. This is really bad from
>> dogfooding
>>>>>> perspective, because it breaks the type validation logic when
>> Select\INL
>>>>>> Join rewrite rules are trying to fire.
>>>>>> However I have an idea how to use them without breaking the
>> validation
>>>>>> logic.
>>>>>>
>>>>>> 2015-08-13 13:11 GMT-07:00 Jochen Wiedmann <jochen.wiedmann@gmail.com
>>> :
>>>>>>
>>>>>> On Thu, Aug 13, 2015 at 10:08 PM, Mike Carey <dt...@gmail.com>
>> wrote:
>>>>>>>
>>>>>>>> So where are we on feeds and metadata - the release blockers?
>>>>>>>> (@Raman? Feed tutorial and API key? @Ildar? Your changes?)
>>>>>>>>
>>>>>>> I'd suggest to complete the "Verify distribution rights" section on
>>>>>>> http://incubator.apache.org/projects/asterixdb.html, before cutting
>> a
>>>>>>> release.
>>>>>>>
>>>>>>> Jochen
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Any world that can produce the Taj Mahal, William Shakespeare,
>>>>>>> and Stripe toothpaste can't be all bad. (C.R. MacNamara, One Two
>> Three)
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>> Ildar
>>>>
>>>
>>>
>>>
>>> --
>>> Amoudi, Abdullah.
>>
>> Best regards,
>> Ildar
>>
>>
>
>
> --
> Amoudi, Abdullah.
Best regards,
Ildar
Re: First Apache incubating release - blockers?
Posted by abdullah alamoudi <ba...@gmail.com>.
I assume you meant “Dataset" record type definition and not "Datatype" and
you're right. they are the same and can be elevated one level higher :-).
For everyone's information, here is a description of the problem:
In our regular secondary indexes, the structure of the tuple consists of
<SecondaryKeys,PrimaryKeys> and the Hyracks jobs get constructed based on
this assumption. However, our Metadata secondary indexes don't follow this
convention and so we end up with runtime errors whenever they are picked
for access.
Cheers,
Abdullah.
On Fri, Aug 14, 2015 at 8:59 AM, Ildar Absalyamov <
ildar.absalyamov@gmail.com> wrote:
> @Abdullah,
> OK, now I see that change. IMHO this is a hack, but I will adopt that
> approach that as well to submit the code review. However this issue really
> needs to be addressed.
>
> Side question, when you introduced external indexes you have added a field
> “ExternalDetails” (which in turn is a nested record) into “Datatype" record
> type definition in addition to existing “InternalDetails” field (BTW I
> believe either of the fields exists at a particular point in time, maybe be
> can use UNION for them?). However both of them have nested field
> “GroupName”, which is used by one of the secondary metadata indexes. Is
> that the same GroupName in both cases? If yes, can we move this field 1
> level up from {External\Internal}Details to avoid duplication?
>
> > On Aug 13, 2015, at 20:48, abdullah alamoudi <ba...@gmail.com> wrote:
> >
> > Ildar,
> > I remember that I had an issue with the secondary indexes of Metadata
> when
> > I enabled the prefix search for our indexes. I believe with the current
> > structure of the secondary indexes, they are unusable in a typical query
> > execution job and so I disabled secondary index access for Metadata
> indexes.
> >
> > I think we should stick to that at the moment at least. I also believe
> that
> > the point of the secondary indexes for Metadata is to validate ddl
> > operations and not to be used for dataset access within Hyracks.
> >
> > Cheers,
> > Abdullah.
> >
> > On Fri, Aug 14, 2015 at 12:10 AM, Ildar Absalyamov <
> > ildar.absalyamov@gmail.com> wrote:
> >
> >> End of this\early next week would be the estimate.
> >>
> >> 2015-08-13 13:53 GMT-07:00 Mike Carey <dt...@gmail.com>:
> >>
> >>> Okay - cool - future improvements needed, clearly.
> >>> Any ETA for the interim solution?
> >>> Thx!
> >>> Mike
> >>>
> >>>
> >>> On 8/13/15 1:42 PM, Ildar Absalyamov wrote:
> >>>
> >>>> With respect to metadata change I am fixing the issue with
> bootstrapped
> >>>> secondary metadata indexes.
> >>>> Strictly speaking they are not indexes per se, more similar to
> >>>> materialized
> >>>> views.
> >>>> Writing\reading to\from those indexes is done manually bypassing the
> >>>> regular index insert\delete codepath. This is really bad from
> dogfooding
> >>>> perspective, because it breaks the type validation logic when
> Select\INL
> >>>> Join rewrite rules are trying to fire.
> >>>> However I have an idea how to use them without breaking the
> validation
> >>>> logic.
> >>>>
> >>>> 2015-08-13 13:11 GMT-07:00 Jochen Wiedmann <jochen.wiedmann@gmail.com
> >:
> >>>>
> >>>> On Thu, Aug 13, 2015 at 10:08 PM, Mike Carey <dt...@gmail.com>
> wrote:
> >>>>>
> >>>>>> So where are we on feeds and metadata - the release blockers?
> >>>>>> (@Raman? Feed tutorial and API key? @Ildar? Your changes?)
> >>>>>>
> >>>>> I'd suggest to complete the "Verify distribution rights" section on
> >>>>> http://incubator.apache.org/projects/asterixdb.html, before cutting
> a
> >>>>> release.
> >>>>>
> >>>>> Jochen
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Any world that can produce the Taj Mahal, William Shakespeare,
> >>>>> and Stripe toothpaste can't be all bad. (C.R. MacNamara, One Two
> Three)
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
> >>
> >> --
> >> Best regards,
> >> Ildar
> >>
> >
> >
> >
> > --
> > Amoudi, Abdullah.
>
> Best regards,
> Ildar
>
>
--
Amoudi, Abdullah.
Re: First Apache incubating release - blockers?
Posted by Ildar Absalyamov <il...@gmail.com>.
@Abdullah,
OK, now I see that change. IMHO this is a hack, but I will adopt that approach that as well to submit the code review. However this issue really needs to be addressed.
Side question, when you introduced external indexes you have added a field “ExternalDetails” (which in turn is a nested record) into “Datatype" record type definition in addition to existing “InternalDetails” field (BTW I believe either of the fields exists at a particular point in time, maybe be can use UNION for them?). However both of them have nested field “GroupName”, which is used by one of the secondary metadata indexes. Is that the same GroupName in both cases? If yes, can we move this field 1 level up from {External\Internal}Details to avoid duplication?
> On Aug 13, 2015, at 20:48, abdullah alamoudi <ba...@gmail.com> wrote:
>
> Ildar,
> I remember that I had an issue with the secondary indexes of Metadata when
> I enabled the prefix search for our indexes. I believe with the current
> structure of the secondary indexes, they are unusable in a typical query
> execution job and so I disabled secondary index access for Metadata indexes.
>
> I think we should stick to that at the moment at least. I also believe that
> the point of the secondary indexes for Metadata is to validate ddl
> operations and not to be used for dataset access within Hyracks.
>
> Cheers,
> Abdullah.
>
> On Fri, Aug 14, 2015 at 12:10 AM, Ildar Absalyamov <
> ildar.absalyamov@gmail.com> wrote:
>
>> End of this\early next week would be the estimate.
>>
>> 2015-08-13 13:53 GMT-07:00 Mike Carey <dt...@gmail.com>:
>>
>>> Okay - cool - future improvements needed, clearly.
>>> Any ETA for the interim solution?
>>> Thx!
>>> Mike
>>>
>>>
>>> On 8/13/15 1:42 PM, Ildar Absalyamov wrote:
>>>
>>>> With respect to metadata change I am fixing the issue with bootstrapped
>>>> secondary metadata indexes.
>>>> Strictly speaking they are not indexes per se, more similar to
>>>> materialized
>>>> views.
>>>> Writing\reading to\from those indexes is done manually bypassing the
>>>> regular index insert\delete codepath. This is really bad from dogfooding
>>>> perspective, because it breaks the type validation logic when Select\INL
>>>> Join rewrite rules are trying to fire.
>>>> However I have an idea how to use them without breaking the validation
>>>> logic.
>>>>
>>>> 2015-08-13 13:11 GMT-07:00 Jochen Wiedmann <jo...@gmail.com>:
>>>>
>>>> On Thu, Aug 13, 2015 at 10:08 PM, Mike Carey <dt...@gmail.com> wrote:
>>>>>
>>>>>> So where are we on feeds and metadata - the release blockers?
>>>>>> (@Raman? Feed tutorial and API key? @Ildar? Your changes?)
>>>>>>
>>>>> I'd suggest to complete the "Verify distribution rights" section on
>>>>> http://incubator.apache.org/projects/asterixdb.html, before cutting a
>>>>> release.
>>>>>
>>>>> Jochen
>>>>>
>>>>>
>>>>> --
>>>>> Any world that can produce the Taj Mahal, William Shakespeare,
>>>>> and Stripe toothpaste can't be all bad. (C.R. MacNamara, One Two Three)
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> Best regards,
>> Ildar
>>
>
>
>
> --
> Amoudi, Abdullah.
Best regards,
Ildar
Re: First Apache incubating release - blockers?
Posted by abdullah alamoudi <ba...@gmail.com>.
Ildar,
I remember that I had an issue with the secondary indexes of Metadata when
I enabled the prefix search for our indexes. I believe with the current
structure of the secondary indexes, they are unusable in a typical query
execution job and so I disabled secondary index access for Metadata indexes.
I think we should stick to that at the moment at least. I also believe that
the point of the secondary indexes for Metadata is to validate ddl
operations and not to be used for dataset access within Hyracks.
Cheers,
Abdullah.
On Fri, Aug 14, 2015 at 12:10 AM, Ildar Absalyamov <
ildar.absalyamov@gmail.com> wrote:
> End of this\early next week would be the estimate.
>
> 2015-08-13 13:53 GMT-07:00 Mike Carey <dt...@gmail.com>:
>
> > Okay - cool - future improvements needed, clearly.
> > Any ETA for the interim solution?
> > Thx!
> > Mike
> >
> >
> > On 8/13/15 1:42 PM, Ildar Absalyamov wrote:
> >
> >> With respect to metadata change I am fixing the issue with bootstrapped
> >> secondary metadata indexes.
> >> Strictly speaking they are not indexes per se, more similar to
> >> materialized
> >> views.
> >> Writing\reading to\from those indexes is done manually bypassing the
> >> regular index insert\delete codepath. This is really bad from dogfooding
> >> perspective, because it breaks the type validation logic when Select\INL
> >> Join rewrite rules are trying to fire.
> >> However I have an idea how to use them without breaking the validation
> >> logic.
> >>
> >> 2015-08-13 13:11 GMT-07:00 Jochen Wiedmann <jo...@gmail.com>:
> >>
> >> On Thu, Aug 13, 2015 at 10:08 PM, Mike Carey <dt...@gmail.com> wrote:
> >>>
> >>>> So where are we on feeds and metadata - the release blockers?
> >>>> (@Raman? Feed tutorial and API key? @Ildar? Your changes?)
> >>>>
> >>> I'd suggest to complete the "Verify distribution rights" section on
> >>> http://incubator.apache.org/projects/asterixdb.html, before cutting a
> >>> release.
> >>>
> >>> Jochen
> >>>
> >>>
> >>> --
> >>> Any world that can produce the Taj Mahal, William Shakespeare,
> >>> and Stripe toothpaste can't be all bad. (C.R. MacNamara, One Two Three)
> >>>
> >>>
> >>
> >>
> >
>
>
> --
> Best regards,
> Ildar
>
--
Amoudi, Abdullah.
Re: First Apache incubating release - blockers?
Posted by Ildar Absalyamov <il...@gmail.com>.
End of this\early next week would be the estimate.
2015-08-13 13:53 GMT-07:00 Mike Carey <dt...@gmail.com>:
> Okay - cool - future improvements needed, clearly.
> Any ETA for the interim solution?
> Thx!
> Mike
>
>
> On 8/13/15 1:42 PM, Ildar Absalyamov wrote:
>
>> With respect to metadata change I am fixing the issue with bootstrapped
>> secondary metadata indexes.
>> Strictly speaking they are not indexes per se, more similar to
>> materialized
>> views.
>> Writing\reading to\from those indexes is done manually bypassing the
>> regular index insert\delete codepath. This is really bad from dogfooding
>> perspective, because it breaks the type validation logic when Select\INL
>> Join rewrite rules are trying to fire.
>> However I have an idea how to use them without breaking the validation
>> logic.
>>
>> 2015-08-13 13:11 GMT-07:00 Jochen Wiedmann <jo...@gmail.com>:
>>
>> On Thu, Aug 13, 2015 at 10:08 PM, Mike Carey <dt...@gmail.com> wrote:
>>>
>>>> So where are we on feeds and metadata - the release blockers?
>>>> (@Raman? Feed tutorial and API key? @Ildar? Your changes?)
>>>>
>>> I'd suggest to complete the "Verify distribution rights" section on
>>> http://incubator.apache.org/projects/asterixdb.html, before cutting a
>>> release.
>>>
>>> Jochen
>>>
>>>
>>> --
>>> Any world that can produce the Taj Mahal, William Shakespeare,
>>> and Stripe toothpaste can't be all bad. (C.R. MacNamara, One Two Three)
>>>
>>>
>>
>>
>
--
Best regards,
Ildar
Re: First Apache incubating release - blockers?
Posted by Mike Carey <dt...@gmail.com>.
Okay - cool - future improvements needed, clearly.
Any ETA for the interim solution?
Thx!
Mike
On 8/13/15 1:42 PM, Ildar Absalyamov wrote:
> With respect to metadata change I am fixing the issue with bootstrapped
> secondary metadata indexes.
> Strictly speaking they are not indexes per se, more similar to materialized
> views.
> Writing\reading to\from those indexes is done manually bypassing the
> regular index insert\delete codepath. This is really bad from dogfooding
> perspective, because it breaks the type validation logic when Select\INL
> Join rewrite rules are trying to fire.
> However I have an idea how to use them without breaking the validation
> logic.
>
> 2015-08-13 13:11 GMT-07:00 Jochen Wiedmann <jo...@gmail.com>:
>
>> On Thu, Aug 13, 2015 at 10:08 PM, Mike Carey <dt...@gmail.com> wrote:
>>> So where are we on feeds and metadata - the release blockers?
>>> (@Raman? Feed tutorial and API key? @Ildar? Your changes?)
>> I'd suggest to complete the "Verify distribution rights" section on
>> http://incubator.apache.org/projects/asterixdb.html, before cutting a
>> release.
>>
>> Jochen
>>
>>
>> --
>> Any world that can produce the Taj Mahal, William Shakespeare,
>> and Stripe toothpaste can't be all bad. (C.R. MacNamara, One Two Three)
>>
>
>
Re: First Apache incubating release - blockers?
Posted by Ildar Absalyamov <il...@gmail.com>.
With respect to metadata change I am fixing the issue with bootstrapped
secondary metadata indexes.
Strictly speaking they are not indexes per se, more similar to materialized
views.
Writing\reading to\from those indexes is done manually bypassing the
regular index insert\delete codepath. This is really bad from dogfooding
perspective, because it breaks the type validation logic when Select\INL
Join rewrite rules are trying to fire.
However I have an idea how to use them without breaking the validation
logic.
2015-08-13 13:11 GMT-07:00 Jochen Wiedmann <jo...@gmail.com>:
> On Thu, Aug 13, 2015 at 10:08 PM, Mike Carey <dt...@gmail.com> wrote:
> > So where are we on feeds and metadata - the release blockers?
> > (@Raman? Feed tutorial and API key? @Ildar? Your changes?)
>
> I'd suggest to complete the "Verify distribution rights" section on
> http://incubator.apache.org/projects/asterixdb.html, before cutting a
> release.
>
> Jochen
>
>
> --
> Any world that can produce the Taj Mahal, William Shakespeare,
> and Stripe toothpaste can't be all bad. (C.R. MacNamara, One Two Three)
>
--
Best regards,
Ildar
Re: First Apache incubating release - blockers?
Posted by Jochen Wiedmann <jo...@gmail.com>.
On Thu, Aug 13, 2015 at 10:08 PM, Mike Carey <dt...@gmail.com> wrote:
> So where are we on feeds and metadata - the release blockers?
> (@Raman? Feed tutorial and API key? @Ildar? Your changes?)
I'd suggest to complete the "Verify distribution rights" section on
http://incubator.apache.org/projects/asterixdb.html, before cutting a
release.
Jochen
--
Any world that can produce the Taj Mahal, William Shakespeare,
and Stripe toothpaste can't be all bad. (C.R. MacNamara, One Two Three)