You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@asterixdb.apache.org by Steven Jacobs <sj...@ucr.edu> on 2016/09/16 17:47:53 UTC
Retain inputs on index-search
Hi,
I have come across an issue where retainInputs is always set to false when
creating an index search in the select case. This was because it was
previously assumed that variables before the search don't need to be kept
after the search. In the nonPure case, this assumption is wrong. I'm now
looking at two possible solution:
1) Add specialization for nonPure cases to set retainInputs to true
2) Always set retainInputs to true. This has performance implications, as
well as plan changes, since now a project will be added above the index
search in most cases, and the index search operator will have extra fields
attached to it's tuples (although in general this will be only single
values).
I wanted to meet with whoever is interested, to decide the best way to
address this. If you are interested in joining the meeting, please reply.
For now I'll tentatively set the meeting for Tuesday at 11 AM.
Steven
Re: Retain inputs on index-search
Posted by Taewoo Kim <wa...@gmail.com>.
+1. At least the one of most important chunk of index-only plan can be
merged soon, then. :-)
Best,
Taewoo
On Fri, Sep 16, 2016 at 1:08 PM, Mike Carey <dt...@gmail.com> wrote:
> Cool - let's break out that bit of logic and maybe do that as a separate
> item/review?
>
>
>
> On 9/16/16 11:34 AM, Steven Jacobs wrote:
>
>> I will plan to work through Taewoo's idea unless there are objections. I
>> think it won't be too much of a "hack" since the code is needed for the
>> index-only plan as well, and it will not be nonPure-specific.
>> Steven
>>
>> On Friday, September 16, 2016, Taewoo Kim <wa...@gmail.com> wrote:
>>
>> I just had a conversation with Steven.
>>>
>>> There is third choice: if a variable is used after SELECT operator (which
>>> will be transformed into an UNNESTMAP operator), then we need to
>>> retainInput as true. Otherwise, we can set it as false. However,
>>> currently,
>>> the IntroduceSelectAccessMethodRule traverses the plan from bottom to
>>> the
>>> top and makes it difficult to check whether a variable is used after a
>>> SELECT operator. So, we need to change the rule that it starts from TOP
>>> level and collects used variables information until it sees SELECT
>>> operator. Actually, that's what I do in the index-only plan. Much of
>>> functionality is already there. And I showed Steven the link to the
>>> IntroduceSelectAccessMethodRule in Index-only plan.
>>>
>>>
>>> Best,
>>> Taewoo
>>>
>>> On Fri, Sep 16, 2016 at 11:21 AM, Mike Carey <dtabass@gmail.com
>>> <javascript:_e(%7B%7D,'cvml','dtabass@gmail.com');>> wrote:
>>>
>>> Ditto...
>>>>
>>>> On Sep 16, 2016 11:15 AM, "Yingyi Bu" <buyingyi@gmail.com
>>>> <javascript:_e(%7B%7D,'cvml','buyingyi@gmail.com');>> wrote:
>>>>
>>>> I'd like to join the discussion.
>>>>>
>>>>> Best,
>>>>> Yingyi
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Sep 16, 2016 at 10:47 AM, Steven Jacobs <sjaco002@ucr.edu
>>>>> <javascript:_e(%7B%7D,'cvml','sjaco002@ucr.edu');>> wrote:
>>>>>
>>>>> Hi,
>>>>>> I have come across an issue where retainInputs is always set to false
>>>>>> when creating an index search in the select case. This was because it
>>>>>> was
>>>>>> previously assumed that variables before the search don't need to be
>>>>>> kept
>>>>>> after the search. In the nonPure case, this assumption is wrong. I'm
>>>>>> now
>>>>>> looking at two possible solution:
>>>>>>
>>>>>> 1) Add specialization for nonPure cases to set retainInputs to true
>>>>>> 2) Always set retainInputs to true. This has performance implications,
>>>>>> as well as plan changes, since now a project will be added above the
>>>>>> index
>>>>>> search in most cases, and the index search operator will have extra
>>>>>> fields
>>>>>> attached to it's tuples (although in general this will be only single
>>>>>> values).
>>>>>>
>>>>>> I wanted to meet with whoever is interested, to decide the best way to
>>>>>> address this. If you are interested in joining the meeting, please
>>>>>> reply.
>>>>>> For now I'll tentatively set the meeting for Tuesday at 11 AM.
>>>>>>
>>>>>> Steven
>>>>>>
>>>>>>
>>>>>
>
Re: Retain inputs on index-search
Posted by Mike Carey <dt...@gmail.com>.
Cool - let's break out that bit of logic and maybe do that as a separate
item/review?
On 9/16/16 11:34 AM, Steven Jacobs wrote:
> I will plan to work through Taewoo's idea unless there are objections. I
> think it won't be too much of a "hack" since the code is needed for the
> index-only plan as well, and it will not be nonPure-specific.
> Steven
>
> On Friday, September 16, 2016, Taewoo Kim <wa...@gmail.com> wrote:
>
>> I just had a conversation with Steven.
>>
>> There is third choice: if a variable is used after SELECT operator (which
>> will be transformed into an UNNESTMAP operator), then we need to
>> retainInput as true. Otherwise, we can set it as false. However, currently,
>> the IntroduceSelectAccessMethodRule traverses the plan from bottom to the
>> top and makes it difficult to check whether a variable is used after a
>> SELECT operator. So, we need to change the rule that it starts from TOP
>> level and collects used variables information until it sees SELECT
>> operator. Actually, that's what I do in the index-only plan. Much of
>> functionality is already there. And I showed Steven the link to the
>> IntroduceSelectAccessMethodRule in Index-only plan.
>>
>>
>> Best,
>> Taewoo
>>
>> On Fri, Sep 16, 2016 at 11:21 AM, Mike Carey <dtabass@gmail.com
>> <javascript:_e(%7B%7D,'cvml','dtabass@gmail.com');>> wrote:
>>
>>> Ditto...
>>>
>>> On Sep 16, 2016 11:15 AM, "Yingyi Bu" <buyingyi@gmail.com
>>> <javascript:_e(%7B%7D,'cvml','buyingyi@gmail.com');>> wrote:
>>>
>>>> I'd like to join the discussion.
>>>>
>>>> Best,
>>>> Yingyi
>>>>
>>>>
>>>>
>>>> On Fri, Sep 16, 2016 at 10:47 AM, Steven Jacobs <sjaco002@ucr.edu
>>>> <javascript:_e(%7B%7D,'cvml','sjaco002@ucr.edu');>> wrote:
>>>>
>>>>> Hi,
>>>>> I have come across an issue where retainInputs is always set to false
>>>>> when creating an index search in the select case. This was because it was
>>>>> previously assumed that variables before the search don't need to be kept
>>>>> after the search. In the nonPure case, this assumption is wrong. I'm now
>>>>> looking at two possible solution:
>>>>>
>>>>> 1) Add specialization for nonPure cases to set retainInputs to true
>>>>> 2) Always set retainInputs to true. This has performance implications,
>>>>> as well as plan changes, since now a project will be added above the index
>>>>> search in most cases, and the index search operator will have extra fields
>>>>> attached to it's tuples (although in general this will be only single
>>>>> values).
>>>>>
>>>>> I wanted to meet with whoever is interested, to decide the best way to
>>>>> address this. If you are interested in joining the meeting, please reply.
>>>>> For now I'll tentatively set the meeting for Tuesday at 11 AM.
>>>>>
>>>>> Steven
>>>>>
>>>>
Re: Retain inputs on index-search
Posted by Steven Jacobs <sj...@ucr.edu>.
I will plan to work through Taewoo's idea unless there are objections. I
think it won't be too much of a "hack" since the code is needed for the
index-only plan as well, and it will not be nonPure-specific.
Steven
On Friday, September 16, 2016, Taewoo Kim <wa...@gmail.com> wrote:
> I just had a conversation with Steven.
>
> There is third choice: if a variable is used after SELECT operator (which
> will be transformed into an UNNESTMAP operator), then we need to
> retainInput as true. Otherwise, we can set it as false. However, currently,
> the IntroduceSelectAccessMethodRule traverses the plan from bottom to the
> top and makes it difficult to check whether a variable is used after a
> SELECT operator. So, we need to change the rule that it starts from TOP
> level and collects used variables information until it sees SELECT
> operator. Actually, that's what I do in the index-only plan. Much of
> functionality is already there. And I showed Steven the link to the
> IntroduceSelectAccessMethodRule in Index-only plan.
>
>
> Best,
> Taewoo
>
> On Fri, Sep 16, 2016 at 11:21 AM, Mike Carey <dtabass@gmail.com
> <javascript:_e(%7B%7D,'cvml','dtabass@gmail.com');>> wrote:
>
>> Ditto...
>>
>> On Sep 16, 2016 11:15 AM, "Yingyi Bu" <buyingyi@gmail.com
>> <javascript:_e(%7B%7D,'cvml','buyingyi@gmail.com');>> wrote:
>>
>>> I'd like to join the discussion.
>>>
>>> Best,
>>> Yingyi
>>>
>>>
>>>
>>> On Fri, Sep 16, 2016 at 10:47 AM, Steven Jacobs <sjaco002@ucr.edu
>>> <javascript:_e(%7B%7D,'cvml','sjaco002@ucr.edu');>> wrote:
>>>
>>>> Hi,
>>>> I have come across an issue where retainInputs is always set to false
>>>> when creating an index search in the select case. This was because it was
>>>> previously assumed that variables before the search don't need to be kept
>>>> after the search. In the nonPure case, this assumption is wrong. I'm now
>>>> looking at two possible solution:
>>>>
>>>> 1) Add specialization for nonPure cases to set retainInputs to true
>>>> 2) Always set retainInputs to true. This has performance implications,
>>>> as well as plan changes, since now a project will be added above the index
>>>> search in most cases, and the index search operator will have extra fields
>>>> attached to it's tuples (although in general this will be only single
>>>> values).
>>>>
>>>> I wanted to meet with whoever is interested, to decide the best way to
>>>> address this. If you are interested in joining the meeting, please reply.
>>>> For now I'll tentatively set the meeting for Tuesday at 11 AM.
>>>>
>>>> Steven
>>>>
>>>
>>>
>
Re: Retain inputs on index-search
Posted by Taewoo Kim <wa...@gmail.com>.
I just had a conversation with Steven.
There is third choice: if a variable is used after SELECT operator (which
will be transformed into an UNNESTMAP operator), then we need to
retainInput as true. Otherwise, we can set it as false. However, currently,
the IntroduceSelectAccessMethodRule traverses the plan from bottom to the
top and makes it difficult to check whether a variable is used after a
SELECT operator. So, we need to change the rule that it starts from TOP
level and collects used variables information until it sees SELECT
operator. Actually, that's what I do in the index-only plan. Much of
functionality is already there. And I showed Steven the link to the
IntroduceSelectAccessMethodRule in Index-only plan.
Best,
Taewoo
On Fri, Sep 16, 2016 at 11:21 AM, Mike Carey <dt...@gmail.com> wrote:
> Ditto...
>
> On Sep 16, 2016 11:15 AM, "Yingyi Bu" <bu...@gmail.com> wrote:
>
>> I'd like to join the discussion.
>>
>> Best,
>> Yingyi
>>
>>
>>
>> On Fri, Sep 16, 2016 at 10:47 AM, Steven Jacobs <sj...@ucr.edu> wrote:
>>
>>> Hi,
>>> I have come across an issue where retainInputs is always set to false
>>> when creating an index search in the select case. This was because it was
>>> previously assumed that variables before the search don't need to be kept
>>> after the search. In the nonPure case, this assumption is wrong. I'm now
>>> looking at two possible solution:
>>>
>>> 1) Add specialization for nonPure cases to set retainInputs to true
>>> 2) Always set retainInputs to true. This has performance implications,
>>> as well as plan changes, since now a project will be added above the index
>>> search in most cases, and the index search operator will have extra fields
>>> attached to it's tuples (although in general this will be only single
>>> values).
>>>
>>> I wanted to meet with whoever is interested, to decide the best way to
>>> address this. If you are interested in joining the meeting, please reply.
>>> For now I'll tentatively set the meeting for Tuesday at 11 AM.
>>>
>>> Steven
>>>
>>
>>
Re: Retain inputs on index-search
Posted by Mike Carey <dt...@gmail.com>.
Ditto...
On Sep 16, 2016 11:15 AM, "Yingyi Bu" <bu...@gmail.com> wrote:
> I'd like to join the discussion.
>
> Best,
> Yingyi
>
>
>
> On Fri, Sep 16, 2016 at 10:47 AM, Steven Jacobs <sj...@ucr.edu> wrote:
>
>> Hi,
>> I have come across an issue where retainInputs is always set to false
>> when creating an index search in the select case. This was because it was
>> previously assumed that variables before the search don't need to be kept
>> after the search. In the nonPure case, this assumption is wrong. I'm now
>> looking at two possible solution:
>>
>> 1) Add specialization for nonPure cases to set retainInputs to true
>> 2) Always set retainInputs to true. This has performance implications, as
>> well as plan changes, since now a project will be added above the index
>> search in most cases, and the index search operator will have extra fields
>> attached to it's tuples (although in general this will be only single
>> values).
>>
>> I wanted to meet with whoever is interested, to decide the best way to
>> address this. If you are interested in joining the meeting, please reply.
>> For now I'll tentatively set the meeting for Tuesday at 11 AM.
>>
>> Steven
>>
>
>
Re: Retain inputs on index-search
Posted by Yingyi Bu <bu...@gmail.com>.
I'd like to join the discussion.
Best,
Yingyi
On Fri, Sep 16, 2016 at 10:47 AM, Steven Jacobs <sj...@ucr.edu> wrote:
> Hi,
> I have come across an issue where retainInputs is always set to false when
> creating an index search in the select case. This was because it was
> previously assumed that variables before the search don't need to be kept
> after the search. In the nonPure case, this assumption is wrong. I'm now
> looking at two possible solution:
>
> 1) Add specialization for nonPure cases to set retainInputs to true
> 2) Always set retainInputs to true. This has performance implications, as
> well as plan changes, since now a project will be added above the index
> search in most cases, and the index search operator will have extra fields
> attached to it's tuples (although in general this will be only single
> values).
>
> I wanted to meet with whoever is interested, to decide the best way to
> address this. If you are interested in joining the meeting, please reply.
> For now I'll tentatively set the meeting for Tuesday at 11 AM.
>
> Steven
>