You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by Stefán Baxter <st...@activitystream.com> on 2016/01/14 23:52:20 UTC
Lucene Plugin :: Join Filter and pushdown
Hi,
I'm working on the Lucene plugin (see previous email) and the focus now is
support for joins with filter push-down to avoid the default table scan
that is provided by default.
I'm fairly new to Drill and in over my head, to be honest, but this is fun
and with this addition the Lucene plugin could be come usable or at least
worth exploring.
Is there anyone here that could assist me a bit?
Current status:
- The lucene plugin is working and join filters are partially
- RelOptRuleOperand is constructed and DrillJoinRel.conditions are
processed by a sceleton class (The "normal" queries are already being
processed fairly well)
There are probably more things involved then I can imagine at this point
and perhaps I'm naive in thinking someone has the time to assist a relative
noob on such a task but examples are also appreciated. The plugins that I
have seen seem to have relatively no join-filter logic so a rich
example/blueprint would also be great.
Regards,
-Stefán
Re: Lucene Plugin :: Join Filter and pushdown
Posted by Stefán Baxter <st...@activitystream.com>.
Hi,
I would be more than happy to pay for this support if someone renders
professional services in this area and is willing to assist us in getting
this going.
What Drill reader/plugins currently support joins based on filters/indexes
that I can study?
Regards,
-Stefan
On Thu, Jan 14, 2016 at 11:08 PM, rahul challapalli <
challapallirahul@gmail.com> wrote:
> Use Case : In the case of a left join between a non-index table and a
> lucene index, it is more efficient to read the join keys from the non-index
> table and push them into the LuceneGroupScan. This way we can avoid reading
> the whole index.
> I was suggesting converting the plan for Q1 into a plan similar to Q2 using
> an optimizer rule.
> Q1.) select * from tbl1 left join tbl2 on tbl1.col1 = tbl2.col1
> Q2.) select * from tbl1 left join (select * from tbl2 where tbl2.col1 in
> (select col1 from tbl1))
>
> Any other suggestions or pointers are appreciated
>
> - Rahul
>
>
> On Thu, Jan 14, 2016 at 2:52 PM, Stefán Baxter <st...@activitystream.com>
> wrote:
>
> > Hi,
> >
> > I'm working on the Lucene plugin (see previous email) and the focus now
> is
> > support for joins with filter push-down to avoid the default table scan
> > that is provided by default.
> >
> > I'm fairly new to Drill and in over my head, to be honest, but this is
> fun
> > and with this addition the Lucene plugin could be come usable or at least
> > worth exploring.
> >
> > Is there anyone here that could assist me a bit?
> >
> > Current status:
> >
> > - The lucene plugin is working and join filters are partially
> > - RelOptRuleOperand is constructed and DrillJoinRel.conditions are
> > processed by a sceleton class (The "normal" queries are already being
> > processed fairly well)
> >
> > There are probably more things involved then I can imagine at this point
> > and perhaps I'm naive in thinking someone has the time to assist a
> relative
> > noob on such a task but examples are also appreciated. The plugins that I
> > have seen seem to have relatively no join-filter logic so a rich
> > example/blueprint would also be great.
> >
> > Regards,
> > -Stefán
> >
>
Re: Lucene Plugin :: Join Filter and pushdown
Posted by Stefán Baxter <st...@activitystream.com>.
Hi,
I would be more than happy to pay for this support if someone renders
professional services in this area and is willing to assist us in getting
this going.
What Drill reader/plugins currently support joins based on filters/indexes
that I can study?
Regards,
-Stefan
On Thu, Jan 14, 2016 at 11:08 PM, rahul challapalli <
challapallirahul@gmail.com> wrote:
> Use Case : In the case of a left join between a non-index table and a
> lucene index, it is more efficient to read the join keys from the non-index
> table and push them into the LuceneGroupScan. This way we can avoid reading
> the whole index.
> I was suggesting converting the plan for Q1 into a plan similar to Q2 using
> an optimizer rule.
> Q1.) select * from tbl1 left join tbl2 on tbl1.col1 = tbl2.col1
> Q2.) select * from tbl1 left join (select * from tbl2 where tbl2.col1 in
> (select col1 from tbl1))
>
> Any other suggestions or pointers are appreciated
>
> - Rahul
>
>
> On Thu, Jan 14, 2016 at 2:52 PM, Stefán Baxter <st...@activitystream.com>
> wrote:
>
> > Hi,
> >
> > I'm working on the Lucene plugin (see previous email) and the focus now
> is
> > support for joins with filter push-down to avoid the default table scan
> > that is provided by default.
> >
> > I'm fairly new to Drill and in over my head, to be honest, but this is
> fun
> > and with this addition the Lucene plugin could be come usable or at least
> > worth exploring.
> >
> > Is there anyone here that could assist me a bit?
> >
> > Current status:
> >
> > - The lucene plugin is working and join filters are partially
> > - RelOptRuleOperand is constructed and DrillJoinRel.conditions are
> > processed by a sceleton class (The "normal" queries are already being
> > processed fairly well)
> >
> > There are probably more things involved then I can imagine at this point
> > and perhaps I'm naive in thinking someone has the time to assist a
> relative
> > noob on such a task but examples are also appreciated. The plugins that I
> > have seen seem to have relatively no join-filter logic so a rich
> > example/blueprint would also be great.
> >
> > Regards,
> > -Stefán
> >
>
Re: Lucene Plugin :: Join Filter and pushdown
Posted by rahul challapalli <ch...@gmail.com>.
Use Case : In the case of a left join between a non-index table and a
lucene index, it is more efficient to read the join keys from the non-index
table and push them into the LuceneGroupScan. This way we can avoid reading
the whole index.
I was suggesting converting the plan for Q1 into a plan similar to Q2 using
an optimizer rule.
Q1.) select * from tbl1 left join tbl2 on tbl1.col1 = tbl2.col1
Q2.) select * from tbl1 left join (select * from tbl2 where tbl2.col1 in
(select col1 from tbl1))
Any other suggestions or pointers are appreciated
- Rahul
On Thu, Jan 14, 2016 at 2:52 PM, Stefán Baxter <st...@activitystream.com>
wrote:
> Hi,
>
> I'm working on the Lucene plugin (see previous email) and the focus now is
> support for joins with filter push-down to avoid the default table scan
> that is provided by default.
>
> I'm fairly new to Drill and in over my head, to be honest, but this is fun
> and with this addition the Lucene plugin could be come usable or at least
> worth exploring.
>
> Is there anyone here that could assist me a bit?
>
> Current status:
>
> - The lucene plugin is working and join filters are partially
> - RelOptRuleOperand is constructed and DrillJoinRel.conditions are
> processed by a sceleton class (The "normal" queries are already being
> processed fairly well)
>
> There are probably more things involved then I can imagine at this point
> and perhaps I'm naive in thinking someone has the time to assist a relative
> noob on such a task but examples are also appreciated. The plugins that I
> have seen seem to have relatively no join-filter logic so a rich
> example/blueprint would also be great.
>
> Regards,
> -Stefán
>
Re: Lucene Plugin :: Join Filter and pushdown
Posted by rahul challapalli <ch...@gmail.com>.
Use Case : In the case of a left join between a non-index table and a
lucene index, it is more efficient to read the join keys from the non-index
table and push them into the LuceneGroupScan. This way we can avoid reading
the whole index.
I was suggesting converting the plan for Q1 into a plan similar to Q2 using
an optimizer rule.
Q1.) select * from tbl1 left join tbl2 on tbl1.col1 = tbl2.col1
Q2.) select * from tbl1 left join (select * from tbl2 where tbl2.col1 in
(select col1 from tbl1))
Any other suggestions or pointers are appreciated
- Rahul
On Thu, Jan 14, 2016 at 2:52 PM, Stefán Baxter <st...@activitystream.com>
wrote:
> Hi,
>
> I'm working on the Lucene plugin (see previous email) and the focus now is
> support for joins with filter push-down to avoid the default table scan
> that is provided by default.
>
> I'm fairly new to Drill and in over my head, to be honest, but this is fun
> and with this addition the Lucene plugin could be come usable or at least
> worth exploring.
>
> Is there anyone here that could assist me a bit?
>
> Current status:
>
> - The lucene plugin is working and join filters are partially
> - RelOptRuleOperand is constructed and DrillJoinRel.conditions are
> processed by a sceleton class (The "normal" queries are already being
> processed fairly well)
>
> There are probably more things involved then I can imagine at this point
> and perhaps I'm naive in thinking someone has the time to assist a relative
> noob on such a task but examples are also appreciated. The plugins that I
> have seen seem to have relatively no join-filter logic so a rich
> example/blueprint would also be great.
>
> Regards,
> -Stefán
>