Access to where clause in a FormatPlugin

[Lee Butts <le...@mahifx.com>]

Hi,

I'm having a go at adding a format plugin to Drill and would like to be
able to access the where clause of a running query in order to skip through
a large data file (currently our data files are in 1 hour chunks and can be
hundreds of MBs).

In my use case the where clause would contain a "between" clause that I
could use to skip/filter rows based on a timestamp column.

I can't see anywhere in the various contexts that a format plugin has
access to that the query AST or something similar is held.

Is that something that drill could ever support?

thanks

Lee

[image: MahiFX] <https://mahifx.com/>

Lee Butts
Technology
Level 3, 50 Victoria Street
Christchurch 8013
New Zealand
Mobile: +64 226 908315
Skype: leemahifx

<https://www.facebook.com/mahifx>   <https://twitter.com/mahiforex>
<https://www.youtube.com/channel/UCxCiKv8XEM8GATA8kFFqsAw>
<https://www.linkedin.com/company/mahifx>
<https://mahifx.com/blog/feed.xml>   <https://plus.google.com/+Mahifx/posts>


[image: MFX Compass] <http://mfxcompass.mahifx.com/>

-- 


------------------------------

*IMPORTANT NOTICE*: MahiFX Limited (*MahiFX*), (company no. 2446590) is 
registered in New Zealand. MahiFX is authorised and regulated by the 
Australian Securities and Investment Commission ARBN 152 535 085, AFSL 
414198. This email, its attachments and any rights attaching hereto are 
confidential and intended exclusively for the person to whom the email is 
addressed. If you are not the intended recipient, do not read, copy, 
disclose or use the contents in any way. Please notify the sender by return 
email and destroy the email and attachments immediately. MahiFX does not 
accept any liability for any changes made to this email or attachments 
after sending by MahiFX. You must scan this email and attachments for 
viruses. The opinions expressed are not necessarily those of MahiFX. MahiFX 
accepts no liability for any loss, damage or consequence, whether caused by 
our own negligence or not, resulting directly or indirectly from the use of 
this email and attachments.

For more information about MahiFX Limited see mahifx.com.

Re: Access to where clause in a FormatPlugin

[Aditya <ad...@gmail.com>]

Hi Lee,

Welcome to Drill!

Yes, Drill does allow the storage/format plugins to inspect the WHERE
clause and modify the query plan using optimizer rules.

Please take a look at HBase storage plugin for similar example

The storage plugin needs to specify the set of optimizer rule[1] and then
parse and modify[2] the query plan in the optimizer rule.

aditya...

[1]
https://github.com/apache/drill/blob/master/contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseStoragePlugin.java#L80
[2]
https://github.com/apache/drill/blob/master/contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBasePushFilterIntoScan.java

On Tue, Mar 22, 2016 at 7:59 PM, Lee Butts <le...@mahifx.com> wrote:

> Hi,
>
> I'm having a go at adding a format plugin to Drill and would like to be
> able to access the where clause of a running query in order to skip through
> a large data file (currently our data files are in 1 hour chunks and can be
> hundreds of MBs).
>
> In my use case the where clause would contain a "between" clause that I
> could use to skip/filter rows based on a timestamp column.
>
> I can't see anywhere in the various contexts that a format plugin has
> access to that the query AST or something similar is held.
>
> Is that something that drill could ever support?
>
> thanks
>
> Lee
>
> [image: MahiFX] <https://mahifx.com/>
>
> Lee Butts
> Technology
> Level 3, 50 Victoria Street
> Christchurch 8013
> New Zealand
> Mobile: +64 226 908315
> Skype: leemahifx
>
> <https://www.facebook.com/mahifx>   <https://twitter.com/mahiforex>
> <https://www.youtube.com/channel/UCxCiKv8XEM8GATA8kFFqsAw>
> <https://www.linkedin.com/company/mahifx>
> <https://mahifx.com/blog/feed.xml>   <
> https://plus.google.com/+Mahifx/posts>
>
>
> [image: MFX Compass] <http://mfxcompass.mahifx.com/>
>
> --
>
>
> ------------------------------
>
> *IMPORTANT NOTICE*: MahiFX Limited (*MahiFX*), (company no. 2446590) is
> registered in New Zealand. MahiFX is authorised and regulated by the
> Australian Securities and Investment Commission ARBN 152 535 085, AFSL
> 414198. This email, its attachments and any rights attaching hereto are
> confidential and intended exclusively for the person to whom the email is
> addressed. If you are not the intended recipient, do not read, copy,
> disclose or use the contents in any way. Please notify the sender by return
> email and destroy the email and attachments immediately. MahiFX does not
> accept any liability for any changes made to this email or attachments
> after sending by MahiFX. You must scan this email and attachments for
> viruses. The opinions expressed are not necessarily those of MahiFX. MahiFX
> accepts no liability for any loss, damage or consequence, whether caused by
> our own negligence or not, resulting directly or indirectly from the use of
> this email and attachments.
>
> For more information about MahiFX Limited see mahifx.com.
>