You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by Sudip Mukherjee <mu...@hotmail.com> on 2015/08/06 11:01:11 UTC
Apache Drill: How does the plug-in know that an aggregate function
is applied
Hi,
I am using apache drill recently with mongodb and trying to write a basic plug-in for apache solr. Wanted to know how does the plug-in know that an count(*) query has been applied so that the query to data-source can be optimized? Can i get it if I extend AbstractExprVisitor class?
Thanks,Sudip Mukherjee
RE: Apache Drill: How does the plug-in know that an aggregate
function is applied
Posted by Sudip Mukherjee <sm...@commvault.com>.
Hi,
Thanks a lot for this. I could use isSkipQuery for the count(*) in sql statement. Need to figure out if I can optimize the solr query (if applicable) for other aggregate functions like SUM where I can use a simple stats query on the given field.
Regards,
Sudip
-----Original Message-----
From: Hanifi Gunes [mailto:hgunes@maprtech.com]
Sent: 10 August 2015 PM 11:27
To: user; dev@drill.apache.org
Subject: Re: Apache Drill: How does the plug-in know that an aggregate function is applied
+dev
+1 to Hakim. AbstractRR#isSkipQuery is the way to go. If you want more
details on this you should check out DRILL-2358[1] that is an umbrella issue targeting to make count(*) queries more efficient per storage plugin.
Currently (I guess) JSON and Mongo(?) readers support it though.
1:
https://github.com/apache/drill/commit/54df129cab544c3df8e75a7dae3f85a91a9ded5a
On Thu, Aug 6, 2015 at 6:50 PM, Abdel Hakim Deneche <ad...@maprtech.com>
wrote:
> Hi Sudip,
>
> I'm not really an expert in this matter but I came recently across
> isSkipQuery() method in AbstractRecordReader, it's javadoc states:
>
> *Returns true if reader should skip all of the columns, reporting
> number of
> > records only. Handling of a skip query is storage plugin-specific.*
>
>
> You can take a look at JSONRecordReader for an example on how to use
> isSkipQuery() to optimize the reading.
>
> Thanks
>
> On Thu, Aug 6, 2015 at 2:01 AM, Sudip Mukherjee <
> mukherjeesudip1@hotmail.com
> > wrote:
>
> > Hi,
> > I am using apache drill recently with mongodb and trying to write a
> > basic plug-in for apache solr. Wanted to know how does the plug-in
> > know that an
> > count(*) query has been applied so that the query to data-source can
> > be optimized? Can i get it if I extend AbstractExprVisitor class?
> >
> > Thanks,Sudip Mukherjee
>
>
>
>
> --
>
> Abdelhakim Deneche
>
> Software Engineer
>
> <http://www.mapr.com/>
>
>
> Now Available - Free Hadoop On-Demand Training <
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm
> _campaign=Free%20available
> >
>
***************************Legal Disclaimer***************************
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**********************************************************************
RE: Apache Drill: How does the plug-in know that an aggregate
function is applied
Posted by Sudip Mukherjee <sm...@commvault.com>.
Hi,
Thanks a lot for this. I could use isSkipQuery for the count(*) in sql statement. Need to figure out if I can optimize the solr query (if applicable) for other aggregate functions like SUM where I can use a simple stats query on the given field.
Regards,
Sudip
-----Original Message-----
From: Hanifi Gunes [mailto:hgunes@maprtech.com]
Sent: 10 August 2015 PM 11:27
To: user; dev@drill.apache.org
Subject: Re: Apache Drill: How does the plug-in know that an aggregate function is applied
+dev
+1 to Hakim. AbstractRR#isSkipQuery is the way to go. If you want more
details on this you should check out DRILL-2358[1] that is an umbrella issue targeting to make count(*) queries more efficient per storage plugin.
Currently (I guess) JSON and Mongo(?) readers support it though.
1:
https://github.com/apache/drill/commit/54df129cab544c3df8e75a7dae3f85a91a9ded5a
On Thu, Aug 6, 2015 at 6:50 PM, Abdel Hakim Deneche <ad...@maprtech.com>
wrote:
> Hi Sudip,
>
> I'm not really an expert in this matter but I came recently across
> isSkipQuery() method in AbstractRecordReader, it's javadoc states:
>
> *Returns true if reader should skip all of the columns, reporting
> number of
> > records only. Handling of a skip query is storage plugin-specific.*
>
>
> You can take a look at JSONRecordReader for an example on how to use
> isSkipQuery() to optimize the reading.
>
> Thanks
>
> On Thu, Aug 6, 2015 at 2:01 AM, Sudip Mukherjee <
> mukherjeesudip1@hotmail.com
> > wrote:
>
> > Hi,
> > I am using apache drill recently with mongodb and trying to write a
> > basic plug-in for apache solr. Wanted to know how does the plug-in
> > know that an
> > count(*) query has been applied so that the query to data-source can
> > be optimized? Can i get it if I extend AbstractExprVisitor class?
> >
> > Thanks,Sudip Mukherjee
>
>
>
>
> --
>
> Abdelhakim Deneche
>
> Software Engineer
>
> <http://www.mapr.com/>
>
>
> Now Available - Free Hadoop On-Demand Training <
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm
> _campaign=Free%20available
> >
>
***************************Legal Disclaimer***************************
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**********************************************************************
Re: Apache Drill: How does the plug-in know that an aggregate
function is applied
Posted by Hanifi Gunes <hg...@maprtech.com>.
+dev
+1 to Hakim. AbstractRR#isSkipQuery is the way to go. If you want more
details on this you should check out DRILL-2358[1] that is an umbrella
issue targeting to make count(*) queries more efficient per storage plugin.
Currently (I guess) JSON and Mongo(?) readers support it though.
1:
https://github.com/apache/drill/commit/54df129cab544c3df8e75a7dae3f85a91a9ded5a
On Thu, Aug 6, 2015 at 6:50 PM, Abdel Hakim Deneche <ad...@maprtech.com>
wrote:
> Hi Sudip,
>
> I'm not really an expert in this matter but I came recently across
> isSkipQuery() method in AbstractRecordReader, it's javadoc states:
>
> *Returns true if reader should skip all of the columns, reporting number of
> > records only. Handling of a skip query is storage plugin-specific.*
>
>
> You can take a look at JSONRecordReader for an example on how to use
> isSkipQuery() to optimize the reading.
>
> Thanks
>
> On Thu, Aug 6, 2015 at 2:01 AM, Sudip Mukherjee <
> mukherjeesudip1@hotmail.com
> > wrote:
>
> > Hi,
> > I am using apache drill recently with mongodb and trying to write a basic
> > plug-in for apache solr. Wanted to know how does the plug-in know that an
> > count(*) query has been applied so that the query to data-source can be
> > optimized? Can i get it if I extend AbstractExprVisitor class?
> >
> > Thanks,Sudip Mukherjee
>
>
>
>
> --
>
> Abdelhakim Deneche
>
> Software Engineer
>
> <http://www.mapr.com/>
>
>
> Now Available - Free Hadoop On-Demand Training
> <
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >
>
Re: Apache Drill: How does the plug-in know that an aggregate
function is applied
Posted by Hanifi Gunes <hg...@maprtech.com>.
+dev
+1 to Hakim. AbstractRR#isSkipQuery is the way to go. If you want more
details on this you should check out DRILL-2358[1] that is an umbrella
issue targeting to make count(*) queries more efficient per storage plugin.
Currently (I guess) JSON and Mongo(?) readers support it though.
1:
https://github.com/apache/drill/commit/54df129cab544c3df8e75a7dae3f85a91a9ded5a
On Thu, Aug 6, 2015 at 6:50 PM, Abdel Hakim Deneche <ad...@maprtech.com>
wrote:
> Hi Sudip,
>
> I'm not really an expert in this matter but I came recently across
> isSkipQuery() method in AbstractRecordReader, it's javadoc states:
>
> *Returns true if reader should skip all of the columns, reporting number of
> > records only. Handling of a skip query is storage plugin-specific.*
>
>
> You can take a look at JSONRecordReader for an example on how to use
> isSkipQuery() to optimize the reading.
>
> Thanks
>
> On Thu, Aug 6, 2015 at 2:01 AM, Sudip Mukherjee <
> mukherjeesudip1@hotmail.com
> > wrote:
>
> > Hi,
> > I am using apache drill recently with mongodb and trying to write a basic
> > plug-in for apache solr. Wanted to know how does the plug-in know that an
> > count(*) query has been applied so that the query to data-source can be
> > optimized? Can i get it if I extend AbstractExprVisitor class?
> >
> > Thanks,Sudip Mukherjee
>
>
>
>
> --
>
> Abdelhakim Deneche
>
> Software Engineer
>
> <http://www.mapr.com/>
>
>
> Now Available - Free Hadoop On-Demand Training
> <
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >
>
Re: Apache Drill: How does the plug-in know that an aggregate
function is applied
Posted by Abdel Hakim Deneche <ad...@maprtech.com>.
Hi Sudip,
I'm not really an expert in this matter but I came recently across
isSkipQuery() method in AbstractRecordReader, it's javadoc states:
*Returns true if reader should skip all of the columns, reporting number of
> records only. Handling of a skip query is storage plugin-specific.*
You can take a look at JSONRecordReader for an example on how to use
isSkipQuery() to optimize the reading.
Thanks
On Thu, Aug 6, 2015 at 2:01 AM, Sudip Mukherjee <mukherjeesudip1@hotmail.com
> wrote:
> Hi,
> I am using apache drill recently with mongodb and trying to write a basic
> plug-in for apache solr. Wanted to know how does the plug-in know that an
> count(*) query has been applied so that the query to data-source can be
> optimized? Can i get it if I extend AbstractExprVisitor class?
>
> Thanks,Sudip Mukherjee
--
Abdelhakim Deneche
Software Engineer
<http://www.mapr.com/>
Now Available - Free Hadoop On-Demand Training
<http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>