You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by Alexandre BECHE <al...@gmail.com> on 2013/05/30 12:07:19 UTC
Min max aggregated function
Dear drill dev,
I was playing with the example "simpe_plan.json", and try other aggregating
function: namely, min and max (In the context of the collapsingaggregate op).
After few failures, I looked in the code and I didn't find them.
Are this functions already implemented? If not, do you currently have
effort in that direction? If not, I would be interested in trying to write
them.
Thanks in advance for your answer,
Cheers,
Alexandre
PS: Do you have "dev meeting" or all the discussion goes through mailing
list?
Re: Min max aggregated function
Posted by Timothy Chen <tn...@gmail.com>.
Hi Alexandre,
Another channel is the IIRC room on freenode #drill.
I know folks (including myself) will hangout there from time to time.
Feel free to drop by and chat.
Tim
On Tue, Jun 4, 2013 at 12:45 PM, Alexandre BECHE
<al...@gmail.com>wrote:
> Thanks a lot for your details, I begin to have a better understanding the
> internal working.Unfortunately, I was not able to join your hangout today,
> hopefuly next week.
>
> Regarding the reference interpreter, do you have any timescale for a HBase
> driver to be integrated?
> Is it related to the #DRILL-15 JIRA request?
>
> Cheers,
> Alex
>
> PS: I have another thousand of question but I will continue the discussion
> on a more appropriate thread
>
>
>
>
>
>
> On Mon, Jun 3, 2013 at 6:43 PM, Jacques Nadeau <ja...@apache.org> wrote:
>
> > See below
> >
> > On Fri, May 31, 2013 at 8:47 AM, Alexandre BECHE
> > <al...@gmail.com> wrote:
> > > Thanks all for your comments,
> > >
> > > Now I would like to understand few things:
> > > If I well understood, the files I created (MinAggregator.java and
> > > MaxAggregator.java under
> > >
> >
> incubator-drill/sandbox/prototype/exec/ref/src/main/java/org/apache/drill/exec/ref/eval/fn)
> > > are part of the Logical plan, is that correct?
> >
> > Yes, that makes sense
> >
> > >
> > > Then, when I execute the example through the maven command : mvn
> > exec:java
> > > -Dexec.mainClass="org.apache.drill.exec.ref.ReferenceInterpreter"
> > > -Dexec.classpathScope=test
> > -Dexec.args="src/test/resources/simple_scan.json"
> > > The query goes to the physical plan (Which tell me how to interact with
> > > json, could be anything else) and is executing on my datasource: json
> in
> > > that case (the logical plan is datasource-agnostic)?
> >
> > The path is the logical plan. Currently, the reference interpreter
> > only has a driver to access json. The full execution engine will have
> > many more.
> >
> > >
> > > Finally, when trying to write this two functions, I found another
> > potential
> > > place in the sources to put them:
> > >
> plan-parser/src/main/java/org/apache/drill/plan/json/ScalarAggregate.java
> > > However, it was not necessary to modify it for the example to behave as
> > > expected.
> > > What is exactly this function, why I didn't had to modify it to get the
> > > example working?
> >
> > That location is for queries written in SQL. Your submitting a
> > logical plan which is a lower level concept and bypasses the SQL
> > parser.
> >
> > Thanks for your contribution, I'm hoping someone will do a first pass
> > review soon.
> >
> > thanks,
> > Jacques
> >
> > >
> > > Thanks in advance for your help,
> > > Cheers,
> > > Alex
> > >
> > >
> > >
> > > On Thu, May 30, 2013 at 10:05 PM, Ted Dunning <te...@gmail.com>
> > wrote:
> > >
> > >> The author is the drill project. This mailing list is the place to go
> > for
> > >> questions.
> > >>
> > >> The person who typed the semi-colons is probably here, bit so are the
> > >> people who designed the API and who worked through the architecture.
> By
> > >> asking everybody you get access to answers you may not have known you
> > >> needed.
> > >>
> > >> Besides that makes it easy for you to remember.
> > >>
> > >> Sent from my iPhone
> > >>
> > >> On May 30, 2013, at 15:51, Alexandre BECHE <alexandre.beche@gmail.com
> >
> > >> wrote:
> > >>
> > >> > Do you know who is the original author of the Count and Sum
> function?
> > I
> > >> > would like to discuss with him on ir if possible.
> > >>
> >
>
Re: Min max aggregated function
Posted by Alexandre BECHE <al...@gmail.com>.
Thanks a lot for your details, I begin to have a better understanding the
internal working.Unfortunately, I was not able to join your hangout today,
hopefuly next week.
Regarding the reference interpreter, do you have any timescale for a HBase
driver to be integrated?
Is it related to the #DRILL-15 JIRA request?
Cheers,
Alex
PS: I have another thousand of question but I will continue the discussion
on a more appropriate thread
On Mon, Jun 3, 2013 at 6:43 PM, Jacques Nadeau <ja...@apache.org> wrote:
> See below
>
> On Fri, May 31, 2013 at 8:47 AM, Alexandre BECHE
> <al...@gmail.com> wrote:
> > Thanks all for your comments,
> >
> > Now I would like to understand few things:
> > If I well understood, the files I created (MinAggregator.java and
> > MaxAggregator.java under
> >
> incubator-drill/sandbox/prototype/exec/ref/src/main/java/org/apache/drill/exec/ref/eval/fn)
> > are part of the Logical plan, is that correct?
>
> Yes, that makes sense
>
> >
> > Then, when I execute the example through the maven command : mvn
> exec:java
> > -Dexec.mainClass="org.apache.drill.exec.ref.ReferenceInterpreter"
> > -Dexec.classpathScope=test
> -Dexec.args="src/test/resources/simple_scan.json"
> > The query goes to the physical plan (Which tell me how to interact with
> > json, could be anything else) and is executing on my datasource: json in
> > that case (the logical plan is datasource-agnostic)?
>
> The path is the logical plan. Currently, the reference interpreter
> only has a driver to access json. The full execution engine will have
> many more.
>
> >
> > Finally, when trying to write this two functions, I found another
> potential
> > place in the sources to put them:
> > plan-parser/src/main/java/org/apache/drill/plan/json/ScalarAggregate.java
> > However, it was not necessary to modify it for the example to behave as
> > expected.
> > What is exactly this function, why I didn't had to modify it to get the
> > example working?
>
> That location is for queries written in SQL. Your submitting a
> logical plan which is a lower level concept and bypasses the SQL
> parser.
>
> Thanks for your contribution, I'm hoping someone will do a first pass
> review soon.
>
> thanks,
> Jacques
>
> >
> > Thanks in advance for your help,
> > Cheers,
> > Alex
> >
> >
> >
> > On Thu, May 30, 2013 at 10:05 PM, Ted Dunning <te...@gmail.com>
> wrote:
> >
> >> The author is the drill project. This mailing list is the place to go
> for
> >> questions.
> >>
> >> The person who typed the semi-colons is probably here, bit so are the
> >> people who designed the API and who worked through the architecture. By
> >> asking everybody you get access to answers you may not have known you
> >> needed.
> >>
> >> Besides that makes it easy for you to remember.
> >>
> >> Sent from my iPhone
> >>
> >> On May 30, 2013, at 15:51, Alexandre BECHE <al...@gmail.com>
> >> wrote:
> >>
> >> > Do you know who is the original author of the Count and Sum function?
> I
> >> > would like to discuss with him on ir if possible.
> >>
>
Re: Min max aggregated function
Posted by Jacques Nadeau <ja...@apache.org>.
See below
On Fri, May 31, 2013 at 8:47 AM, Alexandre BECHE
<al...@gmail.com> wrote:
> Thanks all for your comments,
>
> Now I would like to understand few things:
> If I well understood, the files I created (MinAggregator.java and
> MaxAggregator.java under
> incubator-drill/sandbox/prototype/exec/ref/src/main/java/org/apache/drill/exec/ref/eval/fn)
> are part of the Logical plan, is that correct?
Yes, that makes sense
>
> Then, when I execute the example through the maven command : mvn exec:java
> -Dexec.mainClass="org.apache.drill.exec.ref.ReferenceInterpreter"
> -Dexec.classpathScope=test -Dexec.args="src/test/resources/simple_scan.json"
> The query goes to the physical plan (Which tell me how to interact with
> json, could be anything else) and is executing on my datasource: json in
> that case (the logical plan is datasource-agnostic)?
The path is the logical plan. Currently, the reference interpreter
only has a driver to access json. The full execution engine will have
many more.
>
> Finally, when trying to write this two functions, I found another potential
> place in the sources to put them:
> plan-parser/src/main/java/org/apache/drill/plan/json/ScalarAggregate.java
> However, it was not necessary to modify it for the example to behave as
> expected.
> What is exactly this function, why I didn't had to modify it to get the
> example working?
That location is for queries written in SQL. Your submitting a
logical plan which is a lower level concept and bypasses the SQL
parser.
Thanks for your contribution, I'm hoping someone will do a first pass
review soon.
thanks,
Jacques
>
> Thanks in advance for your help,
> Cheers,
> Alex
>
>
>
> On Thu, May 30, 2013 at 10:05 PM, Ted Dunning <te...@gmail.com> wrote:
>
>> The author is the drill project. This mailing list is the place to go for
>> questions.
>>
>> The person who typed the semi-colons is probably here, bit so are the
>> people who designed the API and who worked through the architecture. By
>> asking everybody you get access to answers you may not have known you
>> needed.
>>
>> Besides that makes it easy for you to remember.
>>
>> Sent from my iPhone
>>
>> On May 30, 2013, at 15:51, Alexandre BECHE <al...@gmail.com>
>> wrote:
>>
>> > Do you know who is the original author of the Count and Sum function? I
>> > would like to discuss with him on ir if possible.
>>
Re: Min max aggregated function
Posted by Alexandre BECHE <al...@gmail.com>.
Thanks all for your comments,
Now I would like to understand few things:
If I well understood, the files I created (MinAggregator.java and
MaxAggregator.java under
incubator-drill/sandbox/prototype/exec/ref/src/main/java/org/apache/drill/exec/ref/eval/fn)
are part of the Logical plan, is that correct?
Then, when I execute the example through the maven command : mvn exec:java
-Dexec.mainClass="org.apache.drill.exec.ref.ReferenceInterpreter"
-Dexec.classpathScope=test -Dexec.args="src/test/resources/simple_scan.json"
The query goes to the physical plan (Which tell me how to interact with
json, could be anything else) and is executing on my datasource: json in
that case (the logical plan is datasource-agnostic)?
Finally, when trying to write this two functions, I found another potential
place in the sources to put them:
plan-parser/src/main/java/org/apache/drill/plan/json/ScalarAggregate.java
However, it was not necessary to modify it for the example to behave as
expected.
What is exactly this function, why I didn't had to modify it to get the
example working?
Thanks in advance for your help,
Cheers,
Alex
On Thu, May 30, 2013 at 10:05 PM, Ted Dunning <te...@gmail.com> wrote:
> The author is the drill project. This mailing list is the place to go for
> questions.
>
> The person who typed the semi-colons is probably here, bit so are the
> people who designed the API and who worked through the architecture. By
> asking everybody you get access to answers you may not have known you
> needed.
>
> Besides that makes it easy for you to remember.
>
> Sent from my iPhone
>
> On May 30, 2013, at 15:51, Alexandre BECHE <al...@gmail.com>
> wrote:
>
> > Do you know who is the original author of the Count and Sum function? I
> > would like to discuss with him on ir if possible.
>
Re: Min max aggregated function
Posted by Ted Dunning <te...@gmail.com>.
The author is the drill project. This mailing list is the place to go for questions.
The person who typed the semi-colons is probably here, bit so are the people who designed the API and who worked through the architecture. By asking everybody you get access to answers you may not have known you needed.
Besides that makes it easy for you to remember.
Sent from my iPhone
On May 30, 2013, at 15:51, Alexandre BECHE <al...@gmail.com> wrote:
> Do you know who is the original author of the Count and Sum function? I
> would like to discuss with him on ir if possible.
Re: Min max aggregated function
Posted by Alexandre BECHE <al...@gmail.com>.
Hello Jacques,
Thanks a lot for your answer,
I created a bug in JIRA with both function (DRILL-67) and submit a first
version of a patch for it.
Do you know who is the original author of the Count and Sum function? I
would like to discuss with him on ir if possible.
Thanks in advance,
Cheers,
Alex
On Thu, May 30, 2013 at 5:36 PM, Jacques Nadeau <ja...@apache.org> wrote:
> Hey Alexandre,
>
> Would love for you to work on those. File a new feature bug in JIRA
> for each and give it a shot! They are reasonably sized things to get
> started with. Feel free to ask questions here and you're also welcome
> to come to the weekly Drill standup on Google Hangout.
>
> Welcome,
> Jacques
>
> On Thu, May 30, 2013 at 3:07 AM, Alexandre BECHE
> <al...@gmail.com> wrote:
> > Dear drill dev,
> >
> > I was playing with the example "simpe_plan.json", and try other
> aggregating
> > function: namely, min and max (In the context of the collapsingaggregate
> op).
> > After few failures, I looked in the code and I didn't find them.
> >
> > Are this functions already implemented? If not, do you currently have
> > effort in that direction? If not, I would be interested in trying to
> write
> > them.
> >
> > Thanks in advance for your answer,
> > Cheers,
> > Alexandre
> >
> > PS: Do you have "dev meeting" or all the discussion goes through mailing
> > list?
>
Re: Min max aggregated function
Posted by Jacques Nadeau <ja...@apache.org>.
Hey Alexandre,
Would love for you to work on those. File a new feature bug in JIRA
for each and give it a shot! They are reasonably sized things to get
started with. Feel free to ask questions here and you're also welcome
to come to the weekly Drill standup on Google Hangout.
Welcome,
Jacques
On Thu, May 30, 2013 at 3:07 AM, Alexandre BECHE
<al...@gmail.com> wrote:
> Dear drill dev,
>
> I was playing with the example "simpe_plan.json", and try other aggregating
> function: namely, min and max (In the context of the collapsingaggregate op).
> After few failures, I looked in the code and I didn't find them.
>
> Are this functions already implemented? If not, do you currently have
> effort in that direction? If not, I would be interested in trying to write
> them.
>
> Thanks in advance for your answer,
> Cheers,
> Alexandre
>
> PS: Do you have "dev meeting" or all the discussion goes through mailing
> list?
Re: Min max aggregated function
Posted by Michael Hausenblas <mi...@gmail.com>.
> Do you have "dev meeting" or all the discussion goes through mailing list?
As per Apache rules all the decisions need to happen on the ML, yes, however, we have a weekly G+ hangout on Tuesday to discuss progress, see [1]—hope to see you there!
BTW, there is also #drill on Freenode/IRC ;)
Cheers,
Michael
[1] http://j.mp/apache-drill-hangouts
--
Michael Hausenblas
Ireland, Europe
http://mhausenblas.info/
On 30 May 2013, at 11:07, Alexandre BECHE <al...@gmail.com> wrote:
> Dear drill dev,
>
> I was playing with the example "simpe_plan.json", and try other aggregating
> function: namely, min and max (In the context of the collapsingaggregate op).
> After few failures, I looked in the code and I didn't find them.
>
> Are this functions already implemented? If not, do you currently have
> effort in that direction? If not, I would be interested in trying to write
> them.
>
> Thanks in advance for your answer,
> Cheers,
> Alexandre
>
> PS: Do you have "dev meeting" or all the discussion goes through mailing
> list?