You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by Jason Altekruse <al...@gmail.com> on 2014/01/22 18:30:54 UTC
Hangout Notes, and a question
Hi everyone,
I have updated the google doc with notes from this weeks meeting.
https://docs.google.com/document/d/1o2GvZUtJvKzN013JdM715ZBzhseT0VyZ9WgmLMeeUUk
In response to my question about how we want to implement setting options
within Drill we had a rather long discussion about storing distributed
state in Drill.
Unfortunately during the beginning of the discussion when I was
participating more actively I wasn't taking complete notes. I kind of lost
the answer to my original question with all of the other topics we
discussed.
Did we want to implement setting options as a component of both the logical
and physical plans, or separated from the usual plan processing path?
-Jason Altekruse
Re: Hangout Notes, and a question
Posted by Jason Altekruse <al...@gmail.com>.
The logical and physical setup makes sense, but I guess the only thing I'm
still unsure about is how we transfer the setting data from the optiq SQL
parse tree into Drill. Currently the SQL tree is transformed directly into
a logical plan, I'm just uncertain if there is another path out of optiq
for passing the options that we are interested in using.
I had seen the plan fragments formatted like physical plans, so I assumed
they were still passed as JSON, I had not realized we had defined protobuf
structs to hold them.
Thanks,
- Jason
On Wed, Jan 22, 2014 at 11:59 AM, Jacques Nadeau <ja...@apache.org> wrote:
> It seems that you're somewhat mixing internal formats and external apis.
> You can submit a query via sql, logical or physical. For sql, you set
> options via separate statements. For logical and physical you set options
> via the head of the plan. Drill should internally manage the options
> information to pass to other nodes as part of query execution. In that
> case, the data will be a protobuf struct as part of PlanFragment as opposed
> to anything to do with logical or physical plans. In the context of
> PlanFragment, options are only relevant to that fragment and no 'session'
> state is maintained on those working nodes.
>
>
>
>
> On Wed, Jan 22, 2014 at 9:38 AM, Jason Altekruse
> <al...@gmail.com>wrote:
>
> > I remember having discussed this. I was thinking this was related to
> > distributing the settings to other drillbits that would need to use
> > settings in their various plan fragments, but I that would work for
> keeping
> > all of the options consistent.
> >
> > In that case if set statements in SQL will be independent statements,
> will
> > we be passing around plans with no bodies to set the options originally?
> >
> > Thanks,
> > Jason
> >
> >
> > On Wed, Jan 22, 2014 at 11:34 AM, Jacques Nadeau <ja...@apache.org>
> > wrote:
> >
> > > My thoughts were as part of the plan but not as part of the operator.
> > Put
> > > it in the head as an additional set of data.
> > >
> > >
> > > On Wed, Jan 22, 2014 at 9:30 AM, Jason Altekruse
> > > <al...@gmail.com>wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > I have updated the google doc with notes from this weeks meeting.
> > > >
> > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1o2GvZUtJvKzN013JdM715ZBzhseT0VyZ9WgmLMeeUUk
> > > >
> > > > In response to my question about how we want to implement setting
> > options
> > > > within Drill we had a rather long discussion about storing
> distributed
> > > > state in Drill.
> > > >
> > > > Unfortunately during the beginning of the discussion when I was
> > > > participating more actively I wasn't taking complete notes. I kind of
> > > lost
> > > > the answer to my original question with all of the other topics we
> > > > discussed.
> > > >
> > > > Did we want to implement setting options as a component of both the
> > > logical
> > > > and physical plans, or separated from the usual plan processing path?
> > > >
> > > > -Jason Altekruse
> > > >
> > >
> >
>
Re: Hangout Notes, and a question
Posted by Jacques Nadeau <ja...@apache.org>.
It seems that you're somewhat mixing internal formats and external apis.
You can submit a query via sql, logical or physical. For sql, you set
options via separate statements. For logical and physical you set options
via the head of the plan. Drill should internally manage the options
information to pass to other nodes as part of query execution. In that
case, the data will be a protobuf struct as part of PlanFragment as opposed
to anything to do with logical or physical plans. In the context of
PlanFragment, options are only relevant to that fragment and no 'session'
state is maintained on those working nodes.
On Wed, Jan 22, 2014 at 9:38 AM, Jason Altekruse
<al...@gmail.com>wrote:
> I remember having discussed this. I was thinking this was related to
> distributing the settings to other drillbits that would need to use
> settings in their various plan fragments, but I that would work for keeping
> all of the options consistent.
>
> In that case if set statements in SQL will be independent statements, will
> we be passing around plans with no bodies to set the options originally?
>
> Thanks,
> Jason
>
>
> On Wed, Jan 22, 2014 at 11:34 AM, Jacques Nadeau <ja...@apache.org>
> wrote:
>
> > My thoughts were as part of the plan but not as part of the operator.
> Put
> > it in the head as an additional set of data.
> >
> >
> > On Wed, Jan 22, 2014 at 9:30 AM, Jason Altekruse
> > <al...@gmail.com>wrote:
> >
> > > Hi everyone,
> > >
> > > I have updated the google doc with notes from this weeks meeting.
> > >
> > >
> > >
> >
> https://docs.google.com/document/d/1o2GvZUtJvKzN013JdM715ZBzhseT0VyZ9WgmLMeeUUk
> > >
> > > In response to my question about how we want to implement setting
> options
> > > within Drill we had a rather long discussion about storing distributed
> > > state in Drill.
> > >
> > > Unfortunately during the beginning of the discussion when I was
> > > participating more actively I wasn't taking complete notes. I kind of
> > lost
> > > the answer to my original question with all of the other topics we
> > > discussed.
> > >
> > > Did we want to implement setting options as a component of both the
> > logical
> > > and physical plans, or separated from the usual plan processing path?
> > >
> > > -Jason Altekruse
> > >
> >
>
Re: Hangout Notes, and a question
Posted by Jason Altekruse <al...@gmail.com>.
I remember having discussed this. I was thinking this was related to
distributing the settings to other drillbits that would need to use
settings in their various plan fragments, but I that would work for keeping
all of the options consistent.
In that case if set statements in SQL will be independent statements, will
we be passing around plans with no bodies to set the options originally?
Thanks,
Jason
On Wed, Jan 22, 2014 at 11:34 AM, Jacques Nadeau <ja...@apache.org> wrote:
> My thoughts were as part of the plan but not as part of the operator. Put
> it in the head as an additional set of data.
>
>
> On Wed, Jan 22, 2014 at 9:30 AM, Jason Altekruse
> <al...@gmail.com>wrote:
>
> > Hi everyone,
> >
> > I have updated the google doc with notes from this weeks meeting.
> >
> >
> >
> https://docs.google.com/document/d/1o2GvZUtJvKzN013JdM715ZBzhseT0VyZ9WgmLMeeUUk
> >
> > In response to my question about how we want to implement setting options
> > within Drill we had a rather long discussion about storing distributed
> > state in Drill.
> >
> > Unfortunately during the beginning of the discussion when I was
> > participating more actively I wasn't taking complete notes. I kind of
> lost
> > the answer to my original question with all of the other topics we
> > discussed.
> >
> > Did we want to implement setting options as a component of both the
> logical
> > and physical plans, or separated from the usual plan processing path?
> >
> > -Jason Altekruse
> >
>
Re: Hangout Notes, and a question
Posted by Jacques Nadeau <ja...@apache.org>.
My thoughts were as part of the plan but not as part of the operator. Put
it in the head as an additional set of data.
On Wed, Jan 22, 2014 at 9:30 AM, Jason Altekruse
<al...@gmail.com>wrote:
> Hi everyone,
>
> I have updated the google doc with notes from this weeks meeting.
>
>
> https://docs.google.com/document/d/1o2GvZUtJvKzN013JdM715ZBzhseT0VyZ9WgmLMeeUUk
>
> In response to my question about how we want to implement setting options
> within Drill we had a rather long discussion about storing distributed
> state in Drill.
>
> Unfortunately during the beginning of the discussion when I was
> participating more actively I wasn't taking complete notes. I kind of lost
> the answer to my original question with all of the other topics we
> discussed.
>
> Did we want to implement setting options as a component of both the logical
> and physical plans, or separated from the usual plan processing path?
>
> -Jason Altekruse
>