contribution & advice needed

[Lisen Mu <im...@gmail.com>]

Hi,

We are building a prototype system aiming to process query against
different data sources, based on drill's prototype. Thanks to all the great
work done so far in drill, it actually works for us. And I'm also pleased
to see all the effort being done to make progress in various aspects of
drill.

In the near future we will continue mainly optimizing our prototype system.
we are also pleased to contribute back to drill project whatever we come
across and could help maturing drill, may it be bug reports, patches,
testcases, etc.

On the other hand, in order to fit our production need ASAP, we have built
some modules in a non-generic way (optimizer and storage engine etc.),
based on drill's current master state. we're planning to adopt future
progress of these modules of drill, and willing to contribute to make that
future nearer.

Also advices is most appreciated: what's your future plan about
prototype/exec/ref? we are going to do some optimization work based on it
soon.

Thanks,

Re: contribution & advice needed

[Lisen Mu <im...@gmail.com>]

Jacques,

Thanks very much.

As for now, I would like to focus on our local query part based on
exec/ref, and keep track of changes as clean as possible in the future at

https://github.com/XingCloud/incubator-drill

Hope we could migrate to distributed exec engine in the future. And I would
feed back problems we solved/unsolved in JIRA if appropriate.

Many thanks!








On Tue, Mar 26, 2013 at 10:00 AM, Jacques Nadeau <ja...@apache.org> wrote:

> snip...
>
> However, if distributed execution engine is your future focus, could you
> > please release the local query part of the engine first? I suppose we'd
> > better start off from it instead of exec/ref.
> >
> >
> I'm initially working on the basic distribution framework for the execution
> engine and a single operator.  From there we can expand up both directions.
>
> J
>
>
>
> >
> >
> > > Thanks,
> > > Jacques
> > >
> > > On Wed, Mar 20, 2013 at 12:32 AM, Lisen Mu <im...@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > We are building a prototype system aiming to process query against
> > > > different data sources, based on drill's prototype. Thanks to all the
> > > great
> > > > work done so far in drill, it actually works for us. And I'm also
> > pleased
> > > > to see all the effort being done to make progress in various aspects
> of
> > > > drill.
> > > >
> > > > In the near future we will continue mainly optimizing our prototype
> > > system.
> > > > we are also pleased to contribute back to drill project whatever we
> > come
> > > > across and could help maturing drill, may it be bug reports, patches,
> > > > testcases, etc.
> > > >
> > > > On the other hand, in order to fit our production need ASAP, we have
> > > built
> > > > some modules in a non-generic way (optimizer and storage engine
> etc.),
> > > > based on drill's current master state. we're planning to adopt future
> > > > progress of these modules of drill, and willing to contribute to make
> > > that
> > > > future nearer.
> > > >
> > > > Also advices is most appreciated: what's your future plan about
> > > > prototype/exec/ref? we are going to do some optimization work based
> on
> > it
> > > > soon.
> > > >
> > > > Thanks,
> > > >
> > >
> >
>

Re: contribution & advice needed

[Jacques Nadeau <ja...@apache.org>]

snip...

However, if distributed execution engine is your future focus, could you
> please release the local query part of the engine first? I suppose we'd
> better start off from it instead of exec/ref.
>
>
I'm initially working on the basic distribution framework for the execution
engine and a single operator.  From there we can expand up both directions.

J



>
>
> > Thanks,
> > Jacques
> >
> > On Wed, Mar 20, 2013 at 12:32 AM, Lisen Mu <im...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > We are building a prototype system aiming to process query against
> > > different data sources, based on drill's prototype. Thanks to all the
> > great
> > > work done so far in drill, it actually works for us. And I'm also
> pleased
> > > to see all the effort being done to make progress in various aspects of
> > > drill.
> > >
> > > In the near future we will continue mainly optimizing our prototype
> > system.
> > > we are also pleased to contribute back to drill project whatever we
> come
> > > across and could help maturing drill, may it be bug reports, patches,
> > > testcases, etc.
> > >
> > > On the other hand, in order to fit our production need ASAP, we have
> > built
> > > some modules in a non-generic way (optimizer and storage engine etc.),
> > > based on drill's current master state. we're planning to adopt future
> > > progress of these modules of drill, and willing to contribute to make
> > that
> > > future nearer.
> > >
> > > Also advices is most appreciated: what's your future plan about
> > > prototype/exec/ref? we are going to do some optimization work based on
> it
> > > soon.
> > >
> > > Thanks,
> > >
> >
>

Re: contribution & advice needed

[Lisen Mu <im...@gmail.com>]

Jacques,

Thanks for the reply.

On Fri, Mar 22, 2013 at 5:54 AM, Jacques Nadeau <ja...@apache.org> wrote:

> Hey there,
>
> This is great news.  I'm glad that your finding value in what we have built
> thus far.  Couple of thoughts:
>
> >> Regarding prototype/exec/ref and future plans
> Exec/ref will probably continue to move along slowly.  We're now starting
> to focus a lot more on the full distributed execution engine and so future
> optimizations will probably be focused on that portion of the project.
>  Hopefully you'll see more code evidence of this soon.
>
>
Really great news, cant wait to see this!

>> Regarding your own work
> Even if you're building stuff that may not be able to be directly
> contributed back, it would be great if you could use a github fork so we
> can see whether anything can be pulled back.
>
> Let me know specific foci you have as well any problems or questions you
> have and we'll try to do our best as community to help you out.
>

Currently we are using drill as part of our local query engine. Our goal is
to support query(Join/Groupby possibly with sub-query) across hbase and
mysql, and possibly other nosql data sources.

One thing is that we are building missing pieces, such as And/Distinct etc.
which we hope could contribute back if helps, and customized storage engine
implementation with some hack into current exec/ref SE API which will not
open sourced and we hope would be replaced by future drill's
implementation.

Another thing is that for better performance, we need to(but not yet)
modify current exec/ref's implementation, DataValue/Join/QueueRSE to name a
few.
However, if distributed execution engine is your future focus, could you
please release the local query part of the engine first? I suppose we'd
better start off from it instead of exec/ref.



> Thanks,
> Jacques
>
> On Wed, Mar 20, 2013 at 12:32 AM, Lisen Mu <im...@gmail.com> wrote:
>
> > Hi,
> >
> > We are building a prototype system aiming to process query against
> > different data sources, based on drill's prototype. Thanks to all the
> great
> > work done so far in drill, it actually works for us. And I'm also pleased
> > to see all the effort being done to make progress in various aspects of
> > drill.
> >
> > In the near future we will continue mainly optimizing our prototype
> system.
> > we are also pleased to contribute back to drill project whatever we come
> > across and could help maturing drill, may it be bug reports, patches,
> > testcases, etc.
> >
> > On the other hand, in order to fit our production need ASAP, we have
> built
> > some modules in a non-generic way (optimizer and storage engine etc.),
> > based on drill's current master state. we're planning to adopt future
> > progress of these modules of drill, and willing to contribute to make
> that
> > future nearer.
> >
> > Also advices is most appreciated: what's your future plan about
> > prototype/exec/ref? we are going to do some optimization work based on it
> > soon.
> >
> > Thanks,
> >
>

Re: contribution & advice needed

[Jacques Nadeau <ja...@apache.org>]

Hey there,

This is great news.  I'm glad that your finding value in what we have built
thus far.  Couple of thoughts:

>> Regarding prototype/exec/ref and future plans
Exec/ref will probably continue to move along slowly.  We're now starting
to focus a lot more on the full distributed execution engine and so future
optimizations will probably be focused on that portion of the project.
 Hopefully you'll see more code evidence of this soon.

>> Regarding your own work
Even if you're building stuff that may not be able to be directly
contributed back, it would be great if you could use a github fork so we
can see whether anything can be pulled back.

Let me know specific foci you have as well any problems or questions you
have and we'll try to do our best as community to help you out.

Thanks,
Jacques

On Wed, Mar 20, 2013 at 12:32 AM, Lisen Mu <im...@gmail.com> wrote:

> Hi,
>
> We are building a prototype system aiming to process query against
> different data sources, based on drill's prototype. Thanks to all the great
> work done so far in drill, it actually works for us. And I'm also pleased
> to see all the effort being done to make progress in various aspects of
> drill.
>
> In the near future we will continue mainly optimizing our prototype system.
> we are also pleased to contribute back to drill project whatever we come
> across and could help maturing drill, may it be bug reports, patches,
> testcases, etc.
>
> On the other hand, in order to fit our production need ASAP, we have built
> some modules in a non-generic way (optimizer and storage engine etc.),
> based on drill's current master state. we're planning to adopt future
> progress of these modules of drill, and willing to contribute to make that
> future nearer.
>
> Also advices is most appreciated: what's your future plan about
> prototype/exec/ref? we are going to do some optimization work based on it
> soon.
>
> Thanks,
>