You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by 徐泷泽 <15...@qq.com> on 2020/05/13 08:11:38 UTC

Question of Calcite Dynamic Code Generation Feature

Hi buddies !
Our team is working on developing a&nbsp; mulit data source sql engine use Calcite.But something confuse us,&nbsp; why calcite generate Java code dynamicly on SQL query, it seen have performance issues in my opinion.
And building dynamic code object is very hard,the code is unreadable and hard to understood, futhermore we can not debug it.


i wander why it was designed like that, what advantage&nbsp; of that .we search on internet,&nbsp;
but nobody can realy explain it(may be we not goot at search, and poor english).
so anybody can tell me the history of this, or give us some article.


applogize for my poor english,&nbsp; hope you know what i'm taking aboult. thank you very very very much !

Re: Question of Calcite Dynamic Code Generation Feature

Posted by Haisheng Yuan <hy...@apache.org>.
The generated code is readable and debuggable.
https://calcite.apache.org/docs/howto.html#debugging-generated-classes-in-intellij
The more data you have, the more benefits you get from Codegen.

On 2020/05/13 19:19:13, Rui Wang <am...@apache.org> wrote: 
> Based on my past experience in distributed computing, there is a difference
> between a compilation time that generates java code and then compiles java
> code into bytecode, and a runtime that runs bytecode by JVM.
> 
> The compilation will take time, but the runtime shouldn't be slower due to
> a good generation of bytecode. It works for distributed computation. For
> example, the query complies in your local machine, slower compilation time
> (a few more minutes?) is ok, and then the bytecode will run on multiple
> workers which can execute bytecode directly (so efficient). I don't have a
> benchmark to show the difference between running java code and executing
> bytecode, but in Apache Beam, after we switched to the dynamic code
> generation, we didn't see a performance regression.
> 
> 
> However I don't have enough experience on if single machine SQL engine can
> benefit from this concept.
> 
> 
> -Rui
> 
> On Wed, May 13, 2020 at 11:54 AM 徐泷泽 <15...@qq.com> wrote:
> 
> > Hi buddies !
> > Our team is working on developing a  mulit data source sql engine use
> > Calcite.But something confuse us,  why calcite generate Java code
> > dynamicly on SQL query, it seen have performance issues in my opinion.
> > And building dynamic code object is very hard,the code is unreadable and
> > hard to understood, futhermore we can not debug it.
> >
> >
> > i wander why it was designed like that, what advantage  of that .we
> > search on internet, 
> > but nobody can realy explain it(may be we not goot at search, and poor
> > english).
> > so anybody can tell me the history of this, or give us some article.
> >
> >
> > applogize for my poor english,  hope you know what i'm taking aboult.
> > thank you very very very much !
> 

Re: Question of Calcite Dynamic Code Generation Feature

Posted by Rui Wang <am...@apache.org>.
Based on my past experience in distributed computing, there is a difference
between a compilation time that generates java code and then compiles java
code into bytecode, and a runtime that runs bytecode by JVM.

The compilation will take time, but the runtime shouldn't be slower due to
a good generation of bytecode. It works for distributed computation. For
example, the query complies in your local machine, slower compilation time
(a few more minutes?) is ok, and then the bytecode will run on multiple
workers which can execute bytecode directly (so efficient). I don't have a
benchmark to show the difference between running java code and executing
bytecode, but in Apache Beam, after we switched to the dynamic code
generation, we didn't see a performance regression.


However I don't have enough experience on if single machine SQL engine can
benefit from this concept.


-Rui

On Wed, May 13, 2020 at 11:54 AM 徐泷泽 <15...@qq.com> wrote:

> Hi buddies !
> Our team is working on developing a&nbsp; mulit data source sql engine use
> Calcite.But something confuse us,&nbsp; why calcite generate Java code
> dynamicly on SQL query, it seen have performance issues in my opinion.
> And building dynamic code object is very hard,the code is unreadable and
> hard to understood, futhermore we can not debug it.
>
>
> i wander why it was designed like that, what advantage&nbsp; of that .we
> search on internet,&nbsp;
> but nobody can realy explain it(may be we not goot at search, and poor
> english).
> so anybody can tell me the history of this, or give us some article.
>
>
> applogize for my poor english,&nbsp; hope you know what i'm taking aboult.
> thank you very very very much !

Re: Question of Calcite Dynamic Code Generation Feature

Posted by Albert <zi...@gmail.com>.
like Haisheng mentioned, it is certainly debuggable, although not as
convenient as plain java code.

my understanding is that, the java code generation is merely a
demonstration of the physical plan implementation.
there are all kinds of ways to implement the logical plan, java code is
merely a `helloworld` of that.  you can see many adapters does that
differently.

On Thu, May 14, 2020 at 2:54 AM 徐泷泽 <15...@qq.com> wrote:

> Hi buddies !
> Our team is working on developing a&nbsp; mulit data source sql engine use
> Calcite.But something confuse us,&nbsp; why calcite generate Java code
> dynamicly on SQL query, it seen have performance issues in my opinion.
> And building dynamic code object is very hard,the code is unreadable and
> hard to understood, futhermore we can not debug it.
>
>
> i wander why it was designed like that, what advantage&nbsp; of that .we
> search on internet,&nbsp;
> but nobody can realy explain it(may be we not goot at search, and poor
> english).
> so anybody can tell me the history of this, or give us some article.
>
>
> applogize for my poor english,&nbsp; hope you know what i'm taking aboult.
> thank you very very very much !



-- 
~~~~~~~~~~~~~~~
no mistakes
~~~~~~~~~~~~~~~~~~

Re: Question of Calcite Dynamic Code Generation Feature

Posted by Feng Zhu <we...@gmail.com>.
 Codegen is a common technique for performance. You can find its history
from some research work (e.g. [1]).

[1] “Efficiently Compiling Efficient Query Plans for Modern Hardware,”
PVLDB, vol. 4, no. 9, pp. 539–550, 2011
https://www.vldb.org/pvldb/vol4/p539-neumann.pdf

徐泷泽 <15...@qq.com> 于2020年5月14日周四 上午2:54写道:

> Hi buddies !
> Our team is working on developing a&nbsp; mulit data source sql engine use
> Calcite.But something confuse us,&nbsp; why calcite generate Java code
> dynamicly on SQL query, it seen have performance issues in my opinion.
> And building dynamic code object is very hard,the code is unreadable and
> hard to understood, futhermore we can not debug it.
>
>
> i wander why it was designed like that, what advantage&nbsp; of that .we
> search on internet,&nbsp;
> but nobody can realy explain it(may be we not goot at search, and poor
> english).
> so anybody can tell me the history of this, or give us some article.
>
>
> applogize for my poor english,&nbsp; hope you know what i'm taking aboult.
> thank you very very very much !