[jira] [Created] (FLINK-11421) Providing more compilation options for code-generated operators

["Liya Fan (JIRA)" <ji...@apache.org>]

Liya Fan created FLINK-11421:
--------------------------------

             Summary: Providing more compilation options for code-generated operators
                 Key: FLINK-11421
                 URL: https://issues.apache.org/jira/browse/FLINK-11421
             Project: Flink
          Issue Type: New Feature
          Components: Core
            Reporter: Liya Fan
            Assignee: Liya Fan


Flink supports some operators (like Calc, Hash Agg, Hash Join, etc.) by code generation. That is, Flink generates their source code dynamically, and then compile it into Java Byte Code, which is load and executed at runtime.

 

By default, Flink compiles the generated source code by Janino. This is fast, as the compilation often finishes in hundreds of milliseconds. The generated Java Byte Code, however, is of poor quality. To illustrate, we use Java Compiler API (JCA) to compile the generated code. Experiments on TPC-H (1 TB) queries show that the E2E time can be more than 10% shorter, when operators are compiled by JCA, despite that it takes more time (a few seconds) to compile with JCA.

 

Therefore, we believe it is beneficial to compile generated code by JCA in the following scenarios: 1) For batch jobs, the E2E time is relatively long, so it is worth of spending more time compiling and generating high quality Java Byte Code. 2) For repeated stream jobs, the generated code will be compiled once and run many times. Therefore, it pays to spend more time compiling for the first time, and enjoy the high byte code qualities for later runs.

 

According to the above observations, we want to provide a compilation option (Janino, JCA, or dynamic) for Flink, so that the user can choose the one suitable for their specific scenario and obtain better performance whenever possible.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)