You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Jingsong Lee (JIRA)" <ji...@apache.org> on 2019/05/06 07:56:00 UTC
[jira] [Commented] (FLINK-11421) Add compilation options to allow
compiling generated code with JDK compiler
[ https://issues.apache.org/jira/browse/FLINK-11421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16833578#comment-16833578 ]
Jingsong Lee commented on FLINK-11421:
--------------------------------------
Hi [~fan_li_ya] , CodeGenerator of blink almost merge done. I have a few questions here:
1.Why is Java Compiler faster than Janino? any technical details and evidence?
2.Do some benchmark to measure how fast E2E run after compiling by JCA?
3.Do some benchmark to measure how slowly JCA compiles?
4.Open the JCA compiler to run all tests of table-planner?
> Add compilation options to allow compiling generated code with JDK compiler
> ----------------------------------------------------------------------------
>
> Key: FLINK-11421
> URL: https://issues.apache.org/jira/browse/FLINK-11421
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / Runtime
> Reporter: Liya Fan
> Assignee: Liya Fan
> Priority: Major
> Labels: pull-request-available
> Original Estimate: 240h
> Time Spent: 20m
> Remaining Estimate: 239h 40m
>
> Flink supports some operators (like Calc, Hash Agg, Hash Join, etc.) by code generation. That is, Flink generates their source code dynamically, and then compile it into Java Byte Code, which is load and executed at runtime.
>
> By default, Flink compiles the generated source code by Janino. This is fast, as the compilation often finishes in hundreds of milliseconds. The generated Java Byte Code, however, is of poor quality. To illustrate, we use Java Compiler API (JCA) to compile the generated code. Experiments on TPC-H (1 TB) queries show that the E2E time can be more than 10% shorter, when operators are compiled by JCA, despite that it takes more time (a few seconds) to compile with JCA.
>
> Therefore, we believe it is beneficial to compile generated code by JCA in the following scenarios: 1) For batch jobs, the E2E time is relatively long, so it is worth of spending more time compiling and generating high quality Java Byte Code. 2) For repeated stream jobs, the generated code will be compiled once and run many times. Therefore, it pays to spend more time compiling for the first time, and enjoy the high byte code qualities for later runs.
>
> According to the above observations, we want to provide a compilation option (Janino, JCA, or dynamic) for Flink, so that the user can choose the one suitable for their specific scenario and obtain better performance whenever possible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)