[jira] [Commented] (FLINK-32650) Added the ability to split flink-protobuf codegen code

["李精卫 (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-32650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17751502#comment-17751502 ] 

李精卫 commented on FLINK-32650:
-----------------------------

[~maosuhan] Thank you for your suggestion, I will continue to pay attention to the problems you mentioned

> Added the ability to split flink-protobuf codegen code
> ------------------------------------------------------
>
>                 Key: FLINK-32650
>                 URL: https://issues.apache.org/jira/browse/FLINK-32650
>             Project: Flink
>          Issue Type: Improvement
>          Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>    Affects Versions: 1.17.1
>            Reporter: 李精卫
>            Assignee: 李精卫
>            Priority: Major
>
> h3. backgroud
> Flink serializes and deserializes protobuf format data by calling the decode or encode method in GeneratedProtoToRow_XXX.java generated by codegen to parse byte[] data into protobuf java objects. The size of the decode/encode codegen method body is strongly related to the number of defined fields in protobuf. When the number of fields exceeds a certain threshold and the compiled method body exceeds 8k, the decode/encode method will not be optimized by JIT, seriously affecting serialization or deserialization performance. Even if the compiled method body exceeds 64k, it will directly cause the task to fail to start.
> h3. solution
> So I proposed Codegen Splitter for protobuf parsing to split the encode/decode method to solve this problem.
> The specific idea is as follows. In the current decode/encode method, each field defined for the protobuf message is placed in the method body. In fact, there are no shared parameters between the fields, so multiple fields can be merged and parsed and written into the split method body. If the number of strings in the current method body exceeds the threshold, a split method will be generated, these fields will be parsed in the split method, and the split method will be called in the decode/encode method. By analogy, the decode/encode method including the split method is finally generated.
> after spilt code example
>  
> {code:java}
> //代码占位符
> public static RowData decode(org.apache.flink.formats.protobuf.testproto.AdProfile.AdProfilePb message){
> RowData rowData=null;
> org.apache.flink.formats.protobuf.testproto.AdProfile.AdProfilePb message1242 = message;
> GenericRowData rowData1242 = new GenericRowData(5);
> split2585(rowData1242, message1242);
> rowData = rowData1242;return rowData;
> }
> {code}
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)