You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "Bo Cui (Jira)" <ji...@apache.org> on 2022/02/25 08:41:00 UTC

[jira] [Commented] (HUDI-3516) Optimize the memory of HoodieDataBlock for Flink

    [ https://issues.apache.org/jira/browse/HUDI-3516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17497981#comment-17497981 ] 

Bo Cui commented on HUDI-3516:
------------------------------

in my cluster, after the pr, the TM memory is reduced from 8 GB to 3 GB.

> Optimize the memory of HoodieDataBlock for Flink
> ------------------------------------------------
>
>                 Key: HUDI-3516
>                 URL: https://issues.apache.org/jira/browse/HUDI-3516
>             Project: Apache Hudi
>          Issue Type: Improvement
>          Components: flink, index, reader-core
>            Reporter: Bo Cui
>            Priority: Major
>              Labels: pull-request-available
>         Attachments: image-2022-02-25-16-19-05-244.png, image-2022-02-25-16-22-49-689.png
>
>
> [https://github.com/apache/hudi/blob/45d1216e91babcfb92470a7b5c314fefa27fce0d/hudi-common/src/main/java/org/apache/hudi/common/table/log/AbstractHoodieLogRecordReader.java#L365]
> the logic uses a lot of memory and has memory leaks..
> !image-2022-02-25-16-19-05-244.png!
>  
> Memory leakage：We used ThreadLocal, but forgot to use ThreadLocal#remove()
> [https://github.com/apache/hudi/blob/45d1216e91babcfb92470a7b5c314fefa27fce0d/hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieAvroDataBlock.java#L63]
> !image-2022-02-25-16-22-49-689.png!



--
This message was sent by Atlassian Jira
(v8.20.1#820001)