You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@hudi.apache.org by Danny Chan <da...@apache.org> on 2023/12/04 02:31:18 UTC

Re: [External] Current state of parquet zstd OOM with hudi

> I would say an entry in the hudi FAQ on this issue would be great, since hard to spot, and marked as fixed on spark side.

Makes sense, welcome to fire a fix to Hudi website.

Best,
Danny

Nicolas Paris <ni...@riseup.net> 于2023年11月22日周三 15:55写道：
>
> We fixed the hudi memory leak by patching parquet 1.12 and rely on gradle to overwrite the transitive dependencies of parquet with that latest version.
>
> I would say an entry in the hudi FAQ on this issue would be great, since hard to spot, and marked as fixed on spark side.
>
> Also we didn't notice the issue on EMR and had issue when we migrated on kubernetes, which did not help to identify the zstd leak.