You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/03/27 23:58:08 UTC

[GitHub] [hudi] srinikandi commented on issue #5054: [SUPPORT] Hudi Failing with out of memory issue on Glue with >300 Mio. Records

srinikandi commented on issue #5054:
URL: https://github.com/apache/hudi/issues/5054#issuecomment-1080048329


   I have been experiencing a similar issue with Glue and Hudi 0.90. However in my case, I did a full load of a table that had close to a billion records and with 30 worker nodes, it took around 16 minutes using a partitioning key on a data column. When I tried to run an upsert operation on the same table with about 1.5 million records, the Glue job fails with no more containers available, which indicates that there is a memory issue/disk spill.  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org