You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/11/04 00:04:18 UTC
[GitHub] [hudi] nsivabalan commented on issue #7062: [SUPPORT] Appeding to files during UPSERT causes executors to die due to memory issues.
nsivabalan commented on issue #7062:
URL: https://github.com/apache/hudi/issues/7062#issuecomment-1302807726
hey @HEPBO3AH : do you mean to say that, even after our fix https://github.com/apache/hudi/pull/6864, your avg record size estimate is wrong in some cases. And as a result your are running into OOM issue.
Can you furnish these details.
- whats the actual avg record size.
- whats parquet max file size =(250Mb?)
- How many records are packed into this file?
- whats the record size that hudi infers (from info logging in driver logs).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org