You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/09/22 07:03:25 UTC

[GitHub] [hudi] xushiyan commented on issue #3697: [SUPPORT] Performance Tuning: How to speed up stages?

xushiyan commented on issue #3697:
URL: https://github.com/apache/hudi/issues/3697#issuecomment-924644754


   looks like there are too many partitions (1100) for a dataset < 2GB. This resulted in many small files. Some suggestions 1) reduce number of partitions to <100 2) double check number executors were created. c5.xlarge is a very small machine also with low memory to core ratio. You set 100 executors but each requires 5g memory 3) also tune parallelism settings in Hudi for the insert and upsert


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org