[GitHub] [hudi] yangdafish commented on issue #5541: [SUPPORT] Upgrade from 0.8.0 to 0.9.0 decreases Upsert performance

[GitBox <gi...@apache.org>]

yangdafish commented on issue #5541:
URL: https://github.com/apache/hudi/issues/5541#issuecomment-1149059891

   Hi, have upgraded our Hudi to 0.11.0 as suggested, using Maven bundle
   
   Performance issue still persists for larger datasets. Upsert job every 10 mins now takes longer than 10 minutes with the same Hudi settings.
   
   Executing Hudi 0.11.0 on AWS EMR 0.66 with Spark 3.2
   
   Included the following documentation suggested Hudi flags, but all other settings remained the same:
   --conf spark.sql.avro.datetimeRebaseModeInWrite=CORRECTED 
   --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog 
   --conf spark.sql.hive.convertMetastoreParquet=false 
   
   Attaching EMR Spark History server snapshot as reference
   
   <img width="1789" alt="Screen Shot 2022-06-07 at 12 05 22 PM" src="https://user-images.githubusercontent.com/32607057/172462410-3cbcd5f8-66ab-4010-8c81-4fae843354a1.png">
   
   <img width="1786" alt="Screen Shot 2022-06-07 at 12 05 36 PM" src="https://user-images.githubusercontent.com/32607057/172462401-ca9e063e-5d3e-411e-ad27-f7a84551018f.png">
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org