You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/05/09 15:57:34 UTC

[GitHub] [hudi] yangdafish opened a new issue, #5541: [SUPPORT] Upgrade from 0.8.0 to 0.9.0 decreases Upsert performance

yangdafish opened a new issue, #5541:
URL: https://github.com/apache/hudi/issues/5541

   **Describe the problem you faced**
   
   Recently, we upgraded our environment from Hudi 0.8.0 to Hudi 0.9.0, and after the upgrade we noticed that upsert jobs for some of our existing tables run much slower compared to how they ran in Hudi 0.8.0.
   
   For our Hudi tables, we ran one bulk_insert job to ingest snapshot, and schedule an upsert job every 10 mins to ingest incremental updates after the completion of bulk_insert job.
   
   To reproduce the issue, we ran upsert job on a table with the size around 1.8T. The job took in 11 tsv files (< 150 MB in total) containing both new records and updates.
   
   In Hudi 0.8.0, the job took 8.5 mins to complete whereas in Hudi 0.9.0, the job took 25+ mins consistently. And we notice that the main difference seemed to come from the steps "Getting small files from partitions".
   
   We don't upgrade to Hudi 0.10 because it is not yet supported by Redshift for our customer purposes
   
   0.8
   ![image](https://user-images.githubusercontent.com/32607057/167448206-e192362f-d9ee-4085-b1aa-0a429d956177.png)
   
   0.9
   <img width="1782" alt="Screen Shot 2022-05-03 at 2 28 43 PM" src="https://user-images.githubusercontent.com/32607057/167448363-51dbdf36-44ec-4d7b-ac80-d166f8b6f656.png">
   
   
   **Environment Description**
   
   * Hudi version : 0.9.0
   
   * Spark version : 3.1.2
   
   * Scala version: 2.12 
   
   * Hadoop version : 2.10.1
   
   * Storage (HDFS/S3/GCS..) : S3
   
   * Running on Docker? (yes/no) : no
   
   * AWS EMR: 6.55.0, 1 master(r6g.16xlarge) with 30 cores(r6g.16xlarge)
   
   
   **Additional context**
   
   Spark configs:
   
   --deploy-mode cluster
   --executor-memory 43g
   --driver-memory 43g
   --executor-cores 6
   --conf spark.serializer=org.apache.spark.serializer.KryoSerializer
   --conf spark.sql.hive.convertMetastoreParquet=false
   --conf spark.hadoop.fs.s3.maxRetries=30
   --conf spark.yarn.executor.memoryOverhead=5g
   
   
   Hudi configs:
   
   hoodie.consistency.check.enabled -> true
   hoodie.datasource.write.table.type -> "COPY_ON_WRITE"
   hoodie.datasource.write.keygenerator.class -> "org.apache.hudi.keygen.ComplexKeyGenerator"
   hoodie.upsert.shuffle.parallelism -> 1500
   hoodie.bulkinsert.shuffle.parallelism -> 1800
   hoodie.parquet.max.file.size -> 500 * 1024 * 1024
   hoodie.datasource.write.operation -> "upsert"
   hoodie.metadata.enable -> true
   hoodie.metadata.validate -> true
   hoodie.fail.on.timeline.archiving -> false
   hoodie.clean.automatic -> true
   hoodie.cleaner.commits.retained: 72
   hoodie.keep.min.commits: 100
   hoodie.keep.max.commits: 150
   hoodie.write.markers.type: timeline_server_based
   
   **Stacktrace**
   
   Getting small files from partitions isEmpty at HoodieSparkSqlWriter.scala:609 +details
   
   org.apache.spark.rdd.RDD.isEmpty(RDD.scala:1557)
   org.apache.hudi.HoodieSparkSqlWriter$.commitAndPerformPostOperations(HoodieSparkSqlWriter.scala:609)
   org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:274)
   org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:164)
   org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:46)
   org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
   org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
   org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:90)
   org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:194)
   org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:232)
   org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
   org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:229)
   org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:190)
   org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:134)
   org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:133)
   org.apache.spark.sql.DataFrameWriter.$anonfun$runCommand$1(DataFrameWriter.scala:989)
   org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
   org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:232)
   org.apache.spark.sql.execution.SQLExecution$.executeQuery$1(SQLExecution.scala:110)
   org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:135)
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] codope commented on issue #5541: [SUPPORT] Upgrade from 0.8.0 to 0.9.0 decreases Upsert performance

Posted by GitBox <gi...@apache.org>.
codope commented on issue #5541:
URL: https://github.com/apache/hudi/issues/5541#issuecomment-1160587714

   @yangdafish As suggested above, could you please upgrade to latest Hudi release 0.11.1 and see if issue is still there?
   @YuweiXiao if you can share the patch which fixed this issue, that would be great.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] xushiyan commented on issue #5541: [SUPPORT] Upgrade from 0.8.0 to 0.9.0 decreases Upsert performance

Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #5541:
URL: https://github.com/apache/hudi/issues/5541#issuecomment-1129882521

   @yangdafish please let us know if this suggestion works
   
   > Maybe related to this PR #4012
   > 
   > Could you try adding this patch and test again? The issue is introduced in 0.9.0 and fixed in 0.11.0.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] YuweiXiao commented on issue #5541: [SUPPORT] Upgrade from 0.8.0 to 0.9.0 decreases Upsert performance

Posted by GitBox <gi...@apache.org>.
YuweiXiao commented on issue #5541:
URL: https://github.com/apache/hudi/issues/5541#issuecomment-1149374796

   @yangdafish Hi, looking at the new spark UI you posted, the issue of 'one task being executed in multiple jobs' has actually been resolved (compared to job 15-19 in the 0.9.0 spark ui).
   
   And in the new spark ui, if we exclude the time used by job 0 (which seems not part of Hudi write), the total costs is around 1.5min.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] YuweiXiao commented on issue #5541: [SUPPORT] Upgrade from 0.8.0 to 0.9.0 decreases Upsert performance

Posted by GitBox <gi...@apache.org>.
YuweiXiao commented on issue #5541:
URL: https://github.com/apache/hudi/issues/5541#issuecomment-1181369576

   > @yangdafish As suggested above, could you please upgrade to latest Hudi release 0.11.1 and see if issue is still there? @YuweiXiao if you can share the patch which fixed this issue, that would be great.
   
   Sorry for the late reply. The patch is posted above, PR #4012


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] yangdafish commented on issue #5541: [SUPPORT] Upgrade from 0.8.0 to 0.9.0 decreases Upsert performance

Posted by GitBox <gi...@apache.org>.
yangdafish commented on issue #5541:
URL: https://github.com/apache/hudi/issues/5541#issuecomment-1149059891

   Hi, have upgraded our Hudi to 0.11.0 as suggested, using Maven bundle
   
   Performance issue still persists for larger datasets. Upsert job every 10 mins now takes longer than 10 minutes with the same Hudi settings.
   
   Executing Hudi 0.11.0 on AWS EMR 0.66 with Spark 3.2
   
   Included the following documentation suggested Hudi flags, but all other settings remained the same:
   --conf spark.sql.avro.datetimeRebaseModeInWrite=CORRECTED 
   --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog 
   --conf spark.sql.hive.convertMetastoreParquet=false 
   
   Attaching EMR Spark History server snapshot as reference
   
   <img width="1789" alt="Screen Shot 2022-06-07 at 12 05 22 PM" src="https://user-images.githubusercontent.com/32607057/172462410-3cbcd5f8-66ab-4010-8c81-4fae843354a1.png">
   
   <img width="1786" alt="Screen Shot 2022-06-07 at 12 05 36 PM" src="https://user-images.githubusercontent.com/32607057/172462401-ca9e063e-5d3e-411e-ad27-f7a84551018f.png">
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] nsivabalan closed issue #5541: [SUPPORT] Upgrade from 0.8.0 to 0.9.0 decreases Upsert performance

Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #5541: [SUPPORT] Upgrade from 0.8.0 to 0.9.0 decreases Upsert performance
URL: https://github.com/apache/hudi/issues/5541


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] nsivabalan commented on issue #5541: [SUPPORT] Upgrade from 0.8.0 to 0.9.0 decreases Upsert performance

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #5541:
URL: https://github.com/apache/hudi/issues/5541#issuecomment-1299684429

   Hey looks like we got the perf fix in https://github.com/apache/hudi/pull/5187 
   Can you try it out w/ 0.11.1 or above and let us know if you are still seeing the issue. 
   Closing the github issue for now. but feel free to reopen if you are still facing issues. 
   would be glad to assist. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] yangdafish commented on issue #5541: [SUPPORT] Upgrade from 0.8.0 to 0.9.0 decreases Upsert performance

Posted by GitBox <gi...@apache.org>.
yangdafish commented on issue #5541:
URL: https://github.com/apache/hudi/issues/5541#issuecomment-1149062702

   Also attempted to add the following Hudi flags for performance optimization, but was still unable to see same performance as pre-upgrade 0.8 Hudi with Spark 2
   
   ```
   "hoodie.metadata.enable" -> "true",
   "spark.memory.storageFraction" -> "0.2",
   "spark.yarn.driver.memoryOverhead" -> "1024",
   "spark.yarn.executor.memoryOverhead" -> "3072",
   "spark.yarn.max.executor.failures" -> "100"
    ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] YuweiXiao commented on issue #5541: [SUPPORT] Upgrade from 0.8.0 to 0.9.0 decreases Upsert performance

Posted by GitBox <gi...@apache.org>.
YuweiXiao commented on issue #5541:
URL: https://github.com/apache/hudi/issues/5541#issuecomment-1121939354

   Maybe related to this PR https://github.com/apache/hudi/pull/4012
   
   Could you try adding this patch and test again? The issue is introduced in 0.9.0 and fixed in 0.11.0.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org