[GitHub] [hudi] SamarthRaval commented on issue #8438: Standalone Compaction Scheduling takes so much time

["SamarthRaval (via GitHub)" <gi...@apache.org>]

SamarthRaval commented on issue #8438:
URL: https://github.com/apache/hudi/issues/8438#issuecomment-1520664120

   > @SamarthRaval Can you paste the screenshots of spark UI for the stages which are taking long. Also if possible provide the logs of the driver and one of the executor.
   
   Hello,
   
   Due to unavailability of spark history server(As its > 1 month) I don't have access to history server.
   
   But executor logs error has been provided above, and job gets finishes successfully but performance is the main concern here.
   
   Compaction stage takes ~ 1.2 hrs
   
   `spark-submit --deploy-mode cluster --class org.apache.hudi.utilities.HoodieCompactor --jars /usr/lib/hudi/hudi-utilities-bundle.jar,/usr/lib/hudi/hudi-spark-bundle.jar,s3://hudi-tier2-extractor-1.2.3.jar /usr/lib/hudi/hudi-utilities-bundle.jar --base-path s3://transactions.all_hudi/ --table-name transactions_all --parallelism 1000 --spark-memory 90g --mode schedule --hoodie-conf hoodie.embed.timeline.server=false --hoodie-conf hoodie.datasource.write.payload.class=com.spark.datalake.NullSafeDefaultHoodieRecordPayload --hoodie-conf hoodie.payload.ordering.field=lastmodifieddate --hoodie-conf hoodie.compaction.logfile.size.threshold=500000`
   
   Execution stage takes ~ 7 mins
   
   `spark-submit --deploy-mode cluster --class org.apache.hudi.utilities.HoodieCompactor --jars /usr/lib/hudi/hudi-utilities-bundle.jar,/usr/lib/hudi/hudi-spark-bundle.jar,s3://hudi-tier2-extractor-1.2.3.jar /usr/lib/hudi/hudi-utilities-bundle.jar --base-path s3://transactions.all_hudi/ --table-name transactions_all --parallelism 1000 --spark-memory 90g --hoodie-conf hoodie.embed.timeline.server=false --hoodie-conf hoodie.datasource.write.payload.class=com.spark.datalake.NullSafeDefaultHoodieRecordPayload --hoodie-conf hoodie.payload.ordering.field=lastmodifieddate --hoodie-conf hoodie.compaction.logfile.size.threshold=500000`
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org