[GitHub] [hudi] BalaMahesh commented on issue #7595: [SUPPORT] Hudi Clean and Delta commits taking ~50 mins to finish frequently

["BalaMahesh (via GitHub)" <gi...@apache.org>]

BalaMahesh commented on issue #7595:
URL: https://github.com/apache/hudi/issues/7595#issuecomment-1464816071

   > > We were running with
   > > hoodie.metadata.index.bloom.filter.enable=false hoodie.metadata.index.column.stats.enable=false
   > > I hope that answers your question. We have set this false because, I have run into this issue when set to true. #7657
   > > I have changed the index type to simple and then restarted the application.
   > > Index look up duration has come down and uniform now. <img alt="Screenshot 2023-01-13 at 10 29 51 AM" width="671" src="https://user-images.githubusercontent.com/25053668/212241200-1284af49-d728-432f-a1ba-a72e1ed50dbe.png">
   > > delta commit durations are uniform too except for the issue I mentioned here #7364. After the restart delta commit gets stuck and then later progresses.
   > > <img alt="Screenshot 2023-01-13 at 10 30 00 AM" width="672" src="https://user-images.githubusercontent.com/25053668/212241373-1f67ca15-7e5d-4bdc-b9a1-ced25a94ba68.png">
   > 
   > @BalaMahesh Hello, BalaMahesh, we met same issue here, I have two questions here, hope you could have chance to response. Thanks in advance !
   > 
   > * how to find charts of "Index lookup duration" ?
   > * simple is the default index type on Spark engine, and we are using simple index type on our Custom Spark job, seems simple index type is not the key solution. do you have any more updates ?
   > 
   > Thanks in advance again !
   
   1. You have to push metrics to either Prometheus push gateway or any other monitoring solution for plotting these charts. 
   2. You can see the spark stages and jobs in the spark web ui to identify which task is taking long. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org