[GitHub] [spark] baohe-zhang edited a comment on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster

[GitBox <gi...@apache.org>]

baohe-zhang edited a comment on pull request #28412:
URL: https://github.com/apache/spark/pull/28412#issuecomment-654455672


   @tgravescs I measured the memory usage of an app that contains 4 jobs, and each job contains 1 stage, each stage contains 100000 tasks (the max retained tasks per stage by default). The number of entities stored in kvstore are 4 jobs, 4 stages, and 400000 tasks.  The log size of this app is 1.1 GB, and the memory usage is 212 MB. I measured another app that contains 1000 jobs (the max retained jobs by default), each job contains 1 stage and each stage contains 400 tasks. The total number of entities for this app are 1000 jobs, 1000 stages, and 400000 tasks, which led to a 1.29 GB log file and 271 MB memory usage. 
   
   The number of entities for the app in https://github.com/apache/spark/pull/28412#issuecomment-653110422 is 400 jobs, 400 stages, and 400000 tasks.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org