[jira] [Commented] (SPARK-33753) Reduce the memory footprint and gc of the cache (hadoopJobMetadata)

["Apache Spark (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-33753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17561327#comment-17561327 ] 

Apache Spark commented on SPARK-33753:
--------------------------------------

User 'mcdull-zhang' has created a pull request for this issue:
https://github.com/apache/spark/pull/37043

> Reduce the memory footprint and gc of the cache (hadoopJobMetadata)
> -------------------------------------------------------------------
>
>                 Key: SPARK-33753
>                 URL: https://issues.apache.org/jira/browse/SPARK-33753
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 3.0.1
>            Reporter: dzcxzl
>            Priority: Minor
>         Attachments: current_gcutil.png, current_job_finish_time.png, current_visual_gc.png, fix_gcutil.png, fix_job_finish_time.png, fix_visual_gc.png, jobconf.png
>
>
>  
> HadoopRDD uses soft-reference map to cache jobconf (rdd_id -> jobconf).
>  When the number of hive partitions read by the driver is large, HadoopRDD.getPartitions will create many jobconfs and add them to the cache.
>  The executor will also create a jobconf, add it to the cache, and share it among exeuctors.
> The number of jobconfs in the driver cache increases the memory pressure. When the driver memory configuration is not high, full gc becoming very frequent, and these jobconfs are hardly reused.
> For example, spark.driver.memory=2560m, the read partition is about 14,000, and a jobconf 96kb.
> !jobconf.png!
>  
> The following is a repair comparison, full gc decreased from 62s to 0.8s, and the number of times decreased from 31 to 5. And the driver applied for less memory (Old Gen 1.667G->968M), the job execution time is also reduced.
>  
> Current:
> !current_job_finish_time.png!
> jstat -gcutil PID 2s
> !current_gcutil.png!
> !current_visual_gc.png!
>  
> Try to change softValues to weakValues
> !fix_job_finish_time.png!
> !fix_gcutil.png!
> !fix_visual_gc.png!
>  
>  
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org