You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2015/11/17 11:51:10 UTC
[jira] [Resolved] (SPARK-11776) Cache fraction hardly reach 100%
[ https://issues.apache.org/jira/browse/SPARK-11776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-11776.
-------------------------------
Resolution: Invalid
I don't think this describes a valid problem. For example, there is no indication that you even have enough data to fill the cache.
> Cache fraction hardly reach 100%
> --------------------------------
>
> Key: SPARK-11776
> URL: https://issues.apache.org/jira/browse/SPARK-11776
> Project: Spark
> Issue Type: Bug
> Reporter: Wenmin Wu
>
> Even with plenty partitions as well as executor memory the cache fraction can hardly reach 100%. What even worse is that those uncached executor has too much records. Here is the summary of the aggregated metrics:
> 1 hd268dg.prod.mediav.com:19344 23 s 1 0 1 339.5 MB / 314 5.7 MB / 1
> 10 hd268dg.prod.mediav.com:2342 23 s 1 0 1 340.0 MB / 315 5.7 MB / 1
> 11 hd258dg.prod.mediav.com:63130 23 s 1 0 1 340.1 MB / 315 5.7 MB / 1
> 12 hd278dg.prod.mediav.com:57601 15 s 1 0 1 340.6 MB / 315 5.7 MB / 1
> 13 hd268dg.prod.mediav.com:38740 25 s 1 0 1 340.0 MB / 315 5.7 MB / 1
> 14 hd258dg.prod.mediav.com:34498 22 s 1 0 1 339.8 MB / 315 5.7 MB / 1
> 16 hd268dg.prod.mediav.com:27714 25 s 1 0 1 341.1 MB / 316 5.7 MB / 1
> 17 hd258dg.prod.mediav.com:36584 22 s 1 0 1 340.7 MB / 315 5.7 MB / 1
> 18 hd278dg.prod.mediav.com:37627 16 s 1 0 1 340.8 MB / 315 5.7 MB / 1
> 19 hd268dg.prod.mediav.com:5938 25 s 1 0 1 339.8 MB / 315 5.7 MB / 1
> 2 hd258dg.prod.mediav.com:46592 2.3 min 1 0 1 190.8 MB / 628778 6.5 MB / 1
> 20 hd258dg.prod.mediav.com:29043 2.2 min 1 0 1 190.5 MB / 627972 6.5 MB / 1
> You can see the records in executor 2 and executor 20 is too much with respect to the storage. I think some bugs must exists during caching stage.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org