You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Liang Lee (JIRA)" <ji...@apache.org> on 2015/08/14 06:00:47 UTC
[jira] [Commented] (SPARK-9498) Some statistical information missed
when the driver is out of the cluster
[ https://issues.apache.org/jira/browse/SPARK-9498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14696430#comment-14696430 ]
Liang Lee commented on SPARK-9498:
----------------------------------
Out of cluster means:
There is a cluster has three node called A,B,C.
The driver is on another node called D.
If the driver is on any of A or B or C, the info is OK.
But if the driver is on node D, some info lost as described above.
Is this clear?
> Some statistical information missed when the driver is out of the cluster
> -------------------------------------------------------------------------
>
> Key: SPARK-9498
> URL: https://issues.apache.org/jira/browse/SPARK-9498
> Project: Spark
> Issue Type: Improvement
> Components: Web UI
> Affects Versions: 1.3.1, 1.4.0
> Reporter: Liang Lee
>
> When an application is submited and the driver is out of the Spark cluster, Some statistical information missed sometimes.
> In stage detail inforamtion page, it will display following info when the driver is in the spark cluster:
> Details for Stage 7
> Total task time across all tasks: 37 min
> Input Size / Records: 55.8 GB / 60488
> Shuffle write: 26.6 GB / 585242962
> But when the dreive is out of the spark cluster, it will sometimes display above info, while sometimes not, just like this:
> Details for Stage 7
> Total task time across all tasks: 37 min
> That is the Input Size and Shuffle data does not display.
> I have check the code and find that when the input size is zero then it will not display.
> And the input size is sent by each Executors and collected by Driver.
> The problem is that the data that should be repored by Executors, is missed.
> But I don't know why. Could anyone help to solve this problem?
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org