[jira] [Created] (SPARK-9498) Some statistical information missed when the driver is out of the cluster

["Liang Lee (JIRA)" <ji...@apache.org>]

Liang Lee created SPARK-9498:
--------------------------------

             Summary: Some statistical information missed when the driver is out of the cluster
                 Key: SPARK-9498
                 URL: https://issues.apache.org/jira/browse/SPARK-9498
             Project: Spark
          Issue Type: Improvement
          Components: Web UI
    Affects Versions: 1.4.0, 1.3.1
            Reporter: Liang Lee


When an application is submited and the driver is out of the Spark cluster, Some statistical information missed sometimes. 
In stage detail inforamtion page, it will display following  info when the driver is in the spark cluster:
Details for Stage 7
    Total task time across all tasks: 37 min
     Input Size / Records: 55.8 GB / 60488
     Shuffle write: 26.6 GB / 585242962 

But when the dreive is out of the spark cluster, it will sometimes  display above info, while sometimes not, just like this:

Details for Stage 7
    Total task time across all tasks: 37 min

That is the Input Size and Shuffle data does not display.

I have check the code and find that when the input size is zero then it will not display. 
And the  input size is sent by each Executors and collected by Driver. 
The problem is that the data that should be repored by Executors, is missed. 
But I don't know why. Could anyone help to solve this problem?
 







--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org