You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Shixiong Zhu (JIRA)" <ji...@apache.org> on 2015/08/12 04:08:46 UTC
[jira] [Issue Comment Deleted] (SPARK-9829) peakExecutionMemory is
not correct
[ https://issues.apache.org/jira/browse/SPARK-9829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shixiong Zhu updated SPARK-9829:
--------------------------------
Comment: was deleted
(was: How many tasks? peakExecutionMemory in Web UI is the sum of peakExecutionMemory in all tasks. This value may be confusing sometimes. E.g., assume we have 2 tasks, at 10:00am, task 1's memory usage is 10G, which is peak, and it finishes at 10:02am; then task 2 starts at 10:03am, and it reaches the peak at 10:04am, which is 10G. Then peakExecutionMemory in Web UI will be 20G, although we have never used more than 10G.
BTW, did you modify the codes? These values should not be shown directly in Web UI.
/cc [~andrewor14])
> peakExecutionMemory is not correct
> ----------------------------------
>
> Key: SPARK-9829
> URL: https://issues.apache.org/jira/browse/SPARK-9829
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Reporter: Davies Liu
> Assignee: Shixiong Zhu
>
> When run a query with 8G memory, the peakExecutionMemory in WebUI said that 40344371200 (40G).
> Alos there are lots of accumulators with the same name, can't know what do they mean
> {code}
> Accumulable Value
> number of output rows 439614
> number of output rows 7711
> number of output rows 965
> number of rows 7829
> number of rows 7711
> number of input rows 965
> number of rows 52
> number of input rows 439614
> number of output rows 30
> number of input rows 7726
> number of rows 277000
> peakExecutionMemory 40344371200
> number of rows 7829
> number of rows 965
> number of rows 7726
> number of rows 30
> number of rows 138000
> number of rows 8028
> number of rows 439614
> number of input rows 30
> {code}
> How to reproduce:
> run TPCDS q19 with scale=5, checkout out the Web UI
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org