You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "sam (Jira)" <ji...@apache.org> on 2021/03/13 16:35:00 UTC
[jira] [Created] (SPARK-34733) Spark UI not showing memory used of
partitions in memory
sam created SPARK-34733:
---------------------------
Summary: Spark UI not showing memory used of partitions in memory
Key: SPARK-34733
URL: https://issues.apache.org/jira/browse/SPARK-34733
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 2.4.4
Environment: EMR AWS emr-6.0.0
Reporter: sam
Attachments: Screenshot 2021-03-13 at 16.31.06.png
We have a job that caches RDDs into memory. We know the code to cache is working as the spark logs correctly point out the caching is happening:
```
21/03/13 16:17:38 INFO BlockManagerInfo: Added rdd_12_413 in memory on ip-172-31-24-152.eu-west-2.compute.internal:43849 (size: 201.4 MB, free: 575.0 GB)
21/03/13 16:17:38 INFO TaskSetManager: Starting task 897.0 in stage 1.0 (TID 10277, ip-172-31-24-152.eu-west-2.compute.internal, executor 8, partition 897, RACK_LOCAL, 9812 bytes)
21/03/13 16:17:38 INFO TaskSetManager: Finished task 413.0 in stage 1.0 (TID 9793) in 1250463 ms on ip-172-31-24-152.eu-west-2.compute.internal (executor 8) (34/2162)
21/03/13 16:17:42 INFO BlockManagerInfo: Added rdd_12_768 in memory on ip-172-31-23-154.eu-west-2.compute.internal:37957 (size: 718.4 MB, free: 574.6 GB)
21/03/13 16:17:43 INFO TaskSetManager: Starting task 898.0 in stage 1.0 (TID 10278, ip-172-31-23-154.eu-west-2.compute.internal, executor 7, partition 898, RACK_LOCAL, 9841 bytes)
21/03/13 16:17:43 INFO TaskSetManager: Finished task 768.0 in stage 1.0 (TID 10148) in 1254945 ms on ip-172-31-23-154.eu-west-2.compute.internal (executor 7) (35/2162)
21/03/13 16:18:44 INFO BlockManagerInfo: Added rdd_12_409 in memory on ip-172-31-21-66.eu-west-2.compute.internal:38921 (size: 177.6 MB, free: 575.1 GB)
21/03/13 16:18:45 INFO TaskSetManager: Starting task 899.0 in stage 1.0 (TID 10279, ip-172-31-21-66.eu-west-2.compute.internal, executor 4, partition 899, RACK_LOCAL, 9828 bytes)
21/03/13 16:18:45 INFO TaskSetManager: Finished task 409.0 in stage 1.0 (TID 9789) in 1316584 ms on ip-172-31-21-66.eu-west-2.compute.internal (executor 4) (36/2162)
21/03/13 16:19:40 INFO BlockManagerInfo: Added rdd_12_400 in memory on ip-172-31-21-66.eu-west-2.compute.internal:38921 (size: 187.9 MB, free: 574.9 GB)
21/03/13 16:19:41 INFO TaskSetManager: Starting task 900.0 in stage 1.0 (TID 10280, ip-172-31-21-66.eu-west-2.compute.internal, executor 4, partition 900, RACK_LOCAL, 9843 bytes)
21/03/13 16:19:41 INFO TaskSetManager: Finished task 400.0 in stage 1.0 (TID 9780) in 1372717 ms on ip-172-31-21-66.eu-west-2.compute.internal (executor 4) (37/2162)
21/03/13 16:20:55 INFO BlockManagerInfo: Added rdd_12_640 in memory on ip-172-31-17-157.eu-west-2.compute.internal:34005 (size: 576.1 MB, free: 574.7 GB)
21/03/13 16:20:58 INFO TaskSetManager: Starting task 901.0 in stage 1.0 (TID 10281, ip-172-31-17-157.eu-west-2.compute.internal, executor 9, partition 901, RACK_LOCAL, 9750 bytes)
21/03/13 16:20:58 INFO TaskSetManager: Finished task 640.0 in stage 1.0 (TID 10020) in 1449618 ms on ip-172-31-17-157.eu-west-2.compute.internal (executor 9) (38/2162)
21/03/13 16:21:07 INFO BlockManagerInfo: Added rdd_12_610 in memory on ip-172-31-30-188.eu-west-2.compute.internal:38111 (size: 582.2 MB, free: 574.7 GB)
```
But when we look in the Spark UI Executors tab it shows 0 B used of the maximum. Please see screenshot:
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org