You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Andy Davidson <An...@SantaCruzIntegration.com> on 2016/07/11 15:52:59 UTC

spark UI what does storage memory x/y mean

My stream app is running into problems It seems to slow down over time. How
can I interpret the storage memory column. I wonder if I have a GC problem?
Any idea how I can get GC stats?

Thanks

Andy

Executors (3)
* Memory: 9.4 GB Used (1533.4 MB Total)
* Disk: 0.0 B Used
Executor IDAddressRDD BlocksStorage MemoryDisk UsedActive TasksFailed
TasksComplete TasksTotal TasksTask TimeInputShuffle ReadShuffle
WriteLogsThread Dump
0ip-172-31-23-202.us-west-1.compute.internal:5245628604.7 GB / 511.1 MB0.0
B04013495793499805.37 h72.9 GB84.0 B5.9 MBstdout
<http://ec2-54-193-94-207.us-west-1.compute.amazonaws.com:8081/logPage/?appI
d=app-20160710171110-0048&executorId=0&logType=stdout>
stderr 
<http://ec2-54-193-94-207.us-west-1.compute.amazonaws.com:8081/logPage/?appI
d=app-20160710171110-0048&executorId=0&logType=stderr>
Thread Dump 
<http://ec2-54-67-13-246.us-west-1.compute.amazonaws.com:4040/executors/thre
adDump/?executorId=0>
1ip-172-31-23-200.us-west-1.compute.internal:5160928544.6 GB / 511.1 MB0.0
B04113493653497765.42 h72.6 GB142.0 B5.9 MBstdout
<http://ec2-54-67-48-49.us-west-1.compute.amazonaws.com:8081/logPage/?appId=
app-20160710171110-0048&executorId=1&logType=stdout>
stderr 
<http://ec2-54-67-48-49.us-west-1.compute.amazonaws.com:8081/logPage/?appId=
app-20160710171110-0048&executorId=1&logType=stderr>
Thread Dump 
<http://ec2-54-67-13-246.us-west-1.compute.amazonaws.com:4040/executors/thre
adDump/?executorId=1>
driver172.31.23.203:4801800.0 B / 511.1 MB0.0 B00000 ms0.0 B0.0 B0.0 BThread
Dump 
<http://ec2-54-67-13-246.us-west-1.compute.amazonaws.com:4040/executors/thre
adDump/?executorId=driver>