Multiple Streaming Apps running on the Spark Cluster

[winter fresh <wi...@gmail.com>]

We do have 4 streaming Apps running on a 3 Node cluster.

No. Of Executors / App = 2
No. Of Cores / Executor = 2
Memory / Executor = 12 GB


Cluster Capacity
----------------------
Cores / Node = 8
Memory / Node = 61 GB

The problem is that the one Heavy Load App just crashes after running for
about ~10 Hours without giving any specific error messages on Log.

We collect Metrics in Graphite and the below graph of "JVM Total Used"
shows that it keeps increasing over time. Could that be the reason for the
Failure? I glanced at the other Metrics too but couldn't find anything
interesting. Are there other Metrics to look at for Executors or Drivers?


[image: Inline image 1]

What is the best way to identify the resources used by Each Application as
we would be adding more Apps to the cluster?

Thank you!