Computing total application vcore/mem demand

[Chris Mullins <cm...@liveramp.com>]

Hello,

In order to track demand for our cluster's resources, I'm trying to figure
out a way to measure instantaneous application demand for vcore and memory.
The motivation is to know when and by how much we are under- or
over-utilizing our resources.

I've tried summing //apps/app/allocatedMB and //apps/app/allocatedVCores
from this RM API endpoint:

http://<url>/ws/v1/cluster/apps?states=RUNNING

This is quite close to what I want, but does not capture the case when
total requested resources exceeds what's available -- I think the above sum
represents resources allocated to currently running containers and excludes
requested resources that are currently not running.

I poked around a bit in the source and if I'm interpreting it correctly, it
seems like this class has the information I need:

https://hadoop.apache.org/docs/r2.5.2/api/org/apache/hadoop/yarn/api/records/ApplicationResourceUsageReport.html#getNeededResources()

but I couldn't find where this information was exposed in the YARN rest
api/cli (seemed plausible this applied to containers and not applications,
but I wasn't sure)

Does anyone know how I might accomplish this? Obviously I'm open to
alternative approaches as well.

Thanks in advance!

Chris Mullins