[GitHub] [aurora] StephanErb opened a new pull request #66: Add observer flag to disable resource metric collection

[GitBox <gi...@apache.org>]

StephanErb opened a new pull request #66: Add observer flag to disable resource metric collection
URL: https://github.com/apache/aurora/pull/66
 
 
   ### Description:
   
   Add observer command line option `--disable_task_resource_collection` to disable the collection of CPU, memory, and disk metrics for observed tasks. This is useful in setups where metrics cannot be gathered reliable (e.g. when using PID namespaces) or when it is expensive due to hundreds of active tasks per host.
   
   Sometimes the hosts are also tightly packed with many small tasks (e.g. ~130 active tasks and ~1000 finished tasks). Even with very relaxed scrape settings of  `--task_process_collection_interval_secs=3000` and `--task_disk_collection_interval_secs=3000` it can take between 150ms-2500ms to render the observer landing page `/main`. This patch reduces this to about 100ms-150ms. 
   
   There is no immediate downside as metrics reporting is broken anyway due to the PID namespacing: We are running our Mesos agents with enabled PID namespaces (i.e.
   `--isolation='namespaces/ipc,namespaces/pid,...')`. In that mode, the PID of the same process is different within the container and outside of it. This breaks the assumption of Thermos that the executor can checkpoint a PID to disk that then can be used by the Observer to show live resource statistics for that PID.
   
   ### Testing Done:
   Running in production for over a year now.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services