You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by "Sam Whitlock (Created) (JIRA)" <ji...@apache.org> on 2011/10/14 20:24:12 UTC
[jira] [Created] (MESOS-38) Executor resource monitoring and local
reporting of usage stats
Executor resource monitoring and local reporting of usage stats
---------------------------------------------------------------
Key: MESOS-38
URL: https://issues.apache.org/jira/browse/MESOS-38
Project: Mesos
Issue Type: Improvement
Environment: Initial executor monitoring for linux only. Dummy monitoring capability (no-op) for OSX, with functionality to be filled in later.
Reporter: Sam Whitlock
Implement reporting of resource usage usage on executors and log them to a local log file (for now). The eventual usage of this will be to report these statistics to the Mesos master in order to build either or both a timeline for the webui and/or a top-like command-line interface. This improvement ticket is just for the local monitoring and log file reporting. A reporting system (to the master node) will be a later improvement ticket.
With the current version of Mesos, it is not possible to monitor individual task. Therefore the best this sort of system can do is monitor the usage of an individual executor and aggregate the resource usage of over the executor's tasks and resource allocations.
Reporting will be available for both lxc isolation and process-based isolation. For lxc isolation the task is easier because of the isolation facilities of lxc. Process-based isolation is more difficult as processes can become re-parented from the process tree of the executor (e.g. double fork). The session ID and the process group ID will likely still be the same as that of the executor except for the uncommon case of the process resetting both of those.
Initial reporting will be to a local log file. This will be a 'heartbeat' style akin to pidstat output (in sysstat library). This may not be incredibly useful, but local monitoring of resource usage is separate from the reporting and timeline building mentioned above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MESOS-38) Executor resource monitoring and
local reporting of usage stats
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MESOS-38?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222006#comment-13222006 ]
jiraposter@reviews.apache.org commented on MESOS-38:
----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4167/
-----------------------------------------------------------
(Updated 2012-03-04 20:40:08.696183)
Review request for mesos, Benjamin Hindman and Charles Reiss.
Changes
-------
Diff from parent:
- reduced logging level for failing to read info from container to INFO (from ERROR), because it is not necessarily an error
- added command-line arg -f for frequency of usage info scraping (1/f is the time passed to the delay call)
Summary
-------
This mega-patch is intended to represent the partial completion of the slave monitoring functionality. It is not intended to be committed. Changes based on comments in this review will be reflected in future reviews that are smaller and more modular.
Proc utils is included in this patch, but is already under review here: https://reviews.apache.org/r/3050/
The relevant design doc can be found here: https://docs.google.com/document/d/14Wj9i6TpMR6cV3LL0ySjLjfZOq5QOQeybvt1gSyaQGs/edit
The following items are ones where specific feedback is requested:
* A better mechanism is needed to control the rate at which the slave asks each executor for its UsageMessage. This is currently hard-coded to be at 1 second intervals, but could potentially be read as a command-line option or from a config file. Is there a better or different way to pass in this value?
* Currently, UsageMessages are passed from a ResourceMonitor to the Slave using the Future construct, and used as containers that hold a snapshot of the latest usage. This is to prevent unnecessary marshalling and extra data structures, since messages will eventually be sent in the standard dispatch style from the slave to the master. Is it fine that we are using Protobuf messages in this way?
There are several changes that are not yet implemented in this patch. These changes are as follows:
* Sufficient tests cases have not yet been written for any component (resource monitor, lxc collector, and process collector).
* Code has not been cleaned up to adhere to all style recommendations.
* Process collector code needs to be updated to prevent CPU usage spikes when monitored sub-processes die.
* Code to send UsageMessages from the slave to the master.
This addresses bug MESOS-38.
https://issues.apache.org/jira/browse/MESOS-38
Diffs (updated)
-----
src/monitoring/process_stats.hpp PRE-CREATION
src/monitoring/resource_collector.hpp PRE-CREATION
src/slave/http.cpp f03815d
src/slave/isolation_module.hpp c896908
src/slave/isolation_module.cpp 5b7b4a2
src/slave/lxc_isolation_module.hpp b7beefe
src/slave/lxc_isolation_module.cpp d544625
src/slave/main.cpp ac780c4
src/slave/process_based_isolation_module.hpp f6f9554
src/slave/process_based_isolation_module.cpp 100b1e3
src/slave/resource_monitor.hpp PRE-CREATION
src/slave/resource_monitor.cpp PRE-CREATION
src/slave/slave.hpp b1a07e9
src/slave/slave.cpp ce8fda5
src/tests/Makefile.in 6f51be4
src/tests/proc_utils_tests.cpp PRE-CREATION
src/tests/process_resource_collector_tests.cpp PRE-CREATION
src/tests/resource_monitor_tests.cpp PRE-CREATION
src/monitoring/process_resource_collector.cpp PRE-CREATION
src/monitoring/process_resource_collector.hpp PRE-CREATION
src/monitoring/linux/proc_utils.cpp PRE-CREATION
src/monitoring/linux/proc_utils.hpp PRE-CREATION
src/monitoring/linux/proc_resource_collector.cpp PRE-CREATION
src/monitoring/linux/proc_resource_collector.hpp PRE-CREATION
src/monitoring/linux/lxc_resource_collector.cpp PRE-CREATION
src/master/master.hpp 53551b0
src/master/master.cpp 1d3961e
src/messages/messages.proto 11a2c41
src/monitoring/linux/lxc_resource_collector.hpp PRE-CREATION
src/master/allocator.hpp 1ac435b
src/master/http.cpp 591433a
src/Makefile.am 1137a3e
Diff: https://reviews.apache.org/r/4167/diff
Testing
-------
Test cases:
* A test case exercising the basic monitoring code with a mocked-out collector.
* The first of several tests for the process resource monitor, with the proc-based collecting mocked out.
Some ad-hoc testing with log statements to ensure that the monitoring works end-to-end from both the container-based and process-based isolation modules.
Thanks,
Sam
> Executor resource monitoring and local reporting of usage stats
> ---------------------------------------------------------------
>
> Key: MESOS-38
> URL: https://issues.apache.org/jira/browse/MESOS-38
> Project: Mesos
> Issue Type: New Feature
> Components: isolation, slave
> Environment: Initial executor monitoring for linux only. Dummy monitoring capability (no-op) for OSX, with functionality to be filled in later.
> Reporter: Sam Whitlock
> Assignee: Sam Whitlock
> Labels: monitoring
>
> Implement reporting of resource usage on executors and log them to a local log file (for now). The eventual usage of this will be to report these statistics to the Mesos master in order to build either or both a timeline for the webui and/or a top-like command-line interface. This improvement ticket is just for the local monitoring and log file reporting. A reporting system (to the master node) will be a later improvement ticket.
> With the current version of Mesos, it is not possible to monitor individual tasks. Therefore the best this sort of system can do is monitor the usage of an individual executor and aggregate the resource usage of over the executor's tasks and resource allocations. If frameworks have a 1-to-1 relationship of a job to an executor, then the aggregate statistics will be more meaningful.
> Reporting will be available for both lxc isolation and process-based isolation. For lxc isolation the task is easier because of the isolation facilities of lxc. Process-based isolation is more difficult as processes can become re-parented from the process tree of the executor (e.g. double fork). The session ID and the process group ID will likely still be the same as that of the executor except for the uncommon case of the process resetting both of those.
> When usage statistics are eventually reported to the Mesos master, it may be possible to use them to oversubscribe slave nodes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MESOS-38) Executor resource monitoring and local
reporting of usage stats
Posted by "Sam Whitlock (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MESOS-38?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sam Whitlock updated MESOS-38:
------------------------------
Issue Type: New Feature (was: Improvement)
> Executor resource monitoring and local reporting of usage stats
> ---------------------------------------------------------------
>
> Key: MESOS-38
> URL: https://issues.apache.org/jira/browse/MESOS-38
> Project: Mesos
> Issue Type: New Feature
> Environment: Initial executor monitoring for linux only. Dummy monitoring capability (no-op) for OSX, with functionality to be filled in later.
> Reporter: Sam Whitlock
> Labels: monitoring
>
> Implement reporting of resource usage usage on executors and log them to a local log file (for now). The eventual usage of this will be to report these statistics to the Mesos master in order to build either or both a timeline for the webui and/or a top-like command-line interface. This improvement ticket is just for the local monitoring and log file reporting. A reporting system (to the master node) will be a later improvement ticket.
> With the current version of Mesos, it is not possible to monitor individual task. Therefore the best this sort of system can do is monitor the usage of an individual executor and aggregate the resource usage of over the executor's tasks and resource allocations.
> Reporting will be available for both lxc isolation and process-based isolation. For lxc isolation the task is easier because of the isolation facilities of lxc. Process-based isolation is more difficult as processes can become re-parented from the process tree of the executor (e.g. double fork). The session ID and the process group ID will likely still be the same as that of the executor except for the uncommon case of the process resetting both of those.
> Initial reporting will be to a local log file. This will be a 'heartbeat' style akin to pidstat output (in sysstat library). This may not be incredibly useful, but local monitoring of resource usage is separate from the reporting and timeline building mentioned above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MESOS-38) Executor resource monitoring and local
reporting of usage stats
Posted by "Sam Whitlock (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MESOS-38?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sam Whitlock updated MESOS-38:
------------------------------
Description:
Implement reporting of resource usage on executors and log them to a local log file (for now). The eventual usage of this will be to report these statistics to the Mesos master in order to build either or both a timeline for the webui and/or a top-like command-line interface. This improvement ticket is just for the local monitoring and log file reporting. A reporting system (to the master node) will be a later improvement ticket.
With the current version of Mesos, it is not possible to monitor individual tasks. Therefore the best this sort of system can do is monitor the usage of an individual executor and aggregate the resource usage of over the executor's tasks and resource allocations. If frameworks have a 1-to-1 relationship of a job to an executor, then the aggregate statistics will be more meaningful.
Reporting will be available for both lxc isolation and process-based isolation. For lxc isolation the task is easier because of the isolation facilities of lxc. Process-based isolation is more difficult as processes can become re-parented from the process tree of the executor (e.g. double fork). The session ID and the process group ID will likely still be the same as that of the executor except for the uncommon case of the process resetting both of those.
When usage statistics are eventually reported to the Mesos master, it may be possible to use them to oversubscribe slave nodes.
was:
Implement reporting of resource usage on executors and log them to a local log file (for now). The eventual usage of this will be to report these statistics to the Mesos master in order to build either or both a timeline for the webui and/or a top-like command-line interface. This improvement ticket is just for the local monitoring and log file reporting. A reporting system (to the master node) will be a later improvement ticket.
With the current version of Mesos, it is not possible to monitor individual tasks. Therefore the best this sort of system can do is monitor the usage of an individual executor and aggregate the resource usage of over the executor's tasks and resource allocations. If frameworks have a 1-to-1 relationship of a job to an executor, then the aggregate statistics will be more meaningful.
Reporting will be available for both lxc isolation and process-based isolation. For lxc isolation the task is easier because of the isolation facilities of lxc. Process-based isolation is more difficult as processes can become re-parented from the process tree of the executor (e.g. double fork). The session ID and the process group ID will likely still be the same as that of the executor except for the uncommon case of the process resetting both of those.
Initial reporting will be to a local log file. This will be a 'heartbeat' style akin to pidstat output (in sysstat library). This may not be incredibly useful, but local monitoring of resource usage is separate from the reporting and timeline building mentioned above.
When usage statistics are eventually reported to the Mesos master, it may be possible to use them to oversubscribe slave nodes.
> Executor resource monitoring and local reporting of usage stats
> ---------------------------------------------------------------
>
> Key: MESOS-38
> URL: https://issues.apache.org/jira/browse/MESOS-38
> Project: Mesos
> Issue Type: New Feature
> Components: isolation, slave
> Environment: Initial executor monitoring for linux only. Dummy monitoring capability (no-op) for OSX, with functionality to be filled in later.
> Reporter: Sam Whitlock
> Assignee: Sam Whitlock
> Labels: monitoring
>
> Implement reporting of resource usage on executors and log them to a local log file (for now). The eventual usage of this will be to report these statistics to the Mesos master in order to build either or both a timeline for the webui and/or a top-like command-line interface. This improvement ticket is just for the local monitoring and log file reporting. A reporting system (to the master node) will be a later improvement ticket.
> With the current version of Mesos, it is not possible to monitor individual tasks. Therefore the best this sort of system can do is monitor the usage of an individual executor and aggregate the resource usage of over the executor's tasks and resource allocations. If frameworks have a 1-to-1 relationship of a job to an executor, then the aggregate statistics will be more meaningful.
> Reporting will be available for both lxc isolation and process-based isolation. For lxc isolation the task is easier because of the isolation facilities of lxc. Process-based isolation is more difficult as processes can become re-parented from the process tree of the executor (e.g. double fork). The session ID and the process group ID will likely still be the same as that of the executor except for the uncommon case of the process resetting both of those.
> When usage statistics are eventually reported to the Mesos master, it may be possible to use them to oversubscribe slave nodes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MESOS-38) Executor resource monitoring and local
reporting of usage stats
Posted by "Sam Whitlock (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MESOS-38?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sam Whitlock reassigned MESOS-38:
---------------------------------
Assignee: Sam Whitlock
> Executor resource monitoring and local reporting of usage stats
> ---------------------------------------------------------------
>
> Key: MESOS-38
> URL: https://issues.apache.org/jira/browse/MESOS-38
> Project: Mesos
> Issue Type: New Feature
> Environment: Initial executor monitoring for linux only. Dummy monitoring capability (no-op) for OSX, with functionality to be filled in later.
> Reporter: Sam Whitlock
> Assignee: Sam Whitlock
> Labels: monitoring
>
> Implement reporting of resource usage on executors and log them to a local log file (for now). The eventual usage of this will be to report these statistics to the Mesos master in order to build either or both a timeline for the webui and/or a top-like command-line interface. This improvement ticket is just for the local monitoring and log file reporting. A reporting system (to the master node) will be a later improvement ticket.
> With the current version of Mesos, it is not possible to monitor individual tasks. Therefore the best this sort of system can do is monitor the usage of an individual executor and aggregate the resource usage of over the executor's tasks and resource allocations. If frameworks have a 1-to-1 relationship of a job to an executor, then the aggregate statistics will be more meaningful.
> Reporting will be available for both lxc isolation and process-based isolation. For lxc isolation the task is easier because of the isolation facilities of lxc. Process-based isolation is more difficult as processes can become re-parented from the process tree of the executor (e.g. double fork). The session ID and the process group ID will likely still be the same as that of the executor except for the uncommon case of the process resetting both of those.
> Initial reporting will be to a local log file. This will be a 'heartbeat' style akin to pidstat output (in sysstat library). This may not be incredibly useful, but local monitoring of resource usage is separate from the reporting and timeline building mentioned above.
> When usage statistics are eventually reported to the Mesos master, it may be possible to use them to oversubscribe slave nodes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MESOS-38) Executor resource monitoring and
local reporting of usage stats
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MESOS-38?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13224754#comment-13224754 ]
jiraposter@reviews.apache.org commented on MESOS-38:
----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4167/
-----------------------------------------------------------
(Updated 2012-03-07 21:53:30.216474)
Review request for mesos, Benjamin Hindman and Charles Reiss.
Changes
-------
changing resource names to match the resources they are monitoring
Summary
-------
This mega-patch is intended to represent the partial completion of the slave monitoring functionality. It is not intended to be committed. Changes based on comments in this review will be reflected in future reviews that are smaller and more modular.
Proc utils is included in this patch, but is already under review here: https://reviews.apache.org/r/3050/
The relevant design doc can be found here: https://docs.google.com/document/d/14Wj9i6TpMR6cV3LL0ySjLjfZOq5QOQeybvt1gSyaQGs/edit
The following items are ones where specific feedback is requested:
* A better mechanism is needed to control the rate at which the slave asks each executor for its UsageMessage. This is currently hard-coded to be at 1 second intervals, but could potentially be read as a command-line option or from a config file. Is there a better or different way to pass in this value?
* Currently, UsageMessages are passed from a ResourceMonitor to the Slave using the Future construct, and used as containers that hold a snapshot of the latest usage. This is to prevent unnecessary marshalling and extra data structures, since messages will eventually be sent in the standard dispatch style from the slave to the master. Is it fine that we are using Protobuf messages in this way?
There are several changes that are not yet implemented in this patch. These changes are as follows:
* Sufficient tests cases have not yet been written for any component (resource monitor, lxc collector, and process collector).
* Code has not been cleaned up to adhere to all style recommendations.
* Process collector code needs to be updated to prevent CPU usage spikes when monitored sub-processes die.
* Code to send UsageMessages from the slave to the master.
This addresses bug MESOS-38.
https://issues.apache.org/jira/browse/MESOS-38
Diffs (updated)
-----
src/Makefile.am 1137a3e
src/master/allocator.hpp 1ac435b
src/master/http.cpp 591433a
src/master/master.hpp 53551b0
src/master/master.cpp 1d3961e
src/messages/messages.proto 11a2c41
src/monitoring/linux/lxc_resource_collector.hpp PRE-CREATION
src/monitoring/linux/lxc_resource_collector.cpp PRE-CREATION
src/monitoring/linux/proc_resource_collector.hpp PRE-CREATION
src/monitoring/linux/proc_resource_collector.cpp PRE-CREATION
src/monitoring/linux/proc_utils.hpp PRE-CREATION
src/monitoring/linux/proc_utils.cpp PRE-CREATION
src/monitoring/process_resource_collector.hpp PRE-CREATION
src/monitoring/process_resource_collector.cpp PRE-CREATION
src/monitoring/process_stats.hpp PRE-CREATION
src/monitoring/resource_collector.hpp PRE-CREATION
src/slave/http.cpp f03815d
src/slave/isolation_module.hpp c896908
src/slave/isolation_module.cpp 5b7b4a2
src/slave/lxc_isolation_module.hpp b7beefe
src/slave/lxc_isolation_module.cpp d544625
src/slave/main.cpp ac780c4
src/slave/process_based_isolation_module.hpp f6f9554
src/slave/process_based_isolation_module.cpp 100b1e3
src/slave/resource_monitor.hpp PRE-CREATION
src/slave/resource_monitor.cpp PRE-CREATION
src/slave/slave.hpp b1a07e9
src/slave/slave.cpp ce8fda5
src/tests/Makefile.in 6f51be4
src/tests/proc_utils_tests.cpp PRE-CREATION
src/tests/process_resource_collector_tests.cpp PRE-CREATION
src/tests/resource_monitor_tests.cpp PRE-CREATION
Diff: https://reviews.apache.org/r/4167/diff
Testing
-------
Test cases:
* A test case exercising the basic monitoring code with a mocked-out collector.
* The first of several tests for the process resource monitor, with the proc-based collecting mocked out.
Some ad-hoc testing with log statements to ensure that the monitoring works end-to-end from both the container-based and process-based isolation modules.
Thanks,
Sam
> Executor resource monitoring and local reporting of usage stats
> ---------------------------------------------------------------
>
> Key: MESOS-38
> URL: https://issues.apache.org/jira/browse/MESOS-38
> Project: Mesos
> Issue Type: New Feature
> Components: isolation, slave
> Environment: Initial executor monitoring for linux only. Dummy monitoring capability (no-op) for OSX, with functionality to be filled in later.
> Reporter: Sam Whitlock
> Assignee: Sam Whitlock
> Labels: monitoring
>
> Implement reporting of resource usage on executors and log them to a local log file (for now). The eventual usage of this will be to report these statistics to the Mesos master in order to build either or both a timeline for the webui and/or a top-like command-line interface. This improvement ticket is just for the local monitoring and log file reporting. A reporting system (to the master node) will be a later improvement ticket.
> With the current version of Mesos, it is not possible to monitor individual tasks. Therefore the best this sort of system can do is monitor the usage of an individual executor and aggregate the resource usage of over the executor's tasks and resource allocations. If frameworks have a 1-to-1 relationship of a job to an executor, then the aggregate statistics will be more meaningful.
> Reporting will be available for both lxc isolation and process-based isolation. For lxc isolation the task is easier because of the isolation facilities of lxc. Process-based isolation is more difficult as processes can become re-parented from the process tree of the executor (e.g. double fork). The session ID and the process group ID will likely still be the same as that of the executor except for the uncommon case of the process resetting both of those.
> When usage statistics are eventually reported to the Mesos master, it may be possible to use them to oversubscribe slave nodes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MESOS-38) Executor resource monitoring and local
reporting of usage stats
Posted by "Sam Whitlock (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MESOS-38?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sam Whitlock updated MESOS-38:
------------------------------
Description:
Implement reporting of resource usage on executors and log them to a local log file (for now). The eventual usage of this will be to report these statistics to the Mesos master in order to build either or both a timeline for the webui and/or a top-like command-line interface. This improvement ticket is just for the local monitoring and log file reporting. A reporting system (to the master node) will be a later improvement ticket.
With the current version of Mesos, it is not possible to monitor individual tasks. Therefore the best this sort of system can do is monitor the usage of an individual executor and aggregate the resource usage of over the executor's tasks and resource allocations. If frameworks have a 1-to-1 relationship of a job to an executor, then the aggregate statistics will be more meaningful.
Reporting will be available for both lxc isolation and process-based isolation. For lxc isolation the task is easier because of the isolation facilities of lxc. Process-based isolation is more difficult as processes can become re-parented from the process tree of the executor (e.g. double fork). The session ID and the process group ID will likely still be the same as that of the executor except for the uncommon case of the process resetting both of those.
Initial reporting will be to a local log file. This will be a 'heartbeat' style akin to pidstat output (in sysstat library). This may not be incredibly useful, but local monitoring of resource usage is separate from the reporting and timeline building mentioned above.
When usage statistics are eventually reported to the Mesos master, it may be possible to use them to oversubscribe slave nodes.
was:
Implement reporting of resource usage on executors and log them to a local log file (for now). The eventual usage of this will be to report these statistics to the Mesos master in order to build either or both a timeline for the webui and/or a top-like command-line interface. This improvement ticket is just for the local monitoring and log file reporting. A reporting system (to the master node) will be a later improvement ticket.
With the current version of Mesos, it is not possible to monitor individual tasks. Therefore the best this sort of system can do is monitor the usage of an individual executor and aggregate the resource usage of over the executor's tasks and resource allocations. If frameworks have a 1-to-1 relationship of a job to an executor, then the aggregate statistics will be more meaningful.
Reporting will be available for both lxc isolation and process-based isolation. For lxc isolation the task is easier because of the isolation facilities of lxc. Process-based isolation is more difficult as processes can become re-parented from the process tree of the executor (e.g. double fork). The session ID and the process group ID will likely still be the same as that of the executor except for the uncommon case of the process resetting both of those.
Initial reporting will be to a local log file. This will be a 'heartbeat' style akin to pidstat output (in sysstat library). This may not be incredibly useful, but local monitoring of resource usage is separate from the reporting and timeline building mentioned above.
> Executor resource monitoring and local reporting of usage stats
> ---------------------------------------------------------------
>
> Key: MESOS-38
> URL: https://issues.apache.org/jira/browse/MESOS-38
> Project: Mesos
> Issue Type: New Feature
> Environment: Initial executor monitoring for linux only. Dummy monitoring capability (no-op) for OSX, with functionality to be filled in later.
> Reporter: Sam Whitlock
> Labels: monitoring
>
> Implement reporting of resource usage on executors and log them to a local log file (for now). The eventual usage of this will be to report these statistics to the Mesos master in order to build either or both a timeline for the webui and/or a top-like command-line interface. This improvement ticket is just for the local monitoring and log file reporting. A reporting system (to the master node) will be a later improvement ticket.
> With the current version of Mesos, it is not possible to monitor individual tasks. Therefore the best this sort of system can do is monitor the usage of an individual executor and aggregate the resource usage of over the executor's tasks and resource allocations. If frameworks have a 1-to-1 relationship of a job to an executor, then the aggregate statistics will be more meaningful.
> Reporting will be available for both lxc isolation and process-based isolation. For lxc isolation the task is easier because of the isolation facilities of lxc. Process-based isolation is more difficult as processes can become re-parented from the process tree of the executor (e.g. double fork). The session ID and the process group ID will likely still be the same as that of the executor except for the uncommon case of the process resetting both of those.
> Initial reporting will be to a local log file. This will be a 'heartbeat' style akin to pidstat output (in sysstat library). This may not be incredibly useful, but local monitoring of resource usage is separate from the reporting and timeline building mentioned above.
> When usage statistics are eventually reported to the Mesos master, it may be possible to use them to oversubscribe slave nodes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MESOS-38) Executor resource monitoring and local
reporting of usage stats
Posted by "Alex Degtiar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MESOS-38?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alex Degtiar updated MESOS-38:
------------------------------
Component/s: slave
isolation
> Executor resource monitoring and local reporting of usage stats
> ---------------------------------------------------------------
>
> Key: MESOS-38
> URL: https://issues.apache.org/jira/browse/MESOS-38
> Project: Mesos
> Issue Type: New Feature
> Components: isolation, slave
> Environment: Initial executor monitoring for linux only. Dummy monitoring capability (no-op) for OSX, with functionality to be filled in later.
> Reporter: Sam Whitlock
> Assignee: Sam Whitlock
> Labels: monitoring
>
> Implement reporting of resource usage on executors and log them to a local log file (for now). The eventual usage of this will be to report these statistics to the Mesos master in order to build either or both a timeline for the webui and/or a top-like command-line interface. This improvement ticket is just for the local monitoring and log file reporting. A reporting system (to the master node) will be a later improvement ticket.
> With the current version of Mesos, it is not possible to monitor individual tasks. Therefore the best this sort of system can do is monitor the usage of an individual executor and aggregate the resource usage of over the executor's tasks and resource allocations. If frameworks have a 1-to-1 relationship of a job to an executor, then the aggregate statistics will be more meaningful.
> Reporting will be available for both lxc isolation and process-based isolation. For lxc isolation the task is easier because of the isolation facilities of lxc. Process-based isolation is more difficult as processes can become re-parented from the process tree of the executor (e.g. double fork). The session ID and the process group ID will likely still be the same as that of the executor except for the uncommon case of the process resetting both of those.
> Initial reporting will be to a local log file. This will be a 'heartbeat' style akin to pidstat output (in sysstat library). This may not be incredibly useful, but local monitoring of resource usage is separate from the reporting and timeline building mentioned above.
> When usage statistics are eventually reported to the Mesos master, it may be possible to use them to oversubscribe slave nodes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MESOS-38) Executor resource monitoring and
local reporting of usage stats
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MESOS-38?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222991#comment-13222991 ]
jiraposter@reviews.apache.org commented on MESOS-38:
----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4167/
-----------------------------------------------------------
(Updated 2012-03-06 04:54:32.861529)
Review request for mesos, Benjamin Hindman and Charles Reiss.
Changes
-------
addressing some of the comments from charles
Summary
-------
This mega-patch is intended to represent the partial completion of the slave monitoring functionality. It is not intended to be committed. Changes based on comments in this review will be reflected in future reviews that are smaller and more modular.
Proc utils is included in this patch, but is already under review here: https://reviews.apache.org/r/3050/
The relevant design doc can be found here: https://docs.google.com/document/d/14Wj9i6TpMR6cV3LL0ySjLjfZOq5QOQeybvt1gSyaQGs/edit
The following items are ones where specific feedback is requested:
* A better mechanism is needed to control the rate at which the slave asks each executor for its UsageMessage. This is currently hard-coded to be at 1 second intervals, but could potentially be read as a command-line option or from a config file. Is there a better or different way to pass in this value?
* Currently, UsageMessages are passed from a ResourceMonitor to the Slave using the Future construct, and used as containers that hold a snapshot of the latest usage. This is to prevent unnecessary marshalling and extra data structures, since messages will eventually be sent in the standard dispatch style from the slave to the master. Is it fine that we are using Protobuf messages in this way?
There are several changes that are not yet implemented in this patch. These changes are as follows:
* Sufficient tests cases have not yet been written for any component (resource monitor, lxc collector, and process collector).
* Code has not been cleaned up to adhere to all style recommendations.
* Process collector code needs to be updated to prevent CPU usage spikes when monitored sub-processes die.
* Code to send UsageMessages from the slave to the master.
This addresses bug MESOS-38.
https://issues.apache.org/jira/browse/MESOS-38
Diffs (updated)
-----
src/Makefile.am 1137a3e
src/master/allocator.hpp 1ac435b
src/master/http.cpp 591433a
src/master/master.hpp 53551b0
src/master/master.cpp 1d3961e
src/messages/messages.proto 11a2c41
src/monitoring/linux/lxc_resource_collector.hpp PRE-CREATION
src/monitoring/linux/lxc_resource_collector.cpp PRE-CREATION
src/monitoring/linux/proc_resource_collector.hpp PRE-CREATION
src/monitoring/linux/proc_resource_collector.cpp PRE-CREATION
src/monitoring/linux/proc_utils.hpp PRE-CREATION
src/monitoring/linux/proc_utils.cpp PRE-CREATION
src/monitoring/process_resource_collector.hpp PRE-CREATION
src/monitoring/process_resource_collector.cpp PRE-CREATION
src/monitoring/process_stats.hpp PRE-CREATION
src/monitoring/resource_collector.hpp PRE-CREATION
src/slave/http.cpp f03815d
src/slave/isolation_module.hpp c896908
src/slave/isolation_module.cpp 5b7b4a2
src/slave/lxc_isolation_module.hpp b7beefe
src/slave/lxc_isolation_module.cpp d544625
src/slave/main.cpp ac780c4
src/slave/process_based_isolation_module.hpp f6f9554
src/slave/process_based_isolation_module.cpp 100b1e3
src/slave/resource_monitor.hpp PRE-CREATION
src/slave/resource_monitor.cpp PRE-CREATION
src/slave/slave.hpp b1a07e9
src/slave/slave.cpp ce8fda5
src/tests/Makefile.in 6f51be4
src/tests/proc_utils_tests.cpp PRE-CREATION
src/tests/process_resource_collector_tests.cpp PRE-CREATION
src/tests/resource_monitor_tests.cpp PRE-CREATION
Diff: https://reviews.apache.org/r/4167/diff
Testing
-------
Test cases:
* A test case exercising the basic monitoring code with a mocked-out collector.
* The first of several tests for the process resource monitor, with the proc-based collecting mocked out.
Some ad-hoc testing with log statements to ensure that the monitoring works end-to-end from both the container-based and process-based isolation modules.
Thanks,
Sam
> Executor resource monitoring and local reporting of usage stats
> ---------------------------------------------------------------
>
> Key: MESOS-38
> URL: https://issues.apache.org/jira/browse/MESOS-38
> Project: Mesos
> Issue Type: New Feature
> Components: isolation, slave
> Environment: Initial executor monitoring for linux only. Dummy monitoring capability (no-op) for OSX, with functionality to be filled in later.
> Reporter: Sam Whitlock
> Assignee: Sam Whitlock
> Labels: monitoring
>
> Implement reporting of resource usage on executors and log them to a local log file (for now). The eventual usage of this will be to report these statistics to the Mesos master in order to build either or both a timeline for the webui and/or a top-like command-line interface. This improvement ticket is just for the local monitoring and log file reporting. A reporting system (to the master node) will be a later improvement ticket.
> With the current version of Mesos, it is not possible to monitor individual tasks. Therefore the best this sort of system can do is monitor the usage of an individual executor and aggregate the resource usage of over the executor's tasks and resource allocations. If frameworks have a 1-to-1 relationship of a job to an executor, then the aggregate statistics will be more meaningful.
> Reporting will be available for both lxc isolation and process-based isolation. For lxc isolation the task is easier because of the isolation facilities of lxc. Process-based isolation is more difficult as processes can become re-parented from the process tree of the executor (e.g. double fork). The session ID and the process group ID will likely still be the same as that of the executor except for the uncommon case of the process resetting both of those.
> When usage statistics are eventually reported to the Mesos master, it may be possible to use them to oversubscribe slave nodes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MESOS-38) Executor resource monitoring and
local reporting of usage stats
Posted by "Sam Whitlock (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MESOS-38?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152236#comment-13152236 ]
Sam Whitlock commented on MESOS-38:
-----------------------------------
Apparently jira doesn't make these components link back to this one as being a submodule. Weird, but I linked them back to this issue.
> Executor resource monitoring and local reporting of usage stats
> ---------------------------------------------------------------
>
> Key: MESOS-38
> URL: https://issues.apache.org/jira/browse/MESOS-38
> Project: Mesos
> Issue Type: New Feature
> Components: isolation, slave
> Environment: Initial executor monitoring for linux only. Dummy monitoring capability (no-op) for OSX, with functionality to be filled in later.
> Reporter: Sam Whitlock
> Assignee: Sam Whitlock
> Labels: monitoring
>
> Implement reporting of resource usage on executors and log them to a local log file (for now). The eventual usage of this will be to report these statistics to the Mesos master in order to build either or both a timeline for the webui and/or a top-like command-line interface. This improvement ticket is just for the local monitoring and log file reporting. A reporting system (to the master node) will be a later improvement ticket.
> With the current version of Mesos, it is not possible to monitor individual tasks. Therefore the best this sort of system can do is monitor the usage of an individual executor and aggregate the resource usage of over the executor's tasks and resource allocations. If frameworks have a 1-to-1 relationship of a job to an executor, then the aggregate statistics will be more meaningful.
> Reporting will be available for both lxc isolation and process-based isolation. For lxc isolation the task is easier because of the isolation facilities of lxc. Process-based isolation is more difficult as processes can become re-parented from the process tree of the executor (e.g. double fork). The session ID and the process group ID will likely still be the same as that of the executor except for the uncommon case of the process resetting both of those.
> Initial reporting will be to a local log file. This will be a 'heartbeat' style akin to pidstat output (in sysstat library). This may not be incredibly useful, but local monitoring of resource usage is separate from the reporting and timeline building mentioned above.
> When usage statistics are eventually reported to the Mesos master, it may be possible to use them to oversubscribe slave nodes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MESOS-38) Executor resource monitoring and
local reporting of usage stats
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MESOS-38?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13221769#comment-13221769 ]
jiraposter@reviews.apache.org commented on MESOS-38:
----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4167/#review5603
-----------------------------------------------------------
There are a couple comments we forgot to add about the work that still needs to be done.
1. We have not yet done any benchmarking to evaluate the performance cost of this additional monitoring.
2. Another configuration option to consider is the ability to enable/disable monitoring. Perhaps a configured sample rate of 0 should disable monitoring?
- Alex
On 2012-03-04 01:15:55, Sam Whitlock wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4167/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-04 01:15:55)
bq.
bq.
bq. Review request for mesos, Benjamin Hindman and Charles Reiss.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. This mega-patch is intended to represent the partial completion of the slave monitoring functionality. It is not intended to be committed. Changes based on comments in this review will be reflected in future reviews that are smaller and more modular.
bq.
bq. Proc utils is included in this patch, but is already under review here: https://reviews.apache.org/r/3050/
bq.
bq. The relevant design doc can be found here: https://docs.google.com/document/d/14Wj9i6TpMR6cV3LL0ySjLjfZOq5QOQeybvt1gSyaQGs/edit
bq.
bq. The following items are ones where specific feedback is requested:
bq.
bq. * A better mechanism is needed to control the rate at which the slave asks each executor for its UsageMessage. This is currently hard-coded to be at 1 second intervals, but could potentially be read as a command-line option or from a config file. Is there a better or different way to pass in this value?
bq. * Currently, UsageMessages are passed from a ResourceMonitor to the Slave using the Future construct, and used as containers that hold a snapshot of the latest usage. This is to prevent unnecessary marshalling and extra data structures, since messages will eventually be sent in the standard dispatch style from the slave to the master. Is it fine that we are using Protobuf messages in this way?
bq.
bq. There are several changes that are not yet implemented in this patch. These changes are as follows:
bq.
bq. * Sufficient tests cases have not yet been written for any component (resource monitor, lxc collector, and process collector).
bq. * Code has not been cleaned up to adhere to all style recommendations.
bq. * Process collector code needs to be updated to prevent CPU usage spikes when monitored sub-processes die.
bq. * Code to send UsageMessages from the slave to the master.
bq.
bq.
bq. This addresses bug MESOS-38.
bq. https://issues.apache.org/jira/browse/MESOS-38
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/Makefile.am 1137a3e
bq. src/master/allocator.hpp 1ac435b
bq. src/master/http.cpp 591433a
bq. src/master/master.hpp 53551b0
bq. src/master/master.cpp 1d3961e
bq. src/messages/messages.proto 11a2c41
bq. src/monitoring/linux/lxc_resource_collector.hpp PRE-CREATION
bq. src/monitoring/linux/lxc_resource_collector.cpp PRE-CREATION
bq. src/monitoring/linux/proc_resource_collector.hpp PRE-CREATION
bq. src/monitoring/linux/proc_resource_collector.cpp PRE-CREATION
bq. src/monitoring/linux/proc_utils.hpp PRE-CREATION
bq. src/monitoring/linux/proc_utils.cpp PRE-CREATION
bq. src/monitoring/process_resource_collector.hpp PRE-CREATION
bq. src/monitoring/process_resource_collector.cpp PRE-CREATION
bq. src/monitoring/process_stats.hpp PRE-CREATION
bq. src/monitoring/resource_collector.hpp PRE-CREATION
bq. src/slave/http.cpp f03815d
bq. src/slave/isolation_module.hpp c896908
bq. src/slave/isolation_module.cpp 5b7b4a2
bq. src/slave/lxc_isolation_module.hpp b7beefe
bq. src/slave/lxc_isolation_module.cpp d544625
bq. src/slave/process_based_isolation_module.hpp f6f9554
bq. src/slave/process_based_isolation_module.cpp 100b1e3
bq. src/slave/resource_monitor.hpp PRE-CREATION
bq. src/slave/resource_monitor.cpp PRE-CREATION
bq. src/slave/slave.hpp b1a07e9
bq. src/slave/slave.cpp ce8fda5
bq. src/tests/Makefile.in 6f51be4
bq. src/tests/proc_utils_tests.cpp PRE-CREATION
bq. src/tests/process_resource_collector_tests.cpp PRE-CREATION
bq. src/tests/resource_monitor_tests.cpp PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/4167/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Test cases:
bq. * A test case exercising the basic monitoring code with a mocked-out collector.
bq. * The first of several tests for the process resource monitor, with the proc-based collecting mocked out.
bq.
bq. Some ad-hoc testing with log statements to ensure that the monitoring works end-to-end from both the container-based and process-based isolation modules.
bq.
bq.
bq. Thanks,
bq.
bq. Sam
bq.
bq.
> Executor resource monitoring and local reporting of usage stats
> ---------------------------------------------------------------
>
> Key: MESOS-38
> URL: https://issues.apache.org/jira/browse/MESOS-38
> Project: Mesos
> Issue Type: New Feature
> Components: isolation, slave
> Environment: Initial executor monitoring for linux only. Dummy monitoring capability (no-op) for OSX, with functionality to be filled in later.
> Reporter: Sam Whitlock
> Assignee: Sam Whitlock
> Labels: monitoring
>
> Implement reporting of resource usage on executors and log them to a local log file (for now). The eventual usage of this will be to report these statistics to the Mesos master in order to build either or both a timeline for the webui and/or a top-like command-line interface. This improvement ticket is just for the local monitoring and log file reporting. A reporting system (to the master node) will be a later improvement ticket.
> With the current version of Mesos, it is not possible to monitor individual tasks. Therefore the best this sort of system can do is monitor the usage of an individual executor and aggregate the resource usage of over the executor's tasks and resource allocations. If frameworks have a 1-to-1 relationship of a job to an executor, then the aggregate statistics will be more meaningful.
> Reporting will be available for both lxc isolation and process-based isolation. For lxc isolation the task is easier because of the isolation facilities of lxc. Process-based isolation is more difficult as processes can become re-parented from the process tree of the executor (e.g. double fork). The session ID and the process group ID will likely still be the same as that of the executor except for the uncommon case of the process resetting both of those.
> When usage statistics are eventually reported to the Mesos master, it may be possible to use them to oversubscribe slave nodes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MESOS-38) Executor resource monitoring and local
reporting of usage stats
Posted by "Sam Whitlock (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MESOS-38?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sam Whitlock updated MESOS-38:
------------------------------
Description:
Implement reporting of resource usage on executors and log them to a local log file (for now). The eventual usage of this will be to report these statistics to the Mesos master in order to build either or both a timeline for the webui and/or a top-like command-line interface. This improvement ticket is just for the local monitoring and log file reporting. A reporting system (to the master node) will be a later improvement ticket.
With the current version of Mesos, it is not possible to monitor individual tasks. Therefore the best this sort of system can do is monitor the usage of an individual executor and aggregate the resource usage of over the executor's tasks and resource allocations. If frameworks have a 1-to-1 relationship of a job to an executor, then the aggregate statistics will be more meaningful.
Reporting will be available for both lxc isolation and process-based isolation. For lxc isolation the task is easier because of the isolation facilities of lxc. Process-based isolation is more difficult as processes can become re-parented from the process tree of the executor (e.g. double fork). The session ID and the process group ID will likely still be the same as that of the executor except for the uncommon case of the process resetting both of those.
Initial reporting will be to a local log file. This will be a 'heartbeat' style akin to pidstat output (in sysstat library). This may not be incredibly useful, but local monitoring of resource usage is separate from the reporting and timeline building mentioned above.
was:
Implement reporting of resource usage on executors and log them to a local log file (for now). The eventual usage of this will be to report these statistics to the Mesos master in order to build either or both a timeline for the webui and/or a top-like command-line interface. This improvement ticket is just for the local monitoring and log file reporting. A reporting system (to the master node) will be a later improvement ticket.
With the current version of Mesos, it is not possible to monitor individual task. Therefore the best this sort of system can do is monitor the usage of an individual executor and aggregate the resource usage of over the executor's tasks and resource allocations.
Reporting will be available for both lxc isolation and process-based isolation. For lxc isolation the task is easier because of the isolation facilities of lxc. Process-based isolation is more difficult as processes can become re-parented from the process tree of the executor (e.g. double fork). The session ID and the process group ID will likely still be the same as that of the executor except for the uncommon case of the process resetting both of those.
Initial reporting will be to a local log file. This will be a 'heartbeat' style akin to pidstat output (in sysstat library). This may not be incredibly useful, but local monitoring of resource usage is separate from the reporting and timeline building mentioned above.
> Executor resource monitoring and local reporting of usage stats
> ---------------------------------------------------------------
>
> Key: MESOS-38
> URL: https://issues.apache.org/jira/browse/MESOS-38
> Project: Mesos
> Issue Type: New Feature
> Environment: Initial executor monitoring for linux only. Dummy monitoring capability (no-op) for OSX, with functionality to be filled in later.
> Reporter: Sam Whitlock
> Labels: monitoring
>
> Implement reporting of resource usage on executors and log them to a local log file (for now). The eventual usage of this will be to report these statistics to the Mesos master in order to build either or both a timeline for the webui and/or a top-like command-line interface. This improvement ticket is just for the local monitoring and log file reporting. A reporting system (to the master node) will be a later improvement ticket.
> With the current version of Mesos, it is not possible to monitor individual tasks. Therefore the best this sort of system can do is monitor the usage of an individual executor and aggregate the resource usage of over the executor's tasks and resource allocations. If frameworks have a 1-to-1 relationship of a job to an executor, then the aggregate statistics will be more meaningful.
> Reporting will be available for both lxc isolation and process-based isolation. For lxc isolation the task is easier because of the isolation facilities of lxc. Process-based isolation is more difficult as processes can become re-parented from the process tree of the executor (e.g. double fork). The session ID and the process group ID will likely still be the same as that of the executor except for the uncommon case of the process resetting both of those.
> Initial reporting will be to a local log file. This will be a 'heartbeat' style akin to pidstat output (in sysstat library). This may not be incredibly useful, but local monitoring of resource usage is separate from the reporting and timeline building mentioned above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MESOS-38) Executor resource monitoring and
local reporting of usage stats
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MESOS-38?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13224755#comment-13224755 ]
jiraposter@reviews.apache.org commented on MESOS-38:
----------------------------------------------------
bq. On 2012-03-05 23:04:17, Charles Reiss wrote:
bq. > src/slave/resource_monitor.cpp, line 67
bq. > <https://reviews.apache.org/r/4167/diff/2/?file=88033#file88033line67>
bq. >
bq. > I assume that you've discussed calling this 'mem_usage' and 'cpu_usage' rather than 'mem' and 'cpus' with Ben? Can you explain the reasoning briefly?
bq.
bq. Sam Whitlock wrote:
bq. I sorta made this choice unilaterally based on its similarity to the naming for cgroups infos.
bq.
bq. I don't really care what it is called and would certainly be willing to change it if it would make it more similar to the rest of mesos.
bq.
bq. Sam Whitlock wrote:
bq. I guess the choice to go with *_usage is because mem and cpus is used elsewhere in mesos for things that are not reported usage, and I wanted to draw a distinction.
bq.
bq. Charles Reiss wrote:
bq. I was assuming that usage measurements would always be kept in separate fields from requests. Is there a usecase where this doesn't look like this will be the case?
bq.
bq. Unless there's a reason to believe it would be deceptive, I'd prefer to name the usage measurements the same as the requested resource they are supposed to measure.
done
- Sam
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4167/#review5605
-----------------------------------------------------------
On 2012-03-07 21:53:30, Sam Whitlock wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4167/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-07 21:53:30)
bq.
bq.
bq. Review request for mesos, Benjamin Hindman and Charles Reiss.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. This mega-patch is intended to represent the partial completion of the slave monitoring functionality. It is not intended to be committed. Changes based on comments in this review will be reflected in future reviews that are smaller and more modular.
bq.
bq. Proc utils is included in this patch, but is already under review here: https://reviews.apache.org/r/3050/
bq.
bq. The relevant design doc can be found here: https://docs.google.com/document/d/14Wj9i6TpMR6cV3LL0ySjLjfZOq5QOQeybvt1gSyaQGs/edit
bq.
bq. The following items are ones where specific feedback is requested:
bq.
bq. * A better mechanism is needed to control the rate at which the slave asks each executor for its UsageMessage. This is currently hard-coded to be at 1 second intervals, but could potentially be read as a command-line option or from a config file. Is there a better or different way to pass in this value?
bq. * Currently, UsageMessages are passed from a ResourceMonitor to the Slave using the Future construct, and used as containers that hold a snapshot of the latest usage. This is to prevent unnecessary marshalling and extra data structures, since messages will eventually be sent in the standard dispatch style from the slave to the master. Is it fine that we are using Protobuf messages in this way?
bq.
bq. There are several changes that are not yet implemented in this patch. These changes are as follows:
bq.
bq. * Sufficient tests cases have not yet been written for any component (resource monitor, lxc collector, and process collector).
bq. * Code has not been cleaned up to adhere to all style recommendations.
bq. * Process collector code needs to be updated to prevent CPU usage spikes when monitored sub-processes die.
bq. * Code to send UsageMessages from the slave to the master.
bq.
bq.
bq. This addresses bug MESOS-38.
bq. https://issues.apache.org/jira/browse/MESOS-38
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/Makefile.am 1137a3e
bq. src/master/allocator.hpp 1ac435b
bq. src/master/http.cpp 591433a
bq. src/master/master.hpp 53551b0
bq. src/master/master.cpp 1d3961e
bq. src/messages/messages.proto 11a2c41
bq. src/monitoring/linux/lxc_resource_collector.hpp PRE-CREATION
bq. src/monitoring/linux/lxc_resource_collector.cpp PRE-CREATION
bq. src/monitoring/linux/proc_resource_collector.hpp PRE-CREATION
bq. src/monitoring/linux/proc_resource_collector.cpp PRE-CREATION
bq. src/monitoring/linux/proc_utils.hpp PRE-CREATION
bq. src/monitoring/linux/proc_utils.cpp PRE-CREATION
bq. src/monitoring/process_resource_collector.hpp PRE-CREATION
bq. src/monitoring/process_resource_collector.cpp PRE-CREATION
bq. src/monitoring/process_stats.hpp PRE-CREATION
bq. src/monitoring/resource_collector.hpp PRE-CREATION
bq. src/slave/http.cpp f03815d
bq. src/slave/isolation_module.hpp c896908
bq. src/slave/isolation_module.cpp 5b7b4a2
bq. src/slave/lxc_isolation_module.hpp b7beefe
bq. src/slave/lxc_isolation_module.cpp d544625
bq. src/slave/main.cpp ac780c4
bq. src/slave/process_based_isolation_module.hpp f6f9554
bq. src/slave/process_based_isolation_module.cpp 100b1e3
bq. src/slave/resource_monitor.hpp PRE-CREATION
bq. src/slave/resource_monitor.cpp PRE-CREATION
bq. src/slave/slave.hpp b1a07e9
bq. src/slave/slave.cpp ce8fda5
bq. src/tests/Makefile.in 6f51be4
bq. src/tests/proc_utils_tests.cpp PRE-CREATION
bq. src/tests/process_resource_collector_tests.cpp PRE-CREATION
bq. src/tests/resource_monitor_tests.cpp PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/4167/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Test cases:
bq. * A test case exercising the basic monitoring code with a mocked-out collector.
bq. * The first of several tests for the process resource monitor, with the proc-based collecting mocked out.
bq.
bq. Some ad-hoc testing with log statements to ensure that the monitoring works end-to-end from both the container-based and process-based isolation modules.
bq.
bq.
bq. Thanks,
bq.
bq. Sam
bq.
bq.
> Executor resource monitoring and local reporting of usage stats
> ---------------------------------------------------------------
>
> Key: MESOS-38
> URL: https://issues.apache.org/jira/browse/MESOS-38
> Project: Mesos
> Issue Type: New Feature
> Components: isolation, slave
> Environment: Initial executor monitoring for linux only. Dummy monitoring capability (no-op) for OSX, with functionality to be filled in later.
> Reporter: Sam Whitlock
> Assignee: Sam Whitlock
> Labels: monitoring
>
> Implement reporting of resource usage on executors and log them to a local log file (for now). The eventual usage of this will be to report these statistics to the Mesos master in order to build either or both a timeline for the webui and/or a top-like command-line interface. This improvement ticket is just for the local monitoring and log file reporting. A reporting system (to the master node) will be a later improvement ticket.
> With the current version of Mesos, it is not possible to monitor individual tasks. Therefore the best this sort of system can do is monitor the usage of an individual executor and aggregate the resource usage of over the executor's tasks and resource allocations. If frameworks have a 1-to-1 relationship of a job to an executor, then the aggregate statistics will be more meaningful.
> Reporting will be available for both lxc isolation and process-based isolation. For lxc isolation the task is easier because of the isolation facilities of lxc. Process-based isolation is more difficult as processes can become re-parented from the process tree of the executor (e.g. double fork). The session ID and the process group ID will likely still be the same as that of the executor except for the uncommon case of the process resetting both of those.
> When usage statistics are eventually reported to the Mesos master, it may be possible to use them to oversubscribe slave nodes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MESOS-38) Executor resource monitoring and local
reporting of usage stats
Posted by "Sam Whitlock (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MESOS-38?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sam Whitlock updated MESOS-38:
------------------------------
Description:
Implement reporting of resource usage on executors and log them to a local log file (for now). The eventual usage of this will be to report these statistics to the Mesos master in order to build either or both a timeline for the webui and/or a top-like command-line interface. This improvement ticket is just for the local monitoring and log file reporting. A reporting system (to the master node) will be a later improvement ticket.
With the current version of Mesos, it is not possible to monitor individual task. Therefore the best this sort of system can do is monitor the usage of an individual executor and aggregate the resource usage of over the executor's tasks and resource allocations.
Reporting will be available for both lxc isolation and process-based isolation. For lxc isolation the task is easier because of the isolation facilities of lxc. Process-based isolation is more difficult as processes can become re-parented from the process tree of the executor (e.g. double fork). The session ID and the process group ID will likely still be the same as that of the executor except for the uncommon case of the process resetting both of those.
Initial reporting will be to a local log file. This will be a 'heartbeat' style akin to pidstat output (in sysstat library). This may not be incredibly useful, but local monitoring of resource usage is separate from the reporting and timeline building mentioned above.
was:
Implement reporting of resource usage usage on executors and log them to a local log file (for now). The eventual usage of this will be to report these statistics to the Mesos master in order to build either or both a timeline for the webui and/or a top-like command-line interface. This improvement ticket is just for the local monitoring and log file reporting. A reporting system (to the master node) will be a later improvement ticket.
With the current version of Mesos, it is not possible to monitor individual task. Therefore the best this sort of system can do is monitor the usage of an individual executor and aggregate the resource usage of over the executor's tasks and resource allocations.
Reporting will be available for both lxc isolation and process-based isolation. For lxc isolation the task is easier because of the isolation facilities of lxc. Process-based isolation is more difficult as processes can become re-parented from the process tree of the executor (e.g. double fork). The session ID and the process group ID will likely still be the same as that of the executor except for the uncommon case of the process resetting both of those.
Initial reporting will be to a local log file. This will be a 'heartbeat' style akin to pidstat output (in sysstat library). This may not be incredibly useful, but local monitoring of resource usage is separate from the reporting and timeline building mentioned above.
> Executor resource monitoring and local reporting of usage stats
> ---------------------------------------------------------------
>
> Key: MESOS-38
> URL: https://issues.apache.org/jira/browse/MESOS-38
> Project: Mesos
> Issue Type: New Feature
> Environment: Initial executor monitoring for linux only. Dummy monitoring capability (no-op) for OSX, with functionality to be filled in later.
> Reporter: Sam Whitlock
> Labels: monitoring
>
> Implement reporting of resource usage on executors and log them to a local log file (for now). The eventual usage of this will be to report these statistics to the Mesos master in order to build either or both a timeline for the webui and/or a top-like command-line interface. This improvement ticket is just for the local monitoring and log file reporting. A reporting system (to the master node) will be a later improvement ticket.
> With the current version of Mesos, it is not possible to monitor individual task. Therefore the best this sort of system can do is monitor the usage of an individual executor and aggregate the resource usage of over the executor's tasks and resource allocations.
> Reporting will be available for both lxc isolation and process-based isolation. For lxc isolation the task is easier because of the isolation facilities of lxc. Process-based isolation is more difficult as processes can become re-parented from the process tree of the executor (e.g. double fork). The session ID and the process group ID will likely still be the same as that of the executor except for the uncommon case of the process resetting both of those.
> Initial reporting will be to a local log file. This will be a 'heartbeat' style akin to pidstat output (in sysstat library). This may not be incredibly useful, but local monitoring of resource usage is separate from the reporting and timeline building mentioned above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MESOS-38) Executor resource monitoring and
local reporting of usage stats
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MESOS-38?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222993#comment-13222993 ]
jiraposter@reviews.apache.org commented on MESOS-38:
----------------------------------------------------
bq. On 2012-03-05 23:04:17, Charles Reiss wrote:
bq. > src/slave/resource_monitor.cpp, line 67
bq. > <https://reviews.apache.org/r/4167/diff/2/?file=88033#file88033line67>
bq. >
bq. > I assume that you've discussed calling this 'mem_usage' and 'cpu_usage' rather than 'mem' and 'cpus' with Ben? Can you explain the reasoning briefly?
bq.
bq. Sam Whitlock wrote:
bq. I sorta made this choice unilaterally based on its similarity to the naming for cgroups infos.
bq.
bq. I don't really care what it is called and would certainly be willing to change it if it would make it more similar to the rest of mesos.
I guess the choice to go with *_usage is because mem and cpus is used elsewhere in mesos for things that are not reported usage, and I wanted to draw a distinction.
- Sam
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4167/#review5605
-----------------------------------------------------------
On 2012-03-06 04:54:32, Sam Whitlock wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4167/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-06 04:54:32)
bq.
bq.
bq. Review request for mesos, Benjamin Hindman and Charles Reiss.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. This mega-patch is intended to represent the partial completion of the slave monitoring functionality. It is not intended to be committed. Changes based on comments in this review will be reflected in future reviews that are smaller and more modular.
bq.
bq. Proc utils is included in this patch, but is already under review here: https://reviews.apache.org/r/3050/
bq.
bq. The relevant design doc can be found here: https://docs.google.com/document/d/14Wj9i6TpMR6cV3LL0ySjLjfZOq5QOQeybvt1gSyaQGs/edit
bq.
bq. The following items are ones where specific feedback is requested:
bq.
bq. * A better mechanism is needed to control the rate at which the slave asks each executor for its UsageMessage. This is currently hard-coded to be at 1 second intervals, but could potentially be read as a command-line option or from a config file. Is there a better or different way to pass in this value?
bq. * Currently, UsageMessages are passed from a ResourceMonitor to the Slave using the Future construct, and used as containers that hold a snapshot of the latest usage. This is to prevent unnecessary marshalling and extra data structures, since messages will eventually be sent in the standard dispatch style from the slave to the master. Is it fine that we are using Protobuf messages in this way?
bq.
bq. There are several changes that are not yet implemented in this patch. These changes are as follows:
bq.
bq. * Sufficient tests cases have not yet been written for any component (resource monitor, lxc collector, and process collector).
bq. * Code has not been cleaned up to adhere to all style recommendations.
bq. * Process collector code needs to be updated to prevent CPU usage spikes when monitored sub-processes die.
bq. * Code to send UsageMessages from the slave to the master.
bq.
bq.
bq. This addresses bug MESOS-38.
bq. https://issues.apache.org/jira/browse/MESOS-38
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/Makefile.am 1137a3e
bq. src/master/allocator.hpp 1ac435b
bq. src/master/http.cpp 591433a
bq. src/master/master.hpp 53551b0
bq. src/master/master.cpp 1d3961e
bq. src/messages/messages.proto 11a2c41
bq. src/monitoring/linux/lxc_resource_collector.hpp PRE-CREATION
bq. src/monitoring/linux/lxc_resource_collector.cpp PRE-CREATION
bq. src/monitoring/linux/proc_resource_collector.hpp PRE-CREATION
bq. src/monitoring/linux/proc_resource_collector.cpp PRE-CREATION
bq. src/monitoring/linux/proc_utils.hpp PRE-CREATION
bq. src/monitoring/linux/proc_utils.cpp PRE-CREATION
bq. src/monitoring/process_resource_collector.hpp PRE-CREATION
bq. src/monitoring/process_resource_collector.cpp PRE-CREATION
bq. src/monitoring/process_stats.hpp PRE-CREATION
bq. src/monitoring/resource_collector.hpp PRE-CREATION
bq. src/slave/http.cpp f03815d
bq. src/slave/isolation_module.hpp c896908
bq. src/slave/isolation_module.cpp 5b7b4a2
bq. src/slave/lxc_isolation_module.hpp b7beefe
bq. src/slave/lxc_isolation_module.cpp d544625
bq. src/slave/main.cpp ac780c4
bq. src/slave/process_based_isolation_module.hpp f6f9554
bq. src/slave/process_based_isolation_module.cpp 100b1e3
bq. src/slave/resource_monitor.hpp PRE-CREATION
bq. src/slave/resource_monitor.cpp PRE-CREATION
bq. src/slave/slave.hpp b1a07e9
bq. src/slave/slave.cpp ce8fda5
bq. src/tests/Makefile.in 6f51be4
bq. src/tests/proc_utils_tests.cpp PRE-CREATION
bq. src/tests/process_resource_collector_tests.cpp PRE-CREATION
bq. src/tests/resource_monitor_tests.cpp PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/4167/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Test cases:
bq. * A test case exercising the basic monitoring code with a mocked-out collector.
bq. * The first of several tests for the process resource monitor, with the proc-based collecting mocked out.
bq.
bq. Some ad-hoc testing with log statements to ensure that the monitoring works end-to-end from both the container-based and process-based isolation modules.
bq.
bq.
bq. Thanks,
bq.
bq. Sam
bq.
bq.
> Executor resource monitoring and local reporting of usage stats
> ---------------------------------------------------------------
>
> Key: MESOS-38
> URL: https://issues.apache.org/jira/browse/MESOS-38
> Project: Mesos
> Issue Type: New Feature
> Components: isolation, slave
> Environment: Initial executor monitoring for linux only. Dummy monitoring capability (no-op) for OSX, with functionality to be filled in later.
> Reporter: Sam Whitlock
> Assignee: Sam Whitlock
> Labels: monitoring
>
> Implement reporting of resource usage on executors and log them to a local log file (for now). The eventual usage of this will be to report these statistics to the Mesos master in order to build either or both a timeline for the webui and/or a top-like command-line interface. This improvement ticket is just for the local monitoring and log file reporting. A reporting system (to the master node) will be a later improvement ticket.
> With the current version of Mesos, it is not possible to monitor individual tasks. Therefore the best this sort of system can do is monitor the usage of an individual executor and aggregate the resource usage of over the executor's tasks and resource allocations. If frameworks have a 1-to-1 relationship of a job to an executor, then the aggregate statistics will be more meaningful.
> Reporting will be available for both lxc isolation and process-based isolation. For lxc isolation the task is easier because of the isolation facilities of lxc. Process-based isolation is more difficult as processes can become re-parented from the process tree of the executor (e.g. double fork). The session ID and the process group ID will likely still be the same as that of the executor except for the uncommon case of the process resetting both of those.
> When usage statistics are eventually reported to the Mesos master, it may be possible to use them to oversubscribe slave nodes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MESOS-38) Executor resource monitoring and
local reporting of usage stats
Posted by "Sam Whitlock (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MESOS-38?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127874#comment-13127874 ]
Sam Whitlock commented on MESOS-38:
-----------------------------------
The way this would probably work is there would be a reporting class(es) associated with the executor. Each of these would be specific to the isolation module.
lxc would doesn't have to a pluggable reporting module, but it might as well so long it make it too abstract.
The process-based isolation will need a pluggable reporting module because the syscall interface differences between OSX and Linux. This would enable the OSX reporting to be implemented at a later date.
> Executor resource monitoring and local reporting of usage stats
> ---------------------------------------------------------------
>
> Key: MESOS-38
> URL: https://issues.apache.org/jira/browse/MESOS-38
> Project: Mesos
> Issue Type: New Feature
> Environment: Initial executor monitoring for linux only. Dummy monitoring capability (no-op) for OSX, with functionality to be filled in later.
> Reporter: Sam Whitlock
> Labels: monitoring
>
> Implement reporting of resource usage on executors and log them to a local log file (for now). The eventual usage of this will be to report these statistics to the Mesos master in order to build either or both a timeline for the webui and/or a top-like command-line interface. This improvement ticket is just for the local monitoring and log file reporting. A reporting system (to the master node) will be a later improvement ticket.
> With the current version of Mesos, it is not possible to monitor individual tasks. Therefore the best this sort of system can do is monitor the usage of an individual executor and aggregate the resource usage of over the executor's tasks and resource allocations. If frameworks have a 1-to-1 relationship of a job to an executor, then the aggregate statistics will be more meaningful.
> Reporting will be available for both lxc isolation and process-based isolation. For lxc isolation the task is easier because of the isolation facilities of lxc. Process-based isolation is more difficult as processes can become re-parented from the process tree of the executor (e.g. double fork). The session ID and the process group ID will likely still be the same as that of the executor except for the uncommon case of the process resetting both of those.
> Initial reporting will be to a local log file. This will be a 'heartbeat' style akin to pidstat output (in sysstat library). This may not be incredibly useful, but local monitoring of resource usage is separate from the reporting and timeline building mentioned above.
> When usage statistics are eventually reported to the Mesos master, it may be possible to use them to oversubscribe slave nodes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MESOS-38) Executor resource monitoring and
local reporting of usage stats
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MESOS-38?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13221761#comment-13221761 ]
jiraposter@reviews.apache.org commented on MESOS-38:
----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4167/
-----------------------------------------------------------
(Updated 2012-03-04 01:15:55.499253)
Review request for mesos, Benjamin Hindman and Charles Reiss.
Changes
-------
forgot to link relevant jira
Summary
-------
This mega-patch is intended to represent the partial completion of the slave monitoring functionality. It is not intended to be committed. Changes based on comments in this review will be reflected in future reviews that are smaller and more modular.
Proc utils is included in this patch, but is already under review here: https://reviews.apache.org/r/3050/
The relevant design doc can be found here: https://docs.google.com/document/d/14Wj9i6TpMR6cV3LL0ySjLjfZOq5QOQeybvt1gSyaQGs/edit
The following items are ones where specific feedback is requested:
* A better mechanism is needed to control the rate at which the slave asks each executor for its UsageMessage. This is currently hard-coded to be at 1 second intervals, but could potentially be read as a command-line option or from a config file. Is there a better or different way to pass in this value?
* Currently, UsageMessages are passed from a ResourceMonitor to the Slave using the Future construct, and used as containers that hold a snapshot of the latest usage. This is to prevent unnecessary marshalling and extra data structures, since messages will eventually be sent in the standard dispatch style from the slave to the master. Is it fine that we are using Protobuf messages in this way?
There are several changes that are not yet implemented in this patch. These changes are as follows:
* Sufficient tests cases have not yet been written for any component (resource monitor, lxc collector, and process collector).
* Code has not been cleaned up to adhere to all style recommendations.
* Process collector code needs to be updated to prevent CPU usage spikes when monitored sub-processes die.
* Code to send UsageMessages from the slave to the master.
This addresses bug MESOS-38.
https://issues.apache.org/jira/browse/MESOS-38
Diffs
-----
src/Makefile.am 1137a3e
src/master/allocator.hpp 1ac435b
src/master/http.cpp 591433a
src/master/master.hpp 53551b0
src/master/master.cpp 1d3961e
src/messages/messages.proto 11a2c41
src/monitoring/linux/lxc_resource_collector.hpp PRE-CREATION
src/monitoring/linux/lxc_resource_collector.cpp PRE-CREATION
src/monitoring/linux/proc_resource_collector.hpp PRE-CREATION
src/monitoring/linux/proc_resource_collector.cpp PRE-CREATION
src/monitoring/linux/proc_utils.hpp PRE-CREATION
src/monitoring/linux/proc_utils.cpp PRE-CREATION
src/monitoring/process_resource_collector.hpp PRE-CREATION
src/monitoring/process_resource_collector.cpp PRE-CREATION
src/monitoring/process_stats.hpp PRE-CREATION
src/monitoring/resource_collector.hpp PRE-CREATION
src/slave/http.cpp f03815d
src/slave/isolation_module.hpp c896908
src/slave/isolation_module.cpp 5b7b4a2
src/slave/lxc_isolation_module.hpp b7beefe
src/slave/lxc_isolation_module.cpp d544625
src/slave/process_based_isolation_module.hpp f6f9554
src/slave/process_based_isolation_module.cpp 100b1e3
src/slave/resource_monitor.hpp PRE-CREATION
src/slave/resource_monitor.cpp PRE-CREATION
src/slave/slave.hpp b1a07e9
src/slave/slave.cpp ce8fda5
src/tests/Makefile.in 6f51be4
src/tests/proc_utils_tests.cpp PRE-CREATION
src/tests/process_resource_collector_tests.cpp PRE-CREATION
src/tests/resource_monitor_tests.cpp PRE-CREATION
Diff: https://reviews.apache.org/r/4167/diff
Testing
-------
Test cases:
* A test case exercising the basic monitoring code with a mocked-out collector.
* The first of several tests for the process resource monitor, with the proc-based collecting mocked out.
Some ad-hoc testing with log statements to ensure that the monitoring works end-to-end from both the container-based and process-based isolation modules.
Thanks,
Sam
> Executor resource monitoring and local reporting of usage stats
> ---------------------------------------------------------------
>
> Key: MESOS-38
> URL: https://issues.apache.org/jira/browse/MESOS-38
> Project: Mesos
> Issue Type: New Feature
> Components: isolation, slave
> Environment: Initial executor monitoring for linux only. Dummy monitoring capability (no-op) for OSX, with functionality to be filled in later.
> Reporter: Sam Whitlock
> Assignee: Sam Whitlock
> Labels: monitoring
>
> Implement reporting of resource usage on executors and log them to a local log file (for now). The eventual usage of this will be to report these statistics to the Mesos master in order to build either or both a timeline for the webui and/or a top-like command-line interface. This improvement ticket is just for the local monitoring and log file reporting. A reporting system (to the master node) will be a later improvement ticket.
> With the current version of Mesos, it is not possible to monitor individual tasks. Therefore the best this sort of system can do is monitor the usage of an individual executor and aggregate the resource usage of over the executor's tasks and resource allocations. If frameworks have a 1-to-1 relationship of a job to an executor, then the aggregate statistics will be more meaningful.
> Reporting will be available for both lxc isolation and process-based isolation. For lxc isolation the task is easier because of the isolation facilities of lxc. Process-based isolation is more difficult as processes can become re-parented from the process tree of the executor (e.g. double fork). The session ID and the process group ID will likely still be the same as that of the executor except for the uncommon case of the process resetting both of those.
> When usage statistics are eventually reported to the Mesos master, it may be possible to use them to oversubscribe slave nodes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MESOS-38) Executor resource monitoring and
local reporting of usage stats
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MESOS-38?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222992#comment-13222992 ]
jiraposter@reviews.apache.org commented on MESOS-38:
----------------------------------------------------
bq. On 2012-03-05 23:04:17, Charles Reiss wrote:
bq. > src/slave/resource_monitor.cpp, line 67
bq. > <https://reviews.apache.org/r/4167/diff/2/?file=88033#file88033line67>
bq. >
bq. > I assume that you've discussed calling this 'mem_usage' and 'cpu_usage' rather than 'mem' and 'cpus' with Ben? Can you explain the reasoning briefly?
I sorta made this choice unilaterally based on its similarity to the naming for cgroups infos.
I don't really care what it is called and would certainly be willing to change it if it would make it more similar to the rest of mesos.
bq. On 2012-03-05 23:04:17, Charles Reiss wrote:
bq. > src/slave/slave.cpp, line 1446
bq. > <https://reviews.apache.org/r/4167/diff/2/?file=88035#file88035line1446>
bq. >
bq. > How is this deleted? Does this actually need to be on the heap?
no. before collect, I was doing things on the heap, but now I am using copying instead. thanks for catching this memory leak!
bq. On 2012-03-05 23:04:17, Charles Reiss wrote:
bq. > src/slave/slave.cpp, line 1448
bq. > <https://reviews.apache.org/r/4167/diff/2/?file=88035#file88035line1448>
bq. >
bq. > Suggest foreachpair
done
bq. On 2012-03-05 23:04:17, Charles Reiss wrote:
bq. > src/slave/slave.cpp, line 1469
bq. > <https://reviews.apache.org/r/4167/diff/2/?file=88035#file88035line1469>
bq. >
bq. > const UsageMessage&
done
bq. On 2012-03-05 23:04:17, Charles Reiss wrote:
bq. > src/slave/http.cpp, line 76
bq. > <https://reviews.apache.org/r/4167/diff/2/?file=88024#file88024line76>
bq. >
bq. > Why would we bother adding this before it does anything useful?
This was the start of displaying the usage through the webui.
I just bundled this in because it is a work-in-progress review (not for committing). For a future review that is more committable, this would not be part of it.
Only stuff in src/monitoring and the non-webui-related changes in src/slave would be part of the eventual committable review.
- Sam
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4167/#review5605
-----------------------------------------------------------
On 2012-03-06 04:54:32, Sam Whitlock wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4167/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-06 04:54:32)
bq.
bq.
bq. Review request for mesos, Benjamin Hindman and Charles Reiss.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. This mega-patch is intended to represent the partial completion of the slave monitoring functionality. It is not intended to be committed. Changes based on comments in this review will be reflected in future reviews that are smaller and more modular.
bq.
bq. Proc utils is included in this patch, but is already under review here: https://reviews.apache.org/r/3050/
bq.
bq. The relevant design doc can be found here: https://docs.google.com/document/d/14Wj9i6TpMR6cV3LL0ySjLjfZOq5QOQeybvt1gSyaQGs/edit
bq.
bq. The following items are ones where specific feedback is requested:
bq.
bq. * A better mechanism is needed to control the rate at which the slave asks each executor for its UsageMessage. This is currently hard-coded to be at 1 second intervals, but could potentially be read as a command-line option or from a config file. Is there a better or different way to pass in this value?
bq. * Currently, UsageMessages are passed from a ResourceMonitor to the Slave using the Future construct, and used as containers that hold a snapshot of the latest usage. This is to prevent unnecessary marshalling and extra data structures, since messages will eventually be sent in the standard dispatch style from the slave to the master. Is it fine that we are using Protobuf messages in this way?
bq.
bq. There are several changes that are not yet implemented in this patch. These changes are as follows:
bq.
bq. * Sufficient tests cases have not yet been written for any component (resource monitor, lxc collector, and process collector).
bq. * Code has not been cleaned up to adhere to all style recommendations.
bq. * Process collector code needs to be updated to prevent CPU usage spikes when monitored sub-processes die.
bq. * Code to send UsageMessages from the slave to the master.
bq.
bq.
bq. This addresses bug MESOS-38.
bq. https://issues.apache.org/jira/browse/MESOS-38
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/Makefile.am 1137a3e
bq. src/master/allocator.hpp 1ac435b
bq. src/master/http.cpp 591433a
bq. src/master/master.hpp 53551b0
bq. src/master/master.cpp 1d3961e
bq. src/messages/messages.proto 11a2c41
bq. src/monitoring/linux/lxc_resource_collector.hpp PRE-CREATION
bq. src/monitoring/linux/lxc_resource_collector.cpp PRE-CREATION
bq. src/monitoring/linux/proc_resource_collector.hpp PRE-CREATION
bq. src/monitoring/linux/proc_resource_collector.cpp PRE-CREATION
bq. src/monitoring/linux/proc_utils.hpp PRE-CREATION
bq. src/monitoring/linux/proc_utils.cpp PRE-CREATION
bq. src/monitoring/process_resource_collector.hpp PRE-CREATION
bq. src/monitoring/process_resource_collector.cpp PRE-CREATION
bq. src/monitoring/process_stats.hpp PRE-CREATION
bq. src/monitoring/resource_collector.hpp PRE-CREATION
bq. src/slave/http.cpp f03815d
bq. src/slave/isolation_module.hpp c896908
bq. src/slave/isolation_module.cpp 5b7b4a2
bq. src/slave/lxc_isolation_module.hpp b7beefe
bq. src/slave/lxc_isolation_module.cpp d544625
bq. src/slave/main.cpp ac780c4
bq. src/slave/process_based_isolation_module.hpp f6f9554
bq. src/slave/process_based_isolation_module.cpp 100b1e3
bq. src/slave/resource_monitor.hpp PRE-CREATION
bq. src/slave/resource_monitor.cpp PRE-CREATION
bq. src/slave/slave.hpp b1a07e9
bq. src/slave/slave.cpp ce8fda5
bq. src/tests/Makefile.in 6f51be4
bq. src/tests/proc_utils_tests.cpp PRE-CREATION
bq. src/tests/process_resource_collector_tests.cpp PRE-CREATION
bq. src/tests/resource_monitor_tests.cpp PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/4167/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Test cases:
bq. * A test case exercising the basic monitoring code with a mocked-out collector.
bq. * The first of several tests for the process resource monitor, with the proc-based collecting mocked out.
bq.
bq. Some ad-hoc testing with log statements to ensure that the monitoring works end-to-end from both the container-based and process-based isolation modules.
bq.
bq.
bq. Thanks,
bq.
bq. Sam
bq.
bq.
> Executor resource monitoring and local reporting of usage stats
> ---------------------------------------------------------------
>
> Key: MESOS-38
> URL: https://issues.apache.org/jira/browse/MESOS-38
> Project: Mesos
> Issue Type: New Feature
> Components: isolation, slave
> Environment: Initial executor monitoring for linux only. Dummy monitoring capability (no-op) for OSX, with functionality to be filled in later.
> Reporter: Sam Whitlock
> Assignee: Sam Whitlock
> Labels: monitoring
>
> Implement reporting of resource usage on executors and log them to a local log file (for now). The eventual usage of this will be to report these statistics to the Mesos master in order to build either or both a timeline for the webui and/or a top-like command-line interface. This improvement ticket is just for the local monitoring and log file reporting. A reporting system (to the master node) will be a later improvement ticket.
> With the current version of Mesos, it is not possible to monitor individual tasks. Therefore the best this sort of system can do is monitor the usage of an individual executor and aggregate the resource usage of over the executor's tasks and resource allocations. If frameworks have a 1-to-1 relationship of a job to an executor, then the aggregate statistics will be more meaningful.
> Reporting will be available for both lxc isolation and process-based isolation. For lxc isolation the task is easier because of the isolation facilities of lxc. Process-based isolation is more difficult as processes can become re-parented from the process tree of the executor (e.g. double fork). The session ID and the process group ID will likely still be the same as that of the executor except for the uncommon case of the process resetting both of those.
> When usage statistics are eventually reported to the Mesos master, it may be possible to use them to oversubscribe slave nodes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MESOS-38) Executor resource monitoring and
local reporting of usage stats
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MESOS-38?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222733#comment-13222733 ]
jiraposter@reviews.apache.org commented on MESOS-38:
----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4167/#review5605
-----------------------------------------------------------
src/slave/http.cpp
<https://reviews.apache.org/r/4167/#comment12223>
Why would we bother adding this before it does anything useful?
src/slave/process_based_isolation_module.cpp
<https://reviews.apache.org/r/4167/#comment12241>
Make sure this entry exists and fail gracefully if it doesn't. (It's probably possible for an executor to die between the time usage is requested and when the ProcessInfo is deleted.)
src/slave/resource_monitor.cpp
<https://reviews.apache.org/r/4167/#comment12242>
I assume that you've discussed calling this 'mem_usage' and 'cpu_usage' rather than 'mem' and 'cpus' with Ben? Can you explain the reasoning briefly?
src/slave/slave.cpp
<https://reviews.apache.org/r/4167/#comment12243>
How is this deleted? Does this actually need to be on the heap?
src/slave/slave.cpp
<https://reviews.apache.org/r/4167/#comment12244>
Suggest foreachpair
src/slave/slave.cpp
<https://reviews.apache.org/r/4167/#comment12245>
const UsageMessage&
- Charles
On 2012-03-04 20:40:08, Sam Whitlock wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4167/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-04 20:40:08)
bq.
bq.
bq. Review request for mesos, Benjamin Hindman and Charles Reiss.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. This mega-patch is intended to represent the partial completion of the slave monitoring functionality. It is not intended to be committed. Changes based on comments in this review will be reflected in future reviews that are smaller and more modular.
bq.
bq. Proc utils is included in this patch, but is already under review here: https://reviews.apache.org/r/3050/
bq.
bq. The relevant design doc can be found here: https://docs.google.com/document/d/14Wj9i6TpMR6cV3LL0ySjLjfZOq5QOQeybvt1gSyaQGs/edit
bq.
bq. The following items are ones where specific feedback is requested:
bq.
bq. * A better mechanism is needed to control the rate at which the slave asks each executor for its UsageMessage. This is currently hard-coded to be at 1 second intervals, but could potentially be read as a command-line option or from a config file. Is there a better or different way to pass in this value?
bq. * Currently, UsageMessages are passed from a ResourceMonitor to the Slave using the Future construct, and used as containers that hold a snapshot of the latest usage. This is to prevent unnecessary marshalling and extra data structures, since messages will eventually be sent in the standard dispatch style from the slave to the master. Is it fine that we are using Protobuf messages in this way?
bq.
bq. There are several changes that are not yet implemented in this patch. These changes are as follows:
bq.
bq. * Sufficient tests cases have not yet been written for any component (resource monitor, lxc collector, and process collector).
bq. * Code has not been cleaned up to adhere to all style recommendations.
bq. * Process collector code needs to be updated to prevent CPU usage spikes when monitored sub-processes die.
bq. * Code to send UsageMessages from the slave to the master.
bq.
bq.
bq. This addresses bug MESOS-38.
bq. https://issues.apache.org/jira/browse/MESOS-38
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/monitoring/process_stats.hpp PRE-CREATION
bq. src/monitoring/resource_collector.hpp PRE-CREATION
bq. src/slave/http.cpp f03815d
bq. src/slave/isolation_module.hpp c896908
bq. src/slave/isolation_module.cpp 5b7b4a2
bq. src/slave/lxc_isolation_module.hpp b7beefe
bq. src/slave/lxc_isolation_module.cpp d544625
bq. src/slave/main.cpp ac780c4
bq. src/slave/process_based_isolation_module.hpp f6f9554
bq. src/slave/process_based_isolation_module.cpp 100b1e3
bq. src/slave/resource_monitor.hpp PRE-CREATION
bq. src/slave/resource_monitor.cpp PRE-CREATION
bq. src/slave/slave.hpp b1a07e9
bq. src/slave/slave.cpp ce8fda5
bq. src/tests/Makefile.in 6f51be4
bq. src/tests/proc_utils_tests.cpp PRE-CREATION
bq. src/tests/process_resource_collector_tests.cpp PRE-CREATION
bq. src/tests/resource_monitor_tests.cpp PRE-CREATION
bq. src/monitoring/process_resource_collector.cpp PRE-CREATION
bq. src/monitoring/process_resource_collector.hpp PRE-CREATION
bq. src/monitoring/linux/proc_utils.cpp PRE-CREATION
bq. src/monitoring/linux/proc_utils.hpp PRE-CREATION
bq. src/monitoring/linux/proc_resource_collector.cpp PRE-CREATION
bq. src/monitoring/linux/proc_resource_collector.hpp PRE-CREATION
bq. src/monitoring/linux/lxc_resource_collector.cpp PRE-CREATION
bq. src/master/master.hpp 53551b0
bq. src/master/master.cpp 1d3961e
bq. src/messages/messages.proto 11a2c41
bq. src/monitoring/linux/lxc_resource_collector.hpp PRE-CREATION
bq. src/master/allocator.hpp 1ac435b
bq. src/master/http.cpp 591433a
bq. src/Makefile.am 1137a3e
bq.
bq. Diff: https://reviews.apache.org/r/4167/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Test cases:
bq. * A test case exercising the basic monitoring code with a mocked-out collector.
bq. * The first of several tests for the process resource monitor, with the proc-based collecting mocked out.
bq.
bq. Some ad-hoc testing with log statements to ensure that the monitoring works end-to-end from both the container-based and process-based isolation modules.
bq.
bq.
bq. Thanks,
bq.
bq. Sam
bq.
bq.
> Executor resource monitoring and local reporting of usage stats
> ---------------------------------------------------------------
>
> Key: MESOS-38
> URL: https://issues.apache.org/jira/browse/MESOS-38
> Project: Mesos
> Issue Type: New Feature
> Components: isolation, slave
> Environment: Initial executor monitoring for linux only. Dummy monitoring capability (no-op) for OSX, with functionality to be filled in later.
> Reporter: Sam Whitlock
> Assignee: Sam Whitlock
> Labels: monitoring
>
> Implement reporting of resource usage on executors and log them to a local log file (for now). The eventual usage of this will be to report these statistics to the Mesos master in order to build either or both a timeline for the webui and/or a top-like command-line interface. This improvement ticket is just for the local monitoring and log file reporting. A reporting system (to the master node) will be a later improvement ticket.
> With the current version of Mesos, it is not possible to monitor individual tasks. Therefore the best this sort of system can do is monitor the usage of an individual executor and aggregate the resource usage of over the executor's tasks and resource allocations. If frameworks have a 1-to-1 relationship of a job to an executor, then the aggregate statistics will be more meaningful.
> Reporting will be available for both lxc isolation and process-based isolation. For lxc isolation the task is easier because of the isolation facilities of lxc. Process-based isolation is more difficult as processes can become re-parented from the process tree of the executor (e.g. double fork). The session ID and the process group ID will likely still be the same as that of the executor except for the uncommon case of the process resetting both of those.
> When usage statistics are eventually reported to the Mesos master, it may be possible to use them to oversubscribe slave nodes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MESOS-38) Executor resource monitoring and
local reporting of usage stats
Posted by "Alex Degtiar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MESOS-38?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13149191#comment-13149191 ]
Alex Degtiar commented on MESOS-38:
-----------------------------------
The organization we are implementing now is that each method of measuring resources will be a separate implementation of the ResourceMonitor class. Such a ResourceMonitor will be a module specific to a method of measurement (e.g. reading container info from cgroups, reading /proc on Linux for processes, etc). The base class would have a single method that collects the (delta) usage since the previous call.
A particular IsolationModule implementation that launches a new executor would also create and initialize a new ResourceMonitor for that executor, specific to that type of IsolationModule. When a slave asynchronously calls sampleUsage() on an isolation module, the isolation module would then collect a usage report from the ResourceMonitor, package it into a UsageMessage, and push it back to the slave.
Currently, Sam is working on the ResourceMonitor for the LxcIsolationModule (reading from cgroups), and I am working on the ResourceMonitor for ProcessBasedIsolationModule (reading from proc on Linux). Much of the measurement functionality is complete, and we will update when we have more to report.
> Executor resource monitoring and local reporting of usage stats
> ---------------------------------------------------------------
>
> Key: MESOS-38
> URL: https://issues.apache.org/jira/browse/MESOS-38
> Project: Mesos
> Issue Type: New Feature
> Environment: Initial executor monitoring for linux only. Dummy monitoring capability (no-op) for OSX, with functionality to be filled in later.
> Reporter: Sam Whitlock
> Labels: monitoring
>
> Implement reporting of resource usage on executors and log them to a local log file (for now). The eventual usage of this will be to report these statistics to the Mesos master in order to build either or both a timeline for the webui and/or a top-like command-line interface. This improvement ticket is just for the local monitoring and log file reporting. A reporting system (to the master node) will be a later improvement ticket.
> With the current version of Mesos, it is not possible to monitor individual tasks. Therefore the best this sort of system can do is monitor the usage of an individual executor and aggregate the resource usage of over the executor's tasks and resource allocations. If frameworks have a 1-to-1 relationship of a job to an executor, then the aggregate statistics will be more meaningful.
> Reporting will be available for both lxc isolation and process-based isolation. For lxc isolation the task is easier because of the isolation facilities of lxc. Process-based isolation is more difficult as processes can become re-parented from the process tree of the executor (e.g. double fork). The session ID and the process group ID will likely still be the same as that of the executor except for the uncommon case of the process resetting both of those.
> Initial reporting will be to a local log file. This will be a 'heartbeat' style akin to pidstat output (in sysstat library). This may not be incredibly useful, but local monitoring of resource usage is separate from the reporting and timeline building mentioned above.
> When usage statistics are eventually reported to the Mesos master, it may be possible to use them to oversubscribe slave nodes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MESOS-38) Executor resource monitoring and
local reporting of usage stats
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MESOS-38?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13224689#comment-13224689 ]
jiraposter@reviews.apache.org commented on MESOS-38:
----------------------------------------------------
bq. On 2012-03-05 23:04:17, Charles Reiss wrote:
bq. > src/slave/resource_monitor.cpp, line 67
bq. > <https://reviews.apache.org/r/4167/diff/2/?file=88033#file88033line67>
bq. >
bq. > I assume that you've discussed calling this 'mem_usage' and 'cpu_usage' rather than 'mem' and 'cpus' with Ben? Can you explain the reasoning briefly?
bq.
bq. Sam Whitlock wrote:
bq. I sorta made this choice unilaterally based on its similarity to the naming for cgroups infos.
bq.
bq. I don't really care what it is called and would certainly be willing to change it if it would make it more similar to the rest of mesos.
bq.
bq. Sam Whitlock wrote:
bq. I guess the choice to go with *_usage is because mem and cpus is used elsewhere in mesos for things that are not reported usage, and I wanted to draw a distinction.
I was assuming that usage measurements would always be kept in separate fields from requests. Is there a usecase where this doesn't look like this will be the case?
Unless there's a reason to believe it would be deceptive, I'd prefer to name the usage measurements the same as the requested resource they are supposed to measure.
- Charles
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4167/#review5605
-----------------------------------------------------------
On 2012-03-06 04:54:32, Sam Whitlock wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4167/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-06 04:54:32)
bq.
bq.
bq. Review request for mesos, Benjamin Hindman and Charles Reiss.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. This mega-patch is intended to represent the partial completion of the slave monitoring functionality. It is not intended to be committed. Changes based on comments in this review will be reflected in future reviews that are smaller and more modular.
bq.
bq. Proc utils is included in this patch, but is already under review here: https://reviews.apache.org/r/3050/
bq.
bq. The relevant design doc can be found here: https://docs.google.com/document/d/14Wj9i6TpMR6cV3LL0ySjLjfZOq5QOQeybvt1gSyaQGs/edit
bq.
bq. The following items are ones where specific feedback is requested:
bq.
bq. * A better mechanism is needed to control the rate at which the slave asks each executor for its UsageMessage. This is currently hard-coded to be at 1 second intervals, but could potentially be read as a command-line option or from a config file. Is there a better or different way to pass in this value?
bq. * Currently, UsageMessages are passed from a ResourceMonitor to the Slave using the Future construct, and used as containers that hold a snapshot of the latest usage. This is to prevent unnecessary marshalling and extra data structures, since messages will eventually be sent in the standard dispatch style from the slave to the master. Is it fine that we are using Protobuf messages in this way?
bq.
bq. There are several changes that are not yet implemented in this patch. These changes are as follows:
bq.
bq. * Sufficient tests cases have not yet been written for any component (resource monitor, lxc collector, and process collector).
bq. * Code has not been cleaned up to adhere to all style recommendations.
bq. * Process collector code needs to be updated to prevent CPU usage spikes when monitored sub-processes die.
bq. * Code to send UsageMessages from the slave to the master.
bq.
bq.
bq. This addresses bug MESOS-38.
bq. https://issues.apache.org/jira/browse/MESOS-38
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/Makefile.am 1137a3e
bq. src/master/allocator.hpp 1ac435b
bq. src/master/http.cpp 591433a
bq. src/master/master.hpp 53551b0
bq. src/master/master.cpp 1d3961e
bq. src/messages/messages.proto 11a2c41
bq. src/monitoring/linux/lxc_resource_collector.hpp PRE-CREATION
bq. src/monitoring/linux/lxc_resource_collector.cpp PRE-CREATION
bq. src/monitoring/linux/proc_resource_collector.hpp PRE-CREATION
bq. src/monitoring/linux/proc_resource_collector.cpp PRE-CREATION
bq. src/monitoring/linux/proc_utils.hpp PRE-CREATION
bq. src/monitoring/linux/proc_utils.cpp PRE-CREATION
bq. src/monitoring/process_resource_collector.hpp PRE-CREATION
bq. src/monitoring/process_resource_collector.cpp PRE-CREATION
bq. src/monitoring/process_stats.hpp PRE-CREATION
bq. src/monitoring/resource_collector.hpp PRE-CREATION
bq. src/slave/http.cpp f03815d
bq. src/slave/isolation_module.hpp c896908
bq. src/slave/isolation_module.cpp 5b7b4a2
bq. src/slave/lxc_isolation_module.hpp b7beefe
bq. src/slave/lxc_isolation_module.cpp d544625
bq. src/slave/main.cpp ac780c4
bq. src/slave/process_based_isolation_module.hpp f6f9554
bq. src/slave/process_based_isolation_module.cpp 100b1e3
bq. src/slave/resource_monitor.hpp PRE-CREATION
bq. src/slave/resource_monitor.cpp PRE-CREATION
bq. src/slave/slave.hpp b1a07e9
bq. src/slave/slave.cpp ce8fda5
bq. src/tests/Makefile.in 6f51be4
bq. src/tests/proc_utils_tests.cpp PRE-CREATION
bq. src/tests/process_resource_collector_tests.cpp PRE-CREATION
bq. src/tests/resource_monitor_tests.cpp PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/4167/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Test cases:
bq. * A test case exercising the basic monitoring code with a mocked-out collector.
bq. * The first of several tests for the process resource monitor, with the proc-based collecting mocked out.
bq.
bq. Some ad-hoc testing with log statements to ensure that the monitoring works end-to-end from both the container-based and process-based isolation modules.
bq.
bq.
bq. Thanks,
bq.
bq. Sam
bq.
bq.
> Executor resource monitoring and local reporting of usage stats
> ---------------------------------------------------------------
>
> Key: MESOS-38
> URL: https://issues.apache.org/jira/browse/MESOS-38
> Project: Mesos
> Issue Type: New Feature
> Components: isolation, slave
> Environment: Initial executor monitoring for linux only. Dummy monitoring capability (no-op) for OSX, with functionality to be filled in later.
> Reporter: Sam Whitlock
> Assignee: Sam Whitlock
> Labels: monitoring
>
> Implement reporting of resource usage on executors and log them to a local log file (for now). The eventual usage of this will be to report these statistics to the Mesos master in order to build either or both a timeline for the webui and/or a top-like command-line interface. This improvement ticket is just for the local monitoring and log file reporting. A reporting system (to the master node) will be a later improvement ticket.
> With the current version of Mesos, it is not possible to monitor individual tasks. Therefore the best this sort of system can do is monitor the usage of an individual executor and aggregate the resource usage of over the executor's tasks and resource allocations. If frameworks have a 1-to-1 relationship of a job to an executor, then the aggregate statistics will be more meaningful.
> Reporting will be available for both lxc isolation and process-based isolation. For lxc isolation the task is easier because of the isolation facilities of lxc. Process-based isolation is more difficult as processes can become re-parented from the process tree of the executor (e.g. double fork). The session ID and the process group ID will likely still be the same as that of the executor except for the uncommon case of the process resetting both of those.
> When usage statistics are eventually reported to the Mesos master, it may be possible to use them to oversubscribe slave nodes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira