You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@aurora.apache.org by Reza Motamedi <re...@gmail.com> on 2017/06/15 16:16:55 UTC
Review Request 60060: Remove restriction on task id length.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60060/
-----------------------------------------------------------
Review request for Aurora, David McLaughlin and Santhosh Kumar Shanmugham.
Repository: aurora
Description
-------
# Introduce `ProcessTreeCache` to stores and updates a copy of `pstree`
My measurements shows that on a busy machine, a single call to `psutil.Process.children` can take up to 20 seconds. Looking at what the library does (Process.children)[https://pythonhosted.org/psutil/_modules/psutil.html#Process.children] I see that it computes a mapping from each parent to its immidiate children and does a BFS on it. This repeated process (calculating the table) takes up a lot of time and can be dones on a slightly obsolete version of pstree.
Diffs
-----
src/main/python/apache/thermos/monitoring/process_collector_psutil.py 3594955c68b45ab65c01426ba0a18ec8a132a27f
src/test/python/apache/thermos/monitoring/test_process_collector_psutil.py 93ff878be578fa7a63d25b65e7d915790dc9ccc6
Diff: https://reviews.apache.org/r/60060/diff/1/
Testing
-------
current test pass. I added a few unit tests but also more be added.
Thanks,
Reza Motamedi
Re: Review Request 60060: Remove restriction on task id length.
Posted by Aurora ReviewBot <wf...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60060/#review178012
-----------------------------------------------------------
Master (0001f90) is red with this patch.
./build-support/jenkins/build.sh
New python executable in /home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/isort.venv/bin/python2.7
Also creating executable in /home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/isort.venv/bin/python
Installing setuptools, pip, wheel...done.
Collecting isort==4.0.0
Downloading isort-4.0.0-py2.py3-none-any.whl
Installing collected packages: isort
Successfully installed isort-4.0.0
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
ERROR: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/thermos/monitoring/process_collector_psutil.py Imports are incorrectly sorted.
--- /home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/thermos/monitoring/process_collector_psutil.py:before 2017-06-15 16:21:05.421873
+++ /home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/thermos/monitoring/process_collector_psutil.py:after 2017-06-15 16:26:51.062056
@@ -14,12 +14,12 @@
""" Sample resource consumption statistics for processes using psutil """
+import threading
from collections import defaultdict
from operator import attrgetter
from time import time
import psutil
-import threading
from psutil import Error as PsutilError
from psutil import AccessDenied, NoSuchProcess, ZombieProcess
from twitter.common import log
ERROR: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/thermos/monitoring/test_process_collector_psutil.py Imports are incorrectly sorted.
--- /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/thermos/monitoring/test_process_collector_psutil.py:before 2017-06-15 16:21:05.421873
+++ /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/thermos/monitoring/test_process_collector_psutil.py:after 2017-06-15 16:26:51.914671
@@ -19,8 +19,8 @@
from apache.thermos.monitoring.process import ProcessSample
from apache.thermos.monitoring.process_collector_psutil import (
- ProcessTreeCache,
- ProcessTreeCollector
+ ProcessTreeCache,
+ ProcessTreeCollector
)
I will refresh this build result if you post a review containing "@ReviewBot retry"
- Aurora ReviewBot
On June 15, 2017, 4:16 p.m., Reza Motamedi wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60060/
> -----------------------------------------------------------
>
> (Updated June 15, 2017, 4:16 p.m.)
>
>
> Review request for Aurora, David McLaughlin and Santhosh Kumar Shanmugham.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> # Introduce `ProcessTreeCache` to stores and updates a copy of `pstree`
>
> My measurements shows that on a busy machine, a single call to `psutil.Process.children` can take up to 20 seconds. Looking at what the library does (Process.children)[https://pythonhosted.org/psutil/_modules/psutil.html#Process.children] I see that it computes a mapping from each parent to its immidiate children and does a BFS on it. This repeated process (calculating the table) takes up a lot of time and can be dones on a slightly obsolete version of pstree.
>
>
> Diffs
> -----
>
> src/main/python/apache/thermos/monitoring/process_collector_psutil.py 3594955c68b45ab65c01426ba0a18ec8a132a27f
> src/test/python/apache/thermos/monitoring/test_process_collector_psutil.py 93ff878be578fa7a63d25b65e7d915790dc9ccc6
>
>
> Diff: https://reviews.apache.org/r/60060/diff/1/
>
>
> Testing
> -------
>
> current test pass. I added a few unit tests but also more be added.
>
>
> Thanks,
>
> Reza Motamedi
>
>