You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@aurora.apache.org by Stephan Erb <se...@apache.org> on 2017/03/19 16:18:52 UTC
Review Request 57757: Make Thermos resource collection intervals
configurable
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57757/
-----------------------------------------------------------
Review request for Aurora, Santhosh Kumar Shanmugham and Zameer Manji.
Bugs: AURORA-1907
https://issues.apache.org/jira/browse/AURORA-1907
Repository: aurora
Description
-------
We have noticed that on hosts with lots of active tasks (~100) the Thermos UI
is not usable. Thermos fully utilizes one core but does not render any requests.
Dumping `/threads` indicates the observer might be backlogged by the hundred
concurrent `TaskResourceMonitor` threads. Due to the Python GIL only one can
make progress at a time though.
This patch is now adding options to control the resource collection interval,
giving operators a possibility to reduce the CPU pressure.
--
Reviewer notes: I dropped the entrypoint test as it was heavily relying on mocks
without significant code coverage.
Diffs
-----
RELEASE-NOTES.md d58d2bd2558c2060aa659402ea88271bd8704cde
docs/README.md 1d679e27cf26122c34a9157194f189ce3035b176
docs/reference/observer-configuration.md PRE-CREATION
src/main/python/apache/aurora/tools/thermos_observer.py 4bba01930434f33150b0413d20d9657b61b364c5
src/main/python/apache/thermos/monitoring/resource.py 53d0ff1a71c27f053c59acca556c35d1e5ac91f0
src/main/python/apache/thermos/observer/detector.py 79bc6329caf73fbfbb4ebf4bbad96bb1c50be1f9
src/main/python/apache/thermos/observer/task_observer.py 1485de8faef52716f11b82a3556064de26c67427
src/test/python/apache/aurora/tools/test_thermos_observer_entry_point.py e1c8dec5f26be1aec931b1a614ecc138ae63d9de
Diff: https://reviews.apache.org/r/57757/diff/1/
Testing
-------
./pants test.pytest src/{test,main}/python:: -- -v
Thanks,
Stephan Erb
Re: Review Request 57757: Make Thermos resource collection intervals
configurable
Posted by Aurora ReviewBot <wf...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57757/#review169379
-----------------------------------------------------------
Master (b8f72d1) is red with this patch.
./build-support/jenkins/build.sh
Test coverage missing for org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl$8
Test coverage missing for org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl$7
Test coverage missing for org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl$9
Test coverage missing for org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl$4
Test coverage missing for org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl$3
Test coverage missing for org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl$6
Test coverage missing for org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl$5
Test coverage missing for org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl$2
Test coverage missing for org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl$1
Test coverage missing for org/apache/aurora/scheduler/storage/log/LogStorage$Settings
Test coverage missing for org/apache/aurora/scheduler/storage/log/LogStorage$ScheduledExecutorSchedulingService
Test coverage missing for org/apache/aurora/scheduler/storage/log/LogStorageModule
Test coverage missing for org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl$SnapshotField
Test coverage missing for org/apache/aurora/scheduler/storage/backup/TemporaryStorage$TemporaryStorageFactory$1
Test coverage missing for org/apache/aurora/scheduler/storage/backup/BackupModule
Test coverage missing for org/apache/aurora/scheduler/storage/backup/Recovery$RecoveryImpl
Test coverage missing for org/apache/aurora/scheduler/storage/backup/TemporaryStorage$TemporaryStorageFactory
Test coverage missing for org/apache/aurora/scheduler/storage/backup/Recovery$RecoveryImpl$PendingRecovery
Test coverage missing for org/apache/aurora/scheduler/TaskVars
Test coverage missing for org/apache/aurora/scheduler/SchedulerLifecycle$DefaultDelayedActions
Test coverage missing for org/apache/aurora/scheduler/TierManager$TierManagerImpl$TierConfig
Test coverage missing for org/apache/aurora/scheduler/TaskVars$Counter
Test coverage missing for org/apache/aurora/scheduler/TaskVars$1
Test coverage missing for org/apache/aurora/scheduler/SchedulerModule$TaskEventBatchWorker
Test coverage missing for org/apache/aurora/scheduler/HostOffer$1
Test coverage missing for org/apache/aurora/scheduler/SchedulerModule
Test coverage missing for org/apache/aurora/scheduler/TaskIdGenerator$TaskIdGeneratorImpl
Test coverage missing for org/apache/aurora/scheduler/SchedulerModule$1
Test coverage missing for org/apache/aurora/scheduler/TaskStatusHandlerImpl
Test coverage missing for org/apache/aurora/scheduler/TaskStatusHandlerImpl$1
Test coverage missing for org/apache/aurora/scheduler/TierModule
Test coverage missing for org/apache/aurora/scheduler/storage/db/typehandlers/VolumeModeTypeHandler
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
==============================================================================
BUILD FAILED
Total time: 1 hrs 0 mins 27.501 secs
I will refresh this build result if you post a review containing "@ReviewBot retry"
- Aurora ReviewBot
On March 19, 2017, 4:34 p.m., Stephan Erb wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57757/
> -----------------------------------------------------------
>
> (Updated March 19, 2017, 4:34 p.m.)
>
>
> Review request for Aurora, Santhosh Kumar Shanmugham and Zameer Manji.
>
>
> Bugs: AURORA-1907
> https://issues.apache.org/jira/browse/AURORA-1907
>
>
> Repository: aurora
>
>
> Description
> -------
>
> We have noticed that on hosts with lots of active tasks (~100) the Thermos UI
> is not usable. Thermos fully utilizes one core but does not render any requests.
>
> Dumping `/threads` indicates the observer might be backlogged by the hundred
> concurrent `TaskResourceMonitor` threads. Due to the Python GIL only one can
> make progress at a time though.
>
> This patch is now adding options to control the resource collection interval,
> giving operators a possibility to reduce the CPU pressure.
>
> --
>
> Reviewer notes: I dropped the entrypoint test as it was heavily relying on mocks
> without significant code coverage.
>
>
> Diffs
> -----
>
> RELEASE-NOTES.md d58d2bd2558c2060aa659402ea88271bd8704cde
> docs/README.md 1d679e27cf26122c34a9157194f189ce3035b176
> docs/reference/observer-configuration.md PRE-CREATION
> src/main/python/apache/aurora/tools/thermos_observer.py 4bba01930434f33150b0413d20d9657b61b364c5
> src/main/python/apache/thermos/monitoring/resource.py 53d0ff1a71c27f053c59acca556c35d1e5ac91f0
> src/main/python/apache/thermos/observer/task_observer.py 1485de8faef52716f11b82a3556064de26c67427
> src/test/python/apache/aurora/tools/test_thermos_observer_entry_point.py e1c8dec5f26be1aec931b1a614ecc138ae63d9de
>
>
> Diff: https://reviews.apache.org/r/57757/diff/2/
>
>
> Testing
> -------
>
> ./pants test.pytest src/{test,main}/python:: -- -v
>
>
> Thanks,
>
> Stephan Erb
>
>
Re: Review Request 57757: Make Thermos observer resource collection
intervals configurable
Posted by Santhosh Kumar Shanmugham <sa...@gmail.com>.
> On March 20, 2017, 9:58 a.m., Santhosh Kumar Shanmugham wrote:
> > src/main/python/apache/thermos/monitoring/resource.py
> > Lines 121-123 (patched)
> > <https://reviews.apache.org/r/57757/diff/2/?file=1668004#file1668004line121>
> >
> > Can we encode the units into the names of the variables?
>
> Stephan Erb wrote:
> Isn't the whole point of the `Amount` class that we don't have to do this?
>
> Would it help understandability if I eliminate one layer of default variables and use the defaults from `TaskResourceMonitor` everywhere, rather than than the redifintions in `TaskObserver`?
>
> Zameer Manji wrote:
> I think removing the layer of defaults would be easier.
That works for me. Thanks.
- Santhosh Kumar
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57757/#review169462
-----------------------------------------------------------
On March 20, 2017, 1:41 p.m., Stephan Erb wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57757/
> -----------------------------------------------------------
>
> (Updated March 20, 2017, 1:41 p.m.)
>
>
> Review request for Aurora, Santhosh Kumar Shanmugham and Zameer Manji.
>
>
> Bugs: AURORA-1907
> https://issues.apache.org/jira/browse/AURORA-1907
>
>
> Repository: aurora
>
>
> Description
> -------
>
> We have noticed that on hosts with lots of active tasks (~100) the observer UI
> is not usable. Thermos fully utilizes one core but does not render any requests.
>
> Dumping `/threads` indicates the observer might be backlogged by the hundred
> concurrent `TaskResourceMonitor` threads. Due to the Python GIL only one can
> make progress at a time though.
>
> This patch is now adding options to control the resource collection interval,
> giving operators a possibility to reduce the CPU pressure.
>
> --
>
> Reviewer notes: I dropped the entrypoint test as it was heavily relying on mocks
> without significant code coverage.
>
>
> Diffs
> -----
>
> RELEASE-NOTES.md d58d2bd2558c2060aa659402ea88271bd8704cde
> docs/README.md 1d679e27cf26122c34a9157194f189ce3035b176
> docs/reference/observer-configuration.md PRE-CREATION
> src/main/python/apache/aurora/tools/thermos_observer.py 4bba01930434f33150b0413d20d9657b61b364c5
> src/main/python/apache/thermos/monitoring/resource.py 53d0ff1a71c27f053c59acca556c35d1e5ac91f0
> src/main/python/apache/thermos/observer/task_observer.py 1485de8faef52716f11b82a3556064de26c67427
> src/test/python/apache/aurora/tools/test_thermos_observer_entry_point.py e1c8dec5f26be1aec931b1a614ecc138ae63d9de
>
>
> Diff: https://reviews.apache.org/r/57757/diff/3/
>
>
> Testing
> -------
>
> ./pants test.pytest src/{test,main}/python:: -- -v
>
>
> Thanks,
>
> Stephan Erb
>
>
Re: Review Request 57757: Make Thermos observer resource collection
intervals configurable
Posted by Zameer Manji <zm...@apache.org>.
> On March 20, 2017, 9:58 a.m., Santhosh Kumar Shanmugham wrote:
> > src/main/python/apache/thermos/monitoring/resource.py
> > Lines 121-123 (patched)
> > <https://reviews.apache.org/r/57757/diff/2/?file=1668004#file1668004line121>
> >
> > Can we encode the units into the names of the variables?
>
> Stephan Erb wrote:
> Isn't the whole point of the `Amount` class that we don't have to do this?
>
> Would it help understandability if I eliminate one layer of default variables and use the defaults from `TaskResourceMonitor` everywhere, rather than than the redifintions in `TaskObserver`?
I think removing the layer of defaults would be easier.
- Zameer
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57757/#review169462
-----------------------------------------------------------
On March 20, 2017, 1:41 p.m., Stephan Erb wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57757/
> -----------------------------------------------------------
>
> (Updated March 20, 2017, 1:41 p.m.)
>
>
> Review request for Aurora, Santhosh Kumar Shanmugham and Zameer Manji.
>
>
> Bugs: AURORA-1907
> https://issues.apache.org/jira/browse/AURORA-1907
>
>
> Repository: aurora
>
>
> Description
> -------
>
> We have noticed that on hosts with lots of active tasks (~100) the observer UI
> is not usable. Thermos fully utilizes one core but does not render any requests.
>
> Dumping `/threads` indicates the observer might be backlogged by the hundred
> concurrent `TaskResourceMonitor` threads. Due to the Python GIL only one can
> make progress at a time though.
>
> This patch is now adding options to control the resource collection interval,
> giving operators a possibility to reduce the CPU pressure.
>
> --
>
> Reviewer notes: I dropped the entrypoint test as it was heavily relying on mocks
> without significant code coverage.
>
>
> Diffs
> -----
>
> RELEASE-NOTES.md d58d2bd2558c2060aa659402ea88271bd8704cde
> docs/README.md 1d679e27cf26122c34a9157194f189ce3035b176
> docs/reference/observer-configuration.md PRE-CREATION
> src/main/python/apache/aurora/tools/thermos_observer.py 4bba01930434f33150b0413d20d9657b61b364c5
> src/main/python/apache/thermos/monitoring/resource.py 53d0ff1a71c27f053c59acca556c35d1e5ac91f0
> src/main/python/apache/thermos/observer/task_observer.py 1485de8faef52716f11b82a3556064de26c67427
> src/test/python/apache/aurora/tools/test_thermos_observer_entry_point.py e1c8dec5f26be1aec931b1a614ecc138ae63d9de
>
>
> Diff: https://reviews.apache.org/r/57757/diff/3/
>
>
> Testing
> -------
>
> ./pants test.pytest src/{test,main}/python:: -- -v
>
>
> Thanks,
>
> Stephan Erb
>
>
Re: Review Request 57757: Make Thermos observer resource collection
intervals configurable
Posted by Stephan Erb <se...@apache.org>.
> On March 20, 2017, 5:58 p.m., Santhosh Kumar Shanmugham wrote:
> > src/main/python/apache/thermos/monitoring/resource.py
> > Lines 121-123 (patched)
> > <https://reviews.apache.org/r/57757/diff/2/?file=1668004#file1668004line121>
> >
> > Can we encode the units into the names of the variables?
Isn't the whole point of the `Amount` class that we don't have to do this?
Would it help understandability if I eliminate one layer of default variables and use the defaults from `TaskResourceMonitor` everywhere, rather than than the redifintions in `TaskObserver`?
> On March 20, 2017, 5:58 p.m., Santhosh Kumar Shanmugham wrote:
> > src/main/python/apache/thermos/observer/task_observer.py
> > Lines 56 (patched)
> > <https://reviews.apache.org/r/57757/diff/2/?file=1668005#file1668005line56>
> >
> > s/TAKS/TASK/
Good catch, thanks.
> On March 20, 2017, 5:58 p.m., Santhosh Kumar Shanmugham wrote:
> > src/main/python/apache/thermos/observer/task_observer.py
> > Lines 56-57 (patched)
> > <https://reviews.apache.org/r/57757/diff/2/?file=1668005#file1668005line56>
> >
> > Can we sync the time units across these variables? It is confusing to figure out the unit of the defaults vs the incoming params.
Good point. I will stick to seconds where it makes sense.
- Stephan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57757/#review169462
-----------------------------------------------------------
On March 19, 2017, 11:50 p.m., Stephan Erb wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57757/
> -----------------------------------------------------------
>
> (Updated March 19, 2017, 11:50 p.m.)
>
>
> Review request for Aurora, Santhosh Kumar Shanmugham and Zameer Manji.
>
>
> Bugs: AURORA-1907
> https://issues.apache.org/jira/browse/AURORA-1907
>
>
> Repository: aurora
>
>
> Description
> -------
>
> We have noticed that on hosts with lots of active tasks (~100) the observer UI
> is not usable. Thermos fully utilizes one core but does not render any requests.
>
> Dumping `/threads` indicates the observer might be backlogged by the hundred
> concurrent `TaskResourceMonitor` threads. Due to the Python GIL only one can
> make progress at a time though.
>
> This patch is now adding options to control the resource collection interval,
> giving operators a possibility to reduce the CPU pressure.
>
> --
>
> Reviewer notes: I dropped the entrypoint test as it was heavily relying on mocks
> without significant code coverage.
>
>
> Diffs
> -----
>
> RELEASE-NOTES.md d58d2bd2558c2060aa659402ea88271bd8704cde
> docs/README.md 1d679e27cf26122c34a9157194f189ce3035b176
> docs/reference/observer-configuration.md PRE-CREATION
> src/main/python/apache/aurora/tools/thermos_observer.py 4bba01930434f33150b0413d20d9657b61b364c5
> src/main/python/apache/thermos/monitoring/resource.py 53d0ff1a71c27f053c59acca556c35d1e5ac91f0
> src/main/python/apache/thermos/observer/task_observer.py 1485de8faef52716f11b82a3556064de26c67427
> src/test/python/apache/aurora/tools/test_thermos_observer_entry_point.py e1c8dec5f26be1aec931b1a614ecc138ae63d9de
>
>
> Diff: https://reviews.apache.org/r/57757/diff/2/
>
>
> Testing
> -------
>
> ./pants test.pytest src/{test,main}/python:: -- -v
>
>
> Thanks,
>
> Stephan Erb
>
>
Re: Review Request 57757: Make Thermos observer resource collection
intervals configurable
Posted by Santhosh Kumar Shanmugham <sa...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57757/#review169462
-----------------------------------------------------------
src/main/python/apache/thermos/monitoring/resource.py
Lines 121-123 (patched)
<https://reviews.apache.org/r/57757/#comment241806>
Can we encode the units into the names of the variables?
src/main/python/apache/thermos/observer/task_observer.py
Lines 56 (patched)
<https://reviews.apache.org/r/57757/#comment241807>
s/TAKS/TASK/
src/main/python/apache/thermos/observer/task_observer.py
Lines 56-57 (patched)
<https://reviews.apache.org/r/57757/#comment241809>
Can we sync the time units across these variables? It is confusing to figure out the unit of the defaults vs the incoming params.
- Santhosh Kumar Shanmugham
On March 19, 2017, 3:50 p.m., Stephan Erb wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57757/
> -----------------------------------------------------------
>
> (Updated March 19, 2017, 3:50 p.m.)
>
>
> Review request for Aurora, Santhosh Kumar Shanmugham and Zameer Manji.
>
>
> Bugs: AURORA-1907
> https://issues.apache.org/jira/browse/AURORA-1907
>
>
> Repository: aurora
>
>
> Description
> -------
>
> We have noticed that on hosts with lots of active tasks (~100) the observer UI
> is not usable. Thermos fully utilizes one core but does not render any requests.
>
> Dumping `/threads` indicates the observer might be backlogged by the hundred
> concurrent `TaskResourceMonitor` threads. Due to the Python GIL only one can
> make progress at a time though.
>
> This patch is now adding options to control the resource collection interval,
> giving operators a possibility to reduce the CPU pressure.
>
> --
>
> Reviewer notes: I dropped the entrypoint test as it was heavily relying on mocks
> without significant code coverage.
>
>
> Diffs
> -----
>
> RELEASE-NOTES.md d58d2bd2558c2060aa659402ea88271bd8704cde
> docs/README.md 1d679e27cf26122c34a9157194f189ce3035b176
> docs/reference/observer-configuration.md PRE-CREATION
> src/main/python/apache/aurora/tools/thermos_observer.py 4bba01930434f33150b0413d20d9657b61b364c5
> src/main/python/apache/thermos/monitoring/resource.py 53d0ff1a71c27f053c59acca556c35d1e5ac91f0
> src/main/python/apache/thermos/observer/task_observer.py 1485de8faef52716f11b82a3556064de26c67427
> src/test/python/apache/aurora/tools/test_thermos_observer_entry_point.py e1c8dec5f26be1aec931b1a614ecc138ae63d9de
>
>
> Diff: https://reviews.apache.org/r/57757/diff/2/
>
>
> Testing
> -------
>
> ./pants test.pytest src/{test,main}/python:: -- -v
>
>
> Thanks,
>
> Stephan Erb
>
>
Re: Review Request 57757: Make Thermos observer resource collection
intervals configurable
Posted by Aurora ReviewBot <wf...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57757/#review169516
-----------------------------------------------------------
Master (b8f72d1) is red with this patch.
./build-support/jenkins/build.sh
Test coverage missing for org/apache/aurora/scheduler/events/WebhookInfo
Test coverage missing for org/apache/aurora/scheduler/storage/log/WriteAheadStorageForwarder
Test coverage missing for org/apache/aurora/scheduler/storage/log/LogStorage$1
Test coverage missing for org/apache/aurora/scheduler/storage/log/StreamManagerImpl$StreamTransactionImpl
Test coverage missing for org/apache/aurora/scheduler/storage/log/EntrySerializer$EntrySerializerImpl$1
Test coverage missing for org/apache/aurora/scheduler/storage/log/LogStorage$Settings
Test coverage missing for org/apache/aurora/scheduler/storage/log/StreamManagerImpl
Test coverage missing for org/apache/aurora/scheduler/storage/log/EntrySerializer$EntrySerializerImpl
Test coverage missing for org/apache/aurora/scheduler/storage/log/LogStorage$ScheduledExecutorSchedulingService
Test coverage missing for org/apache/aurora/scheduler/storage/log/LogStorageModule
Test coverage missing for org/apache/aurora/scheduler/storage/log/LogStorage
Test coverage missing for org/apache/aurora/scheduler/storage/log/SnapshotDeduplicator$SnapshotDeduplicatorImpl
Test coverage missing for org/apache/aurora/scheduler/storage/log/Entries
Test coverage missing for org/apache/aurora/scheduler/storage/log/LogManager
Test coverage missing for org/apache/aurora/scheduler/storage/backup/StorageBackup$StorageBackupImpl$BackupConfig
Test coverage missing for org/apache/aurora/scheduler/storage/backup/BackupModule
Test coverage missing for org/apache/aurora/scheduler/SchedulerLifecycle$7
Test coverage missing for org/apache/aurora/scheduler/SchedulerLifecycle$6
Test coverage missing for org/apache/aurora/scheduler/SchedulerLifecycle$5
Test coverage missing for org/apache/aurora/scheduler/SchedulerLifecycle$4
Test coverage missing for org/apache/aurora/scheduler/SchedulerLifecycle$3
Test coverage missing for org/apache/aurora/scheduler/SchedulerLifecycle$2
Test coverage missing for org/apache/aurora/scheduler/SchedulerLifecycle$1
Test coverage missing for org/apache/aurora/scheduler/TaskVars
Test coverage missing for org/apache/aurora/scheduler/SchedulerLifecycle$SchedulerCandidateImpl
Test coverage missing for org/apache/aurora/scheduler/SchedulerLifecycle$DefaultDelayedActions
Test coverage missing for org/apache/aurora/scheduler/TierManager$TierManagerImpl$TierConfig
Test coverage missing for org/apache/aurora/scheduler/TaskVars$Counter
Test coverage missing for org/apache/aurora/scheduler/TaskVars$1
Test coverage missing for org/apache/aurora/scheduler/SchedulerLifecycle
Test coverage missing for org/apache/aurora/scheduler/storage/db/typehandlers/VolumeModeTypeHandler
Test coverage missing for org/apache/aurora/scheduler/storage/db/typehandlers/MaintenanceModeTypeHandler
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
==============================================================================
BUILD FAILED
Total time: 7 mins 47.963 secs
I will refresh this build result if you post a review containing "@ReviewBot retry"
- Aurora ReviewBot
On March 21, 2017, 12:36 a.m., Stephan Erb wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57757/
> -----------------------------------------------------------
>
> (Updated March 21, 2017, 12:36 a.m.)
>
>
> Review request for Aurora, Santhosh Kumar Shanmugham and Zameer Manji.
>
>
> Bugs: AURORA-1907
> https://issues.apache.org/jira/browse/AURORA-1907
>
>
> Repository: aurora
>
>
> Description
> -------
>
> We have noticed that on hosts with lots of active tasks (~100) the observer UI
> is not usable. Thermos fully utilizes one core but does not render any requests.
>
> Dumping `/threads` indicates the observer might be backlogged by the hundred
> concurrent `TaskResourceMonitor` threads. Due to the Python GIL only one can
> make progress at a time though.
>
> This patch is now adding options to control the resource collection interval,
> giving operators a possibility to reduce the CPU pressure.
>
> --
>
> Reviewer notes: I dropped the entrypoint test as it was heavily relying on mocks
> without significant code coverage.
>
>
> Diffs
> -----
>
> RELEASE-NOTES.md d58d2bd2558c2060aa659402ea88271bd8704cde
> docs/README.md 1d679e27cf26122c34a9157194f189ce3035b176
> docs/reference/observer-configuration.md PRE-CREATION
> src/main/python/apache/aurora/tools/thermos_observer.py 4bba01930434f33150b0413d20d9657b61b364c5
> src/main/python/apache/thermos/monitoring/resource.py 53d0ff1a71c27f053c59acca556c35d1e5ac91f0
> src/main/python/apache/thermos/observer/task_observer.py 1485de8faef52716f11b82a3556064de26c67427
> src/test/python/apache/aurora/tools/test_thermos_observer_entry_point.py e1c8dec5f26be1aec931b1a614ecc138ae63d9de
>
>
> Diff: https://reviews.apache.org/r/57757/diff/4/
>
>
> Testing
> -------
>
> ./pants test.pytest src/{test,main}/python:: -- -v
>
>
> Thanks,
>
> Stephan Erb
>
>
Re: Review Request 57757: Make Thermos observer resource collection
intervals configurable
Posted by Stephan Erb <se...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57757/
-----------------------------------------------------------
(Updated March 20, 2017, 11:36 p.m.)
Review request for Aurora, Santhosh Kumar Shanmugham and Zameer Manji.
Changes
-------
Remove additonal layer of defaults.
Bugs: AURORA-1907
https://issues.apache.org/jira/browse/AURORA-1907
Repository: aurora
Description
-------
We have noticed that on hosts with lots of active tasks (~100) the observer UI
is not usable. Thermos fully utilizes one core but does not render any requests.
Dumping `/threads` indicates the observer might be backlogged by the hundred
concurrent `TaskResourceMonitor` threads. Due to the Python GIL only one can
make progress at a time though.
This patch is now adding options to control the resource collection interval,
giving operators a possibility to reduce the CPU pressure.
--
Reviewer notes: I dropped the entrypoint test as it was heavily relying on mocks
without significant code coverage.
Diffs (updated)
-----
RELEASE-NOTES.md d58d2bd2558c2060aa659402ea88271bd8704cde
docs/README.md 1d679e27cf26122c34a9157194f189ce3035b176
docs/reference/observer-configuration.md PRE-CREATION
src/main/python/apache/aurora/tools/thermos_observer.py 4bba01930434f33150b0413d20d9657b61b364c5
src/main/python/apache/thermos/monitoring/resource.py 53d0ff1a71c27f053c59acca556c35d1e5ac91f0
src/main/python/apache/thermos/observer/task_observer.py 1485de8faef52716f11b82a3556064de26c67427
src/test/python/apache/aurora/tools/test_thermos_observer_entry_point.py e1c8dec5f26be1aec931b1a614ecc138ae63d9de
Diff: https://reviews.apache.org/r/57757/diff/4/
Changes: https://reviews.apache.org/r/57757/diff/3-4/
Testing
-------
./pants test.pytest src/{test,main}/python:: -- -v
Thanks,
Stephan Erb
Re: Review Request 57757: Make Thermos observer resource collection
intervals configurable
Posted by Zameer Manji <zm...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57757/#review169501
-----------------------------------------------------------
Ship it!
Ship It!
- Zameer Manji
On March 20, 2017, 1:41 p.m., Stephan Erb wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57757/
> -----------------------------------------------------------
>
> (Updated March 20, 2017, 1:41 p.m.)
>
>
> Review request for Aurora, Santhosh Kumar Shanmugham and Zameer Manji.
>
>
> Bugs: AURORA-1907
> https://issues.apache.org/jira/browse/AURORA-1907
>
>
> Repository: aurora
>
>
> Description
> -------
>
> We have noticed that on hosts with lots of active tasks (~100) the observer UI
> is not usable. Thermos fully utilizes one core but does not render any requests.
>
> Dumping `/threads` indicates the observer might be backlogged by the hundred
> concurrent `TaskResourceMonitor` threads. Due to the Python GIL only one can
> make progress at a time though.
>
> This patch is now adding options to control the resource collection interval,
> giving operators a possibility to reduce the CPU pressure.
>
> --
>
> Reviewer notes: I dropped the entrypoint test as it was heavily relying on mocks
> without significant code coverage.
>
>
> Diffs
> -----
>
> RELEASE-NOTES.md d58d2bd2558c2060aa659402ea88271bd8704cde
> docs/README.md 1d679e27cf26122c34a9157194f189ce3035b176
> docs/reference/observer-configuration.md PRE-CREATION
> src/main/python/apache/aurora/tools/thermos_observer.py 4bba01930434f33150b0413d20d9657b61b364c5
> src/main/python/apache/thermos/monitoring/resource.py 53d0ff1a71c27f053c59acca556c35d1e5ac91f0
> src/main/python/apache/thermos/observer/task_observer.py 1485de8faef52716f11b82a3556064de26c67427
> src/test/python/apache/aurora/tools/test_thermos_observer_entry_point.py e1c8dec5f26be1aec931b1a614ecc138ae63d9de
>
>
> Diff: https://reviews.apache.org/r/57757/diff/3/
>
>
> Testing
> -------
>
> ./pants test.pytest src/{test,main}/python:: -- -v
>
>
> Thanks,
>
> Stephan Erb
>
>
Re: Review Request 57757: Make Thermos observer resource collection
intervals configurable
Posted by Santhosh Kumar Shanmugham <sa...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57757/#review169511
-----------------------------------------------------------
Ship it!
Ship It!
- Santhosh Kumar Shanmugham
On March 20, 2017, 1:41 p.m., Stephan Erb wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57757/
> -----------------------------------------------------------
>
> (Updated March 20, 2017, 1:41 p.m.)
>
>
> Review request for Aurora, Santhosh Kumar Shanmugham and Zameer Manji.
>
>
> Bugs: AURORA-1907
> https://issues.apache.org/jira/browse/AURORA-1907
>
>
> Repository: aurora
>
>
> Description
> -------
>
> We have noticed that on hosts with lots of active tasks (~100) the observer UI
> is not usable. Thermos fully utilizes one core but does not render any requests.
>
> Dumping `/threads` indicates the observer might be backlogged by the hundred
> concurrent `TaskResourceMonitor` threads. Due to the Python GIL only one can
> make progress at a time though.
>
> This patch is now adding options to control the resource collection interval,
> giving operators a possibility to reduce the CPU pressure.
>
> --
>
> Reviewer notes: I dropped the entrypoint test as it was heavily relying on mocks
> without significant code coverage.
>
>
> Diffs
> -----
>
> RELEASE-NOTES.md d58d2bd2558c2060aa659402ea88271bd8704cde
> docs/README.md 1d679e27cf26122c34a9157194f189ce3035b176
> docs/reference/observer-configuration.md PRE-CREATION
> src/main/python/apache/aurora/tools/thermos_observer.py 4bba01930434f33150b0413d20d9657b61b364c5
> src/main/python/apache/thermos/monitoring/resource.py 53d0ff1a71c27f053c59acca556c35d1e5ac91f0
> src/main/python/apache/thermos/observer/task_observer.py 1485de8faef52716f11b82a3556064de26c67427
> src/test/python/apache/aurora/tools/test_thermos_observer_entry_point.py e1c8dec5f26be1aec931b1a614ecc138ae63d9de
>
>
> Diff: https://reviews.apache.org/r/57757/diff/3/
>
>
> Testing
> -------
>
> ./pants test.pytest src/{test,main}/python:: -- -v
>
>
> Thanks,
>
> Stephan Erb
>
>
Re: Review Request 57757: Make Thermos observer resource collection
intervals configurable
Posted by Aurora ReviewBot <wf...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57757/#review169502
-----------------------------------------------------------
Master (b8f72d1) is red with this patch.
./build-support/jenkins/build.sh
Test coverage missing for org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl$8
Test coverage missing for org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl$7
Test coverage missing for org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl$9
Test coverage missing for org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl$4
Test coverage missing for org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl$3
Test coverage missing for org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl$6
Test coverage missing for org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl$5
Test coverage missing for org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl$2
Test coverage missing for org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl$1
Test coverage missing for org/apache/aurora/scheduler/storage/log/LogStorage$Settings
Test coverage missing for org/apache/aurora/scheduler/storage/log/LogStorage$ScheduledExecutorSchedulingService
Test coverage missing for org/apache/aurora/scheduler/storage/log/LogStorageModule
Test coverage missing for org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl$SnapshotField
Test coverage missing for org/apache/aurora/scheduler/storage/backup/TemporaryStorage$TemporaryStorageFactory$1
Test coverage missing for org/apache/aurora/scheduler/storage/backup/BackupModule
Test coverage missing for org/apache/aurora/scheduler/storage/backup/Recovery$RecoveryImpl
Test coverage missing for org/apache/aurora/scheduler/storage/backup/TemporaryStorage$TemporaryStorageFactory
Test coverage missing for org/apache/aurora/scheduler/storage/backup/Recovery$RecoveryImpl$PendingRecovery
Test coverage missing for org/apache/aurora/scheduler/TaskVars
Test coverage missing for org/apache/aurora/scheduler/SchedulerLifecycle$DefaultDelayedActions
Test coverage missing for org/apache/aurora/scheduler/TierManager$TierManagerImpl$TierConfig
Test coverage missing for org/apache/aurora/scheduler/TaskVars$Counter
Test coverage missing for org/apache/aurora/scheduler/TaskVars$1
Test coverage missing for org/apache/aurora/scheduler/SchedulerModule$TaskEventBatchWorker
Test coverage missing for org/apache/aurora/scheduler/HostOffer$1
Test coverage missing for org/apache/aurora/scheduler/SchedulerModule
Test coverage missing for org/apache/aurora/scheduler/TaskIdGenerator$TaskIdGeneratorImpl
Test coverage missing for org/apache/aurora/scheduler/SchedulerModule$1
Test coverage missing for org/apache/aurora/scheduler/TaskStatusHandlerImpl
Test coverage missing for org/apache/aurora/scheduler/TaskStatusHandlerImpl$1
Test coverage missing for org/apache/aurora/scheduler/TierModule
Test coverage missing for org/apache/aurora/scheduler/storage/db/typehandlers/VolumeModeTypeHandler
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
==============================================================================
BUILD FAILED
Total time: 6 mins 41.365 secs
I will refresh this build result if you post a review containing "@ReviewBot retry"
- Aurora ReviewBot
On March 20, 2017, 8:41 p.m., Stephan Erb wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57757/
> -----------------------------------------------------------
>
> (Updated March 20, 2017, 8:41 p.m.)
>
>
> Review request for Aurora, Santhosh Kumar Shanmugham and Zameer Manji.
>
>
> Bugs: AURORA-1907
> https://issues.apache.org/jira/browse/AURORA-1907
>
>
> Repository: aurora
>
>
> Description
> -------
>
> We have noticed that on hosts with lots of active tasks (~100) the observer UI
> is not usable. Thermos fully utilizes one core but does not render any requests.
>
> Dumping `/threads` indicates the observer might be backlogged by the hundred
> concurrent `TaskResourceMonitor` threads. Due to the Python GIL only one can
> make progress at a time though.
>
> This patch is now adding options to control the resource collection interval,
> giving operators a possibility to reduce the CPU pressure.
>
> --
>
> Reviewer notes: I dropped the entrypoint test as it was heavily relying on mocks
> without significant code coverage.
>
>
> Diffs
> -----
>
> RELEASE-NOTES.md d58d2bd2558c2060aa659402ea88271bd8704cde
> docs/README.md 1d679e27cf26122c34a9157194f189ce3035b176
> docs/reference/observer-configuration.md PRE-CREATION
> src/main/python/apache/aurora/tools/thermos_observer.py 4bba01930434f33150b0413d20d9657b61b364c5
> src/main/python/apache/thermos/monitoring/resource.py 53d0ff1a71c27f053c59acca556c35d1e5ac91f0
> src/main/python/apache/thermos/observer/task_observer.py 1485de8faef52716f11b82a3556064de26c67427
> src/test/python/apache/aurora/tools/test_thermos_observer_entry_point.py e1c8dec5f26be1aec931b1a614ecc138ae63d9de
>
>
> Diff: https://reviews.apache.org/r/57757/diff/3/
>
>
> Testing
> -------
>
> ./pants test.pytest src/{test,main}/python:: -- -v
>
>
> Thanks,
>
> Stephan Erb
>
>
Re: Review Request 57757: Make Thermos observer resource collection
intervals configurable
Posted by Zameer Manji <zm...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57757/#review169497
-----------------------------------------------------------
Ship it!
Ship It!
- Zameer Manji
On March 20, 2017, 1:41 p.m., Stephan Erb wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57757/
> -----------------------------------------------------------
>
> (Updated March 20, 2017, 1:41 p.m.)
>
>
> Review request for Aurora, Santhosh Kumar Shanmugham and Zameer Manji.
>
>
> Bugs: AURORA-1907
> https://issues.apache.org/jira/browse/AURORA-1907
>
>
> Repository: aurora
>
>
> Description
> -------
>
> We have noticed that on hosts with lots of active tasks (~100) the observer UI
> is not usable. Thermos fully utilizes one core but does not render any requests.
>
> Dumping `/threads` indicates the observer might be backlogged by the hundred
> concurrent `TaskResourceMonitor` threads. Due to the Python GIL only one can
> make progress at a time though.
>
> This patch is now adding options to control the resource collection interval,
> giving operators a possibility to reduce the CPU pressure.
>
> --
>
> Reviewer notes: I dropped the entrypoint test as it was heavily relying on mocks
> without significant code coverage.
>
>
> Diffs
> -----
>
> RELEASE-NOTES.md d58d2bd2558c2060aa659402ea88271bd8704cde
> docs/README.md 1d679e27cf26122c34a9157194f189ce3035b176
> docs/reference/observer-configuration.md PRE-CREATION
> src/main/python/apache/aurora/tools/thermos_observer.py 4bba01930434f33150b0413d20d9657b61b364c5
> src/main/python/apache/thermos/monitoring/resource.py 53d0ff1a71c27f053c59acca556c35d1e5ac91f0
> src/main/python/apache/thermos/observer/task_observer.py 1485de8faef52716f11b82a3556064de26c67427
> src/test/python/apache/aurora/tools/test_thermos_observer_entry_point.py e1c8dec5f26be1aec931b1a614ecc138ae63d9de
>
>
> Diff: https://reviews.apache.org/r/57757/diff/3/
>
>
> Testing
> -------
>
> ./pants test.pytest src/{test,main}/python:: -- -v
>
>
> Thanks,
>
> Stephan Erb
>
>
Re: Review Request 57757: Make Thermos observer resource collection
intervals configurable
Posted by Stephan Erb <se...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57757/
-----------------------------------------------------------
(Updated March 20, 2017, 9:41 p.m.)
Review request for Aurora, Santhosh Kumar Shanmugham and Zameer Manji.
Bugs: AURORA-1907
https://issues.apache.org/jira/browse/AURORA-1907
Repository: aurora
Description
-------
We have noticed that on hosts with lots of active tasks (~100) the observer UI
is not usable. Thermos fully utilizes one core but does not render any requests.
Dumping `/threads` indicates the observer might be backlogged by the hundred
concurrent `TaskResourceMonitor` threads. Due to the Python GIL only one can
make progress at a time though.
This patch is now adding options to control the resource collection interval,
giving operators a possibility to reduce the CPU pressure.
--
Reviewer notes: I dropped the entrypoint test as it was heavily relying on mocks
without significant code coverage.
Diffs (updated)
-----
RELEASE-NOTES.md d58d2bd2558c2060aa659402ea88271bd8704cde
docs/README.md 1d679e27cf26122c34a9157194f189ce3035b176
docs/reference/observer-configuration.md PRE-CREATION
src/main/python/apache/aurora/tools/thermos_observer.py 4bba01930434f33150b0413d20d9657b61b364c5
src/main/python/apache/thermos/monitoring/resource.py 53d0ff1a71c27f053c59acca556c35d1e5ac91f0
src/main/python/apache/thermos/observer/task_observer.py 1485de8faef52716f11b82a3556064de26c67427
src/test/python/apache/aurora/tools/test_thermos_observer_entry_point.py e1c8dec5f26be1aec931b1a614ecc138ae63d9de
Diff: https://reviews.apache.org/r/57757/diff/3/
Changes: https://reviews.apache.org/r/57757/diff/2-3/
Testing
-------
./pants test.pytest src/{test,main}/python:: -- -v
Thanks,
Stephan Erb
Re: Review Request 57757: Make Thermos observer resource collection
intervals configurable
Posted by Stephan Erb <se...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57757/
-----------------------------------------------------------
(Updated March 19, 2017, 11:50 p.m.)
Review request for Aurora, Santhosh Kumar Shanmugham and Zameer Manji.
Summary (updated)
-----------------
Make Thermos observer resource collection intervals configurable
Bugs: AURORA-1907
https://issues.apache.org/jira/browse/AURORA-1907
Repository: aurora
Description (updated)
-------
We have noticed that on hosts with lots of active tasks (~100) the observer UI
is not usable. Thermos fully utilizes one core but does not render any requests.
Dumping `/threads` indicates the observer might be backlogged by the hundred
concurrent `TaskResourceMonitor` threads. Due to the Python GIL only one can
make progress at a time though.
This patch is now adding options to control the resource collection interval,
giving operators a possibility to reduce the CPU pressure.
--
Reviewer notes: I dropped the entrypoint test as it was heavily relying on mocks
without significant code coverage.
Diffs
-----
RELEASE-NOTES.md d58d2bd2558c2060aa659402ea88271bd8704cde
docs/README.md 1d679e27cf26122c34a9157194f189ce3035b176
docs/reference/observer-configuration.md PRE-CREATION
src/main/python/apache/aurora/tools/thermos_observer.py 4bba01930434f33150b0413d20d9657b61b364c5
src/main/python/apache/thermos/monitoring/resource.py 53d0ff1a71c27f053c59acca556c35d1e5ac91f0
src/main/python/apache/thermos/observer/task_observer.py 1485de8faef52716f11b82a3556064de26c67427
src/test/python/apache/aurora/tools/test_thermos_observer_entry_point.py e1c8dec5f26be1aec931b1a614ecc138ae63d9de
Diff: https://reviews.apache.org/r/57757/diff/2/
Testing
-------
./pants test.pytest src/{test,main}/python:: -- -v
Thanks,
Stephan Erb
Re: Review Request 57757: Make Thermos resource collection intervals
configurable
Posted by Aurora ReviewBot <wf...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57757/#review169380
-----------------------------------------------------------
Ship it!
Master (b8f72d1) is green with this patch.
./build-support/jenkins/build.sh
I will refresh this build result if you post a review containing "@ReviewBot retry"
- Aurora ReviewBot
On March 19, 2017, 4:34 p.m., Stephan Erb wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57757/
> -----------------------------------------------------------
>
> (Updated March 19, 2017, 4:34 p.m.)
>
>
> Review request for Aurora, Santhosh Kumar Shanmugham and Zameer Manji.
>
>
> Bugs: AURORA-1907
> https://issues.apache.org/jira/browse/AURORA-1907
>
>
> Repository: aurora
>
>
> Description
> -------
>
> We have noticed that on hosts with lots of active tasks (~100) the Thermos UI
> is not usable. Thermos fully utilizes one core but does not render any requests.
>
> Dumping `/threads` indicates the observer might be backlogged by the hundred
> concurrent `TaskResourceMonitor` threads. Due to the Python GIL only one can
> make progress at a time though.
>
> This patch is now adding options to control the resource collection interval,
> giving operators a possibility to reduce the CPU pressure.
>
> --
>
> Reviewer notes: I dropped the entrypoint test as it was heavily relying on mocks
> without significant code coverage.
>
>
> Diffs
> -----
>
> RELEASE-NOTES.md d58d2bd2558c2060aa659402ea88271bd8704cde
> docs/README.md 1d679e27cf26122c34a9157194f189ce3035b176
> docs/reference/observer-configuration.md PRE-CREATION
> src/main/python/apache/aurora/tools/thermos_observer.py 4bba01930434f33150b0413d20d9657b61b364c5
> src/main/python/apache/thermos/monitoring/resource.py 53d0ff1a71c27f053c59acca556c35d1e5ac91f0
> src/main/python/apache/thermos/observer/task_observer.py 1485de8faef52716f11b82a3556064de26c67427
> src/test/python/apache/aurora/tools/test_thermos_observer_entry_point.py e1c8dec5f26be1aec931b1a614ecc138ae63d9de
>
>
> Diff: https://reviews.apache.org/r/57757/diff/2/
>
>
> Testing
> -------
>
> ./pants test.pytest src/{test,main}/python:: -- -v
>
>
> Thanks,
>
> Stephan Erb
>
>
Re: Review Request 57757: Make Thermos resource collection intervals
configurable
Posted by Stephan Erb <se...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57757/
-----------------------------------------------------------
(Updated March 19, 2017, 5:34 p.m.)
Review request for Aurora, Santhosh Kumar Shanmugham and Zameer Manji.
Changes
-------
Style fixes
Bugs: AURORA-1907
https://issues.apache.org/jira/browse/AURORA-1907
Repository: aurora
Description
-------
We have noticed that on hosts with lots of active tasks (~100) the Thermos UI
is not usable. Thermos fully utilizes one core but does not render any requests.
Dumping `/threads` indicates the observer might be backlogged by the hundred
concurrent `TaskResourceMonitor` threads. Due to the Python GIL only one can
make progress at a time though.
This patch is now adding options to control the resource collection interval,
giving operators a possibility to reduce the CPU pressure.
--
Reviewer notes: I dropped the entrypoint test as it was heavily relying on mocks
without significant code coverage.
Diffs (updated)
-----
RELEASE-NOTES.md d58d2bd2558c2060aa659402ea88271bd8704cde
docs/README.md 1d679e27cf26122c34a9157194f189ce3035b176
docs/reference/observer-configuration.md PRE-CREATION
src/main/python/apache/aurora/tools/thermos_observer.py 4bba01930434f33150b0413d20d9657b61b364c5
src/main/python/apache/thermos/monitoring/resource.py 53d0ff1a71c27f053c59acca556c35d1e5ac91f0
src/main/python/apache/thermos/observer/task_observer.py 1485de8faef52716f11b82a3556064de26c67427
src/test/python/apache/aurora/tools/test_thermos_observer_entry_point.py e1c8dec5f26be1aec931b1a614ecc138ae63d9de
Diff: https://reviews.apache.org/r/57757/diff/2/
Changes: https://reviews.apache.org/r/57757/diff/1-2/
Testing
-------
./pants test.pytest src/{test,main}/python:: -- -v
Thanks,
Stephan Erb
Re: Review Request 57757: Make Thermos resource collection intervals
configurable
Posted by Aurora ReviewBot <wf...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57757/#review169376
-----------------------------------------------------------
Master (b8f72d1) is red with this patch.
./build-support/jenkins/build.sh
virtualenv-15.0.2/virtualenv_embedded/activate.ps1
virtualenv-15.0.2/virtualenv_embedded/activate.sh
virtualenv-15.0.2/virtualenv_embedded/activate_this.py
virtualenv-15.0.2/virtualenv_embedded/deactivate.bat
virtualenv-15.0.2/virtualenv_embedded/distutils-init.py
virtualenv-15.0.2/virtualenv_embedded/distutils.cfg
virtualenv-15.0.2/virtualenv_embedded/python-config
virtualenv-15.0.2/virtualenv_embedded/site.py
virtualenv-15.0.2/virtualenv_support/
virtualenv-15.0.2/virtualenv_support/__init__.py
virtualenv-15.0.2/virtualenv_support/argparse-1.4.0-py2.py3-none-any.whl
virtualenv-15.0.2/virtualenv_support/pip-8.1.2-py2.py3-none-any.whl
virtualenv-15.0.2/virtualenv_support/setuptools-21.2.1-py2.py3-none-any.whl
virtualenv-15.0.2/virtualenv_support/wheel-0.29.0-py2.py3-none-any.whl
+ touch virtualenv-15.0.2/BOOTSTRAPPED
+ popd
/home/jenkins/jenkins-slave/workspace/AuroraBot
+ exec /usr/bin/python2.7 /home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/virtualenv-15.0.2/virtualenv.py --no-download /home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/isort.venv
New python executable in /home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/isort.venv/bin/python2.7
Also creating executable in /home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/isort.venv/bin/python
Installing setuptools, pip, wheel...done.
Collecting isort==4.0.0
Downloading isort-4.0.0-py2.py3-none-any.whl
Installing collected packages: isort
Successfully installed isort-4.0.0
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
ERROR: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/thermos/observer/task_observer.py Imports are incorrectly sorted.
--- /home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/thermos/observer/task_observer.py:before 2017-03-19 16:21:04.909082
+++ /home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/thermos/observer/task_observer.py:after 2017-03-19 16:26:20.269092
@@ -20,8 +20,8 @@
"""
import os
+import threading
import time
-import threading
from operator import attrgetter
from twitter.common import log
I will refresh this build result if you post a review containing "@ReviewBot retry"
- Aurora ReviewBot
On March 19, 2017, 4:18 p.m., Stephan Erb wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57757/
> -----------------------------------------------------------
>
> (Updated March 19, 2017, 4:18 p.m.)
>
>
> Review request for Aurora, Santhosh Kumar Shanmugham and Zameer Manji.
>
>
> Bugs: AURORA-1907
> https://issues.apache.org/jira/browse/AURORA-1907
>
>
> Repository: aurora
>
>
> Description
> -------
>
> We have noticed that on hosts with lots of active tasks (~100) the Thermos UI
> is not usable. Thermos fully utilizes one core but does not render any requests.
>
> Dumping `/threads` indicates the observer might be backlogged by the hundred
> concurrent `TaskResourceMonitor` threads. Due to the Python GIL only one can
> make progress at a time though.
>
> This patch is now adding options to control the resource collection interval,
> giving operators a possibility to reduce the CPU pressure.
>
> --
>
> Reviewer notes: I dropped the entrypoint test as it was heavily relying on mocks
> without significant code coverage.
>
>
> Diffs
> -----
>
> RELEASE-NOTES.md d58d2bd2558c2060aa659402ea88271bd8704cde
> docs/README.md 1d679e27cf26122c34a9157194f189ce3035b176
> docs/reference/observer-configuration.md PRE-CREATION
> src/main/python/apache/aurora/tools/thermos_observer.py 4bba01930434f33150b0413d20d9657b61b364c5
> src/main/python/apache/thermos/monitoring/resource.py 53d0ff1a71c27f053c59acca556c35d1e5ac91f0
> src/main/python/apache/thermos/observer/detector.py 79bc6329caf73fbfbb4ebf4bbad96bb1c50be1f9
> src/main/python/apache/thermos/observer/task_observer.py 1485de8faef52716f11b82a3556064de26c67427
> src/test/python/apache/aurora/tools/test_thermos_observer_entry_point.py e1c8dec5f26be1aec931b1a614ecc138ae63d9de
>
>
> Diff: https://reviews.apache.org/r/57757/diff/1/
>
>
> Testing
> -------
>
> ./pants test.pytest src/{test,main}/python:: -- -v
>
>
> Thanks,
>
> Stephan Erb
>
>