You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@aurora.apache.org by David McLaughlin <da...@dmclaughlin.com> on 2018/04/26 17:51:39 UTC
Review Request 66806: Breakdown resource stats by role
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66806/
-----------------------------------------------------------
Review request for Aurora, Jordan Ly and Santhosh Kumar Shanmugham.
Repository: aurora
Description
-------
Currently Aurora exports total quota and resource reservation over time. This can be very useful to see changes in trends of production and free tier capacity. One challenge (particularly in a self-serve capacity environment) is identifying and tracking where large deltas came from. This change exports both quota and resource usage per role to help with this.
It is possible to make this more efficient by refactoring the current abstractions to do both totals and per-role metrics in a single pass. But given this only runs once per hour, I went for the cleaner/simpler approach.
Diffs
-----
src/main/java/org/apache/aurora/scheduler/stats/ResourceCounter.java a3e9bc75d5ce48cc3d64aa6b650df708f2f7c916
src/main/java/org/apache/aurora/scheduler/stats/TaskStatCalculator.java ac5cf2462cb0f383493296b541348bff40e99025
src/test/java/org/apache/aurora/scheduler/stats/ResourceCounterTest.java a30d74e73774a9fd4224839d08ebfe7fb0df095a
Diff: https://reviews.apache.org/r/66806/diff/1/
Testing
-------
/vars in Vagrant:
quota_per_role_www-data_cpu_cores 10
quota_per_role_www-data_disk_mb 102400
quota_per_role_www-data_ram_mb 102400
resources_allocated_quota_cpu_cores 0
resources_allocated_quota_disk_mb 0
resources_allocated_quota_ram_mb 0
resources_dedicated_consumed_cpu_cores 0
resources_dedicated_consumed_disk_mb 0
resources_dedicated_consumed_ram_mb 0
resources_free_pool_consumed_cpu_cores 1
resources_free_pool_consumed_disk_mb 8
resources_free_pool_consumed_ram_mb 1
resources_per_role_free_pool_consumed_www-data_cpu_cores 1
resources_per_role_free_pool_consumed_www-data_disk_mb 8
resources_per_role_free_pool_consumed_www-data_ram_mb 1
resources_per_role_total_consumed_www-data_cpu_cores 1
resources_per_role_total_consumed_www-data_disk_mb 8
resources_per_role_total_consumed_www-data_ram_mb 1
resources_quota_consumed_cpu_cores 0
resources_quota_consumed_disk_mb 0
resources_quota_consumed_ram_mb 0
resources_total_consumed_cpu_cores 1
resources_total_consumed_disk_mb 8
resources_total_consumed_ram_mb 1
Thanks,
David McLaughlin
Re: Review Request 66806: Breakdown resource stats by role
Posted by Santhosh Kumar Shanmugham <sa...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66806/#review202021
-----------------------------------------------------------
Ship it!
Ship It!
- Santhosh Kumar Shanmugham
On April 26, 2018, 10:51 a.m., David McLaughlin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66806/
> -----------------------------------------------------------
>
> (Updated April 26, 2018, 10:51 a.m.)
>
>
> Review request for Aurora, Jordan Ly and Santhosh Kumar Shanmugham.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> Currently Aurora exports total quota and resource reservation over time. This can be very useful to see changes in trends of production and free tier capacity. One challenge (particularly in a self-serve capacity environment) is identifying and tracking where large deltas came from. This change exports both quota and resource usage per role to help with this.
>
> It is possible to make this more efficient by refactoring the current abstractions to do both totals and per-role metrics in a single pass. But given this only runs once per hour, I went for the cleaner/simpler approach.
>
>
> Diffs
> -----
>
> src/main/java/org/apache/aurora/scheduler/stats/ResourceCounter.java a3e9bc75d5ce48cc3d64aa6b650df708f2f7c916
> src/main/java/org/apache/aurora/scheduler/stats/TaskStatCalculator.java ac5cf2462cb0f383493296b541348bff40e99025
> src/test/java/org/apache/aurora/scheduler/stats/ResourceCounterTest.java a30d74e73774a9fd4224839d08ebfe7fb0df095a
>
>
> Diff: https://reviews.apache.org/r/66806/diff/1/
>
>
> Testing
> -------
>
> /vars in Vagrant:
>
> quota_per_role_www-data_cpu_cores 10
> quota_per_role_www-data_disk_mb 102400
> quota_per_role_www-data_ram_mb 102400
> resources_allocated_quota_cpu_cores 0
> resources_allocated_quota_disk_mb 0
> resources_allocated_quota_ram_mb 0
> resources_dedicated_consumed_cpu_cores 0
> resources_dedicated_consumed_disk_mb 0
> resources_dedicated_consumed_ram_mb 0
> resources_free_pool_consumed_cpu_cores 1
> resources_free_pool_consumed_disk_mb 8
> resources_free_pool_consumed_ram_mb 1
> resources_per_role_free_pool_consumed_www-data_cpu_cores 1
> resources_per_role_free_pool_consumed_www-data_disk_mb 8
> resources_per_role_free_pool_consumed_www-data_ram_mb 1
> resources_per_role_total_consumed_www-data_cpu_cores 1
> resources_per_role_total_consumed_www-data_disk_mb 8
> resources_per_role_total_consumed_www-data_ram_mb 1
> resources_quota_consumed_cpu_cores 0
> resources_quota_consumed_disk_mb 0
> resources_quota_consumed_ram_mb 0
> resources_total_consumed_cpu_cores 1
> resources_total_consumed_disk_mb 8
> resources_total_consumed_ram_mb 1
>
>
> Thanks,
>
> David McLaughlin
>
>
Re: Review Request 66806: Breakdown resource stats by role
Posted by David McLaughlin <da...@dmclaughlin.com>.
> On April 26, 2018, 6:17 p.m., Jordan Ly wrote:
> > src/main/java/org/apache/aurora/scheduler/stats/ResourceCounter.java
> > Lines 101-103 (patched)
> > <https://reviews.apache.org/r/66806/diff/1/?file=2012427#file2012427line101>
> >
> > super nit:
> >
> > to match with the above, this should probably be:
> >
> > ```
> > Computes quota allocations by role.
> >
> > @returns Allocated quota by role.
> > ```
> >
> > Not a huge deal though
Fixed.
> On April 26, 2018, 6:17 p.m., Jordan Ly wrote:
> > src/main/java/org/apache/aurora/scheduler/stats/TaskStatCalculator.java
> > Lines 64 (patched)
> > <https://reviews.apache.org/r/66806/diff/1/?file=2012428#file2012428line64>
> >
> > rename from `test` to something more descriptive
Removed intermediate variable.
- David
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66806/#review202020
-----------------------------------------------------------
On April 26, 2018, 9:10 p.m., David McLaughlin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66806/
> -----------------------------------------------------------
>
> (Updated April 26, 2018, 9:10 p.m.)
>
>
> Review request for Aurora, Jordan Ly and Santhosh Kumar Shanmugham.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> Currently Aurora exports total quota and resource reservation over time. This can be very useful to see changes in trends of production and free tier capacity. One challenge (particularly in a self-serve capacity environment) is identifying and tracking where large deltas came from. This change exports both quota and resource usage per role to help with this.
>
> It is possible to make this more efficient by refactoring the current abstractions to do both totals and per-role metrics in a single pass. But given this only runs once per hour, I went for the cleaner/simpler approach.
>
>
> Diffs
> -----
>
> src/main/java/org/apache/aurora/scheduler/stats/ResourceCounter.java a3e9bc75d5ce48cc3d64aa6b650df708f2f7c916
> src/main/java/org/apache/aurora/scheduler/stats/TaskStatCalculator.java ac5cf2462cb0f383493296b541348bff40e99025
> src/test/java/org/apache/aurora/scheduler/stats/ResourceCounterTest.java a30d74e73774a9fd4224839d08ebfe7fb0df095a
>
>
> Diff: https://reviews.apache.org/r/66806/diff/2/
>
>
> Testing
> -------
>
> /vars in Vagrant:
>
> quota_per_role_www-data_cpu_cores 10
> quota_per_role_www-data_disk_mb 102400
> quota_per_role_www-data_ram_mb 102400
> resources_allocated_quota_cpu_cores 0
> resources_allocated_quota_disk_mb 0
> resources_allocated_quota_ram_mb 0
> resources_dedicated_consumed_cpu_cores 0
> resources_dedicated_consumed_disk_mb 0
> resources_dedicated_consumed_ram_mb 0
> resources_free_pool_consumed_cpu_cores 1
> resources_free_pool_consumed_disk_mb 8
> resources_free_pool_consumed_ram_mb 1
> resources_per_role_free_pool_consumed_www-data_cpu_cores 1
> resources_per_role_free_pool_consumed_www-data_disk_mb 8
> resources_per_role_free_pool_consumed_www-data_ram_mb 1
> resources_per_role_total_consumed_www-data_cpu_cores 1
> resources_per_role_total_consumed_www-data_disk_mb 8
> resources_per_role_total_consumed_www-data_ram_mb 1
> resources_quota_consumed_cpu_cores 0
> resources_quota_consumed_disk_mb 0
> resources_quota_consumed_ram_mb 0
> resources_total_consumed_cpu_cores 1
> resources_total_consumed_disk_mb 8
> resources_total_consumed_ram_mb 1
>
>
> Thanks,
>
> David McLaughlin
>
>
Re: Review Request 66806: Breakdown resource stats by role
Posted by Jordan Ly <jo...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66806/#review202020
-----------------------------------------------------------
Fix it, then Ship it!
src/main/java/org/apache/aurora/scheduler/stats/ResourceCounter.java
Lines 101-103 (patched)
<https://reviews.apache.org/r/66806/#comment283680>
super nit:
to match with the above, this should probably be:
```
Computes quota allocations by role.
@returns Allocated quota by role.
```
Not a huge deal though
src/main/java/org/apache/aurora/scheduler/stats/TaskStatCalculator.java
Lines 64 (patched)
<https://reviews.apache.org/r/66806/#comment283681>
rename from `test` to something more descriptive
- Jordan Ly
On April 26, 2018, 5:51 p.m., David McLaughlin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66806/
> -----------------------------------------------------------
>
> (Updated April 26, 2018, 5:51 p.m.)
>
>
> Review request for Aurora, Jordan Ly and Santhosh Kumar Shanmugham.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> Currently Aurora exports total quota and resource reservation over time. This can be very useful to see changes in trends of production and free tier capacity. One challenge (particularly in a self-serve capacity environment) is identifying and tracking where large deltas came from. This change exports both quota and resource usage per role to help with this.
>
> It is possible to make this more efficient by refactoring the current abstractions to do both totals and per-role metrics in a single pass. But given this only runs once per hour, I went for the cleaner/simpler approach.
>
>
> Diffs
> -----
>
> src/main/java/org/apache/aurora/scheduler/stats/ResourceCounter.java a3e9bc75d5ce48cc3d64aa6b650df708f2f7c916
> src/main/java/org/apache/aurora/scheduler/stats/TaskStatCalculator.java ac5cf2462cb0f383493296b541348bff40e99025
> src/test/java/org/apache/aurora/scheduler/stats/ResourceCounterTest.java a30d74e73774a9fd4224839d08ebfe7fb0df095a
>
>
> Diff: https://reviews.apache.org/r/66806/diff/1/
>
>
> Testing
> -------
>
> /vars in Vagrant:
>
> quota_per_role_www-data_cpu_cores 10
> quota_per_role_www-data_disk_mb 102400
> quota_per_role_www-data_ram_mb 102400
> resources_allocated_quota_cpu_cores 0
> resources_allocated_quota_disk_mb 0
> resources_allocated_quota_ram_mb 0
> resources_dedicated_consumed_cpu_cores 0
> resources_dedicated_consumed_disk_mb 0
> resources_dedicated_consumed_ram_mb 0
> resources_free_pool_consumed_cpu_cores 1
> resources_free_pool_consumed_disk_mb 8
> resources_free_pool_consumed_ram_mb 1
> resources_per_role_free_pool_consumed_www-data_cpu_cores 1
> resources_per_role_free_pool_consumed_www-data_disk_mb 8
> resources_per_role_free_pool_consumed_www-data_ram_mb 1
> resources_per_role_total_consumed_www-data_cpu_cores 1
> resources_per_role_total_consumed_www-data_disk_mb 8
> resources_per_role_total_consumed_www-data_ram_mb 1
> resources_quota_consumed_cpu_cores 0
> resources_quota_consumed_disk_mb 0
> resources_quota_consumed_ram_mb 0
> resources_total_consumed_cpu_cores 1
> resources_total_consumed_disk_mb 8
> resources_total_consumed_ram_mb 1
>
>
> Thanks,
>
> David McLaughlin
>
>
Re: Review Request 66806: Breakdown resource stats by role
Posted by Aurora ReviewBot <wf...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66806/#review202023
-----------------------------------------------------------
Master (18103fe) is red with this patch.
./build-support/jenkins/build.sh
at org.apache.aurora.scheduler.TaskStatusHandlerImpl.lambda$run$0(TaskStatusHandlerImpl.java:158)
at org.apache.aurora.scheduler.storage.Storage$MutateWork$NoResult.apply(Storage.java:144)
at org.apache.aurora.scheduler.storage.Storage$MutateWork$NoResult.apply(Storage.java:139)
at org.apache.aurora.scheduler.storage.testing.StorageTestUtil.lambda$expectWrite$1(StorageTestUtil.java:83)
at org.easymock.internal.Result.answer(Result.java:106)
at org.easymock.internal.ReplayState.invokeInner(ReplayState.java:60)
at org.easymock.internal.ReplayState.invoke(ReplayState.java:46)
at org.easymock.internal.MockInvocationHandler.invoke(MockInvocationHandler.java:40)
at org.easymock.internal.ObjectMethodsFilter.invoke(ObjectMethodsFilter.java:94)
at com.sun.proxy.$Proxy20.write(Unknown Source)
at org.apache.aurora.scheduler.TaskStatusHandlerImpl.run(TaskStatusHandlerImpl.java:154)
at com.google.common.util.concurrent.AbstractExecutionThreadService$1$2.run(AbstractExecutionThreadService.java:66)
at com.google.common.util.concurrent.Callables$4.run(Callables.java:122)
at java.lang.Thread.run(Thread.java:748)
org.apache.aurora.scheduler.events.WebhookTest > testTaskChangedWithOldStateError FAILED
java.lang.AssertionError at WebhookTest.java:251
I0426 18:23:51.059 [ShutdownHook, SchedulerMain] Stopping scheduler services.
1082 tests completed, 1 failed, 1 skipped
:test FAILED
:jacocoTestReport
Coverage report generated: file:///home/jenkins/jenkins-slave/workspace/AuroraBot/dist/reports/jacoco/test/html/index.html
:jacocoTestCoverageVerification
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':test'.
> There were failing tests. See the report at: file:///home/jenkins/jenkins-slave/workspace/AuroraBot/dist/reports/tests/test/index.html
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
BUILD FAILED in 7m 39s
45 actionable tasks: 36 executed, 9 up-to-date
I will refresh this build result if you post a review containing "@ReviewBot retry"
- Aurora ReviewBot
On April 26, 2018, 10:51 a.m., David McLaughlin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66806/
> -----------------------------------------------------------
>
> (Updated April 26, 2018, 10:51 a.m.)
>
>
> Review request for Aurora, Jordan Ly and Santhosh Kumar Shanmugham.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> Currently Aurora exports total quota and resource reservation over time. This can be very useful to see changes in trends of production and free tier capacity. One challenge (particularly in a self-serve capacity environment) is identifying and tracking where large deltas came from. This change exports both quota and resource usage per role to help with this.
>
> It is possible to make this more efficient by refactoring the current abstractions to do both totals and per-role metrics in a single pass. But given this only runs once per hour, I went for the cleaner/simpler approach.
>
>
> Diffs
> -----
>
> src/main/java/org/apache/aurora/scheduler/stats/ResourceCounter.java a3e9bc75d5ce48cc3d64aa6b650df708f2f7c916
> src/main/java/org/apache/aurora/scheduler/stats/TaskStatCalculator.java ac5cf2462cb0f383493296b541348bff40e99025
> src/test/java/org/apache/aurora/scheduler/stats/ResourceCounterTest.java a30d74e73774a9fd4224839d08ebfe7fb0df095a
>
>
> Diff: https://reviews.apache.org/r/66806/diff/1/
>
>
> Testing
> -------
>
> /vars in Vagrant:
>
> quota_per_role_www-data_cpu_cores 10
> quota_per_role_www-data_disk_mb 102400
> quota_per_role_www-data_ram_mb 102400
> resources_allocated_quota_cpu_cores 0
> resources_allocated_quota_disk_mb 0
> resources_allocated_quota_ram_mb 0
> resources_dedicated_consumed_cpu_cores 0
> resources_dedicated_consumed_disk_mb 0
> resources_dedicated_consumed_ram_mb 0
> resources_free_pool_consumed_cpu_cores 1
> resources_free_pool_consumed_disk_mb 8
> resources_free_pool_consumed_ram_mb 1
> resources_per_role_free_pool_consumed_www-data_cpu_cores 1
> resources_per_role_free_pool_consumed_www-data_disk_mb 8
> resources_per_role_free_pool_consumed_www-data_ram_mb 1
> resources_per_role_total_consumed_www-data_cpu_cores 1
> resources_per_role_total_consumed_www-data_disk_mb 8
> resources_per_role_total_consumed_www-data_ram_mb 1
> resources_quota_consumed_cpu_cores 0
> resources_quota_consumed_disk_mb 0
> resources_quota_consumed_ram_mb 0
> resources_total_consumed_cpu_cores 1
> resources_total_consumed_disk_mb 8
> resources_total_consumed_ram_mb 1
>
>
> Thanks,
>
> David McLaughlin
>
>
Re: Review Request 66806: Breakdown resource stats by role
Posted by Aurora ReviewBot <wf...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66806/#review202033
-----------------------------------------------------------
Master (18103fe) is red with this patch.
./build-support/jenkins/build.sh
[191] ./src/main/resources/source-sans-pro.css 1.05 kB {0} [built]
[223] ./src/main/js/index.js 3.43 kB {0} [built]
+ 268 hidden modules
:processResources
:classes
:jar
:moreStartScripts
:startScripts
:distTar
:distZip
:assemble
:compileTestJavaNote: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/java/org/apache/aurora/scheduler/storage/durability/DurableStorageTest.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
:processTestResources
:testClasses
:compileJmhJavaNote: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/jmh/java/org/apache/aurora/benchmark/fakes/FakeSchedulerDriver.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
:processJmhResources NO-SOURCE
:jmhClasses
:checkstyleJmh
:checkstyleMain[ant:checkstyle] [ERROR] /home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/stats/TaskStatCalculator.java:16:8: Unused import - java.util.Map. [UnusedImports]
FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':checkstyleMain'.
> Checkstyle rule violations were found. See the report at: file:///home/jenkins/jenkins-slave/workspace/AuroraBot/dist/reports/checkstyle/main.html
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
BUILD FAILED in 5m 4s
32 actionable tasks: 26 executed, 6 up-to-date
I will refresh this build result if you post a review containing "@ReviewBot retry"
- Aurora ReviewBot
On April 26, 2018, 2:10 p.m., David McLaughlin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66806/
> -----------------------------------------------------------
>
> (Updated April 26, 2018, 2:10 p.m.)
>
>
> Review request for Aurora, Jordan Ly and Santhosh Kumar Shanmugham.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> Currently Aurora exports total quota and resource reservation over time. This can be very useful to see changes in trends of production and free tier capacity. One challenge (particularly in a self-serve capacity environment) is identifying and tracking where large deltas came from. This change exports both quota and resource usage per role to help with this.
>
> It is possible to make this more efficient by refactoring the current abstractions to do both totals and per-role metrics in a single pass. But given this only runs once per hour, I went for the cleaner/simpler approach.
>
>
> Diffs
> -----
>
> src/main/java/org/apache/aurora/scheduler/stats/ResourceCounter.java a3e9bc75d5ce48cc3d64aa6b650df708f2f7c916
> src/main/java/org/apache/aurora/scheduler/stats/TaskStatCalculator.java ac5cf2462cb0f383493296b541348bff40e99025
> src/test/java/org/apache/aurora/scheduler/stats/ResourceCounterTest.java a30d74e73774a9fd4224839d08ebfe7fb0df095a
>
>
> Diff: https://reviews.apache.org/r/66806/diff/2/
>
>
> Testing
> -------
>
> /vars in Vagrant:
>
> quota_per_role_www-data_cpu_cores 10
> quota_per_role_www-data_disk_mb 102400
> quota_per_role_www-data_ram_mb 102400
> resources_allocated_quota_cpu_cores 0
> resources_allocated_quota_disk_mb 0
> resources_allocated_quota_ram_mb 0
> resources_dedicated_consumed_cpu_cores 0
> resources_dedicated_consumed_disk_mb 0
> resources_dedicated_consumed_ram_mb 0
> resources_free_pool_consumed_cpu_cores 1
> resources_free_pool_consumed_disk_mb 8
> resources_free_pool_consumed_ram_mb 1
> resources_per_role_free_pool_consumed_www-data_cpu_cores 1
> resources_per_role_free_pool_consumed_www-data_disk_mb 8
> resources_per_role_free_pool_consumed_www-data_ram_mb 1
> resources_per_role_total_consumed_www-data_cpu_cores 1
> resources_per_role_total_consumed_www-data_disk_mb 8
> resources_per_role_total_consumed_www-data_ram_mb 1
> resources_quota_consumed_cpu_cores 0
> resources_quota_consumed_disk_mb 0
> resources_quota_consumed_ram_mb 0
> resources_total_consumed_cpu_cores 1
> resources_total_consumed_disk_mb 8
> resources_total_consumed_ram_mb 1
>
>
> Thanks,
>
> David McLaughlin
>
>
Re: Review Request 66806: Breakdown resource stats by role
Posted by Aurora ReviewBot <wf...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66806/#review202040
-----------------------------------------------------------
Ship it!
Master (18103fe) is green with this patch.
./build-support/jenkins/build.sh
I will refresh this build result if you post a review containing "@ReviewBot retry"
- Aurora ReviewBot
On April 26, 2018, 2:27 p.m., David McLaughlin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66806/
> -----------------------------------------------------------
>
> (Updated April 26, 2018, 2:27 p.m.)
>
>
> Review request for Aurora, Jordan Ly and Santhosh Kumar Shanmugham.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> Currently Aurora exports total quota and resource reservation over time. This can be very useful to see changes in trends of production and free tier capacity. One challenge (particularly in a self-serve capacity environment) is identifying and tracking where large deltas came from. This change exports both quota and resource usage per role to help with this.
>
> It is possible to make this more efficient by refactoring the current abstractions to do both totals and per-role metrics in a single pass. But given this only runs once per hour, I went for the cleaner/simpler approach.
>
>
> Diffs
> -----
>
> src/main/java/org/apache/aurora/scheduler/stats/ResourceCounter.java a3e9bc75d5ce48cc3d64aa6b650df708f2f7c916
> src/main/java/org/apache/aurora/scheduler/stats/TaskStatCalculator.java ac5cf2462cb0f383493296b541348bff40e99025
> src/test/java/org/apache/aurora/scheduler/stats/ResourceCounterTest.java a30d74e73774a9fd4224839d08ebfe7fb0df095a
>
>
> Diff: https://reviews.apache.org/r/66806/diff/3/
>
>
> Testing
> -------
>
> /vars in Vagrant:
>
> quota_per_role_www-data_cpu_cores 10
> quota_per_role_www-data_disk_mb 102400
> quota_per_role_www-data_ram_mb 102400
> resources_allocated_quota_cpu_cores 0
> resources_allocated_quota_disk_mb 0
> resources_allocated_quota_ram_mb 0
> resources_dedicated_consumed_cpu_cores 0
> resources_dedicated_consumed_disk_mb 0
> resources_dedicated_consumed_ram_mb 0
> resources_free_pool_consumed_cpu_cores 1
> resources_free_pool_consumed_disk_mb 8
> resources_free_pool_consumed_ram_mb 1
> resources_per_role_free_pool_consumed_www-data_cpu_cores 1
> resources_per_role_free_pool_consumed_www-data_disk_mb 8
> resources_per_role_free_pool_consumed_www-data_ram_mb 1
> resources_per_role_total_consumed_www-data_cpu_cores 1
> resources_per_role_total_consumed_www-data_disk_mb 8
> resources_per_role_total_consumed_www-data_ram_mb 1
> resources_quota_consumed_cpu_cores 0
> resources_quota_consumed_disk_mb 0
> resources_quota_consumed_ram_mb 0
> resources_total_consumed_cpu_cores 1
> resources_total_consumed_disk_mb 8
> resources_total_consumed_ram_mb 1
>
>
> Thanks,
>
> David McLaughlin
>
>
Re: Review Request 66806: Breakdown resource stats by role
Posted by David McLaughlin <da...@dmclaughlin.com>.
> On April 26, 2018, 9:38 p.m., Aurora ReviewBot wrote:
> > Master (18103fe) is red with this patch.
> > ./build-support/jenkins/build.sh
> >
> > Pass 2: Analyzing classes (320 / 333) - 96% complete
> > Pass 2: Analyzing classes (321 / 333) - 96% complete
> > Pass 2: Analyzing classes (322 / 333) - 96% complete
> > Pass 2: Analyzing classes (323 / 333) - 96% complete
> > Pass 2: Analyzing classes (324 / 333) - 97% complete
> > Pass 2: Analyzing classes (325 / 333) - 97% complete
> > Pass 2: Analyzing classes (326 / 333) - 97% complete
> > Pass 2: Analyzing classes (327 / 333) - 98% complete
> > Pass 2: Analyzing classes (328 / 333) - 98% complete
> > Pass 2: Analyzing classes (329 / 333) - 98% complete
> > Pass 2: Analyzing classes (330 / 333) - 99% complete
> > Pass 2: Analyzing classes (331 / 333) - 99% complete
> > Pass 2: Analyzing classes (332 / 333) - 99% complete
> > Pass 2: Analyzing classes (333 / 333) - 100% complete
> > Done with analysis
> > :test
> >
> > org.apache.aurora.scheduler.events.WebhookTest > testTaskChangedWithOldStateError FAILED
> > java.lang.AssertionError at WebhookTest.java:251
> > I0426 21:37:40.001 [ShutdownHook, SchedulerMain] Stopping scheduler services.
> >
> > 1082 tests completed, 1 failed, 1 skipped
> > :test FAILED
> > :jacocoTestReport
> > Coverage report generated: file:///home/jenkins/jenkins-slave/workspace/AuroraBot/dist/reports/jacoco/test/html/index.html
> > :jacocoTestCoverageVerification
> >
> > FAILURE: Build failed with an exception.
> >
> > * What went wrong:
> > Execution failed for task ':test'.
> > > There were failing tests. See the report at: file:///home/jenkins/jenkins-slave/workspace/AuroraBot/dist/reports/tests/test/index.html
> >
> > * Try:
> > Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
> >
> > * Get more help at https://help.gradle.org
> >
> > BUILD FAILED in 6m 51s
> > 45 actionable tasks: 36 executed, 9 up-to-date
> >
> >
> > I will refresh this build result if you post a review containing "@ReviewBot retry"
@ReviewBot retry
- David
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66806/#review202034
-----------------------------------------------------------
On April 26, 2018, 9:27 p.m., David McLaughlin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66806/
> -----------------------------------------------------------
>
> (Updated April 26, 2018, 9:27 p.m.)
>
>
> Review request for Aurora, Jordan Ly and Santhosh Kumar Shanmugham.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> Currently Aurora exports total quota and resource reservation over time. This can be very useful to see changes in trends of production and free tier capacity. One challenge (particularly in a self-serve capacity environment) is identifying and tracking where large deltas came from. This change exports both quota and resource usage per role to help with this.
>
> It is possible to make this more efficient by refactoring the current abstractions to do both totals and per-role metrics in a single pass. But given this only runs once per hour, I went for the cleaner/simpler approach.
>
>
> Diffs
> -----
>
> src/main/java/org/apache/aurora/scheduler/stats/ResourceCounter.java a3e9bc75d5ce48cc3d64aa6b650df708f2f7c916
> src/main/java/org/apache/aurora/scheduler/stats/TaskStatCalculator.java ac5cf2462cb0f383493296b541348bff40e99025
> src/test/java/org/apache/aurora/scheduler/stats/ResourceCounterTest.java a30d74e73774a9fd4224839d08ebfe7fb0df095a
>
>
> Diff: https://reviews.apache.org/r/66806/diff/3/
>
>
> Testing
> -------
>
> /vars in Vagrant:
>
> quota_per_role_www-data_cpu_cores 10
> quota_per_role_www-data_disk_mb 102400
> quota_per_role_www-data_ram_mb 102400
> resources_allocated_quota_cpu_cores 0
> resources_allocated_quota_disk_mb 0
> resources_allocated_quota_ram_mb 0
> resources_dedicated_consumed_cpu_cores 0
> resources_dedicated_consumed_disk_mb 0
> resources_dedicated_consumed_ram_mb 0
> resources_free_pool_consumed_cpu_cores 1
> resources_free_pool_consumed_disk_mb 8
> resources_free_pool_consumed_ram_mb 1
> resources_per_role_free_pool_consumed_www-data_cpu_cores 1
> resources_per_role_free_pool_consumed_www-data_disk_mb 8
> resources_per_role_free_pool_consumed_www-data_ram_mb 1
> resources_per_role_total_consumed_www-data_cpu_cores 1
> resources_per_role_total_consumed_www-data_disk_mb 8
> resources_per_role_total_consumed_www-data_ram_mb 1
> resources_quota_consumed_cpu_cores 0
> resources_quota_consumed_disk_mb 0
> resources_quota_consumed_ram_mb 0
> resources_total_consumed_cpu_cores 1
> resources_total_consumed_disk_mb 8
> resources_total_consumed_ram_mb 1
>
>
> Thanks,
>
> David McLaughlin
>
>
Re: Review Request 66806: Breakdown resource stats by role
Posted by Aurora ReviewBot <wf...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66806/#review202034
-----------------------------------------------------------
Master (18103fe) is red with this patch.
./build-support/jenkins/build.sh
Pass 2: Analyzing classes (320 / 333) - 96% complete
Pass 2: Analyzing classes (321 / 333) - 96% complete
Pass 2: Analyzing classes (322 / 333) - 96% complete
Pass 2: Analyzing classes (323 / 333) - 96% complete
Pass 2: Analyzing classes (324 / 333) - 97% complete
Pass 2: Analyzing classes (325 / 333) - 97% complete
Pass 2: Analyzing classes (326 / 333) - 97% complete
Pass 2: Analyzing classes (327 / 333) - 98% complete
Pass 2: Analyzing classes (328 / 333) - 98% complete
Pass 2: Analyzing classes (329 / 333) - 98% complete
Pass 2: Analyzing classes (330 / 333) - 99% complete
Pass 2: Analyzing classes (331 / 333) - 99% complete
Pass 2: Analyzing classes (332 / 333) - 99% complete
Pass 2: Analyzing classes (333 / 333) - 100% complete
Done with analysis
:test
org.apache.aurora.scheduler.events.WebhookTest > testTaskChangedWithOldStateError FAILED
java.lang.AssertionError at WebhookTest.java:251
I0426 21:37:40.001 [ShutdownHook, SchedulerMain] Stopping scheduler services.
1082 tests completed, 1 failed, 1 skipped
:test FAILED
:jacocoTestReport
Coverage report generated: file:///home/jenkins/jenkins-slave/workspace/AuroraBot/dist/reports/jacoco/test/html/index.html
:jacocoTestCoverageVerification
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':test'.
> There were failing tests. See the report at: file:///home/jenkins/jenkins-slave/workspace/AuroraBot/dist/reports/tests/test/index.html
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
BUILD FAILED in 6m 51s
45 actionable tasks: 36 executed, 9 up-to-date
I will refresh this build result if you post a review containing "@ReviewBot retry"
- Aurora ReviewBot
On April 26, 2018, 2:27 p.m., David McLaughlin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66806/
> -----------------------------------------------------------
>
> (Updated April 26, 2018, 2:27 p.m.)
>
>
> Review request for Aurora, Jordan Ly and Santhosh Kumar Shanmugham.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> Currently Aurora exports total quota and resource reservation over time. This can be very useful to see changes in trends of production and free tier capacity. One challenge (particularly in a self-serve capacity environment) is identifying and tracking where large deltas came from. This change exports both quota and resource usage per role to help with this.
>
> It is possible to make this more efficient by refactoring the current abstractions to do both totals and per-role metrics in a single pass. But given this only runs once per hour, I went for the cleaner/simpler approach.
>
>
> Diffs
> -----
>
> src/main/java/org/apache/aurora/scheduler/stats/ResourceCounter.java a3e9bc75d5ce48cc3d64aa6b650df708f2f7c916
> src/main/java/org/apache/aurora/scheduler/stats/TaskStatCalculator.java ac5cf2462cb0f383493296b541348bff40e99025
> src/test/java/org/apache/aurora/scheduler/stats/ResourceCounterTest.java a30d74e73774a9fd4224839d08ebfe7fb0df095a
>
>
> Diff: https://reviews.apache.org/r/66806/diff/3/
>
>
> Testing
> -------
>
> /vars in Vagrant:
>
> quota_per_role_www-data_cpu_cores 10
> quota_per_role_www-data_disk_mb 102400
> quota_per_role_www-data_ram_mb 102400
> resources_allocated_quota_cpu_cores 0
> resources_allocated_quota_disk_mb 0
> resources_allocated_quota_ram_mb 0
> resources_dedicated_consumed_cpu_cores 0
> resources_dedicated_consumed_disk_mb 0
> resources_dedicated_consumed_ram_mb 0
> resources_free_pool_consumed_cpu_cores 1
> resources_free_pool_consumed_disk_mb 8
> resources_free_pool_consumed_ram_mb 1
> resources_per_role_free_pool_consumed_www-data_cpu_cores 1
> resources_per_role_free_pool_consumed_www-data_disk_mb 8
> resources_per_role_free_pool_consumed_www-data_ram_mb 1
> resources_per_role_total_consumed_www-data_cpu_cores 1
> resources_per_role_total_consumed_www-data_disk_mb 8
> resources_per_role_total_consumed_www-data_ram_mb 1
> resources_quota_consumed_cpu_cores 0
> resources_quota_consumed_disk_mb 0
> resources_quota_consumed_ram_mb 0
> resources_total_consumed_cpu_cores 1
> resources_total_consumed_disk_mb 8
> resources_total_consumed_ram_mb 1
>
>
> Thanks,
>
> David McLaughlin
>
>
Re: Review Request 66806: Breakdown resource stats by role
Posted by David McLaughlin <da...@dmclaughlin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66806/#review202038
-----------------------------------------------------------
@ReviewBot retry
- David McLaughlin
On April 26, 2018, 9:27 p.m., David McLaughlin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66806/
> -----------------------------------------------------------
>
> (Updated April 26, 2018, 9:27 p.m.)
>
>
> Review request for Aurora, Jordan Ly and Santhosh Kumar Shanmugham.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> Currently Aurora exports total quota and resource reservation over time. This can be very useful to see changes in trends of production and free tier capacity. One challenge (particularly in a self-serve capacity environment) is identifying and tracking where large deltas came from. This change exports both quota and resource usage per role to help with this.
>
> It is possible to make this more efficient by refactoring the current abstractions to do both totals and per-role metrics in a single pass. But given this only runs once per hour, I went for the cleaner/simpler approach.
>
>
> Diffs
> -----
>
> src/main/java/org/apache/aurora/scheduler/stats/ResourceCounter.java a3e9bc75d5ce48cc3d64aa6b650df708f2f7c916
> src/main/java/org/apache/aurora/scheduler/stats/TaskStatCalculator.java ac5cf2462cb0f383493296b541348bff40e99025
> src/test/java/org/apache/aurora/scheduler/stats/ResourceCounterTest.java a30d74e73774a9fd4224839d08ebfe7fb0df095a
>
>
> Diff: https://reviews.apache.org/r/66806/diff/3/
>
>
> Testing
> -------
>
> /vars in Vagrant:
>
> quota_per_role_www-data_cpu_cores 10
> quota_per_role_www-data_disk_mb 102400
> quota_per_role_www-data_ram_mb 102400
> resources_allocated_quota_cpu_cores 0
> resources_allocated_quota_disk_mb 0
> resources_allocated_quota_ram_mb 0
> resources_dedicated_consumed_cpu_cores 0
> resources_dedicated_consumed_disk_mb 0
> resources_dedicated_consumed_ram_mb 0
> resources_free_pool_consumed_cpu_cores 1
> resources_free_pool_consumed_disk_mb 8
> resources_free_pool_consumed_ram_mb 1
> resources_per_role_free_pool_consumed_www-data_cpu_cores 1
> resources_per_role_free_pool_consumed_www-data_disk_mb 8
> resources_per_role_free_pool_consumed_www-data_ram_mb 1
> resources_per_role_total_consumed_www-data_cpu_cores 1
> resources_per_role_total_consumed_www-data_disk_mb 8
> resources_per_role_total_consumed_www-data_ram_mb 1
> resources_quota_consumed_cpu_cores 0
> resources_quota_consumed_disk_mb 0
> resources_quota_consumed_ram_mb 0
> resources_total_consumed_cpu_cores 1
> resources_total_consumed_disk_mb 8
> resources_total_consumed_ram_mb 1
>
>
> Thanks,
>
> David McLaughlin
>
>
Re: Review Request 66806: Breakdown resource stats by role
Posted by David McLaughlin <da...@dmclaughlin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66806/
-----------------------------------------------------------
(Updated April 26, 2018, 9:27 p.m.)
Review request for Aurora, Jordan Ly and Santhosh Kumar Shanmugham.
Repository: aurora
Description
-------
Currently Aurora exports total quota and resource reservation over time. This can be very useful to see changes in trends of production and free tier capacity. One challenge (particularly in a self-serve capacity environment) is identifying and tracking where large deltas came from. This change exports both quota and resource usage per role to help with this.
It is possible to make this more efficient by refactoring the current abstractions to do both totals and per-role metrics in a single pass. But given this only runs once per hour, I went for the cleaner/simpler approach.
Diffs (updated)
-----
src/main/java/org/apache/aurora/scheduler/stats/ResourceCounter.java a3e9bc75d5ce48cc3d64aa6b650df708f2f7c916
src/main/java/org/apache/aurora/scheduler/stats/TaskStatCalculator.java ac5cf2462cb0f383493296b541348bff40e99025
src/test/java/org/apache/aurora/scheduler/stats/ResourceCounterTest.java a30d74e73774a9fd4224839d08ebfe7fb0df095a
Diff: https://reviews.apache.org/r/66806/diff/3/
Changes: https://reviews.apache.org/r/66806/diff/2-3/
Testing
-------
/vars in Vagrant:
quota_per_role_www-data_cpu_cores 10
quota_per_role_www-data_disk_mb 102400
quota_per_role_www-data_ram_mb 102400
resources_allocated_quota_cpu_cores 0
resources_allocated_quota_disk_mb 0
resources_allocated_quota_ram_mb 0
resources_dedicated_consumed_cpu_cores 0
resources_dedicated_consumed_disk_mb 0
resources_dedicated_consumed_ram_mb 0
resources_free_pool_consumed_cpu_cores 1
resources_free_pool_consumed_disk_mb 8
resources_free_pool_consumed_ram_mb 1
resources_per_role_free_pool_consumed_www-data_cpu_cores 1
resources_per_role_free_pool_consumed_www-data_disk_mb 8
resources_per_role_free_pool_consumed_www-data_ram_mb 1
resources_per_role_total_consumed_www-data_cpu_cores 1
resources_per_role_total_consumed_www-data_disk_mb 8
resources_per_role_total_consumed_www-data_ram_mb 1
resources_quota_consumed_cpu_cores 0
resources_quota_consumed_disk_mb 0
resources_quota_consumed_ram_mb 0
resources_total_consumed_cpu_cores 1
resources_total_consumed_disk_mb 8
resources_total_consumed_ram_mb 1
Thanks,
David McLaughlin
Re: Review Request 66806: Breakdown resource stats by role
Posted by David McLaughlin <da...@dmclaughlin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66806/
-----------------------------------------------------------
(Updated April 26, 2018, 9:10 p.m.)
Review request for Aurora, Jordan Ly and Santhosh Kumar Shanmugham.
Changes
-------
feedback
Repository: aurora
Description
-------
Currently Aurora exports total quota and resource reservation over time. This can be very useful to see changes in trends of production and free tier capacity. One challenge (particularly in a self-serve capacity environment) is identifying and tracking where large deltas came from. This change exports both quota and resource usage per role to help with this.
It is possible to make this more efficient by refactoring the current abstractions to do both totals and per-role metrics in a single pass. But given this only runs once per hour, I went for the cleaner/simpler approach.
Diffs (updated)
-----
src/main/java/org/apache/aurora/scheduler/stats/ResourceCounter.java a3e9bc75d5ce48cc3d64aa6b650df708f2f7c916
src/main/java/org/apache/aurora/scheduler/stats/TaskStatCalculator.java ac5cf2462cb0f383493296b541348bff40e99025
src/test/java/org/apache/aurora/scheduler/stats/ResourceCounterTest.java a30d74e73774a9fd4224839d08ebfe7fb0df095a
Diff: https://reviews.apache.org/r/66806/diff/2/
Changes: https://reviews.apache.org/r/66806/diff/1-2/
Testing
-------
/vars in Vagrant:
quota_per_role_www-data_cpu_cores 10
quota_per_role_www-data_disk_mb 102400
quota_per_role_www-data_ram_mb 102400
resources_allocated_quota_cpu_cores 0
resources_allocated_quota_disk_mb 0
resources_allocated_quota_ram_mb 0
resources_dedicated_consumed_cpu_cores 0
resources_dedicated_consumed_disk_mb 0
resources_dedicated_consumed_ram_mb 0
resources_free_pool_consumed_cpu_cores 1
resources_free_pool_consumed_disk_mb 8
resources_free_pool_consumed_ram_mb 1
resources_per_role_free_pool_consumed_www-data_cpu_cores 1
resources_per_role_free_pool_consumed_www-data_disk_mb 8
resources_per_role_free_pool_consumed_www-data_ram_mb 1
resources_per_role_total_consumed_www-data_cpu_cores 1
resources_per_role_total_consumed_www-data_disk_mb 8
resources_per_role_total_consumed_www-data_ram_mb 1
resources_quota_consumed_cpu_cores 0
resources_quota_consumed_disk_mb 0
resources_quota_consumed_ram_mb 0
resources_total_consumed_cpu_cores 1
resources_total_consumed_disk_mb 8
resources_total_consumed_ram_mb 1
Thanks,
David McLaughlin