You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@aurora.apache.org by Renan DelValle <re...@apache.org> on 2018/08/27 22:44:05 UTC
Re: Review Request 66192: Variable group size updates
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66192/
-----------------------------------------------------------
(Updated Aug. 27, 2018, 3:44 p.m.)
Review request for Aurora, David McLaughlin, Jordan Ly, Santhosh Kumar Shanmugham, and Stephan Erb.
Changes
-------
Updating tests done to reflect end to end tests passed.
Summary (updated)
-----------------
Variable group size updates
Repository: aurora
Description
-------
Adding support for variable group sizes when executing an update.
Design doc for this change is here:
https://docs.google.com/document/d/1xGk4ueH8YlmJCk6hQJh85u4to4M1VQD0l630IOchvgY/edit#heading=h.lg3hty82f5cz
I opted for the path of least resistance with regards to the Thrift changes as I didn't see any benefit in making the larger changes required to make the interfaces a bit more flexible.
Requesting feedback on these changes and the approach from the community before I proceed.
Tests will be added after the community approves of the direciton and approach.
Note to reviewers: Changes made in ActiveLimitedStrategy.java were made to move towards getting rid of FluentIterable. I figured since I was touching that code, it wouldn't hurt to test the Java 8 equivalent of it. I can get rid of the change here and make it in a separate patch if desired.
Diffs
-----
RELEASE-NOTES.md e6edbdeb577450f1119183a247e54a00e8a964d1
api/src/main/thrift/org/apache/aurora/gen/api.thrift dac2267dfe7e2e7563568adf8f04f6e9edc598d4
docs/reference/configuration.md 0632559e83ab1b547541c4087ae503318d2590d5
examples/jobs/hello_world_variable_update.aurora PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/durability/ThriftBackfill.java 41a2f0b5b897c4fc3437cc8fd052478a4de8a426
src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 74820c99ef769083a247608691dd7dea117a673a
src/main/java/org/apache/aurora/scheduler/updater/UpdateFactory.java bc8008ec5dcba98923703d0d35881f3fea7c5b2d
src/main/java/org/apache/aurora/scheduler/updater/strategy/ActiveLimitedStrategy.java 855ea9c20788b51695b7eff5ac0970f0d52a9546
src/main/java/org/apache/aurora/scheduler/updater/strategy/VariableBatchStrategy.java PRE-CREATION
src/main/python/apache/aurora/client/api/__init__.py 34822bcbacbba7ca6c3cd6ebf7f63a5f9d1c820f
src/main/python/apache/aurora/client/api/updater_util.py 5c2d95384a639a08cb0f2dece83c671de7e54764
src/main/python/apache/aurora/config/schema/base.py bf75660f44240ec78f27f8fc76418968bd0e2fc7
src/main/python/apache/aurora/config/thrift.py 2ffa5b6905ecfbd1835df60cfe861a13f08a6492
src/test/java/org/apache/aurora/scheduler/storage/AbstractJobUpdateStoreTest.java 3a93650635fb92fea306498cb86e35a0e25cf847
src/test/java/org/apache/aurora/scheduler/storage/durability/DataCompatibilityTest.java 3372ceccbd91629d5fc3f45ac1ebdebc19326b32
src/test/java/org/apache/aurora/scheduler/storage/durability/DurableStorageTest.java ddb9d06928de8ba73b6592b79cc741e5e97f0ff0
src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotterImplIT.java 4c1918fc33b5123dc522c49739dcf5690bdec3b0
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 334fd5d321d202cd9dfc10941c43fd07e0296aeb
src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java c96def1a51d354d0a55cbb6f574306ef2b5237c2
src/test/java/org/apache/aurora/scheduler/updater/UpdateFactoryImplTest.java 01bdcf1f0613b9c0297321762e0d2b032dd4841c
src/test/java/org/apache/aurora/scheduler/updater/strategy/VariableBatchStrategyTest.java PRE-CREATION
src/test/python/apache/aurora/client/api/test_api.py 9d6d9de903bf0cb9ac2b6415e76f8fb70e051514
src/test/python/apache/aurora/client/api/test_updater_util.py 48926c3695dbcf38abd33b2d58a64d8e1ea1cdd3
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate 3876767610f1d535d95b7dbe5d27dd8188b42f05
Diff: https://reviews.apache.org/r/66192/diff/4/
Testing (updated)
-------
Jenkins integration test passed.
End to end tests passed.
Thanks,
Renan DelValle
Re: Review Request 66192: Variable group size updates
Posted by Aurora ReviewBot <wf...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66192/#review208213
-----------------------------------------------------------
Master (80191a2) is red with this patch.
./build-support/jenkins/build.sh
52 errors, 0 warnings potentially fixable with the `--fix` option.
npm ERR! Linux 4.4.0-131-generic
npm ERR! argv "/home/jenkins/jenkins-slave/workspace/AuroraBot/ui/.gradle/nodejs/node-v6.9.1-linux-x64/bin/node" "/home/jenkins/jenkins-slave/workspace/AuroraBot/ui/.gradle/nodejs/node-v6.9.1-linux-x64/bin/npm" "run" "lint"
npm ERR! node v6.9.1
npm ERR! npm v3.10.8
npm ERR! code ELIFECYCLE
npm ERR! apache-aurora@1.0.0 lint: `eslint src/main/js plugin/ --ext .js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the apache-aurora@1.0.0 lint script 'eslint src/main/js plugin/ --ext .js'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the apache-aurora package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! eslint src/main/js plugin/ --ext .js
npm ERR! You can get information on how to open an issue for this project with:
npm ERR! npm bugs apache-aurora
npm ERR! Or if that isn't available, you can get their info via:
npm ERR! npm owner ls apache-aurora
npm ERR! There is likely additional logging output above.
npm ERR! Please include the following file with any support request:
npm ERR! /home/jenkins/jenkins-slave/workspace/AuroraBot/ui/npm-debug.log
:ui:lint FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':ui:lint'.
> Process 'command '/home/jenkins/jenkins-slave/workspace/AuroraBot/ui/.gradle/nodejs/node-v6.9.1-linux-x64/bin/npm'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
BUILD FAILED in 8m 2s
56 actionable tasks: 45 executed, 11 up-to-date
I will refresh this build result if you post a review containing "@ReviewBot retry"
- Aurora ReviewBot
On Sept. 1, 2018, 2:01 a.m., Renan DelValle wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66192/
> -----------------------------------------------------------
>
> (Updated Sept. 1, 2018, 2:01 a.m.)
>
>
> Review request for Aurora, David McLaughlin, Jordan Ly, Santhosh Kumar Shanmugham, and Stephan Erb.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> Adding support for variable group sizes when executing an update.
>
> Design doc for this change is here:
> https://docs.google.com/document/d/1xGk4ueH8YlmJCk6hQJh85u4to4M1VQD0l630IOchvgY/edit#heading=h.lg3hty82f5cz
>
>
> Diffs
> -----
>
> RELEASE-NOTES.md e6edbdeb577450f1119183a247e54a00e8a964d1
> api/src/main/thrift/org/apache/aurora/gen/api.thrift dac2267dfe7e2e7563568adf8f04f6e9edc598d4
> docs/reference/configuration.md 0632559e83ab1b547541c4087ae503318d2590d5
> examples/jobs/hello_world_variable_update.aurora PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/durability/ThriftBackfill.java 41a2f0b5b897c4fc3437cc8fd052478a4de8a426
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 74820c99ef769083a247608691dd7dea117a673a
> src/main/java/org/apache/aurora/scheduler/updater/UpdateFactory.java bc8008ec5dcba98923703d0d35881f3fea7c5b2d
> src/main/java/org/apache/aurora/scheduler/updater/strategy/ActiveLimitedStrategy.java 855ea9c20788b51695b7eff5ac0970f0d52a9546
> src/main/java/org/apache/aurora/scheduler/updater/strategy/VariableBatchStrategy.java PRE-CREATION
> src/main/python/apache/aurora/client/api/__init__.py 34822bcbacbba7ca6c3cd6ebf7f63a5f9d1c820f
> src/main/python/apache/aurora/client/api/updater_util.py 5c2d95384a639a08cb0f2dece83c671de7e54764
> src/main/python/apache/aurora/config/schema/base.py bf75660f44240ec78f27f8fc76418968bd0e2fc7
> src/main/python/apache/aurora/config/thrift.py 2ffa5b6905ecfbd1835df60cfe861a13f08a6492
> src/test/java/org/apache/aurora/scheduler/storage/AbstractJobUpdateStoreTest.java 3a93650635fb92fea306498cb86e35a0e25cf847
> src/test/java/org/apache/aurora/scheduler/storage/durability/DataCompatibilityTest.java 3372ceccbd91629d5fc3f45ac1ebdebc19326b32
> src/test/java/org/apache/aurora/scheduler/storage/durability/DurableStorageTest.java ddb9d06928de8ba73b6592b79cc741e5e97f0ff0
> src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotterImplIT.java 4c1918fc33b5123dc522c49739dcf5690bdec3b0
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 334fd5d321d202cd9dfc10941c43fd07e0296aeb
> src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java c96def1a51d354d0a55cbb6f574306ef2b5237c2
> src/test/java/org/apache/aurora/scheduler/updater/UpdateFactoryImplTest.java 01bdcf1f0613b9c0297321762e0d2b032dd4841c
> src/test/java/org/apache/aurora/scheduler/updater/strategy/VariableBatchStrategyTest.java PRE-CREATION
> src/test/python/apache/aurora/client/api/test_api.py 9d6d9de903bf0cb9ac2b6415e76f8fb70e051514
> src/test/python/apache/aurora/client/api/test_updater_util.py 48926c3695dbcf38abd33b2d58a64d8e1ea1cdd3
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate 3876767610f1d535d95b7dbe5d27dd8188b42f05
> ui/src/main/js/components/UpdateSettings.js d7fbe00859d89cc0523a2aa604abba9585000662
>
>
> Diff: https://reviews.apache.org/r/66192/diff/5/
>
>
> Testing
> -------
>
> Jenkins integration test passed.
>
> End to end tests passed.
>
>
> File Attachments
> ----------------
>
> Queue update strategy
> https://reviews.apache.org/media/uploaded/files/2018/09/01/dbd016f3-95e8-49f9-9e7b-e0c6d62eab97__Screenshot_from_2018-08-31_18-57-47.png
> Batch Update Strategy
> https://reviews.apache.org/media/uploaded/files/2018/09/01/f28728fc-6eae-471e-b69f-bb6fdffdd342__updatebatch.png
> Variable Batch Strategy
> https://reviews.apache.org/media/uploaded/files/2018/09/01/7b153c90-213a-4267-ac64-6bbbccb14d73__updatestrategyvarbatch.png
>
>
> Thanks,
>
> Renan DelValle
>
>
Re: Review Request 66192: Variable group size updates
Posted by Renan DelValle <re...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66192/
-----------------------------------------------------------
(Updated Aug. 31, 2018, 7:01 p.m.)
Review request for Aurora, David McLaughlin, Jordan Ly, Santhosh Kumar Shanmugham, and Stephan Erb.
Changes
-------
Added a python tests to make sure thrift conversion from pystachio works.
Changed Update Web UI to show update strategy being used.
Still to do:
Discuss wether it would be a good idea to merge VariableBatchUpdate code with BatchUpdate code since BatchUpdate is a specific case of VariableBatchUpdate.
Add a few end to end tests.
Add update strategy field to UI tests.
Repository: aurora
Description (updated)
-------
Adding support for variable group sizes when executing an update.
Design doc for this change is here:
https://docs.google.com/document/d/1xGk4ueH8YlmJCk6hQJh85u4to4M1VQD0l630IOchvgY/edit#heading=h.lg3hty82f5cz
Diffs (updated)
-----
RELEASE-NOTES.md e6edbdeb577450f1119183a247e54a00e8a964d1
api/src/main/thrift/org/apache/aurora/gen/api.thrift dac2267dfe7e2e7563568adf8f04f6e9edc598d4
docs/reference/configuration.md 0632559e83ab1b547541c4087ae503318d2590d5
examples/jobs/hello_world_variable_update.aurora PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/durability/ThriftBackfill.java 41a2f0b5b897c4fc3437cc8fd052478a4de8a426
src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 74820c99ef769083a247608691dd7dea117a673a
src/main/java/org/apache/aurora/scheduler/updater/UpdateFactory.java bc8008ec5dcba98923703d0d35881f3fea7c5b2d
src/main/java/org/apache/aurora/scheduler/updater/strategy/ActiveLimitedStrategy.java 855ea9c20788b51695b7eff5ac0970f0d52a9546
src/main/java/org/apache/aurora/scheduler/updater/strategy/VariableBatchStrategy.java PRE-CREATION
src/main/python/apache/aurora/client/api/__init__.py 34822bcbacbba7ca6c3cd6ebf7f63a5f9d1c820f
src/main/python/apache/aurora/client/api/updater_util.py 5c2d95384a639a08cb0f2dece83c671de7e54764
src/main/python/apache/aurora/config/schema/base.py bf75660f44240ec78f27f8fc76418968bd0e2fc7
src/main/python/apache/aurora/config/thrift.py 2ffa5b6905ecfbd1835df60cfe861a13f08a6492
src/test/java/org/apache/aurora/scheduler/storage/AbstractJobUpdateStoreTest.java 3a93650635fb92fea306498cb86e35a0e25cf847
src/test/java/org/apache/aurora/scheduler/storage/durability/DataCompatibilityTest.java 3372ceccbd91629d5fc3f45ac1ebdebc19326b32
src/test/java/org/apache/aurora/scheduler/storage/durability/DurableStorageTest.java ddb9d06928de8ba73b6592b79cc741e5e97f0ff0
src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotterImplIT.java 4c1918fc33b5123dc522c49739dcf5690bdec3b0
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 334fd5d321d202cd9dfc10941c43fd07e0296aeb
src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java c96def1a51d354d0a55cbb6f574306ef2b5237c2
src/test/java/org/apache/aurora/scheduler/updater/UpdateFactoryImplTest.java 01bdcf1f0613b9c0297321762e0d2b032dd4841c
src/test/java/org/apache/aurora/scheduler/updater/strategy/VariableBatchStrategyTest.java PRE-CREATION
src/test/python/apache/aurora/client/api/test_api.py 9d6d9de903bf0cb9ac2b6415e76f8fb70e051514
src/test/python/apache/aurora/client/api/test_updater_util.py 48926c3695dbcf38abd33b2d58a64d8e1ea1cdd3
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate 3876767610f1d535d95b7dbe5d27dd8188b42f05
ui/src/main/js/components/UpdateSettings.js d7fbe00859d89cc0523a2aa604abba9585000662
Diff: https://reviews.apache.org/r/66192/diff/5/
Changes: https://reviews.apache.org/r/66192/diff/4-5/
Testing
-------
Jenkins integration test passed.
End to end tests passed.
File Attachments (updated)
----------------
Queue update strategy
https://reviews.apache.org/media/uploaded/files/2018/09/01/dbd016f3-95e8-49f9-9e7b-e0c6d62eab97__Screenshot_from_2018-08-31_18-57-47.png
Batch Update Strategy
https://reviews.apache.org/media/uploaded/files/2018/09/01/f28728fc-6eae-471e-b69f-bb6fdffdd342__updatebatch.png
Variable Batch Strategy
https://reviews.apache.org/media/uploaded/files/2018/09/01/7b153c90-213a-4267-ac64-6bbbccb14d73__updatestrategyvarbatch.png
Thanks,
Renan DelValle