You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@aurora.apache.org by Steve Salevan <st...@gmail.com> on 2018/05/03 01:06:52 UTC
Re: Review Request 66537: Adding enhancements to Docker functionality
and client support for FetcherURIs
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66537/
-----------------------------------------------------------
(Updated May 3, 2018, 1:06 a.m.)
Review request for Aurora, Renan DelValle and Stephan Erb.
Changes
-------
Addressing RB comments
Repository: aurora
Description
-------
Hey there!
Over here at Spine we've added a few enhancements to Aurora's Docker functionality to support several of our use cases. We'd like to submit these back up to the upstream to support the great work y'all are doing in this space.
Here's what's included in this RB:
* Support for the force_pull flag to help ensure container freshness
* Overrides for a Docker --entrypoint which can be specified on the job
* Support for alternative Docker networks (defaults to HOST)
* Support for user Docker networks
We currently use Aurora to schedule Docker containers without the use of Thermos, so we've added support for server-side templating of common Thermos variables into Docker executor's parameters for this purpose.
This change modifies Aurora's api.thrift with several new optional fields, and all added code handles their absence gracefully, so no backfills have been added. We've threaded these schema changes through to the Python Aurora client alongside support for the Mesos Fetcher URIs already supported server-side.
Let me know what you think and thanks!
Diffs (updated)
-----
.gitignore 9ce74ebbbc57b77d912eaa573a8fb18ed4aa3c15
api/src/main/thrift/org/apache/aurora/gen/api.thrift ef754e32172e7490a47a13e7b526f243ffa3efeb
src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 4073229b74d0e0e7fd31552bd96894ceb8a0971a
src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java bcb2bbf882f43d813dd26c746d806e78bae6bcf3
src/main/python/apache/aurora/config/schema/base.py a629bcd1261e5959da0a8458a55545d4e2c2a7a5
src/main/python/apache/aurora/config/thrift.py 6d2dde6e964daa68bf6f0e5bbbffecc5bd8c0431
src/test/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 778148a7c033cba9004954cabc33a2b1d003dccf
src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java 749ffeac6cb851f32bba7606390203d7a046a0e6
src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 686087ef858b8a5a8e956d82a7bd692f7be28b12
src/test/java/org/apache/aurora/scheduler/thrift/ThriftIT.java 40851c419e4d62e6545959eebc0ce144fdecc697
src/test/python/apache/aurora/client/cli/test_inspect.py e4f43d0573c7862adc9bc679f4cea40cc76eac38
src/test/python/apache/aurora/config/test_thrift.py 8e1d0e177959af12b97bdd1cd47845b72bc12fe1
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveCronJob 88e1c36a1aa2d192b95963f7aa36e243a447e4af
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate 32fdcdacde58345cdd6c4b449b82c0c90c2b2aae
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveTasks 4323031ec6bd128576c2a43ebc11f04a9f046e2f
Diff: https://reviews.apache.org/r/66537/diff/4/
Changes: https://reviews.apache.org/r/66537/diff/3-4/
Testing
-------
Unit tests for all new functionality have been added and run cleanly alongside checkstyle/static analysis tooling.
Operational/functional testing completed within Spine infrastructure.
Thanks,
Steve Salevan
Re: Review Request 66537: Adding enhancements to Docker functionality
and client support for FetcherURIs
Posted by Renan DelValle <re...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66537/#review202370
-----------------------------------------------------------
@ReviewBot retry
- Renan DelValle
On May 2, 2018, 6:06 p.m., Steve Salevan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66537/
> -----------------------------------------------------------
>
> (Updated May 2, 2018, 6:06 p.m.)
>
>
> Review request for Aurora, Renan DelValle and Stephan Erb.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> Hey there!
>
> Over here at Spine we've added a few enhancements to Aurora's Docker functionality to support several of our use cases. We'd like to submit these back up to the upstream to support the great work y'all are doing in this space.
>
> Here's what's included in this RB:
>
> * Support for the force_pull flag to help ensure container freshness
> * Overrides for a Docker --entrypoint which can be specified on the job
> * Support for alternative Docker networks (defaults to HOST)
> * Support for user Docker networks
>
> We currently use Aurora to schedule Docker containers without the use of Thermos, so we've added support for server-side templating of common Thermos variables into Docker executor's parameters for this purpose.
>
> This change modifies Aurora's api.thrift with several new optional fields, and all added code handles their absence gracefully, so no backfills have been added. We've threaded these schema changes through to the Python Aurora client alongside support for the Mesos Fetcher URIs already supported server-side.
>
> Let me know what you think and thanks!
>
>
> Diffs
> -----
>
> .gitignore 9ce74ebbbc57b77d912eaa573a8fb18ed4aa3c15
> api/src/main/thrift/org/apache/aurora/gen/api.thrift ef754e32172e7490a47a13e7b526f243ffa3efeb
> src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 4073229b74d0e0e7fd31552bd96894ceb8a0971a
> src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java bcb2bbf882f43d813dd26c746d806e78bae6bcf3
> src/main/python/apache/aurora/config/schema/base.py a629bcd1261e5959da0a8458a55545d4e2c2a7a5
> src/main/python/apache/aurora/config/thrift.py 6d2dde6e964daa68bf6f0e5bbbffecc5bd8c0431
> src/test/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 778148a7c033cba9004954cabc33a2b1d003dccf
> src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java 749ffeac6cb851f32bba7606390203d7a046a0e6
> src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 686087ef858b8a5a8e956d82a7bd692f7be28b12
> src/test/java/org/apache/aurora/scheduler/thrift/ThriftIT.java 40851c419e4d62e6545959eebc0ce144fdecc697
> src/test/python/apache/aurora/client/cli/test_inspect.py e4f43d0573c7862adc9bc679f4cea40cc76eac38
> src/test/python/apache/aurora/config/test_thrift.py 8e1d0e177959af12b97bdd1cd47845b72bc12fe1
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveCronJob 88e1c36a1aa2d192b95963f7aa36e243a447e4af
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate 32fdcdacde58345cdd6c4b449b82c0c90c2b2aae
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveTasks 4323031ec6bd128576c2a43ebc11f04a9f046e2f
>
>
> Diff: https://reviews.apache.org/r/66537/diff/4/
>
>
> Testing
> -------
>
> Unit tests for all new functionality have been added and run cleanly alongside checkstyle/static analysis tooling.
>
> Operational/functional testing completed within Spine infrastructure.
>
>
> Thanks,
>
> Steve Salevan
>
>
Re: Review Request 66537: Adding enhancements to Docker functionality
and client support for FetcherURIs
Posted by Renan DelValle <re...@apache.org>.
> On May 3, 2018, 2:41 p.m., Renan DelValle wrote:
> > Ship It!
Actually, Steve, if you could add some docs here https://github.com/apache/aurora/blob/master/docs/reference/configuration.md I can get Santhosh to review and then hopefully we can get this shipped quick!
- Renan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66537/#review202384
-----------------------------------------------------------
On May 2, 2018, 6:06 p.m., Steve Salevan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66537/
> -----------------------------------------------------------
>
> (Updated May 2, 2018, 6:06 p.m.)
>
>
> Review request for Aurora, Renan DelValle and Stephan Erb.
>
>
> Bugs: AURORA-1982
> https://issues.apache.org/jira/browse/AURORA-1982
>
>
> Repository: aurora
>
>
> Description
> -------
>
> Hey there!
>
> Over here at Spine we've added a few enhancements to Aurora's Docker functionality to support several of our use cases. We'd like to submit these back up to the upstream to support the great work y'all are doing in this space.
>
> Here's what's included in this RB:
>
> * Support for the force_pull flag to help ensure container freshness
> * Overrides for a Docker --entrypoint which can be specified on the job
> * Support for alternative Docker networks (defaults to HOST)
> * Support for user Docker networks
>
> We currently use Aurora to schedule Docker containers without the use of Thermos, so we've added support for server-side templating of common Thermos variables into Docker executor's parameters for this purpose.
>
> This change modifies Aurora's api.thrift with several new optional fields, and all added code handles their absence gracefully, so no backfills have been added. We've threaded these schema changes through to the Python Aurora client alongside support for the Mesos Fetcher URIs already supported server-side.
>
> Let me know what you think and thanks!
>
>
> Diffs
> -----
>
> .gitignore 9ce74ebbbc57b77d912eaa573a8fb18ed4aa3c15
> api/src/main/thrift/org/apache/aurora/gen/api.thrift ef754e32172e7490a47a13e7b526f243ffa3efeb
> src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 4073229b74d0e0e7fd31552bd96894ceb8a0971a
> src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java bcb2bbf882f43d813dd26c746d806e78bae6bcf3
> src/main/python/apache/aurora/config/schema/base.py a629bcd1261e5959da0a8458a55545d4e2c2a7a5
> src/main/python/apache/aurora/config/thrift.py 6d2dde6e964daa68bf6f0e5bbbffecc5bd8c0431
> src/test/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 778148a7c033cba9004954cabc33a2b1d003dccf
> src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java 749ffeac6cb851f32bba7606390203d7a046a0e6
> src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 686087ef858b8a5a8e956d82a7bd692f7be28b12
> src/test/java/org/apache/aurora/scheduler/thrift/ThriftIT.java 40851c419e4d62e6545959eebc0ce144fdecc697
> src/test/python/apache/aurora/client/cli/test_inspect.py e4f43d0573c7862adc9bc679f4cea40cc76eac38
> src/test/python/apache/aurora/config/test_thrift.py 8e1d0e177959af12b97bdd1cd47845b72bc12fe1
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveCronJob 88e1c36a1aa2d192b95963f7aa36e243a447e4af
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate 32fdcdacde58345cdd6c4b449b82c0c90c2b2aae
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveTasks 4323031ec6bd128576c2a43ebc11f04a9f046e2f
>
>
> Diff: https://reviews.apache.org/r/66537/diff/4/
>
>
> Testing
> -------
>
> Unit tests for all new functionality have been added and run cleanly alongside checkstyle/static analysis tooling.
>
> Operational/functional testing completed within Spine infrastructure.
>
>
> Thanks,
>
> Steve Salevan
>
>
Re: Review Request 66537: Adding enhancements to Docker functionality
and client support for FetcherURIs
Posted by Renan DelValle <re...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66537/#review202384
-----------------------------------------------------------
Ship it!
Ship It!
- Renan DelValle
On May 2, 2018, 6:06 p.m., Steve Salevan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66537/
> -----------------------------------------------------------
>
> (Updated May 2, 2018, 6:06 p.m.)
>
>
> Review request for Aurora, Renan DelValle and Stephan Erb.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> Hey there!
>
> Over here at Spine we've added a few enhancements to Aurora's Docker functionality to support several of our use cases. We'd like to submit these back up to the upstream to support the great work y'all are doing in this space.
>
> Here's what's included in this RB:
>
> * Support for the force_pull flag to help ensure container freshness
> * Overrides for a Docker --entrypoint which can be specified on the job
> * Support for alternative Docker networks (defaults to HOST)
> * Support for user Docker networks
>
> We currently use Aurora to schedule Docker containers without the use of Thermos, so we've added support for server-side templating of common Thermos variables into Docker executor's parameters for this purpose.
>
> This change modifies Aurora's api.thrift with several new optional fields, and all added code handles their absence gracefully, so no backfills have been added. We've threaded these schema changes through to the Python Aurora client alongside support for the Mesos Fetcher URIs already supported server-side.
>
> Let me know what you think and thanks!
>
>
> Diffs
> -----
>
> .gitignore 9ce74ebbbc57b77d912eaa573a8fb18ed4aa3c15
> api/src/main/thrift/org/apache/aurora/gen/api.thrift ef754e32172e7490a47a13e7b526f243ffa3efeb
> src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 4073229b74d0e0e7fd31552bd96894ceb8a0971a
> src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java bcb2bbf882f43d813dd26c746d806e78bae6bcf3
> src/main/python/apache/aurora/config/schema/base.py a629bcd1261e5959da0a8458a55545d4e2c2a7a5
> src/main/python/apache/aurora/config/thrift.py 6d2dde6e964daa68bf6f0e5bbbffecc5bd8c0431
> src/test/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 778148a7c033cba9004954cabc33a2b1d003dccf
> src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java 749ffeac6cb851f32bba7606390203d7a046a0e6
> src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 686087ef858b8a5a8e956d82a7bd692f7be28b12
> src/test/java/org/apache/aurora/scheduler/thrift/ThriftIT.java 40851c419e4d62e6545959eebc0ce144fdecc697
> src/test/python/apache/aurora/client/cli/test_inspect.py e4f43d0573c7862adc9bc679f4cea40cc76eac38
> src/test/python/apache/aurora/config/test_thrift.py 8e1d0e177959af12b97bdd1cd47845b72bc12fe1
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveCronJob 88e1c36a1aa2d192b95963f7aa36e243a447e4af
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate 32fdcdacde58345cdd6c4b449b82c0c90c2b2aae
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveTasks 4323031ec6bd128576c2a43ebc11f04a9f046e2f
>
>
> Diff: https://reviews.apache.org/r/66537/diff/4/
>
>
> Testing
> -------
>
> Unit tests for all new functionality have been added and run cleanly alongside checkstyle/static analysis tooling.
>
> Operational/functional testing completed within Spine infrastructure.
>
>
> Thanks,
>
> Steve Salevan
>
>
Re: Review Request 66537: Adding enhancements to Docker functionality
and client support for FetcherURIs
Posted by Aurora ReviewBot <wf...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66537/#review202372
-----------------------------------------------------------
Ship it!
Master (a3d596e) is green with this patch.
./build-support/jenkins/build.sh
I will refresh this build result if you post a review containing "@ReviewBot retry"
- Aurora ReviewBot
On May 2, 2018, 6:06 p.m., Steve Salevan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66537/
> -----------------------------------------------------------
>
> (Updated May 2, 2018, 6:06 p.m.)
>
>
> Review request for Aurora, Renan DelValle and Stephan Erb.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> Hey there!
>
> Over here at Spine we've added a few enhancements to Aurora's Docker functionality to support several of our use cases. We'd like to submit these back up to the upstream to support the great work y'all are doing in this space.
>
> Here's what's included in this RB:
>
> * Support for the force_pull flag to help ensure container freshness
> * Overrides for a Docker --entrypoint which can be specified on the job
> * Support for alternative Docker networks (defaults to HOST)
> * Support for user Docker networks
>
> We currently use Aurora to schedule Docker containers without the use of Thermos, so we've added support for server-side templating of common Thermos variables into Docker executor's parameters for this purpose.
>
> This change modifies Aurora's api.thrift with several new optional fields, and all added code handles their absence gracefully, so no backfills have been added. We've threaded these schema changes through to the Python Aurora client alongside support for the Mesos Fetcher URIs already supported server-side.
>
> Let me know what you think and thanks!
>
>
> Diffs
> -----
>
> .gitignore 9ce74ebbbc57b77d912eaa573a8fb18ed4aa3c15
> api/src/main/thrift/org/apache/aurora/gen/api.thrift ef754e32172e7490a47a13e7b526f243ffa3efeb
> src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 4073229b74d0e0e7fd31552bd96894ceb8a0971a
> src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java bcb2bbf882f43d813dd26c746d806e78bae6bcf3
> src/main/python/apache/aurora/config/schema/base.py a629bcd1261e5959da0a8458a55545d4e2c2a7a5
> src/main/python/apache/aurora/config/thrift.py 6d2dde6e964daa68bf6f0e5bbbffecc5bd8c0431
> src/test/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 778148a7c033cba9004954cabc33a2b1d003dccf
> src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java 749ffeac6cb851f32bba7606390203d7a046a0e6
> src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 686087ef858b8a5a8e956d82a7bd692f7be28b12
> src/test/java/org/apache/aurora/scheduler/thrift/ThriftIT.java 40851c419e4d62e6545959eebc0ce144fdecc697
> src/test/python/apache/aurora/client/cli/test_inspect.py e4f43d0573c7862adc9bc679f4cea40cc76eac38
> src/test/python/apache/aurora/config/test_thrift.py 8e1d0e177959af12b97bdd1cd47845b72bc12fe1
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveCronJob 88e1c36a1aa2d192b95963f7aa36e243a447e4af
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate 32fdcdacde58345cdd6c4b449b82c0c90c2b2aae
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveTasks 4323031ec6bd128576c2a43ebc11f04a9f046e2f
>
>
> Diff: https://reviews.apache.org/r/66537/diff/4/
>
>
> Testing
> -------
>
> Unit tests for all new functionality have been added and run cleanly alongside checkstyle/static analysis tooling.
>
> Operational/functional testing completed within Spine infrastructure.
>
>
> Thanks,
>
> Steve Salevan
>
>
Re: Review Request 66537: Adding enhancements to Docker functionality
and client support for FetcherURIs
Posted by Renan DelValle <re...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66537/#review202336
-----------------------------------------------------------
@ReviewBot retry
- Renan DelValle
On May 2, 2018, 6:06 p.m., Steve Salevan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66537/
> -----------------------------------------------------------
>
> (Updated May 2, 2018, 6:06 p.m.)
>
>
> Review request for Aurora, Renan DelValle and Stephan Erb.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> Hey there!
>
> Over here at Spine we've added a few enhancements to Aurora's Docker functionality to support several of our use cases. We'd like to submit these back up to the upstream to support the great work y'all are doing in this space.
>
> Here's what's included in this RB:
>
> * Support for the force_pull flag to help ensure container freshness
> * Overrides for a Docker --entrypoint which can be specified on the job
> * Support for alternative Docker networks (defaults to HOST)
> * Support for user Docker networks
>
> We currently use Aurora to schedule Docker containers without the use of Thermos, so we've added support for server-side templating of common Thermos variables into Docker executor's parameters for this purpose.
>
> This change modifies Aurora's api.thrift with several new optional fields, and all added code handles their absence gracefully, so no backfills have been added. We've threaded these schema changes through to the Python Aurora client alongside support for the Mesos Fetcher URIs already supported server-side.
>
> Let me know what you think and thanks!
>
>
> Diffs
> -----
>
> .gitignore 9ce74ebbbc57b77d912eaa573a8fb18ed4aa3c15
> api/src/main/thrift/org/apache/aurora/gen/api.thrift ef754e32172e7490a47a13e7b526f243ffa3efeb
> src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 4073229b74d0e0e7fd31552bd96894ceb8a0971a
> src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java bcb2bbf882f43d813dd26c746d806e78bae6bcf3
> src/main/python/apache/aurora/config/schema/base.py a629bcd1261e5959da0a8458a55545d4e2c2a7a5
> src/main/python/apache/aurora/config/thrift.py 6d2dde6e964daa68bf6f0e5bbbffecc5bd8c0431
> src/test/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 778148a7c033cba9004954cabc33a2b1d003dccf
> src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java 749ffeac6cb851f32bba7606390203d7a046a0e6
> src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 686087ef858b8a5a8e956d82a7bd692f7be28b12
> src/test/java/org/apache/aurora/scheduler/thrift/ThriftIT.java 40851c419e4d62e6545959eebc0ce144fdecc697
> src/test/python/apache/aurora/client/cli/test_inspect.py e4f43d0573c7862adc9bc679f4cea40cc76eac38
> src/test/python/apache/aurora/config/test_thrift.py 8e1d0e177959af12b97bdd1cd47845b72bc12fe1
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveCronJob 88e1c36a1aa2d192b95963f7aa36e243a447e4af
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate 32fdcdacde58345cdd6c4b449b82c0c90c2b2aae
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveTasks 4323031ec6bd128576c2a43ebc11f04a9f046e2f
>
>
> Diff: https://reviews.apache.org/r/66537/diff/4/
>
>
> Testing
> -------
>
> Unit tests for all new functionality have been added and run cleanly alongside checkstyle/static analysis tooling.
>
> Operational/functional testing completed within Spine infrastructure.
>
>
> Thanks,
>
> Steve Salevan
>
>
Re: Review Request 66537: Adding enhancements to Docker functionality
and client support for FetcherURIs
Posted by Aurora ReviewBot <wf...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66537/#review202339
-----------------------------------------------------------
Master (a3d596e) is red with this patch.
./build-support/jenkins/build.sh
at org.apache.aurora.scheduler.TaskStatusHandlerImpl.lambda$run$0(TaskStatusHandlerImpl.java:158)
at org.apache.aurora.scheduler.storage.Storage$MutateWork$NoResult.apply(Storage.java:144)
at org.apache.aurora.scheduler.storage.Storage$MutateWork$NoResult.apply(Storage.java:139)
at org.apache.aurora.scheduler.storage.testing.StorageTestUtil.lambda$expectWrite$1(StorageTestUtil.java:83)
at org.easymock.internal.Result.answer(Result.java:106)
at org.easymock.internal.ReplayState.invokeInner(ReplayState.java:60)
at org.easymock.internal.ReplayState.invoke(ReplayState.java:46)
at org.easymock.internal.MockInvocationHandler.invoke(MockInvocationHandler.java:40)
at org.easymock.internal.ObjectMethodsFilter.invoke(ObjectMethodsFilter.java:94)
at com.sun.proxy.$Proxy20.write(Unknown Source)
at org.apache.aurora.scheduler.TaskStatusHandlerImpl.run(TaskStatusHandlerImpl.java:154)
at com.google.common.util.concurrent.AbstractExecutionThreadService$1$2.run(AbstractExecutionThreadService.java:66)
at com.google.common.util.concurrent.Callables$4.run(Callables.java:122)
at java.lang.Thread.run(Thread.java:748)
org.apache.aurora.scheduler.events.WebhookTest > testTaskChangedWithOldStateError FAILED
java.lang.AssertionError at WebhookTest.java:251
I0503 02:42:16.381 [ShutdownHook, SchedulerMain] Stopping scheduler services.
1087 tests completed, 1 failed, 1 skipped
:test FAILED
:jacocoTestReport
Coverage report generated: file:///home/jenkins/jenkins-slave/workspace/AuroraBot/dist/reports/jacoco/test/html/index.html
:jacocoTestCoverageVerification
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':test'.
> There were failing tests. See the report at: file:///home/jenkins/jenkins-slave/workspace/AuroraBot/dist/reports/tests/test/index.html
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
BUILD FAILED in 8m 23s
45 actionable tasks: 36 executed, 9 up-to-date
I will refresh this build result if you post a review containing "@ReviewBot retry"
- Aurora ReviewBot
On May 3, 2018, 9:06 a.m., Steve Salevan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66537/
> -----------------------------------------------------------
>
> (Updated May 3, 2018, 9:06 a.m.)
>
>
> Review request for Aurora, Renan DelValle and Stephan Erb.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> Hey there!
>
> Over here at Spine we've added a few enhancements to Aurora's Docker functionality to support several of our use cases. We'd like to submit these back up to the upstream to support the great work y'all are doing in this space.
>
> Here's what's included in this RB:
>
> * Support for the force_pull flag to help ensure container freshness
> * Overrides for a Docker --entrypoint which can be specified on the job
> * Support for alternative Docker networks (defaults to HOST)
> * Support for user Docker networks
>
> We currently use Aurora to schedule Docker containers without the use of Thermos, so we've added support for server-side templating of common Thermos variables into Docker executor's parameters for this purpose.
>
> This change modifies Aurora's api.thrift with several new optional fields, and all added code handles their absence gracefully, so no backfills have been added. We've threaded these schema changes through to the Python Aurora client alongside support for the Mesos Fetcher URIs already supported server-side.
>
> Let me know what you think and thanks!
>
>
> Diffs
> -----
>
> .gitignore 9ce74ebbbc57b77d912eaa573a8fb18ed4aa3c15
> api/src/main/thrift/org/apache/aurora/gen/api.thrift ef754e32172e7490a47a13e7b526f243ffa3efeb
> src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 4073229b74d0e0e7fd31552bd96894ceb8a0971a
> src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java bcb2bbf882f43d813dd26c746d806e78bae6bcf3
> src/main/python/apache/aurora/config/schema/base.py a629bcd1261e5959da0a8458a55545d4e2c2a7a5
> src/main/python/apache/aurora/config/thrift.py 6d2dde6e964daa68bf6f0e5bbbffecc5bd8c0431
> src/test/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 778148a7c033cba9004954cabc33a2b1d003dccf
> src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java 749ffeac6cb851f32bba7606390203d7a046a0e6
> src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 686087ef858b8a5a8e956d82a7bd692f7be28b12
> src/test/java/org/apache/aurora/scheduler/thrift/ThriftIT.java 40851c419e4d62e6545959eebc0ce144fdecc697
> src/test/python/apache/aurora/client/cli/test_inspect.py e4f43d0573c7862adc9bc679f4cea40cc76eac38
> src/test/python/apache/aurora/config/test_thrift.py 8e1d0e177959af12b97bdd1cd47845b72bc12fe1
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveCronJob 88e1c36a1aa2d192b95963f7aa36e243a447e4af
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate 32fdcdacde58345cdd6c4b449b82c0c90c2b2aae
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveTasks 4323031ec6bd128576c2a43ebc11f04a9f046e2f
>
>
> Diff: https://reviews.apache.org/r/66537/diff/4/
>
>
> Testing
> -------
>
> Unit tests for all new functionality have been added and run cleanly alongside checkstyle/static analysis tooling.
>
> Operational/functional testing completed within Spine infrastructure.
>
>
> Thanks,
>
> Steve Salevan
>
>
Re: Review Request 66537: Adding enhancements to Docker functionality
and client support for FetcherURIs
Posted by Aurora ReviewBot <wf...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66537/#review202335
-----------------------------------------------------------
Master (a3d596e) is red with this patch.
./build-support/jenkins/build.sh
Pass 2: Analyzing classes (320 / 333) - 96% complete
Pass 2: Analyzing classes (321 / 333) - 96% complete
Pass 2: Analyzing classes (322 / 333) - 96% complete
Pass 2: Analyzing classes (323 / 333) - 96% complete
Pass 2: Analyzing classes (324 / 333) - 97% complete
Pass 2: Analyzing classes (325 / 333) - 97% complete
Pass 2: Analyzing classes (326 / 333) - 97% complete
Pass 2: Analyzing classes (327 / 333) - 98% complete
Pass 2: Analyzing classes (328 / 333) - 98% complete
Pass 2: Analyzing classes (329 / 333) - 98% complete
Pass 2: Analyzing classes (330 / 333) - 99% complete
Pass 2: Analyzing classes (331 / 333) - 99% complete
Pass 2: Analyzing classes (332 / 333) - 99% complete
Pass 2: Analyzing classes (333 / 333) - 100% complete
Done with analysis
:test
org.apache.aurora.scheduler.events.WebhookTest > testTaskChangedWithOldStateError FAILED
java.lang.AssertionError at WebhookTest.java:251
I0503 01:42:21.319 [ShutdownHook, SchedulerMain] Stopping scheduler services.
1087 tests completed, 1 failed, 1 skipped
:test FAILED
:jacocoTestReport
Coverage report generated: file:///home/jenkins/jenkins-slave/workspace/AuroraBot/dist/reports/jacoco/test/html/index.html
:jacocoTestCoverageVerification
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':test'.
> There were failing tests. See the report at: file:///home/jenkins/jenkins-slave/workspace/AuroraBot/dist/reports/tests/test/index.html
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
BUILD FAILED in 8m 46s
45 actionable tasks: 36 executed, 9 up-to-date
I will refresh this build result if you post a review containing "@ReviewBot retry"
- Aurora ReviewBot
On May 3, 2018, 1:06 a.m., Steve Salevan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66537/
> -----------------------------------------------------------
>
> (Updated May 3, 2018, 1:06 a.m.)
>
>
> Review request for Aurora, Renan DelValle and Stephan Erb.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> Hey there!
>
> Over here at Spine we've added a few enhancements to Aurora's Docker functionality to support several of our use cases. We'd like to submit these back up to the upstream to support the great work y'all are doing in this space.
>
> Here's what's included in this RB:
>
> * Support for the force_pull flag to help ensure container freshness
> * Overrides for a Docker --entrypoint which can be specified on the job
> * Support for alternative Docker networks (defaults to HOST)
> * Support for user Docker networks
>
> We currently use Aurora to schedule Docker containers without the use of Thermos, so we've added support for server-side templating of common Thermos variables into Docker executor's parameters for this purpose.
>
> This change modifies Aurora's api.thrift with several new optional fields, and all added code handles their absence gracefully, so no backfills have been added. We've threaded these schema changes through to the Python Aurora client alongside support for the Mesos Fetcher URIs already supported server-side.
>
> Let me know what you think and thanks!
>
>
> Diffs
> -----
>
> .gitignore 9ce74ebbbc57b77d912eaa573a8fb18ed4aa3c15
> api/src/main/thrift/org/apache/aurora/gen/api.thrift ef754e32172e7490a47a13e7b526f243ffa3efeb
> src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 4073229b74d0e0e7fd31552bd96894ceb8a0971a
> src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java bcb2bbf882f43d813dd26c746d806e78bae6bcf3
> src/main/python/apache/aurora/config/schema/base.py a629bcd1261e5959da0a8458a55545d4e2c2a7a5
> src/main/python/apache/aurora/config/thrift.py 6d2dde6e964daa68bf6f0e5bbbffecc5bd8c0431
> src/test/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 778148a7c033cba9004954cabc33a2b1d003dccf
> src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java 749ffeac6cb851f32bba7606390203d7a046a0e6
> src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 686087ef858b8a5a8e956d82a7bd692f7be28b12
> src/test/java/org/apache/aurora/scheduler/thrift/ThriftIT.java 40851c419e4d62e6545959eebc0ce144fdecc697
> src/test/python/apache/aurora/client/cli/test_inspect.py e4f43d0573c7862adc9bc679f4cea40cc76eac38
> src/test/python/apache/aurora/config/test_thrift.py 8e1d0e177959af12b97bdd1cd47845b72bc12fe1
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveCronJob 88e1c36a1aa2d192b95963f7aa36e243a447e4af
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate 32fdcdacde58345cdd6c4b449b82c0c90c2b2aae
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveTasks 4323031ec6bd128576c2a43ebc11f04a9f046e2f
>
>
> Diff: https://reviews.apache.org/r/66537/diff/4/
>
>
> Testing
> -------
>
> Unit tests for all new functionality have been added and run cleanly alongside checkstyle/static analysis tooling.
>
> Operational/functional testing completed within Spine infrastructure.
>
>
> Thanks,
>
> Steve Salevan
>
>
Re: Review Request 66537: Adding enhancements to Docker functionality
and client support for FetcherURIs
Posted by Santhosh Kumar Shanmugham <sa...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66537/#review203817
-----------------------------------------------------------
Can you include an end-to-end test?
https://github.com/apache/aurora/blob/master/src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
api/src/main/thrift/org/apache/aurora/gen/api.thrift
Lines 191 (patched)
<https://reviews.apache.org/r/66537/#comment286149>
Should we just make this a string so that future network types supported by Docker does not need an Aurora update? Since Mesos's `DockerInfo` already type checks via enums this is probably leaking abstraction and is not needed?
api/src/main/thrift/org/apache/aurora/gen/api.thrift
Lines 238 (patched)
<https://reviews.apache.org/r/66537/#comment286130>
nit - s/Hub/registry/
src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
Lines 221-227 (patched)
<https://reviews.apache.org/r/66537/#comment286147>
Not an expert on how Docker entry points work. The Mesos documentation on invoking a Docker entry-point seems to differ here? Or I am not understanding this.
http://mesos.apache.org/documentation/latest/docker-containerizer/#commandinfo-to-run-docker-images
src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
Lines 293 (patched)
<https://reviews.apache.org/r/66537/#comment286148>
Should this be resolved when interpolating the pystachio schema in the client?
What about the other templated namespaces? http://aurora.apache.org/documentation/latest/reference/configuration/#template-namespaces
- Santhosh Kumar Shanmugham
On May 2, 2018, 6:06 p.m., Steve Salevan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66537/
> -----------------------------------------------------------
>
> (Updated May 2, 2018, 6:06 p.m.)
>
>
> Review request for Aurora, Renan DelValle and Stephan Erb.
>
>
> Bugs: AURORA-1982
> https://issues.apache.org/jira/browse/AURORA-1982
>
>
> Repository: aurora
>
>
> Description
> -------
>
> Hey there!
>
> Over here at Spine we've added a few enhancements to Aurora's Docker functionality to support several of our use cases. We'd like to submit these back up to the upstream to support the great work y'all are doing in this space.
>
> Here's what's included in this RB:
>
> * Support for the force_pull flag to help ensure container freshness
> * Overrides for a Docker --entrypoint which can be specified on the job
> * Support for alternative Docker networks (defaults to HOST)
> * Support for user Docker networks
>
> We currently use Aurora to schedule Docker containers without the use of Thermos, so we've added support for server-side templating of common Thermos variables into Docker executor's parameters for this purpose.
>
> This change modifies Aurora's api.thrift with several new optional fields, and all added code handles their absence gracefully, so no backfills have been added. We've threaded these schema changes through to the Python Aurora client alongside support for the Mesos Fetcher URIs already supported server-side.
>
> Let me know what you think and thanks!
>
>
> Diffs
> -----
>
> .gitignore 9ce74ebbbc57b77d912eaa573a8fb18ed4aa3c15
> api/src/main/thrift/org/apache/aurora/gen/api.thrift ef754e32172e7490a47a13e7b526f243ffa3efeb
> src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 4073229b74d0e0e7fd31552bd96894ceb8a0971a
> src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java bcb2bbf882f43d813dd26c746d806e78bae6bcf3
> src/main/python/apache/aurora/config/schema/base.py a629bcd1261e5959da0a8458a55545d4e2c2a7a5
> src/main/python/apache/aurora/config/thrift.py 6d2dde6e964daa68bf6f0e5bbbffecc5bd8c0431
> src/test/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 778148a7c033cba9004954cabc33a2b1d003dccf
> src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java 749ffeac6cb851f32bba7606390203d7a046a0e6
> src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 686087ef858b8a5a8e956d82a7bd692f7be28b12
> src/test/java/org/apache/aurora/scheduler/thrift/ThriftIT.java 40851c419e4d62e6545959eebc0ce144fdecc697
> src/test/python/apache/aurora/client/cli/test_inspect.py e4f43d0573c7862adc9bc679f4cea40cc76eac38
> src/test/python/apache/aurora/config/test_thrift.py 8e1d0e177959af12b97bdd1cd47845b72bc12fe1
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveCronJob 88e1c36a1aa2d192b95963f7aa36e243a447e4af
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate 32fdcdacde58345cdd6c4b449b82c0c90c2b2aae
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveTasks 4323031ec6bd128576c2a43ebc11f04a9f046e2f
>
>
> Diff: https://reviews.apache.org/r/66537/diff/4/
>
>
> Testing
> -------
>
> Unit tests for all new functionality have been added and run cleanly alongside checkstyle/static analysis tooling.
>
> Operational/functional testing completed within Spine infrastructure.
>
>
> Thanks,
>
> Steve Salevan
>
>