You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@aurora.apache.org by Bill Farner <wf...@apache.org> on 2017/12/12 01:35:41 UTC
Review Request 64519: Add a test to detect incompatible storage
changes
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64519/
-----------------------------------------------------------
Review request for Aurora, David McLaughlin, Jordan Ly, and Stephan Erb.
Repository: aurora
Description
-------
This is intended as a safeguard against future compatibility regressions like [AURORA-1959](https://issues.apache.org/jira/browse/AURORA-1959).
I approached this with a few goals:
- golden files should be text-based and human-readable. This allows for non-opaque code reviews, and simpler remedy when it's necessary to update the goldens (i.e. copy-pasteable)
- guidance for schema evolution should be included directly in test failures
- separate detection of 'what the scheduler _can_ read' and 'what the scheduler writes'
- reasonably-complete schema coverage with minimal manual labor. These tests auto-generate structs to mitigate maintenance burden of test code as schemas evolve.
This is not a replacement for vigilance with data compatibility, but it should at least
1. mitigate unintentional breakages in compatibility, especially for new contributors
2. draw code reviewer attention to compatibility changes in a patch (signaled by changes to golden files)
Diffs
-----
src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 2bf7e7ba414d36c99a49599fdc8cf3abdc945dc9
src/main/java/org/apache/aurora/scheduler/config/CliOptions.java b7f43e0d6efbddcac640c3d39c7bc56400e68e68
src/main/java/org/apache/aurora/scheduler/storage/durability/DurableStorageModule.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/storage/log/LogStorageModule.java 75ec42aad0b822d6c3dcd5b1307a4fcb86caa5c0
src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 4929ecdec90a1ccbcafa4857dea83cec1e2d7fd4
src/test/java/org/apache/aurora/scheduler/storage/durability/DataCompatibilityTest.java PRE-CREATION
src/test/java/org/apache/aurora/scheduler/storage/durability/Generator.java PRE-CREATION
src/test/java/org/apache/aurora/scheduler/storage/log/NonVolatileStorageTest.java eb966d722dc01d1760566bc57358afac722d5fec
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/pruneJobUpdateHistory PRE-CREATION
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/removeJob PRE-CREATION
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/removeJobUpdate PRE-CREATION
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/removeLock PRE-CREATION
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/removeQuota PRE-CREATION
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/removeTasks PRE-CREATION
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveCronJob PRE-CREATION
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveFrameworkId PRE-CREATION
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveHostAttributes PRE-CREATION
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobInstanceUpdateEvent PRE-CREATION
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate PRE-CREATION
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdateEvent PRE-CREATION
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveLock PRE-CREATION
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveQuota PRE-CREATION
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveTasks PRE-CREATION
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/1-pruneJobUpdateHistory PRE-CREATION
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/10-saveJobUpdate PRE-CREATION
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/11-saveJobUpdateEvent PRE-CREATION
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/12-saveJobInstanceUpdateEvent PRE-CREATION
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/13-saveLock PRE-CREATION
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/14-saveQuota PRE-CREATION
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/15-saveTasks PRE-CREATION
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/2-removeJob PRE-CREATION
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/3-removeJobUpdate PRE-CREATION
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/4-removeLock PRE-CREATION
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/5-removeQuota PRE-CREATION
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/6-removeTasks PRE-CREATION
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/7-saveCronJob PRE-CREATION
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/8-saveFrameworkId PRE-CREATION
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/9-saveHostAttributes PRE-CREATION
Diff: https://reviews.apache.org/r/64519/diff/1/
Testing
-------
Thanks,
Bill Farner
Re: Review Request 64519: Add a test to detect incompatible storage
changes
Posted by Aurora ReviewBot <wf...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64519/#review193977
-----------------------------------------------------------
Ship it!
Master (2e1ca42) is green with this patch.
./build-support/jenkins/build.sh
I will refresh this build result if you post a review containing "@ReviewBot retry"
- Aurora ReviewBot
On Dec. 12, 2017, 1:35 a.m., Bill Farner wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64519/
> -----------------------------------------------------------
>
> (Updated Dec. 12, 2017, 1:35 a.m.)
>
>
> Review request for Aurora, David McLaughlin, Jordan Ly, and Stephan Erb.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> This is intended as a safeguard against future compatibility regressions like [AURORA-1959](https://issues.apache.org/jira/browse/AURORA-1959).
>
> I approached this with a few goals:
>
> - golden files should be text-based and human-readable. This allows for non-opaque code reviews, and simpler remedy when it's necessary to update the goldens (i.e. copy-pasteable)
> - guidance for schema evolution should be included directly in test failures
> - separate detection of 'what the scheduler _can_ read' and 'what the scheduler writes'
> - reasonably-complete schema coverage with minimal manual labor. These tests auto-generate structs to mitigate maintenance burden of test code as schemas evolve.
>
> This is not a replacement for vigilance with data compatibility, but it should at least
>
> 1. mitigate unintentional breakages in compatibility, especially for new contributors
> 2. draw code reviewer attention to compatibility changes in a patch (signaled by changes to golden files)
>
>
> Diffs
> -----
>
> src/test/java/org/apache/aurora/scheduler/storage/durability/DataCompatibilityTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/storage/durability/Generator.java PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/pruneJobUpdateHistory PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/removeJob PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/removeJobUpdate PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/removeLock PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/removeQuota PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/removeTasks PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveCronJob PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveFrameworkId PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveHostAttributes PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobInstanceUpdateEvent PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdateEvent PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveLock PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveQuota PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveTasks PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/1-pruneJobUpdateHistory PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/10-saveJobUpdate PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/11-saveJobUpdateEvent PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/12-saveJobInstanceUpdateEvent PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/13-saveLock PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/14-saveQuota PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/15-saveTasks PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/2-removeJob PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/3-removeJobUpdate PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/4-removeLock PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/5-removeQuota PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/6-removeTasks PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/7-saveCronJob PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/8-saveFrameworkId PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/9-saveHostAttributes PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/64519/diff/4/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Bill Farner
>
>
Re: Review Request 64519: Add a test to detect incompatible storage
changes
Posted by Bill Farner <wf...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64519/#review193479
-----------------------------------------------------------
Reviewer notes
src/main/java/org/apache/aurora/scheduler/storage/durability/DurableStorageModule.java
Lines 27 (patched)
<https://reviews.apache.org/r/64519/#comment272036>
This refactor was ported from [r/64288/](https://reviews.apache.org/r/64288/) to reduce the necessary surface area in `DataCompatibilityTest`.
src/test/java/org/apache/aurora/scheduler/storage/durability/DataCompatibilityTest.java
Lines 113 (patched)
<https://reviews.apache.org/r/64519/#comment272037>
This (hopefully) solves the minimization of manual labor. I was not pleased with the idea of hand-rolling fully-described objects for each type, as they would be surely become outdated.
src/test/java/org/apache/aurora/scheduler/storage/durability/DataCompatibilityTest.java
Lines 126 (patched)
<https://reviews.apache.org/r/64519/#comment272038>
Some types have specific requirements for recovery. In this case, recovery will fail if a `SaveQuota` does not have (cpu, ram, disk) resources. To accommodate this, it's possible to mix-and-match manually-described structs and auto-generated ones.
src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/pruneJobUpdateHistory
Lines 1 (patched)
<https://reviews.apache.org/r/64519/#comment272040>
Here begins the golden files. These are all pretty-printed `TJSONProtocol`. While it's not easy to look at one of these and know the structure, it is fairly straightforward to map a thrift change to a golden file change.
- Bill Farner
On Dec. 11, 2017, 5:35 p.m., Bill Farner wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64519/
> -----------------------------------------------------------
>
> (Updated Dec. 11, 2017, 5:35 p.m.)
>
>
> Review request for Aurora, David McLaughlin, Jordan Ly, and Stephan Erb.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> This is intended as a safeguard against future compatibility regressions like [AURORA-1959](https://issues.apache.org/jira/browse/AURORA-1959).
>
> I approached this with a few goals:
>
> - golden files should be text-based and human-readable. This allows for non-opaque code reviews, and simpler remedy when it's necessary to update the goldens (i.e. copy-pasteable)
> - guidance for schema evolution should be included directly in test failures
> - separate detection of 'what the scheduler _can_ read' and 'what the scheduler writes'
> - reasonably-complete schema coverage with minimal manual labor. These tests auto-generate structs to mitigate maintenance burden of test code as schemas evolve.
>
> This is not a replacement for vigilance with data compatibility, but it should at least
>
> 1. mitigate unintentional breakages in compatibility, especially for new contributors
> 2. draw code reviewer attention to compatibility changes in a patch (signaled by changes to golden files)
>
>
> Diffs
> -----
>
> src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 2bf7e7ba414d36c99a49599fdc8cf3abdc945dc9
> src/main/java/org/apache/aurora/scheduler/config/CliOptions.java b7f43e0d6efbddcac640c3d39c7bc56400e68e68
> src/main/java/org/apache/aurora/scheduler/storage/durability/DurableStorageModule.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/log/LogStorageModule.java 75ec42aad0b822d6c3dcd5b1307a4fcb86caa5c0
> src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 4929ecdec90a1ccbcafa4857dea83cec1e2d7fd4
> src/test/java/org/apache/aurora/scheduler/storage/durability/DataCompatibilityTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/storage/durability/Generator.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/storage/log/NonVolatileStorageTest.java eb966d722dc01d1760566bc57358afac722d5fec
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/pruneJobUpdateHistory PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/removeJob PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/removeJobUpdate PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/removeLock PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/removeQuota PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/removeTasks PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveCronJob PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveFrameworkId PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveHostAttributes PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobInstanceUpdateEvent PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdateEvent PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveLock PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveQuota PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveTasks PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/1-pruneJobUpdateHistory PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/10-saveJobUpdate PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/11-saveJobUpdateEvent PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/12-saveJobInstanceUpdateEvent PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/13-saveLock PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/14-saveQuota PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/15-saveTasks PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/2-removeJob PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/3-removeJobUpdate PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/4-removeLock PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/5-removeQuota PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/6-removeTasks PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/7-saveCronJob PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/8-saveFrameworkId PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/9-saveHostAttributes PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/64519/diff/1/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Bill Farner
>
>
Re: Review Request 64519: Add a test to detect incompatible storage
changes
Posted by Stephan Erb <se...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64519/#review193802
-----------------------------------------------------------
I am traveling for the next 5 days, so my review will be delayed. Feel free to ship without me though.
- Stephan Erb
On Dec. 12, 2017, 2:35 a.m., Bill Farner wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64519/
> -----------------------------------------------------------
>
> (Updated Dec. 12, 2017, 2:35 a.m.)
>
>
> Review request for Aurora, David McLaughlin, Jordan Ly, and Stephan Erb.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> This is intended as a safeguard against future compatibility regressions like [AURORA-1959](https://issues.apache.org/jira/browse/AURORA-1959).
>
> I approached this with a few goals:
>
> - golden files should be text-based and human-readable. This allows for non-opaque code reviews, and simpler remedy when it's necessary to update the goldens (i.e. copy-pasteable)
> - guidance for schema evolution should be included directly in test failures
> - separate detection of 'what the scheduler _can_ read' and 'what the scheduler writes'
> - reasonably-complete schema coverage with minimal manual labor. These tests auto-generate structs to mitigate maintenance burden of test code as schemas evolve.
>
> This is not a replacement for vigilance with data compatibility, but it should at least
>
> 1. mitigate unintentional breakages in compatibility, especially for new contributors
> 2. draw code reviewer attention to compatibility changes in a patch (signaled by changes to golden files)
>
>
> Diffs
> -----
>
> src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 2bf7e7ba414d36c99a49599fdc8cf3abdc945dc9
> src/main/java/org/apache/aurora/scheduler/config/CliOptions.java b7f43e0d6efbddcac640c3d39c7bc56400e68e68
> src/main/java/org/apache/aurora/scheduler/storage/durability/DurableStorageModule.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/log/LogStorageModule.java 75ec42aad0b822d6c3dcd5b1307a4fcb86caa5c0
> src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 4929ecdec90a1ccbcafa4857dea83cec1e2d7fd4
> src/test/java/org/apache/aurora/scheduler/storage/durability/DataCompatibilityTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/storage/durability/Generator.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/storage/log/NonVolatileStorageTest.java eb966d722dc01d1760566bc57358afac722d5fec
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/pruneJobUpdateHistory PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/removeJob PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/removeJobUpdate PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/removeLock PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/removeQuota PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/removeTasks PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveCronJob PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveFrameworkId PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveHostAttributes PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobInstanceUpdateEvent PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdateEvent PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveLock PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveQuota PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveTasks PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/1-pruneJobUpdateHistory PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/10-saveJobUpdate PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/11-saveJobUpdateEvent PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/12-saveJobInstanceUpdateEvent PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/13-saveLock PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/14-saveQuota PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/15-saveTasks PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/2-removeJob PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/3-removeJobUpdate PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/4-removeLock PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/5-removeQuota PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/6-removeTasks PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/7-saveCronJob PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/8-saveFrameworkId PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/9-saveHostAttributes PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/64519/diff/2/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Bill Farner
>
>
Re: Review Request 64519: Add a test to detect incompatible storage
changes
Posted by Aurora ReviewBot <wf...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64519/#review193482
-----------------------------------------------------------
Ship it!
Master (ef24c2c) is green with this patch.
./build-support/jenkins/build.sh
I will refresh this build result if you post a review containing "@ReviewBot retry"
- Aurora ReviewBot
On Dec. 12, 2017, 1:35 a.m., Bill Farner wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64519/
> -----------------------------------------------------------
>
> (Updated Dec. 12, 2017, 1:35 a.m.)
>
>
> Review request for Aurora, David McLaughlin, Jordan Ly, and Stephan Erb.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> This is intended as a safeguard against future compatibility regressions like [AURORA-1959](https://issues.apache.org/jira/browse/AURORA-1959).
>
> I approached this with a few goals:
>
> - golden files should be text-based and human-readable. This allows for non-opaque code reviews, and simpler remedy when it's necessary to update the goldens (i.e. copy-pasteable)
> - guidance for schema evolution should be included directly in test failures
> - separate detection of 'what the scheduler _can_ read' and 'what the scheduler writes'
> - reasonably-complete schema coverage with minimal manual labor. These tests auto-generate structs to mitigate maintenance burden of test code as schemas evolve.
>
> This is not a replacement for vigilance with data compatibility, but it should at least
>
> 1. mitigate unintentional breakages in compatibility, especially for new contributors
> 2. draw code reviewer attention to compatibility changes in a patch (signaled by changes to golden files)
>
>
> Diffs
> -----
>
> src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 2bf7e7ba414d36c99a49599fdc8cf3abdc945dc9
> src/main/java/org/apache/aurora/scheduler/config/CliOptions.java b7f43e0d6efbddcac640c3d39c7bc56400e68e68
> src/main/java/org/apache/aurora/scheduler/storage/durability/DurableStorageModule.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/storage/log/LogStorageModule.java 75ec42aad0b822d6c3dcd5b1307a4fcb86caa5c0
> src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 4929ecdec90a1ccbcafa4857dea83cec1e2d7fd4
> src/test/java/org/apache/aurora/scheduler/storage/durability/DataCompatibilityTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/storage/durability/Generator.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/storage/log/NonVolatileStorageTest.java eb966d722dc01d1760566bc57358afac722d5fec
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/pruneJobUpdateHistory PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/removeJob PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/removeJobUpdate PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/removeLock PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/removeQuota PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/removeTasks PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveCronJob PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveFrameworkId PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveHostAttributes PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobInstanceUpdateEvent PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdateEvent PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveLock PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveQuota PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveTasks PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/1-pruneJobUpdateHistory PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/10-saveJobUpdate PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/11-saveJobUpdateEvent PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/12-saveJobInstanceUpdateEvent PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/13-saveLock PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/14-saveQuota PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/15-saveTasks PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/2-removeJob PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/3-removeJobUpdate PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/4-removeLock PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/5-removeQuota PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/6-removeTasks PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/7-saveCronJob PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/8-saveFrameworkId PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/9-saveHostAttributes PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/64519/diff/2/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Bill Farner
>
>
Re: Review Request 64519: Add a test to detect incompatible storage
changes
Posted by Jordan Ly <jo...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64519/#review194729
-----------------------------------------------------------
Ship it!
This LGTM!
- Jordan Ly
On Dec. 12, 2017, 1:35 a.m., Bill Farner wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64519/
> -----------------------------------------------------------
>
> (Updated Dec. 12, 2017, 1:35 a.m.)
>
>
> Review request for Aurora, David McLaughlin, Jordan Ly, and Stephan Erb.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> This is intended as a safeguard against future compatibility regressions like [AURORA-1959](https://issues.apache.org/jira/browse/AURORA-1959).
>
> I approached this with a few goals:
>
> - golden files should be text-based and human-readable. This allows for non-opaque code reviews, and simpler remedy when it's necessary to update the goldens (i.e. copy-pasteable)
> - guidance for schema evolution should be included directly in test failures
> - separate detection of 'what the scheduler _can_ read' and 'what the scheduler writes'
> - reasonably-complete schema coverage with minimal manual labor. These tests auto-generate structs to mitigate maintenance burden of test code as schemas evolve.
>
> This is not a replacement for vigilance with data compatibility, but it should at least
>
> 1. mitigate unintentional breakages in compatibility, especially for new contributors
> 2. draw code reviewer attention to compatibility changes in a patch (signaled by changes to golden files)
>
>
> Diffs
> -----
>
> src/test/java/org/apache/aurora/scheduler/storage/durability/DataCompatibilityTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/storage/durability/Generator.java PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/pruneJobUpdateHistory PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/removeJob PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/removeJobUpdate PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/removeLock PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/removeQuota PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/removeTasks PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveCronJob PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveFrameworkId PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveHostAttributes PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobInstanceUpdateEvent PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdateEvent PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveLock PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveQuota PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveTasks PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/1-pruneJobUpdateHistory PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/10-saveJobUpdate PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/11-saveJobUpdateEvent PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/12-saveJobInstanceUpdateEvent PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/13-saveLock PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/14-saveQuota PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/15-saveTasks PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/2-removeJob PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/3-removeJobUpdate PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/4-removeLock PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/5-removeQuota PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/6-removeTasks PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/7-saveCronJob PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/8-saveFrameworkId PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/9-saveHostAttributes PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/64519/diff/4/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Bill Farner
>
>
Re: Review Request 64519: Add a test to detect incompatible storage
changes
Posted by Aurora ReviewBot <wf...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64519/#review193974
-----------------------------------------------------------
Master (2e1ca42) is red with this patch.
./build-support/jenkins/build.sh
[253] ./~/react-router-dom/es/Redirect.js 137 bytes {0} [built]
+ 262 hidden modules
:processResources
:classes
:jar
:moreStartScripts
:startScripts
:distTar
:distZip
:assemble
:compileTestJava/home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/java/org/apache/aurora/scheduler/storage/durability/DataCompatibilityTest.java:326: error: TestPersistence is not abstract and does not override abstract method recover() in Persistence
private static class TestPersistence implements Persistence {
^
/home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/java/org/apache/aurora/scheduler/storage/durability/DataCompatibilityTest.java:339: error: recover() in TestPersistence cannot implement recover() in Persistence
public Stream<Op> recover() {
^
return type Stream<Op> is not compatible with Stream<Edit>
/home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/java/org/apache/aurora/scheduler/storage/durability/DataCompatibilityTest.java:338: error: method does not override or implement a method from a supertype
@Override
^
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/java/org/apache/aurora/scheduler/storage/durability/DurableStorageTest.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
3 errors
FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':compileTestJava'.
> Compilation failed; see the compiler error output for details.
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
BUILD FAILED in 4m 30s
28 actionable tasks: 22 executed, 6 up-to-date
I will refresh this build result if you post a review containing "@ReviewBot retry"
- Aurora ReviewBot
On Dec. 12, 2017, 1:35 a.m., Bill Farner wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64519/
> -----------------------------------------------------------
>
> (Updated Dec. 12, 2017, 1:35 a.m.)
>
>
> Review request for Aurora, David McLaughlin, Jordan Ly, and Stephan Erb.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> This is intended as a safeguard against future compatibility regressions like [AURORA-1959](https://issues.apache.org/jira/browse/AURORA-1959).
>
> I approached this with a few goals:
>
> - golden files should be text-based and human-readable. This allows for non-opaque code reviews, and simpler remedy when it's necessary to update the goldens (i.e. copy-pasteable)
> - guidance for schema evolution should be included directly in test failures
> - separate detection of 'what the scheduler _can_ read' and 'what the scheduler writes'
> - reasonably-complete schema coverage with minimal manual labor. These tests auto-generate structs to mitigate maintenance burden of test code as schemas evolve.
>
> This is not a replacement for vigilance with data compatibility, but it should at least
>
> 1. mitigate unintentional breakages in compatibility, especially for new contributors
> 2. draw code reviewer attention to compatibility changes in a patch (signaled by changes to golden files)
>
>
> Diffs
> -----
>
> src/test/java/org/apache/aurora/scheduler/storage/durability/DataCompatibilityTest.java PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/storage/durability/Generator.java PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/pruneJobUpdateHistory PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/removeJob PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/removeJobUpdate PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/removeLock PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/removeQuota PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/removeTasks PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveCronJob PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveFrameworkId PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveHostAttributes PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobInstanceUpdateEvent PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdate PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveJobUpdateEvent PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveLock PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveQuota PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/current/saveTasks PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/1-pruneJobUpdateHistory PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/10-saveJobUpdate PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/11-saveJobUpdateEvent PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/12-saveJobInstanceUpdateEvent PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/13-saveLock PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/14-saveQuota PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/15-saveTasks PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/2-removeJob PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/3-removeJobUpdate PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/4-removeLock PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/5-removeQuota PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/6-removeTasks PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/7-saveCronJob PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/8-saveFrameworkId PRE-CREATION
> src/test/resources/org/apache/aurora/scheduler/storage/durability/goldens/read-compatible/9-saveHostAttributes PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/64519/diff/3/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Bill Farner
>
>