You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@geode.apache.org by Owen Nichols <on...@vmware.com> on 2021/12/23 03:13:47 UTC
[DISCUSS] proposal to pare down old-version testing
Since adopting our N-2 support policy, the list of released versions in /settings.gradle has ballooned to over 30 entries [1].
CI tests use this list to confirm that we don’t break rolling upgrade ability or compatibility with older clients, but some of these tests don’t seem to scale well: PR#7203 to add the most recent 3 releases (bringing the total to 33) is unable to pass CI after 8 tries.
Possible solutions fall into two categories: keep the full list and throw developers and/or more hardware at the struggling tests, or concede that testing every version is not a scalable approach and find ways to shorten the list, e.g. randomly select a subset of old versions at runtime, or manually pare down the list.
I propose to shorten the list [2] by keeping only the latest patch for each minor (unless the client or server protocol version has changed, so also keep the patch prior to 1.12.1 and prior to 1.13.2). As long as a patch release doesn’t change the client or server protocol version, I see low value in testing upgrades from every patch version to every future version forever. The months between patch releases already provide plenty of upgrade coverage on that specific patch, then we can move on to the next…even if there could somehow be a corner-case where transitive property of upgradability doesn’t hold, most users probably take the latest-to-latest upgrade path anyway, which will always be tested.
Let’s keep discussion open until 3PM PST Jan 5. In case of no response, I will assume lazy consensus and update settings.gradle as proposed [2].
[1] Current list from https://github.com/apache/geode/blob/develop/settings.gradle#L72-L101 :
1.0.0-incubating
1.1.0
1.1.1
1.2.0
1.3.0
1.4.0
1.5.0
1.6.0
1.7.0
1.8.0
1.9.0
1.9.1
1.9.2
1.10.0
1.11.0
1.12.0
1.12.1
1.12.2
1.12.3
1.12.4
1.12.5
1.12.6
1.12.7*
1.13.0
1.13.1
1.13.2
1.13.3
1.13.4
1.13.5
1.13.6*
1.14.0
1.14.1
1.14.2*
*=released, but not yet added to settings.gradle due to PR#7203 not able to pass CI due to size of version list
[2] Proposed shortlist:
1.1.1
1.2.0
1.3.0
1.4.0
1.5.0
1.6.0
1.7.0
1.8.0
1.9.2
1.10.0
1.11.0
1.12.0
1.12.7
1.13.1
1.13.6
1.14.2
Re: [DISCUSS] proposal to pare down old-version testing
Posted by Owen Nichols <on...@vmware.com>.
Sounds like there is consensus to implement this proposal.
Support branches and develop have been updated. Release scripts will automatically check whether the protocol has changed and maintain the list accordingly.
On 1/4/22, 11:21 AM, "Alexander Murmann" <am...@vmware.com> wrote:
+1 Great compromise! Let's not forget about this whenever we change the protocol
--
Please provide anonymous feedback for me here<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fforms.gle%2FkpUiaty3jt8X9y4H9&data=04%7C01%7Conichols%40vmware.com%7C951690bb3afc4c13b6ab08d9cfb75ec2%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637769208757591829%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=%2FSdd%2ByOW%2B0sf1u46uQr27AVPKejKY6M8cexjeC22WVk%3D&reserved=0>.
________________________________
From: Anilkumar Gingade <ag...@vmware.com>
Sent: Tuesday, January 4, 2022 07:09
To: dev@geode.apache.org <de...@geode.apache.org>
Subject: Re: [DISCUSS] proposal to pare down old-version testing
+1
Thanks for bringing this and taking care of this.
-Anil.
On 1/3/22, 10:41 AM, "Dan Smith" <da...@vmware.com> wrote:
Looking at KnownVersion.java - we did make protocol changes in 1.12.1 and 1.13.2. So, my suggestion would be to keep 1.12.0 and 1.13.1, but dop all the other patch versions that aren't the latest.
-Dan
________________________________
From: Dan Smith <da...@vmware.com>
Sent: Monday, January 3, 2022 10:37 AM
To: dev@geode.apache.org <de...@geode.apache.org>
Subject: Re: [DISCUSS] proposal to pare down old-version testing
+1 - this seems reasonable to me. If we do make a protocol change in a patch, we could potentially keep around an older patch version just in that specific case, but otherwise I think this makes sense.
-Dan
________________________________
From: Anthony Baker <ba...@vmware.com>
Sent: Thursday, December 23, 2021 8:53 AM
To: dev@geode.apache.org <de...@geode.apache.org>
Subject: Re: [DISCUSS] proposal to pare down old-version testing
Interesting data point: 40% of maven central downloads last month were for version 1.4.0. Of course those numbers can be easily skewed by CI bots, but still!
@Owen, I think your suggestion nicely improves practicality while continuing to support strong compatibility. In many cases it’s quite a bit easier to upgrade the Geode server cluster compared to potentially many, many client applications. Supporting older client versions gives users time to upgrade, quicker access to bug fixes, and helps avoids downtime.
+1
Anthony
> On Dec 22, 2021, at 7:13 PM, Owen Nichols <on...@vmware.com> wrote:
>
> Since adopting our N-2 support policy, the list of released versions in /settings.gradle has ballooned to over 30 entries [1].
>
> CI tests use this list to confirm that we don’t break rolling upgrade ability or compatibility with older clients, but some of these tests don’t seem to scale well: PR#7203 to add the most recent 3 releases (bringing the total to 33) is unable to pass CI after 8 tries.
>
> Possible solutions fall into two categories: keep the full list and throw developers and/or more hardware at the struggling tests, or concede that testing every version is not a scalable approach and find ways to shorten the list, e.g. randomly select a subset of old versions at runtime, or manually pare down the list.
>
> I propose to shorten the list [2] by keeping only the latest patch for each minor (unless the client or server protocol version has changed, so also keep the patch prior to 1.12.1 and prior to 1.13.2). As long as a patch release doesn’t change the client or server protocol version, I see low value in testing upgrades from every patch version to every future version forever. The months between patch releases already provide plenty of upgrade coverage on that specific patch, then we can move on to the next…even if there could somehow be a corner-case where transitive property of upgradability doesn’t hold, most users probably take the latest-to-latest upgrade path anyway, which will always be tested.
>
> Let’s keep discussion open until 3PM PST Jan 5. In case of no response, I will assume lazy consensus and update settings.gradle as proposed [2].
>
>
>
> [1] Current list from https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fgeode%2Fblob%2Fdevelop%2Fsettings.gradle%23L72-L101&data=04%7C01%7Conichols%40vmware.com%7C951690bb3afc4c13b6ab08d9cfb75ec2%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637769208757591829%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=eZ3X5mtDxcSRIb%2FJwFmfwnIhK7zO1byNKC133qlZdDY%3D&reserved=0 :
> 1.0.0-incubating
> 1.1.0
> 1.1.1
> 1.2.0
> 1.3.0
> 1.4.0
> 1.5.0
> 1.6.0
> 1.7.0
> 1.8.0
> 1.9.0
> 1.9.1
> 1.9.2
> 1.10.0
> 1.11.0
> 1.12.0
> 1.12.1
> 1.12.2
> 1.12.3
> 1.12.4
> 1.12.5
> 1.12.6
> 1.12.7*
> 1.13.0
> 1.13.1
> 1.13.2
> 1.13.3
> 1.13.4
> 1.13.5
> 1.13.6*
> 1.14.0
> 1.14.1
> 1.14.2*
> *=released, but not yet added to settings.gradle due to PR#7203 not able to pass CI due to size of version list
>
> [2] Proposed shortlist:
> 1.1.1
> 1.2.0
> 1.3.0
> 1.4.0
> 1.5.0
> 1.6.0
> 1.7.0
> 1.8.0
> 1.9.2
> 1.10.0
> 1.11.0
> 1.12.0
> 1.12.7
> 1.13.1
> 1.13.6
> 1.14.2
>
Re: [DISCUSS] proposal to pare down old-version testing
Posted by Alexander Murmann <am...@vmware.com>.
+1 Great compromise! Let's not forget about this whenever we change the protocol
--
Please provide anonymous feedback for me here<https://forms.gle/kpUiaty3jt8X9y4H9>.
________________________________
From: Anilkumar Gingade <ag...@vmware.com>
Sent: Tuesday, January 4, 2022 07:09
To: dev@geode.apache.org <de...@geode.apache.org>
Subject: Re: [DISCUSS] proposal to pare down old-version testing
+1
Thanks for bringing this and taking care of this.
-Anil.
On 1/3/22, 10:41 AM, "Dan Smith" <da...@vmware.com> wrote:
Looking at KnownVersion.java - we did make protocol changes in 1.12.1 and 1.13.2. So, my suggestion would be to keep 1.12.0 and 1.13.1, but dop all the other patch versions that aren't the latest.
-Dan
________________________________
From: Dan Smith <da...@vmware.com>
Sent: Monday, January 3, 2022 10:37 AM
To: dev@geode.apache.org <de...@geode.apache.org>
Subject: Re: [DISCUSS] proposal to pare down old-version testing
+1 - this seems reasonable to me. If we do make a protocol change in a patch, we could potentially keep around an older patch version just in that specific case, but otherwise I think this makes sense.
-Dan
________________________________
From: Anthony Baker <ba...@vmware.com>
Sent: Thursday, December 23, 2021 8:53 AM
To: dev@geode.apache.org <de...@geode.apache.org>
Subject: Re: [DISCUSS] proposal to pare down old-version testing
Interesting data point: 40% of maven central downloads last month were for version 1.4.0. Of course those numbers can be easily skewed by CI bots, but still!
@Owen, I think your suggestion nicely improves practicality while continuing to support strong compatibility. In many cases it’s quite a bit easier to upgrade the Geode server cluster compared to potentially many, many client applications. Supporting older client versions gives users time to upgrade, quicker access to bug fixes, and helps avoids downtime.
+1
Anthony
> On Dec 22, 2021, at 7:13 PM, Owen Nichols <on...@vmware.com> wrote:
>
> Since adopting our N-2 support policy, the list of released versions in /settings.gradle has ballooned to over 30 entries [1].
>
> CI tests use this list to confirm that we don’t break rolling upgrade ability or compatibility with older clients, but some of these tests don’t seem to scale well: PR#7203 to add the most recent 3 releases (bringing the total to 33) is unable to pass CI after 8 tries.
>
> Possible solutions fall into two categories: keep the full list and throw developers and/or more hardware at the struggling tests, or concede that testing every version is not a scalable approach and find ways to shorten the list, e.g. randomly select a subset of old versions at runtime, or manually pare down the list.
>
> I propose to shorten the list [2] by keeping only the latest patch for each minor (unless the client or server protocol version has changed, so also keep the patch prior to 1.12.1 and prior to 1.13.2). As long as a patch release doesn’t change the client or server protocol version, I see low value in testing upgrades from every patch version to every future version forever. The months between patch releases already provide plenty of upgrade coverage on that specific patch, then we can move on to the next…even if there could somehow be a corner-case where transitive property of upgradability doesn’t hold, most users probably take the latest-to-latest upgrade path anyway, which will always be tested.
>
> Let’s keep discussion open until 3PM PST Jan 5. In case of no response, I will assume lazy consensus and update settings.gradle as proposed [2].
>
>
>
> [1] Current list from https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fgeode%2Fblob%2Fdevelop%2Fsettings.gradle%23L72-L101&data=04%7C01%7Camurmann%40vmware.com%7C579b50b0cacc49ed54df08d9cf9436a0%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637769057757912491%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=8%2F%2BuhyHnyI0ZPbY4Y%2F0vRG6O%2Bi3wuSf3%2BeISAvg0DR8%3D&reserved=0 :
> 1.0.0-incubating
> 1.1.0
> 1.1.1
> 1.2.0
> 1.3.0
> 1.4.0
> 1.5.0
> 1.6.0
> 1.7.0
> 1.8.0
> 1.9.0
> 1.9.1
> 1.9.2
> 1.10.0
> 1.11.0
> 1.12.0
> 1.12.1
> 1.12.2
> 1.12.3
> 1.12.4
> 1.12.5
> 1.12.6
> 1.12.7*
> 1.13.0
> 1.13.1
> 1.13.2
> 1.13.3
> 1.13.4
> 1.13.5
> 1.13.6*
> 1.14.0
> 1.14.1
> 1.14.2*
> *=released, but not yet added to settings.gradle due to PR#7203 not able to pass CI due to size of version list
>
> [2] Proposed shortlist:
> 1.1.1
> 1.2.0
> 1.3.0
> 1.4.0
> 1.5.0
> 1.6.0
> 1.7.0
> 1.8.0
> 1.9.2
> 1.10.0
> 1.11.0
> 1.12.0
> 1.12.7
> 1.13.1
> 1.13.6
> 1.14.2
>
Re: [DISCUSS] proposal to pare down old-version testing
Posted by Anilkumar Gingade <ag...@vmware.com>.
+1
Thanks for bringing this and taking care of this.
-Anil.
On 1/3/22, 10:41 AM, "Dan Smith" <da...@vmware.com> wrote:
Looking at KnownVersion.java - we did make protocol changes in 1.12.1 and 1.13.2. So, my suggestion would be to keep 1.12.0 and 1.13.1, but dop all the other patch versions that aren't the latest.
-Dan
________________________________
From: Dan Smith <da...@vmware.com>
Sent: Monday, January 3, 2022 10:37 AM
To: dev@geode.apache.org <de...@geode.apache.org>
Subject: Re: [DISCUSS] proposal to pare down old-version testing
+1 - this seems reasonable to me. If we do make a protocol change in a patch, we could potentially keep around an older patch version just in that specific case, but otherwise I think this makes sense.
-Dan
________________________________
From: Anthony Baker <ba...@vmware.com>
Sent: Thursday, December 23, 2021 8:53 AM
To: dev@geode.apache.org <de...@geode.apache.org>
Subject: Re: [DISCUSS] proposal to pare down old-version testing
Interesting data point: 40% of maven central downloads last month were for version 1.4.0. Of course those numbers can be easily skewed by CI bots, but still!
@Owen, I think your suggestion nicely improves practicality while continuing to support strong compatibility. In many cases it’s quite a bit easier to upgrade the Geode server cluster compared to potentially many, many client applications. Supporting older client versions gives users time to upgrade, quicker access to bug fixes, and helps avoids downtime.
+1
Anthony
> On Dec 22, 2021, at 7:13 PM, Owen Nichols <on...@vmware.com> wrote:
>
> Since adopting our N-2 support policy, the list of released versions in /settings.gradle has ballooned to over 30 entries [1].
>
> CI tests use this list to confirm that we don’t break rolling upgrade ability or compatibility with older clients, but some of these tests don’t seem to scale well: PR#7203 to add the most recent 3 releases (bringing the total to 33) is unable to pass CI after 8 tries.
>
> Possible solutions fall into two categories: keep the full list and throw developers and/or more hardware at the struggling tests, or concede that testing every version is not a scalable approach and find ways to shorten the list, e.g. randomly select a subset of old versions at runtime, or manually pare down the list.
>
> I propose to shorten the list [2] by keeping only the latest patch for each minor (unless the client or server protocol version has changed, so also keep the patch prior to 1.12.1 and prior to 1.13.2). As long as a patch release doesn’t change the client or server protocol version, I see low value in testing upgrades from every patch version to every future version forever. The months between patch releases already provide plenty of upgrade coverage on that specific patch, then we can move on to the next…even if there could somehow be a corner-case where transitive property of upgradability doesn’t hold, most users probably take the latest-to-latest upgrade path anyway, which will always be tested.
>
> Let’s keep discussion open until 3PM PST Jan 5. In case of no response, I will assume lazy consensus and update settings.gradle as proposed [2].
>
>
>
> [1] Current list from https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fgeode%2Fblob%2Fdevelop%2Fsettings.gradle%23L72-L101&data=04%7C01%7Cagingade%40vmware.com%7C8b7e0a65ccc34f37b8a608d9cee8b189%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637768321075908672%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Kla9OIPhvSC54LCeIQ%2FhrjOcpn7K39cVBHfAWJlbVh8%3D&reserved=0 :
> 1.0.0-incubating
> 1.1.0
> 1.1.1
> 1.2.0
> 1.3.0
> 1.4.0
> 1.5.0
> 1.6.0
> 1.7.0
> 1.8.0
> 1.9.0
> 1.9.1
> 1.9.2
> 1.10.0
> 1.11.0
> 1.12.0
> 1.12.1
> 1.12.2
> 1.12.3
> 1.12.4
> 1.12.5
> 1.12.6
> 1.12.7*
> 1.13.0
> 1.13.1
> 1.13.2
> 1.13.3
> 1.13.4
> 1.13.5
> 1.13.6*
> 1.14.0
> 1.14.1
> 1.14.2*
> *=released, but not yet added to settings.gradle due to PR#7203 not able to pass CI due to size of version list
>
> [2] Proposed shortlist:
> 1.1.1
> 1.2.0
> 1.3.0
> 1.4.0
> 1.5.0
> 1.6.0
> 1.7.0
> 1.8.0
> 1.9.2
> 1.10.0
> 1.11.0
> 1.12.0
> 1.12.7
> 1.13.1
> 1.13.6
> 1.14.2
>
Re: [DISCUSS] proposal to pare down old-version testing
Posted by Dan Smith <da...@vmware.com>.
Looking at KnownVersion.java - we did make protocol changes in 1.12.1 and 1.13.2. So, my suggestion would be to keep 1.12.0 and 1.13.1, but dop all the other patch versions that aren't the latest.
-Dan
________________________________
From: Dan Smith <da...@vmware.com>
Sent: Monday, January 3, 2022 10:37 AM
To: dev@geode.apache.org <de...@geode.apache.org>
Subject: Re: [DISCUSS] proposal to pare down old-version testing
+1 - this seems reasonable to me. If we do make a protocol change in a patch, we could potentially keep around an older patch version just in that specific case, but otherwise I think this makes sense.
-Dan
________________________________
From: Anthony Baker <ba...@vmware.com>
Sent: Thursday, December 23, 2021 8:53 AM
To: dev@geode.apache.org <de...@geode.apache.org>
Subject: Re: [DISCUSS] proposal to pare down old-version testing
Interesting data point: 40% of maven central downloads last month were for version 1.4.0. Of course those numbers can be easily skewed by CI bots, but still!
@Owen, I think your suggestion nicely improves practicality while continuing to support strong compatibility. In many cases it’s quite a bit easier to upgrade the Geode server cluster compared to potentially many, many client applications. Supporting older client versions gives users time to upgrade, quicker access to bug fixes, and helps avoids downtime.
+1
Anthony
> On Dec 22, 2021, at 7:13 PM, Owen Nichols <on...@vmware.com> wrote:
>
> Since adopting our N-2 support policy, the list of released versions in /settings.gradle has ballooned to over 30 entries [1].
>
> CI tests use this list to confirm that we don’t break rolling upgrade ability or compatibility with older clients, but some of these tests don’t seem to scale well: PR#7203 to add the most recent 3 releases (bringing the total to 33) is unable to pass CI after 8 tries.
>
> Possible solutions fall into two categories: keep the full list and throw developers and/or more hardware at the struggling tests, or concede that testing every version is not a scalable approach and find ways to shorten the list, e.g. randomly select a subset of old versions at runtime, or manually pare down the list.
>
> I propose to shorten the list [2] by keeping only the latest patch for each minor (unless the client or server protocol version has changed, so also keep the patch prior to 1.12.1 and prior to 1.13.2). As long as a patch release doesn’t change the client or server protocol version, I see low value in testing upgrades from every patch version to every future version forever. The months between patch releases already provide plenty of upgrade coverage on that specific patch, then we can move on to the next…even if there could somehow be a corner-case where transitive property of upgradability doesn’t hold, most users probably take the latest-to-latest upgrade path anyway, which will always be tested.
>
> Let’s keep discussion open until 3PM PST Jan 5. In case of no response, I will assume lazy consensus and update settings.gradle as proposed [2].
>
>
>
> [1] Current list from https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fgeode%2Fblob%2Fdevelop%2Fsettings.gradle%23L72-L101&data=04%7C01%7Cdasmith%40vmware.com%7Cff2c00ff1af142cd1bbd08d9cee82048%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637768318660654753%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=4ykbbBS3aRaZe2HwN%2BZ3kXINab%2F%2FfyLx%2Fld5TFH%2B1zc%3D&reserved=0 :
> 1.0.0-incubating
> 1.1.0
> 1.1.1
> 1.2.0
> 1.3.0
> 1.4.0
> 1.5.0
> 1.6.0
> 1.7.0
> 1.8.0
> 1.9.0
> 1.9.1
> 1.9.2
> 1.10.0
> 1.11.0
> 1.12.0
> 1.12.1
> 1.12.2
> 1.12.3
> 1.12.4
> 1.12.5
> 1.12.6
> 1.12.7*
> 1.13.0
> 1.13.1
> 1.13.2
> 1.13.3
> 1.13.4
> 1.13.5
> 1.13.6*
> 1.14.0
> 1.14.1
> 1.14.2*
> *=released, but not yet added to settings.gradle due to PR#7203 not able to pass CI due to size of version list
>
> [2] Proposed shortlist:
> 1.1.1
> 1.2.0
> 1.3.0
> 1.4.0
> 1.5.0
> 1.6.0
> 1.7.0
> 1.8.0
> 1.9.2
> 1.10.0
> 1.11.0
> 1.12.0
> 1.12.7
> 1.13.1
> 1.13.6
> 1.14.2
>
Re: [DISCUSS] proposal to pare down old-version testing
Posted by Dan Smith <da...@vmware.com>.
+1 - this seems reasonable to me. If we do make a protocol change in a patch, we could potentially keep around an older patch version just in that specific case, but otherwise I think this makes sense.
-Dan
________________________________
From: Anthony Baker <ba...@vmware.com>
Sent: Thursday, December 23, 2021 8:53 AM
To: dev@geode.apache.org <de...@geode.apache.org>
Subject: Re: [DISCUSS] proposal to pare down old-version testing
Interesting data point: 40% of maven central downloads last month were for version 1.4.0. Of course those numbers can be easily skewed by CI bots, but still!
@Owen, I think your suggestion nicely improves practicality while continuing to support strong compatibility. In many cases it’s quite a bit easier to upgrade the Geode server cluster compared to potentially many, many client applications. Supporting older client versions gives users time to upgrade, quicker access to bug fixes, and helps avoids downtime.
+1
Anthony
> On Dec 22, 2021, at 7:13 PM, Owen Nichols <on...@vmware.com> wrote:
>
> Since adopting our N-2 support policy, the list of released versions in /settings.gradle has ballooned to over 30 entries [1].
>
> CI tests use this list to confirm that we don’t break rolling upgrade ability or compatibility with older clients, but some of these tests don’t seem to scale well: PR#7203 to add the most recent 3 releases (bringing the total to 33) is unable to pass CI after 8 tries.
>
> Possible solutions fall into two categories: keep the full list and throw developers and/or more hardware at the struggling tests, or concede that testing every version is not a scalable approach and find ways to shorten the list, e.g. randomly select a subset of old versions at runtime, or manually pare down the list.
>
> I propose to shorten the list [2] by keeping only the latest patch for each minor (unless the client or server protocol version has changed, so also keep the patch prior to 1.12.1 and prior to 1.13.2). As long as a patch release doesn’t change the client or server protocol version, I see low value in testing upgrades from every patch version to every future version forever. The months between patch releases already provide plenty of upgrade coverage on that specific patch, then we can move on to the next…even if there could somehow be a corner-case where transitive property of upgradability doesn’t hold, most users probably take the latest-to-latest upgrade path anyway, which will always be tested.
>
> Let’s keep discussion open until 3PM PST Jan 5. In case of no response, I will assume lazy consensus and update settings.gradle as proposed [2].
>
>
>
> [1] Current list from https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fgeode%2Fblob%2Fdevelop%2Fsettings.gradle%23L72-L101&data=04%7C01%7Cdasmith%40vmware.com%7Ca2aa64ae94d145b7f0d008d9c634ca53%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637758752301700154%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=m09xNUntRb4V3YW8zbFTq1XCYMNnqR38%2B2goSoDlxcY%3D&reserved=0 :
> 1.0.0-incubating
> 1.1.0
> 1.1.1
> 1.2.0
> 1.3.0
> 1.4.0
> 1.5.0
> 1.6.0
> 1.7.0
> 1.8.0
> 1.9.0
> 1.9.1
> 1.9.2
> 1.10.0
> 1.11.0
> 1.12.0
> 1.12.1
> 1.12.2
> 1.12.3
> 1.12.4
> 1.12.5
> 1.12.6
> 1.12.7*
> 1.13.0
> 1.13.1
> 1.13.2
> 1.13.3
> 1.13.4
> 1.13.5
> 1.13.6*
> 1.14.0
> 1.14.1
> 1.14.2*
> *=released, but not yet added to settings.gradle due to PR#7203 not able to pass CI due to size of version list
>
> [2] Proposed shortlist:
> 1.1.1
> 1.2.0
> 1.3.0
> 1.4.0
> 1.5.0
> 1.6.0
> 1.7.0
> 1.8.0
> 1.9.2
> 1.10.0
> 1.11.0
> 1.12.0
> 1.12.7
> 1.13.1
> 1.13.6
> 1.14.2
>
Re: [DISCUSS] proposal to pare down old-version testing
Posted by Anthony Baker <ba...@vmware.com>.
Interesting data point: 40% of maven central downloads last month were for version 1.4.0. Of course those numbers can be easily skewed by CI bots, but still!
@Owen, I think your suggestion nicely improves practicality while continuing to support strong compatibility. In many cases it’s quite a bit easier to upgrade the Geode server cluster compared to potentially many, many client applications. Supporting older client versions gives users time to upgrade, quicker access to bug fixes, and helps avoids downtime.
+1
Anthony
> On Dec 22, 2021, at 7:13 PM, Owen Nichols <on...@vmware.com> wrote:
>
> Since adopting our N-2 support policy, the list of released versions in /settings.gradle has ballooned to over 30 entries [1].
>
> CI tests use this list to confirm that we don’t break rolling upgrade ability or compatibility with older clients, but some of these tests don’t seem to scale well: PR#7203 to add the most recent 3 releases (bringing the total to 33) is unable to pass CI after 8 tries.
>
> Possible solutions fall into two categories: keep the full list and throw developers and/or more hardware at the struggling tests, or concede that testing every version is not a scalable approach and find ways to shorten the list, e.g. randomly select a subset of old versions at runtime, or manually pare down the list.
>
> I propose to shorten the list [2] by keeping only the latest patch for each minor (unless the client or server protocol version has changed, so also keep the patch prior to 1.12.1 and prior to 1.13.2). As long as a patch release doesn’t change the client or server protocol version, I see low value in testing upgrades from every patch version to every future version forever. The months between patch releases already provide plenty of upgrade coverage on that specific patch, then we can move on to the next…even if there could somehow be a corner-case where transitive property of upgradability doesn’t hold, most users probably take the latest-to-latest upgrade path anyway, which will always be tested.
>
> Let’s keep discussion open until 3PM PST Jan 5. In case of no response, I will assume lazy consensus and update settings.gradle as proposed [2].
>
>
>
> [1] Current list from https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fgeode%2Fblob%2Fdevelop%2Fsettings.gradle%23L72-L101&data=04%7C01%7Cbakera%40vmware.com%7C803bd50439204bf790a208d9c5c24361%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637758261648196421%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=X3SpCCjuvQo9JO01Di8HM29eKpXPzH9mr2vvpt2fSww%3D&reserved=0 :
> 1.0.0-incubating
> 1.1.0
> 1.1.1
> 1.2.0
> 1.3.0
> 1.4.0
> 1.5.0
> 1.6.0
> 1.7.0
> 1.8.0
> 1.9.0
> 1.9.1
> 1.9.2
> 1.10.0
> 1.11.0
> 1.12.0
> 1.12.1
> 1.12.2
> 1.12.3
> 1.12.4
> 1.12.5
> 1.12.6
> 1.12.7*
> 1.13.0
> 1.13.1
> 1.13.2
> 1.13.3
> 1.13.4
> 1.13.5
> 1.13.6*
> 1.14.0
> 1.14.1
> 1.14.2*
> *=released, but not yet added to settings.gradle due to PR#7203 not able to pass CI due to size of version list
>
> [2] Proposed shortlist:
> 1.1.1
> 1.2.0
> 1.3.0
> 1.4.0
> 1.5.0
> 1.6.0
> 1.7.0
> 1.8.0
> 1.9.2
> 1.10.0
> 1.11.0
> 1.12.0
> 1.12.7
> 1.13.1
> 1.13.6
> 1.14.2
>
Re: [DISCUSS] proposal to pare down old-version testing
Posted by Jacob Barrett <ja...@vmware.com>.
Yes! This! 100%!
> On Dec 22, 2021, at 7:14 PM, Owen Nichols <on...@vmware.com> wrote:
>
> Since adopting our N-2 support policy, the list of released versions in /settings.gradle has ballooned to over 30 entries [1].
>
> CI tests use this list to confirm that we don’t break rolling upgrade ability or compatibility with older clients, but some of these tests don’t seem to scale well: PR#7203 to add the most recent 3 releases (bringing the total to 33) is unable to pass CI after 8 tries.
>
> Possible solutions fall into two categories: keep the full list and throw developers and/or more hardware at the struggling tests, or concede that testing every version is not a scalable approach and find ways to shorten the list, e.g. randomly select a subset of old versions at runtime, or manually pare down the list.
>
> I propose to shorten the list [2] by keeping only the latest patch for each minor (unless the client or server protocol version has changed, so also keep the patch prior to 1.12.1 and prior to 1.13.2). As long as a patch release doesn’t change the client or server protocol version, I see low value in testing upgrades from every patch version to every future version forever. The months between patch releases already provide plenty of upgrade coverage on that specific patch, then we can move on to the next…even if there could somehow be a corner-case where transitive property of upgradability doesn’t hold, most users probably take the latest-to-latest upgrade path anyway, which will always be tested.
>
> Let’s keep discussion open until 3PM PST Jan 5. In case of no response, I will assume lazy consensus and update settings.gradle as proposed [2].
>
>
>
> [1] Current list from https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fgeode%2Fblob%2Fdevelop%2Fsettings.gradle%23L72-L101&data=04%7C01%7Cjabarrett%40vmware.com%7C1699e45bec75493de2b008d9c5c243c5%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637758260417697506%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=qdmkkxmOrSZrjWEzS%2Fmjx6kEqBENrBxKBfBHE7ZiA30%3D&reserved=0 :
> 1.0.0-incubating
> 1.1.0
> 1.1.1
> 1.2.0
> 1.3.0
> 1.4.0
> 1.5.0
> 1.6.0
> 1.7.0
> 1.8.0
> 1.9.0
> 1.9.1
> 1.9.2
> 1.10.0
> 1.11.0
> 1.12.0
> 1.12.1
> 1.12.2
> 1.12.3
> 1.12.4
> 1.12.5
> 1.12.6
> 1.12.7*
> 1.13.0
> 1.13.1
> 1.13.2
> 1.13.3
> 1.13.4
> 1.13.5
> 1.13.6*
> 1.14.0
> 1.14.1
> 1.14.2*
> *=released, but not yet added to settings.gradle due to PR#7203 not able to pass CI due to size of version list
>
> [2] Proposed shortlist:
> 1.1.1
> 1.2.0
> 1.3.0
> 1.4.0
> 1.5.0
> 1.6.0
> 1.7.0
> 1.8.0
> 1.9.2
> 1.10.0
> 1.11.0
> 1.12.0
> 1.12.7
> 1.13.1
> 1.13.6
> 1.14.2
>