You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Andrey Anshin <an...@taragol.is> on 2023/02/10 12:47:58 UTC
[Discussion] DB backend versions policy
Hi devs!
That discussion about whether we need any policy for supported DB backends
versions or not. And when we should drop specific version of DB backend
It is continuation of discussion in Slack about when we potentially could
drop support of MySQL 5.7 -
https://apache-airflow.slack.com/archives/CCPRP7943/p1675423229599569
Just as remainder, currently supported DB and versions
- PostgreSQL: 11, 12, 13, 14, 15
- MySQL: 5.7, 8
- SQLite: 3.15.0+
- MS SQL Server (Experimental): 2017, 2019
We drop support for PostgreSQL 10 in the November 2022 (as soon as
community support is over, see: https://github.com/apache/airflow/pull/27594)
and add support for PostgreSQL 15 (
https://github.com/apache/airflow/pull/27444)
Should we do the same with MySQL 5.7? Unfortunately I can't find any
information about how long Oracle provides community support for MySQL,
only for the commercial version: 5 year Premier Support + 3 Year Extended
(see: https://www.mysql.com/support/). As soon as extended over no new
patches/bug/updates fixes provided. So potentially we also could drop
support of MySQL 5.7 after October 2023 and remove all related to MySQL 5.7
code from Airflow Core.
Another option is to create policy based on how long Cloud Providers or
DBaaS providers support specific versions. I did some quick investigation
for AWS, GCP and Azure
*PostgreSQL*
Life Cycle is 5 years after initial release. One major release each year,
usually a new version released in October.
Community EOL PostgreSQL 11 in October-November 2023
Next version PostgreSQL 16
*AWS RDS (exclude Aurora)*
https://docs.aws.amazon.com/AmazonRDS/latest/PostgreSQLReleaseNotes/postgresql-release-calendar.html
- Supported versions: 10, 11, 12, 13, 14
- EOL for PostgreSQL 10 is April 2023 (see
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html#PostgreSQL.Concepts.General.DBVersions.Deprecation10
)
- EOL for PostgreSQL 11 (minimal formally supported by Airflow): N/A, my
assumption is April 2024
*Google Cloud SQL*
https://cloud.google.com/sql/docs/postgres/db-versions
- Supported versions: 9.6, 10, 11, 12, 13, 14
- I can't find any info about date of EOL for PostgreSQL 9.6-11 in
Google Cloud SQL, only notice period 12 months before sustain
*Azure Database for PostgreSQL - Flexible Server*
https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-supported-versions
- Supported versions: 11, 12, 13, 14
- Retirement for PostgreSQL 11 and 12 is November 2024. See:
https://learn.microsoft.com/en-us/azure/postgresql/single-server/concepts-version-policy#major-version-retirement-policy
*MySQL*
*AWS RDS (exclude Aurora)*
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/MySQL.Concepts.VersionMgmt.html
- Supported versions: 5.7, 8
- EOL 5.7 is October 2023, see:
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/MySQL.Concepts.VersionMgmt.html#MySQL.Concepts.VersionMgmt.ReleaseCalendar
*Google Cloud SQL*
https://cloud.google.com/sql/docs/mysql/db-versions
- Supported versions 5.6, 5.7, 8
- I can't find any info about date of EOL for MySQL 5.6 and 5.7 in
Google Cloud SQL, only notice period 12 months before sustain
*Azure Database for MySQL - Flexible Server*
https://learn.microsoft.com/en-us/azure/mysql/single-server/overview#azure-database-for-mysql---flexible-server
- Supported versions 5.7, 8
- Retirement for MySQL 5.7 is October 2023, see:
https://learn.microsoft.com/en-us/azure/mysql/concepts-version-policy#major-version-retirement-policy
I do not include information about MS SQL Server, because it has
experimental support.
And also I do not think that we require any policy for SQLite, since it is
only for development purposes.
----
Best Wishes
*Andrey Anshin*
Re: [Discussion] DB backend versions policy
Posted by Jarek Potiuk <ja...@potiuk.com>.
I do not think we need more time than given by DB providers. I wonder
(and this is more of a guess - because I have no data to back it up)
extra time is not going to be used anyway. But I am not very strong on
that. As long as we have "a" policy and we communicate it clearly, I
am good.
Why do I think we do not need the extra 9 months? Even if enterprises
move slowly, this means that they are likely behind with Airflow. And
the policy will apply to new minor versions of Airflow only. I
seriously doubt that migrating the DB version that has no security
patches is more problematic than migrating Airflow to be honest. IMHO
the Enterprise users who are not upgrading their DB will also refrain
from upgrading to the latest Airflow version.
J.
On Fri, Feb 10, 2023 at 9:01 PM John Thomas
<jo...@astronomer.io.invalid> wrote:
>
> At the time of the last community survey in spring 2022, only 9 out of 196 respondents were on MySQL 5 (about 5%), down from 27 out of 184 (about 13%) in 2020. Reading the responses carefully it looks like at least half of the remaining users are at large companies probably working on legacy infra (3 of them were still using airflow 1.10.15 or older, and three more were on 2.0 at time of asking). I think aligning the retirement of MySQL 5 with the EOL from other DB providers makes sense, but we may want to give a longer lead time than 9 months since large companies move slowly.
>
> I'll make sure to include a question on the coming survey this spring about why people are using the DB version they're currently on, and if they have plans to upgrade or change in the near future.
>
> Best,
> John
>
> On Fri, Feb 10, 2023 at 6:21 AM Andrey Anshin <an...@taragol.is> wrote:
>>
>> Hi devs!
>>
>> That discussion about whether we need any policy for supported DB backends versions or not. And when we should drop specific version of DB backend
>>
>> It is continuation of discussion in Slack about when we potentially could drop support of MySQL 5.7 - https://apache-airflow.slack.com/archives/CCPRP7943/p1675423229599569
>>
>> Just as remainder, currently supported DB and versions
>>
>> PostgreSQL: 11, 12, 13, 14, 15
>> MySQL: 5.7, 8
>> SQLite: 3.15.0+
>> MS SQL Server (Experimental): 2017, 2019
>>
>> We drop support for PostgreSQL 10 in the November 2022 (as soon as community support is over, see: https://github.com/apache/airflow/pull/27594) and add support for PostgreSQL 15 (https://github.com/apache/airflow/pull/27444)
>>
>> Should we do the same with MySQL 5.7? Unfortunately I can't find any information about how long Oracle provides community support for MySQL, only for the commercial version: 5 year Premier Support + 3 Year Extended (see: https://www.mysql.com/support/). As soon as extended over no new patches/bug/updates fixes provided. So potentially we also could drop support of MySQL 5.7 after October 2023 and remove all related to MySQL 5.7 code from Airflow Core.
>>
>> Another option is to create policy based on how long Cloud Providers or DBaaS providers support specific versions. I did some quick investigation for AWS, GCP and Azure
>>
>>
>> PostgreSQL
>>
>> Life Cycle is 5 years after initial release. One major release each year, usually a new version released in October.
>> Community EOL PostgreSQL 11 in October-November 2023
>> Next version PostgreSQL 16
>>
>> AWS RDS (exclude Aurora)
>> https://docs.aws.amazon.com/AmazonRDS/latest/PostgreSQLReleaseNotes/postgresql-release-calendar.html
>>
>> Supported versions: 10, 11, 12, 13, 14
>> EOL for PostgreSQL 10 is April 2023 (see https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html#PostgreSQL.Concepts.General.DBVersions.Deprecation10)
>> EOL for PostgreSQL 11 (minimal formally supported by Airflow): N/A, my assumption is April 2024
>>
>> Google Cloud SQL
>> https://cloud.google.com/sql/docs/postgres/db-versions
>>
>> Supported versions: 9.6, 10, 11, 12, 13, 14
>> I can't find any info about date of EOL for PostgreSQL 9.6-11 in Google Cloud SQL, only notice period 12 months before sustain
>>
>> Azure Database for PostgreSQL - Flexible Server
>> https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-supported-versions
>>
>> Supported versions: 11, 12, 13, 14
>> Retirement for PostgreSQL 11 and 12 is November 2024. See: https://learn.microsoft.com/en-us/azure/postgresql/single-server/concepts-version-policy#major-version-retirement-policy
>>
>> MySQL
>>
>> AWS RDS (exclude Aurora)
>> https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/MySQL.Concepts.VersionMgmt.html
>>
>> Supported versions: 5.7, 8
>> EOL 5.7 is October 2023, see: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/MySQL.Concepts.VersionMgmt.html#MySQL.Concepts.VersionMgmt.ReleaseCalendar
>>
>> Google Cloud SQL
>> https://cloud.google.com/sql/docs/mysql/db-versions
>>
>> Supported versions 5.6, 5.7, 8
>> I can't find any info about date of EOL for MySQL 5.6 and 5.7 in Google Cloud SQL, only notice period 12 months before sustain
>>
>>
>> Azure Database for MySQL - Flexible Server
>> https://learn.microsoft.com/en-us/azure/mysql/single-server/overview#azure-database-for-mysql---flexible-server
>>
>> Supported versions 5.7, 8
>> Retirement for MySQL 5.7 is October 2023, see: https://learn.microsoft.com/en-us/azure/mysql/concepts-version-policy#major-version-retirement-policy
>>
>>
>> I do not include information about MS SQL Server, because it has experimental support.
>>
>> And also I do not think that we require any policy for SQLite, since it is only for development purposes.
>>
>> ----
>> Best Wishes
>> Andrey Anshin
>>
Re: [Discussion] DB backend versions policy
Posted by John Thomas <jo...@astronomer.io.INVALID>.
At the time of the last community survey
<https://airflow.apache.org/blog/airflow-survey-2022/#which-metadata-database-do-you-use-single-choice>
in spring 2022, only 9 out of 196 respondents were on MySQL 5 (about 5%),
down from 27 out of 184 (about 13%) in 2020
<https://airflow.apache.org/blog/airflow-survey-2020/>. Reading the
responses carefully it looks like at least half of the remaining users are
at large companies probably working on legacy infra (3 of them were still
using airflow 1.10.15 or older, and three more were on 2.0 at time of
asking). I think aligning the retirement of MySQL 5 with the EOL from
other DB providers makes sense, but we may want to give a longer lead time
than 9 months since large companies move slowly.
I'll make sure to include a question on the coming survey this spring about
why people are using the DB version they're currently on, and if they have
plans to upgrade or change in the near future.
Best,
John
On Fri, Feb 10, 2023 at 6:21 AM Andrey Anshin <an...@taragol.is>
wrote:
> Hi devs!
>
> That discussion about whether we need any policy for supported DB backends
> versions or not. And when we should drop specific version of DB backend
>
> It is continuation of discussion in Slack about when we potentially could
> drop support of MySQL 5.7 -
> https://apache-airflow.slack.com/archives/CCPRP7943/p1675423229599569
>
> Just as remainder, currently supported DB and versions
>
> - PostgreSQL: 11, 12, 13, 14, 15
> - MySQL: 5.7, 8
> - SQLite: 3.15.0+
> - MS SQL Server (Experimental): 2017, 2019
>
> We drop support for PostgreSQL 10 in the November 2022 (as soon as
> community support is over, see:
> https://github.com/apache/airflow/pull/27594) and add support for
> PostgreSQL 15 (https://github.com/apache/airflow/pull/27444)
>
> Should we do the same with MySQL 5.7? Unfortunately I can't find any
> information about how long Oracle provides community support for MySQL,
> only for the commercial version: 5 year Premier Support + 3 Year Extended
> (see: https://www.mysql.com/support/). As soon as extended over no new
> patches/bug/updates fixes provided. So potentially we also could drop
> support of MySQL 5.7 after October 2023 and remove all related to MySQL 5.7
> code from Airflow Core.
>
> Another option is to create policy based on how long Cloud Providers or
> DBaaS providers support specific versions. I did some quick investigation
> for AWS, GCP and Azure
>
>
> *PostgreSQL*
>
> Life Cycle is 5 years after initial release. One major release each year,
> usually a new version released in October.
> Community EOL PostgreSQL 11 in October-November 2023
> Next version PostgreSQL 16
>
> *AWS RDS (exclude Aurora)*
>
> https://docs.aws.amazon.com/AmazonRDS/latest/PostgreSQLReleaseNotes/postgresql-release-calendar.html
>
> - Supported versions: 10, 11, 12, 13, 14
> - EOL for PostgreSQL 10 is April 2023 (see
> https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html#PostgreSQL.Concepts.General.DBVersions.Deprecation10
> )
> - EOL for PostgreSQL 11 (minimal formally supported by Airflow): N/A,
> my assumption is April 2024
>
> *Google Cloud SQL*
> https://cloud.google.com/sql/docs/postgres/db-versions
>
>
> - Supported versions: 9.6, 10, 11, 12, 13, 14
> - I can't find any info about date of EOL for PostgreSQL 9.6-11 in
> Google Cloud SQL, only notice period 12 months before sustain
>
> *Azure Database for PostgreSQL - Flexible Server*
>
> https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-supported-versions
>
>
> - Supported versions: 11, 12, 13, 14
> - Retirement for PostgreSQL 11 and 12 is November 2024. See:
> https://learn.microsoft.com/en-us/azure/postgresql/single-server/concepts-version-policy#major-version-retirement-policy
>
> *MySQL*
>
> *AWS RDS (exclude Aurora)*
>
> https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/MySQL.Concepts.VersionMgmt.html
>
>
> - Supported versions: 5.7, 8
> - EOL 5.7 is October 2023, see:
> https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/MySQL.Concepts.VersionMgmt.html#MySQL.Concepts.VersionMgmt.ReleaseCalendar
>
> *Google Cloud SQL*
> https://cloud.google.com/sql/docs/mysql/db-versions
>
> - Supported versions 5.6, 5.7, 8
> - I can't find any info about date of EOL for MySQL 5.6 and 5.7 in
> Google Cloud SQL, only notice period 12 months before sustain
>
>
> *Azure Database for MySQL - Flexible Server*
>
> https://learn.microsoft.com/en-us/azure/mysql/single-server/overview#azure-database-for-mysql---flexible-server
>
> - Supported versions 5.7, 8
> - Retirement for MySQL 5.7 is October 2023, see:
> https://learn.microsoft.com/en-us/azure/mysql/concepts-version-policy#major-version-retirement-policy
>
>
> I do not include information about MS SQL Server, because it has
> experimental support.
>
> And also I do not think that we require any policy for SQLite, since it is
> only for development purposes.
>
> ----
> Best Wishes
> *Andrey Anshin*
>
>
Re: [Discussion] DB backend versions policy
Posted by Jarek Potiuk <ja...@potiuk.com>.
I think we should just rely on the commercial support.
The fact is that if someone will use MySQL community and will relay on the
patches being available from Oracle, this is the relation between them
and.Oracle. Airflow has nothing to do with it.
There is nothing special from our side to do if Oracle does not release
patches for 5.7.
We are not promising in our policy that MySQL 5.7 will get patches. We are
promising that as long as it will get patches, we will support it. So EOL
for commercial version is the maximum when the user of community version
can possibly expect patches (in best case). We know that after that date
there will be no patches for sure, so we drop it then .
J.
pt., 17 lut 2023, 10:33 użytkownik Andrey Anshin <an...@taragol.is>
napisał:
> There is only one question left to consider, what is actually mean 'as
> soon as the "owner" of the DB drop support' in case of MySQL, I can't
> find any information about policy for community version (maybe I'm blind),
> only for Commercial one (https://www.mysql.com/support/): 5 + 3 years
> There only information I could find here
> https://endoflife.date/mysql#community-edition is:
>
> > Historically, patches have been released at the same time as for the
> commercial offerings, but no official commitment is made that such a policy
> will remain.
>
> There is no problem with Postgres based on Postgres Community version
> policy (https://www.postgresql.org/support/versioning/) version supported
> for 5 years since initial release + 1 patch. And also Postgres Community
> provide the list of currently supported version of Postgres and when latest
> patch plan to release / released for specific major version of Postgres 🧡
>
> ----
> Best Wishes
> *Andrey Anshin*
>
>
>
> On Thu, 16 Feb 2023 at 23:39, Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> Yeah. I think we only made "cloud" exception for K8S version because we
>> thought it is really something of a deployment environment. And I am still
>> not 100% convinced we should do it. So yeah - dropping support as soon as
>> the "owner" of the DB drops it should be fine :)
>>
>> On Thu, Feb 16, 2023 at 8:03 PM Andrey Anshin <an...@taragol.is>
>> wrote:
>>
>>> I also check that Oracle not provide MySQL 5.7 server binaries for
>>> modern Linux distributions, see:
>>> https://www.mysql.com/support/supportedplatforms/database.html
>>>
>>> So I agree that it would be strange if we would support 5.7 for new
>>> (after October 2023) versions of Airflow.
>>>
>>> ----
>>> Best Wishes
>>> *Andrey Anshin*
>>>
>>>
>>>
>>> On Mon, 13 Feb 2023 at 23:51, Ferruzzi, Dennis
>>> <fe...@amazon.com.invalid> wrote:
>>>
>>>> Very detailed, thanks. I think I want to lean towards whatever the
>>>> official support for the package is and not measure ourselves by what the
>>>> various SaaS options are doing. I think there will always be some cloud
>>>> provider lagging or keeping some old legacy version alive well beyond it's
>>>> lifecycle and we should focus on whatever the actual lifecycle is as
>>>> defined by the package itself (ie drop a MySQL version after 8 years when
>>>> it's no longer supported, etc). It seems to me that any cloud
>>>> providers/SaaS options would have newer options so worst case we're not
>>>> removing them from the ecosystem, just potentially forcing some users to
>>>> upgrade off of an EoL product.
>>>>
>>>>
>>>> - ferruzzi
>>>>
>>>>
>>>> ------------------------------
>>>> *From:* Andrey Anshin <an...@taragol.is>
>>>> *Sent:* Friday, February 10, 2023 4:47 AM
>>>> *To:* dev@airflow.apache.org
>>>> *Subject:* [EXTERNAL] [Discussion] DB backend versions policy
>>>>
>>>>
>>>> *CAUTION*: This email originated from outside of the organization. Do
>>>> not click links or open attachments unless you can confirm the sender and
>>>> know the content is safe.
>>>>
>>>> Hi devs!
>>>>
>>>> That discussion about whether we need any policy for supported DB
>>>> backends versions or not. And when we should drop specific version of DB
>>>> backend
>>>>
>>>> It is continuation of discussion in Slack about when we potentially
>>>> could drop support of MySQL 5.7 -
>>>> https://apache-airflow.slack.com/archives/CCPRP7943/p1675423229599569
>>>>
>>>> Just as remainder, currently supported DB and versions
>>>>
>>>> - PostgreSQL: 11, 12, 13, 14, 15
>>>> - MySQL: 5.7, 8
>>>> - SQLite: 3.15.0+
>>>> - MS SQL Server (Experimental): 2017, 2019
>>>>
>>>> We drop support for PostgreSQL 10 in the November 2022 (as soon as
>>>> community support is over, see:
>>>> https://github.com/apache/airflow/pull/27594) and add support for
>>>> PostgreSQL 15 (https://github.com/apache/airflow/pull/27444)
>>>>
>>>> Should we do the same with MySQL 5.7? Unfortunately I can't find any
>>>> information about how long Oracle provides community support for MySQL,
>>>> only for the commercial version: 5 year Premier Support + 3 Year Extended
>>>> (see: https://www.mysql.com/support/). As soon as extended over no new
>>>> patches/bug/updates fixes provided. So potentially we also could drop
>>>> support of MySQL 5.7 after October 2023 and remove all related to MySQL 5.7
>>>> code from Airflow Core.
>>>>
>>>> Another option is to create policy based on how long Cloud Providers or
>>>> DBaaS providers support specific versions. I did some quick investigation
>>>> for AWS, GCP and Azure
>>>>
>>>>
>>>> *PostgreSQL*
>>>>
>>>> Life Cycle is 5 years after initial release. One major release each
>>>> year, usually a new version released in October.
>>>> Community EOL PostgreSQL 11 in October-November 2023
>>>> Next version PostgreSQL 16
>>>>
>>>> *AWS RDS (exclude Aurora)*
>>>>
>>>> https://docs.aws.amazon.com/AmazonRDS/latest/PostgreSQLReleaseNotes/postgresql-release-calendar.html
>>>>
>>>> - Supported versions: 10, 11, 12, 13, 14
>>>> - EOL for PostgreSQL 10 is April 2023 (see
>>>> https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html#PostgreSQL.Concepts.General.DBVersions.Deprecation10
>>>> )
>>>> - EOL for PostgreSQL 11 (minimal formally supported by Airflow):
>>>> N/A, my assumption is April 2024
>>>>
>>>> *Google Cloud SQL*
>>>> https://cloud.google.com/sql/docs/postgres/db-versions
>>>>
>>>>
>>>> - Supported versions: 9.6, 10, 11, 12, 13, 14
>>>> - I can't find any info about date of EOL for PostgreSQL 9.6-11 in
>>>> Google Cloud SQL, only notice period 12 months before sustain
>>>>
>>>> *Azure Database for PostgreSQL - Flexible Server*
>>>>
>>>> https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-supported-versions
>>>>
>>>>
>>>> - Supported versions: 11, 12, 13, 14
>>>> - Retirement for PostgreSQL 11 and 12 is November 2024. See:
>>>> https://learn.microsoft.com/en-us/azure/postgresql/single-server/concepts-version-policy#major-version-retirement-policy
>>>>
>>>> *MySQL*
>>>>
>>>> *AWS RDS (exclude Aurora)*
>>>>
>>>> https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/MySQL.Concepts.VersionMgmt.html
>>>>
>>>>
>>>> - Supported versions: 5.7, 8
>>>> - EOL 5.7 is October 2023, see:
>>>> https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/MySQL.Concepts.VersionMgmt.html#MySQL.Concepts.VersionMgmt.ReleaseCalendar
>>>>
>>>> *Google Cloud SQL*
>>>> https://cloud.google.com/sql/docs/mysql/db-versions
>>>>
>>>> - Supported versions 5.6, 5.7, 8
>>>> - I can't find any info about date of EOL for MySQL 5.6 and 5.7 in
>>>> Google Cloud SQL, only notice period 12 months before sustain
>>>>
>>>>
>>>> *Azure Database for MySQL - Flexible Server*
>>>>
>>>> https://learn.microsoft.com/en-us/azure/mysql/single-server/overview#azure-database-for-mysql---flexible-server
>>>>
>>>> - Supported versions 5.7, 8
>>>> - Retirement for MySQL 5.7 is October 2023, see:
>>>> https://learn.microsoft.com/en-us/azure/mysql/concepts-version-policy#major-version-retirement-policy
>>>>
>>>>
>>>> I do not include information about MS SQL Server, because it has
>>>> experimental support.
>>>>
>>>> And also I do not think that we require any policy for SQLite, since it
>>>> is only for development purposes.
>>>>
>>>> ----
>>>> Best Wishes
>>>> *Andrey Anshin*
>>>>
>>>>
Re: [Discussion] DB backend versions policy
Posted by Andrey Anshin <an...@taragol.is>.
There is only one question left to consider, what is actually mean 'as soon
as the "owner" of the DB drop support' in case of MySQL, I can't find any
information about policy for community version (maybe I'm blind), only for
Commercial one (https://www.mysql.com/support/): 5 + 3 years
There only information I could find here
https://endoflife.date/mysql#community-edition is:
> Historically, patches have been released at the same time as for the
commercial offerings, but no official commitment is made that such a policy
will remain.
There is no problem with Postgres based on Postgres Community version
policy (https://www.postgresql.org/support/versioning/) version supported
for 5 years since initial release + 1 patch. And also Postgres Community
provide the list of currently supported version of Postgres and when latest
patch plan to release / released for specific major version of Postgres 🧡
----
Best Wishes
*Andrey Anshin*
On Thu, 16 Feb 2023 at 23:39, Jarek Potiuk <ja...@potiuk.com> wrote:
> Yeah. I think we only made "cloud" exception for K8S version because we
> thought it is really something of a deployment environment. And I am still
> not 100% convinced we should do it. So yeah - dropping support as soon as
> the "owner" of the DB drops it should be fine :)
>
> On Thu, Feb 16, 2023 at 8:03 PM Andrey Anshin <an...@taragol.is>
> wrote:
>
>> I also check that Oracle not provide MySQL 5.7 server binaries for modern
>> Linux distributions, see:
>> https://www.mysql.com/support/supportedplatforms/database.html
>>
>> So I agree that it would be strange if we would support 5.7 for new
>> (after October 2023) versions of Airflow.
>>
>> ----
>> Best Wishes
>> *Andrey Anshin*
>>
>>
>>
>> On Mon, 13 Feb 2023 at 23:51, Ferruzzi, Dennis
>> <fe...@amazon.com.invalid> wrote:
>>
>>> Very detailed, thanks. I think I want to lean towards whatever the
>>> official support for the package is and not measure ourselves by what the
>>> various SaaS options are doing. I think there will always be some cloud
>>> provider lagging or keeping some old legacy version alive well beyond it's
>>> lifecycle and we should focus on whatever the actual lifecycle is as
>>> defined by the package itself (ie drop a MySQL version after 8 years when
>>> it's no longer supported, etc). It seems to me that any cloud
>>> providers/SaaS options would have newer options so worst case we're not
>>> removing them from the ecosystem, just potentially forcing some users to
>>> upgrade off of an EoL product.
>>>
>>>
>>> - ferruzzi
>>>
>>>
>>> ------------------------------
>>> *From:* Andrey Anshin <an...@taragol.is>
>>> *Sent:* Friday, February 10, 2023 4:47 AM
>>> *To:* dev@airflow.apache.org
>>> *Subject:* [EXTERNAL] [Discussion] DB backend versions policy
>>>
>>>
>>> *CAUTION*: This email originated from outside of the organization. Do
>>> not click links or open attachments unless you can confirm the sender and
>>> know the content is safe.
>>>
>>> Hi devs!
>>>
>>> That discussion about whether we need any policy for supported DB
>>> backends versions or not. And when we should drop specific version of DB
>>> backend
>>>
>>> It is continuation of discussion in Slack about when we potentially
>>> could drop support of MySQL 5.7 -
>>> https://apache-airflow.slack.com/archives/CCPRP7943/p1675423229599569
>>>
>>> Just as remainder, currently supported DB and versions
>>>
>>> - PostgreSQL: 11, 12, 13, 14, 15
>>> - MySQL: 5.7, 8
>>> - SQLite: 3.15.0+
>>> - MS SQL Server (Experimental): 2017, 2019
>>>
>>> We drop support for PostgreSQL 10 in the November 2022 (as soon as
>>> community support is over, see:
>>> https://github.com/apache/airflow/pull/27594) and add support for
>>> PostgreSQL 15 (https://github.com/apache/airflow/pull/27444)
>>>
>>> Should we do the same with MySQL 5.7? Unfortunately I can't find any
>>> information about how long Oracle provides community support for MySQL,
>>> only for the commercial version: 5 year Premier Support + 3 Year Extended
>>> (see: https://www.mysql.com/support/). As soon as extended over no new
>>> patches/bug/updates fixes provided. So potentially we also could drop
>>> support of MySQL 5.7 after October 2023 and remove all related to MySQL 5.7
>>> code from Airflow Core.
>>>
>>> Another option is to create policy based on how long Cloud Providers or
>>> DBaaS providers support specific versions. I did some quick investigation
>>> for AWS, GCP and Azure
>>>
>>>
>>> *PostgreSQL*
>>>
>>> Life Cycle is 5 years after initial release. One major release each
>>> year, usually a new version released in October.
>>> Community EOL PostgreSQL 11 in October-November 2023
>>> Next version PostgreSQL 16
>>>
>>> *AWS RDS (exclude Aurora)*
>>>
>>> https://docs.aws.amazon.com/AmazonRDS/latest/PostgreSQLReleaseNotes/postgresql-release-calendar.html
>>>
>>> - Supported versions: 10, 11, 12, 13, 14
>>> - EOL for PostgreSQL 10 is April 2023 (see
>>> https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html#PostgreSQL.Concepts.General.DBVersions.Deprecation10
>>> )
>>> - EOL for PostgreSQL 11 (minimal formally supported by Airflow):
>>> N/A, my assumption is April 2024
>>>
>>> *Google Cloud SQL*
>>> https://cloud.google.com/sql/docs/postgres/db-versions
>>>
>>>
>>> - Supported versions: 9.6, 10, 11, 12, 13, 14
>>> - I can't find any info about date of EOL for PostgreSQL 9.6-11 in
>>> Google Cloud SQL, only notice period 12 months before sustain
>>>
>>> *Azure Database for PostgreSQL - Flexible Server*
>>>
>>> https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-supported-versions
>>>
>>>
>>> - Supported versions: 11, 12, 13, 14
>>> - Retirement for PostgreSQL 11 and 12 is November 2024. See:
>>> https://learn.microsoft.com/en-us/azure/postgresql/single-server/concepts-version-policy#major-version-retirement-policy
>>>
>>> *MySQL*
>>>
>>> *AWS RDS (exclude Aurora)*
>>>
>>> https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/MySQL.Concepts.VersionMgmt.html
>>>
>>>
>>> - Supported versions: 5.7, 8
>>> - EOL 5.7 is October 2023, see:
>>> https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/MySQL.Concepts.VersionMgmt.html#MySQL.Concepts.VersionMgmt.ReleaseCalendar
>>>
>>> *Google Cloud SQL*
>>> https://cloud.google.com/sql/docs/mysql/db-versions
>>>
>>> - Supported versions 5.6, 5.7, 8
>>> - I can't find any info about date of EOL for MySQL 5.6 and 5.7 in
>>> Google Cloud SQL, only notice period 12 months before sustain
>>>
>>>
>>> *Azure Database for MySQL - Flexible Server*
>>>
>>> https://learn.microsoft.com/en-us/azure/mysql/single-server/overview#azure-database-for-mysql---flexible-server
>>>
>>> - Supported versions 5.7, 8
>>> - Retirement for MySQL 5.7 is October 2023, see:
>>> https://learn.microsoft.com/en-us/azure/mysql/concepts-version-policy#major-version-retirement-policy
>>>
>>>
>>> I do not include information about MS SQL Server, because it has
>>> experimental support.
>>>
>>> And also I do not think that we require any policy for SQLite, since it
>>> is only for development purposes.
>>>
>>> ----
>>> Best Wishes
>>> *Andrey Anshin*
>>>
>>>
Re: [Discussion] DB backend versions policy
Posted by Jarek Potiuk <ja...@potiuk.com>.
Yeah. I think we only made "cloud" exception for K8S version because we
thought it is really something of a deployment environment. And I am still
not 100% convinced we should do it. So yeah - dropping support as soon as
the "owner" of the DB drops it should be fine :)
On Thu, Feb 16, 2023 at 8:03 PM Andrey Anshin <an...@taragol.is>
wrote:
> I also check that Oracle not provide MySQL 5.7 server binaries for modern
> Linux distributions, see:
> https://www.mysql.com/support/supportedplatforms/database.html
>
> So I agree that it would be strange if we would support 5.7 for new (after
> October 2023) versions of Airflow.
>
> ----
> Best Wishes
> *Andrey Anshin*
>
>
>
> On Mon, 13 Feb 2023 at 23:51, Ferruzzi, Dennis <fe...@amazon.com.invalid>
> wrote:
>
>> Very detailed, thanks. I think I want to lean towards whatever the
>> official support for the package is and not measure ourselves by what the
>> various SaaS options are doing. I think there will always be some cloud
>> provider lagging or keeping some old legacy version alive well beyond it's
>> lifecycle and we should focus on whatever the actual lifecycle is as
>> defined by the package itself (ie drop a MySQL version after 8 years when
>> it's no longer supported, etc). It seems to me that any cloud
>> providers/SaaS options would have newer options so worst case we're not
>> removing them from the ecosystem, just potentially forcing some users to
>> upgrade off of an EoL product.
>>
>>
>> - ferruzzi
>>
>>
>> ------------------------------
>> *From:* Andrey Anshin <an...@taragol.is>
>> *Sent:* Friday, February 10, 2023 4:47 AM
>> *To:* dev@airflow.apache.org
>> *Subject:* [EXTERNAL] [Discussion] DB backend versions policy
>>
>>
>> *CAUTION*: This email originated from outside of the organization. Do
>> not click links or open attachments unless you can confirm the sender and
>> know the content is safe.
>>
>> Hi devs!
>>
>> That discussion about whether we need any policy for supported DB
>> backends versions or not. And when we should drop specific version of DB
>> backend
>>
>> It is continuation of discussion in Slack about when we potentially could
>> drop support of MySQL 5.7 -
>> https://apache-airflow.slack.com/archives/CCPRP7943/p1675423229599569
>>
>> Just as remainder, currently supported DB and versions
>>
>> - PostgreSQL: 11, 12, 13, 14, 15
>> - MySQL: 5.7, 8
>> - SQLite: 3.15.0+
>> - MS SQL Server (Experimental): 2017, 2019
>>
>> We drop support for PostgreSQL 10 in the November 2022 (as soon as
>> community support is over, see:
>> https://github.com/apache/airflow/pull/27594) and add support for
>> PostgreSQL 15 (https://github.com/apache/airflow/pull/27444)
>>
>> Should we do the same with MySQL 5.7? Unfortunately I can't find any
>> information about how long Oracle provides community support for MySQL,
>> only for the commercial version: 5 year Premier Support + 3 Year Extended
>> (see: https://www.mysql.com/support/). As soon as extended over no new
>> patches/bug/updates fixes provided. So potentially we also could drop
>> support of MySQL 5.7 after October 2023 and remove all related to MySQL 5.7
>> code from Airflow Core.
>>
>> Another option is to create policy based on how long Cloud Providers or
>> DBaaS providers support specific versions. I did some quick investigation
>> for AWS, GCP and Azure
>>
>>
>> *PostgreSQL*
>>
>> Life Cycle is 5 years after initial release. One major release each year,
>> usually a new version released in October.
>> Community EOL PostgreSQL 11 in October-November 2023
>> Next version PostgreSQL 16
>>
>> *AWS RDS (exclude Aurora)*
>>
>> https://docs.aws.amazon.com/AmazonRDS/latest/PostgreSQLReleaseNotes/postgresql-release-calendar.html
>>
>> - Supported versions: 10, 11, 12, 13, 14
>> - EOL for PostgreSQL 10 is April 2023 (see
>> https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html#PostgreSQL.Concepts.General.DBVersions.Deprecation10
>> )
>> - EOL for PostgreSQL 11 (minimal formally supported by Airflow): N/A,
>> my assumption is April 2024
>>
>> *Google Cloud SQL*
>> https://cloud.google.com/sql/docs/postgres/db-versions
>>
>>
>> - Supported versions: 9.6, 10, 11, 12, 13, 14
>> - I can't find any info about date of EOL for PostgreSQL 9.6-11 in
>> Google Cloud SQL, only notice period 12 months before sustain
>>
>> *Azure Database for PostgreSQL - Flexible Server*
>>
>> https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-supported-versions
>>
>>
>> - Supported versions: 11, 12, 13, 14
>> - Retirement for PostgreSQL 11 and 12 is November 2024. See:
>> https://learn.microsoft.com/en-us/azure/postgresql/single-server/concepts-version-policy#major-version-retirement-policy
>>
>> *MySQL*
>>
>> *AWS RDS (exclude Aurora)*
>>
>> https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/MySQL.Concepts.VersionMgmt.html
>>
>>
>> - Supported versions: 5.7, 8
>> - EOL 5.7 is October 2023, see:
>> https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/MySQL.Concepts.VersionMgmt.html#MySQL.Concepts.VersionMgmt.ReleaseCalendar
>>
>> *Google Cloud SQL*
>> https://cloud.google.com/sql/docs/mysql/db-versions
>>
>> - Supported versions 5.6, 5.7, 8
>> - I can't find any info about date of EOL for MySQL 5.6 and 5.7 in
>> Google Cloud SQL, only notice period 12 months before sustain
>>
>>
>> *Azure Database for MySQL - Flexible Server*
>>
>> https://learn.microsoft.com/en-us/azure/mysql/single-server/overview#azure-database-for-mysql---flexible-server
>>
>> - Supported versions 5.7, 8
>> - Retirement for MySQL 5.7 is October 2023, see:
>> https://learn.microsoft.com/en-us/azure/mysql/concepts-version-policy#major-version-retirement-policy
>>
>>
>> I do not include information about MS SQL Server, because it has
>> experimental support.
>>
>> And also I do not think that we require any policy for SQLite, since it
>> is only for development purposes.
>>
>> ----
>> Best Wishes
>> *Andrey Anshin*
>>
>>
Re: [Discussion] DB backend versions policy
Posted by Andrey Anshin <an...@taragol.is>.
I also check that Oracle not provide MySQL 5.7 server binaries for modern
Linux distributions, see:
https://www.mysql.com/support/supportedplatforms/database.html
So I agree that it would be strange if we would support 5.7 for new (after
October 2023) versions of Airflow.
----
Best Wishes
*Andrey Anshin*
On Mon, 13 Feb 2023 at 23:51, Ferruzzi, Dennis <fe...@amazon.com.invalid>
wrote:
> Very detailed, thanks. I think I want to lean towards whatever the
> official support for the package is and not measure ourselves by what the
> various SaaS options are doing. I think there will always be some cloud
> provider lagging or keeping some old legacy version alive well beyond it's
> lifecycle and we should focus on whatever the actual lifecycle is as
> defined by the package itself (ie drop a MySQL version after 8 years when
> it's no longer supported, etc). It seems to me that any cloud
> providers/SaaS options would have newer options so worst case we're not
> removing them from the ecosystem, just potentially forcing some users to
> upgrade off of an EoL product.
>
>
> - ferruzzi
>
>
> ------------------------------
> *From:* Andrey Anshin <an...@taragol.is>
> *Sent:* Friday, February 10, 2023 4:47 AM
> *To:* dev@airflow.apache.org
> *Subject:* [EXTERNAL] [Discussion] DB backend versions policy
>
>
> *CAUTION*: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
> Hi devs!
>
> That discussion about whether we need any policy for supported DB backends
> versions or not. And when we should drop specific version of DB backend
>
> It is continuation of discussion in Slack about when we potentially could
> drop support of MySQL 5.7 -
> https://apache-airflow.slack.com/archives/CCPRP7943/p1675423229599569
>
> Just as remainder, currently supported DB and versions
>
> - PostgreSQL: 11, 12, 13, 14, 15
> - MySQL: 5.7, 8
> - SQLite: 3.15.0+
> - MS SQL Server (Experimental): 2017, 2019
>
> We drop support for PostgreSQL 10 in the November 2022 (as soon as
> community support is over, see:
> https://github.com/apache/airflow/pull/27594) and add support for
> PostgreSQL 15 (https://github.com/apache/airflow/pull/27444)
>
> Should we do the same with MySQL 5.7? Unfortunately I can't find any
> information about how long Oracle provides community support for MySQL,
> only for the commercial version: 5 year Premier Support + 3 Year Extended
> (see: https://www.mysql.com/support/). As soon as extended over no new
> patches/bug/updates fixes provided. So potentially we also could drop
> support of MySQL 5.7 after October 2023 and remove all related to MySQL 5.7
> code from Airflow Core.
>
> Another option is to create policy based on how long Cloud Providers or
> DBaaS providers support specific versions. I did some quick investigation
> for AWS, GCP and Azure
>
>
> *PostgreSQL*
>
> Life Cycle is 5 years after initial release. One major release each year,
> usually a new version released in October.
> Community EOL PostgreSQL 11 in October-November 2023
> Next version PostgreSQL 16
>
> *AWS RDS (exclude Aurora)*
>
> https://docs.aws.amazon.com/AmazonRDS/latest/PostgreSQLReleaseNotes/postgresql-release-calendar.html
>
> - Supported versions: 10, 11, 12, 13, 14
> - EOL for PostgreSQL 10 is April 2023 (see
> https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html#PostgreSQL.Concepts.General.DBVersions.Deprecation10
> )
> - EOL for PostgreSQL 11 (minimal formally supported by Airflow): N/A,
> my assumption is April 2024
>
> *Google Cloud SQL*
> https://cloud.google.com/sql/docs/postgres/db-versions
>
>
> - Supported versions: 9.6, 10, 11, 12, 13, 14
> - I can't find any info about date of EOL for PostgreSQL 9.6-11 in
> Google Cloud SQL, only notice period 12 months before sustain
>
> *Azure Database for PostgreSQL - Flexible Server*
>
> https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-supported-versions
>
>
> - Supported versions: 11, 12, 13, 14
> - Retirement for PostgreSQL 11 and 12 is November 2024. See:
> https://learn.microsoft.com/en-us/azure/postgresql/single-server/concepts-version-policy#major-version-retirement-policy
>
> *MySQL*
>
> *AWS RDS (exclude Aurora)*
>
> https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/MySQL.Concepts.VersionMgmt.html
>
>
> - Supported versions: 5.7, 8
> - EOL 5.7 is October 2023, see:
> https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/MySQL.Concepts.VersionMgmt.html#MySQL.Concepts.VersionMgmt.ReleaseCalendar
>
> *Google Cloud SQL*
> https://cloud.google.com/sql/docs/mysql/db-versions
>
> - Supported versions 5.6, 5.7, 8
> - I can't find any info about date of EOL for MySQL 5.6 and 5.7 in
> Google Cloud SQL, only notice period 12 months before sustain
>
>
> *Azure Database for MySQL - Flexible Server*
>
> https://learn.microsoft.com/en-us/azure/mysql/single-server/overview#azure-database-for-mysql---flexible-server
>
> - Supported versions 5.7, 8
> - Retirement for MySQL 5.7 is October 2023, see:
> https://learn.microsoft.com/en-us/azure/mysql/concepts-version-policy#major-version-retirement-policy
>
>
> I do not include information about MS SQL Server, because it has
> experimental support.
>
> And also I do not think that we require any policy for SQLite, since it is
> only for development purposes.
>
> ----
> Best Wishes
> *Andrey Anshin*
>
>
Re: [Discussion] DB backend versions policy
Posted by "Ferruzzi, Dennis" <fe...@amazon.com.INVALID>.
Very detailed, thanks. I think I want to lean towards whatever the official support for the package is and not measure ourselves by what the various SaaS options are doing. I think there will always be some cloud provider lagging or keeping some old legacy version alive well beyond it's lifecycle and we should focus on whatever the actual lifecycle is as defined by the package itself (ie drop a MySQL version after 8 years when it's no longer supported, etc). It seems to me that any cloud providers/SaaS options would have newer options so worst case we're not removing them from the ecosystem, just potentially forcing some users to upgrade off of an EoL product.
- ferruzzi
________________________________
From: Andrey Anshin <an...@taragol.is>
Sent: Friday, February 10, 2023 4:47 AM
To: dev@airflow.apache.org
Subject: [EXTERNAL] [Discussion] DB backend versions policy
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Hi devs!
That discussion about whether we need any policy for supported DB backends versions or not. And when we should drop specific version of DB backend
It is continuation of discussion in Slack about when we potentially could drop support of MySQL 5.7 - https://apache-airflow.slack.com/archives/CCPRP7943/p1675423229599569
Just as remainder, currently supported DB and versions
* PostgreSQL: 11, 12, 13, 14, 15
* MySQL: 5.7, 8
* SQLite: 3.15.0+
* MS SQL Server (Experimental): 2017, 2019
We drop support for PostgreSQL 10 in the November 2022 (as soon as community support is over, see: https://github.com/apache/airflow/pull/27594) and add support for PostgreSQL 15 (https://github.com/apache/airflow/pull/27444)
Should we do the same with MySQL 5.7? Unfortunately I can't find any information about how long Oracle provides community support for MySQL, only for the commercial version: 5 year Premier Support + 3 Year Extended (see: https://www.mysql.com/support/). As soon as extended over no new patches/bug/updates fixes provided. So potentially we also could drop support of MySQL 5.7 after October 2023 and remove all related to MySQL 5.7 code from Airflow Core.
Another option is to create policy based on how long Cloud Providers or DBaaS providers support specific versions. I did some quick investigation for AWS, GCP and Azure
PostgreSQL
Life Cycle is 5 years after initial release. One major release each year, usually a new version released in October.
Community EOL PostgreSQL 11 in October-November 2023
Next version PostgreSQL 16
AWS RDS (exclude Aurora)
https://docs.aws.amazon.com/AmazonRDS/latest/PostgreSQLReleaseNotes/postgresql-release-calendar.html
* Supported versions: 10, 11, 12, 13, 14
* EOL for PostgreSQL 10 is April 2023 (see https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html#PostgreSQL.Concepts.General.DBVersions.Deprecation10)
* EOL for PostgreSQL 11 (minimal formally supported by Airflow): N/A, my assumption is April 2024
Google Cloud SQL
https://cloud.google.com/sql/docs/postgres/db-versions
* Supported versions: 9.6, 10, 11, 12, 13, 14
* I can't find any info about date of EOL for PostgreSQL 9.6-11 in Google Cloud SQL, only notice period 12 months before sustain
Azure Database for PostgreSQL - Flexible Server
https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-supported-versions
* Supported versions: 11, 12, 13, 14
* Retirement for PostgreSQL 11 and 12 is November 2024. See: https://learn.microsoft.com/en-us/azure/postgresql/single-server/concepts-version-policy#major-version-retirement-policy
MySQL
AWS RDS (exclude Aurora)
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/MySQL.Concepts.VersionMgmt.html
* Supported versions: 5.7, 8
* EOL 5.7 is October 2023, see: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/MySQL.Concepts.VersionMgmt.html#MySQL.Concepts.VersionMgmt.ReleaseCalendar
Google Cloud SQL
https://cloud.google.com/sql/docs/mysql/db-versions
* Supported versions 5.6, 5.7, 8
* I can't find any info about date of EOL for MySQL 5.6 and 5.7 in Google Cloud SQL, only notice period 12 months before sustain
Azure Database for MySQL - Flexible Server
https://learn.microsoft.com/en-us/azure/mysql/single-server/overview#azure-database-for-mysql---flexible-server
* Supported versions 5.7, 8
* Retirement for MySQL 5.7 is October 2023, see: https://learn.microsoft.com/en-us/azure/mysql/concepts-version-policy#major-version-retirement-policy
I do not include information about MS SQL Server, because it has experimental support.
And also I do not think that we require any policy for SQLite, since it is only for development purposes.
----
Best Wishes
Andrey Anshin