You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Jarek Potiuk <ja...@potiuk.com> on 2022/02/14 13:12:36 UTC
[DISCUSS] Buster / Bullseye Debian support
Hello everyone,
I am just about to complete preparation to make our images (and CI) work on
Debian 11 images (Bullseye) vs Debian 10 (Buster). All looks (almost) good,
but I would like to tap into the collective wisdom of the group here to
decide on next steps (and maybe propose a policy we might be following in
the future). The last change that allows to easily switch between the two
is in reviee: https://github.com/apache/airflow/pull/21546
Context/where we are:
For quite a while we used Buster (Debian 10) as our base image. Buster is a
stable release and it is being replaced with the next
stable release Bullseye (Debian 11).
The Release schedule is here: https://wiki.debian.org/DebianReleases
The most important facts from those:
* Bullseye was introduced in August 2021 (6 months ago) and has no
end-of-life yet
* Buster was introduced in July 2019 and its end of life is ~ August 2020
(approx. 6 months from now)
Python default images are (as of a few months), However they provide (as
usual -buster and -bullseye) specific images.
The Bullseye switch is really needed to support ARM (M1) images because of
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=989604 (still unresolved
on Buster).
It seems appropriate to switch to Bullseye now, however there is one
problem with it - MSSQL drivers do not yet support Bullseye. I've reached
out to the maintainers and await their answer when it might be available:
https://github.com/MicrosoftDocs/sql-docs/issues/7255#issuecomment-1037097131
. This is the last blocker that prevents full switch (failing PR for a full
switch is https://github.com/apache/airflow/pull/21378). All tests pass
except MSSQL.
But I think we need to proceed, regardless of the timeline and switch to
Bullseye.
Question:
We need to decide on what support we give our users in the images now, in
the 2.3.0 release and in the future. We have no policies for that yet, so
it might be a good time to decide now. Some of the users might "rely" on
the Buster being used by the images as they might have some incompatible
libraries (similarly to MSSQL). And I believe we need to support Buster
till at least the end of life of it.
This support might be twofold:
* we can publish in https://hub.docker.com/r/apache/airflow our binary
images with Buster and/or Bullseye
* we can only publish one of those (and change from Buster to Bullseye for
2.3.0) but users will be able to build their own custom images using our
Dockerfile for either of those.
Proposal:
I think there are multiple reasons why we should support both Bullseye and
Buster for the coming 6 months. We might build both images in CI and add a
new "matrix" dimension and run most tests on Bullseye but some tests on
Buster (specifically MSSQL until Bullseye is supported).
We can publish both types of images (with -buster, -bullseye suffixes same
as Python) but the default image should be changed to Bullseye.
This will add some complexity (and build overhead) to our CI, but I think
it's worth it.
Then we could drop support and release the -buster images after it reaches
end of life (but we will leave the users possibility of building their own
buster images without guarantee that it will work).
I think this might become our Policy also for the future (6 months before
end-of-life we switch by default to the new stable and provide built images
until end-of-life).
WDYT? Does it look like a good proposal?
J.
Re: [DISCUSS] Buster / Bullseye Debian support
Posted by Jarek Potiuk <ja...@potiuk.com>.
Hey everyone,
We have the Bullseye - based builds successful - including all tests - so
we are ready to switch:
Looking forward to the reviews - there are three PRs:
1) https://github.com/apache/airflow/pull/21546 - allows to switch easily
between Buster/Bullseye
2) https://github.com/apache/airflow/pull/21378 - actually Switch to Debian
11 (bullseye) as base for our dockerfiles
3) https://github.com/apache/airflow/pull/21697 - proposed Draft to
policy we should follow (based on the discussion above)
Once we merge those and we consent on the policy, I will add another one
where we will be building both Bullseye/Buster in our CI and run
verification for both. We are close to switching to Python version of
builds (as the internship work on switching Breeze to Python is close to
conclusion) so I want to do it after we will actually switch in order to
avoid duplication of the work.
Looking forward to comments.
J.
On Wed, Feb 16, 2022 at 8:48 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> See this PR: https://github.com/apache/airflow/pull/21378 (Particularly
> last commit).
>
> Currently failing on pyodbc.OperationalError: ('08001', '[08001]
> [Microsoft][ODBC Driver 18 for SQL Server]SSL Provider: [error:1416F086:SSL
> routines:tls_process_server_certificate:certificate verify failed:self
> signed certificate] (-1) (SQLDriverConnect)' but that will fixed in the nex
> fixup.
>
>
> On Wed, Feb 16, 2022 at 8:45 PM Alex Begg <al...@gmail.com> wrote:
>
>> Not sure if this is done already, but I think places with "ODBC Driver 17
>> for SQL Server" hardcoded in might need to change from 17 to 18 as well:
>> https://github.com/apache/airflow/search?q=ODBC+Driver+17+for+SQL+Server
>>
>>
>> On Wed, Feb 16, 2022 at 12:06 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>>> Yep. Already did :)
>>>
>>> On Wed, Feb 16, 2022 at 6:22 AM Alex Begg <al...@gmail.com> wrote:
>>>
>>>> I wanted to point out that not only did Microsoft add Debian Bullseye
>>>> support for MSSQL today, to do so they did a major version
>>>> upgrade, from msodbcsql17 to msodbcsql18. So in addition to updating the
>>>> URL used to populate mssql-release.list, you also need to change to
>>>> msodbcsql18. I see msodbcsql17 is in multiple places in the Airflow code
>>>> base.
>>>>
>>>> Instructions for installing msodbcsql18 in Debian are here:
>>>>
>>>> https://docs.microsoft.com/en-us/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server?view=sql-server-ver15#debian18
>>>>
>>>> Alex
>>>>
>>>>
>>>> On Tue, Feb 15, 2022 at 1:32 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>
>>>>> Yeah. That would definitely be a simpler approach for us.
>>>>>
>>>>> What we can also do is we can still make sure that you can actually
>>>>> "build" Buster images if you want and we could run the simple tests we
>>>>> already have for those images (those tests just check very basic stuff
>>>>> - like if expected packages are importable, if Airflow commands work,
>>>>> if you can run airflow as root user etc..
>>>>> This would be very little to no complication of the CI and we could
>>>>> provide a recipe for the users how to build their own buster image -
>>>>> but without guarantees that everything works there.
>>>>>
>>>>> This way might provide the users with a "workable solution" while we
>>>>> could keep it simple for us and put a strong emphasis on "If you want
>>>>> to get full support - also upgrade to the latest stable OS".
>>>>>
>>>>> J.
>>>>>
>>>>>
>>>>> On Tue, Feb 15, 2022 at 9:53 AM Ash Berlin-Taylor <as...@apache.org>
>>>>> wrote:
>>>>> >
>>>>> > My vote is to just move forward and only release Bullseye images for
>>>>> 2.3 onwards
>>>>> >
>>>>> > On 15 February 2022 08:43:50 GMT, Jarek Potiuk <ja...@potiuk.com>
>>>>> wrote:
>>>>> >>
>>>>> >> Good news from today - I got the response from Microsoft and it
>>>>> seems
>>>>> >> that MSSQL Bullseye support is going to be released today :) . So at
>>>>> >> least that problem is going to be solved. Still the question of
>>>>> >> whether we should build/release both Bullseye and Buster ?
>>>>> >>
>>>>> >> J.
>>>>> >>
>>>>> >> On Mon, Feb 14, 2022 at 2:14 PM Jarek Potiuk <ja...@potiuk.com>
>>>>> wrote:
>>>>> >>>
>>>>> >>>
>>>>> >>> And Python default images are Bullseye now.
>>>>> >>>
>>>>> >>> On Mon, Feb 14, 2022 at 2:13 PM Jarek Potiuk <ja...@potiuk.com>
>>>>> wrote:
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> Buster end of life is August 2022 of course :)
>>>>> >>>>
>>>>> >>>> On Mon, Feb 14, 2022 at 2:12 PM Jarek Potiuk <ja...@potiuk.com>
>>>>> wrote:
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>> Hello everyone,
>>>>> >>>>>
>>>>> >>>>> I am just about to complete preparation to make our images (and
>>>>> CI) work on Debian 11 images (Bullseye) vs Debian 10 (Buster). All looks
>>>>> (almost) good, but I would like to tap into the collective wisdom of the
>>>>> group here to decide on next steps (and maybe propose a policy we might be
>>>>> following in the future). The last change that allows to easily switch
>>>>> between the two is in reviee:
>>>>> https://github.com/apache/airflow/pull/21546
>>>>> >>>>>
>>>>> >>>>> Context/where we are:
>>>>> >>>>>
>>>>> >>>>> For quite a while we used Buster (Debian 10) as our base image.
>>>>> Buster is a stable release and it is being replaced with the next stable
>>>>> release Bullseye (Debian 11).
>>>>> >>>>>
>>>>> >>>>> The Release schedule is here:
>>>>> https://wiki.debian.org/DebianReleases
>>>>> >>>>>
>>>>> >>>>> The most important facts from those:
>>>>> >>>>> * Bullseye was introduced in August 2021 (6 months ago) and has
>>>>> no end-of-life yet
>>>>> >>>>> * Buster was introduced in July 2019 and its end of life is ~
>>>>> August 2020 (approx. 6 months from now)
>>>>> >>>>>
>>>>> >>>>> Python default images are (as of a few months), However they
>>>>> provide (as usual -buster and -bullseye) specific images.
>>>>> >>>>>
>>>>> >>>>> The Bullseye switch is really needed to support ARM (M1) images
>>>>> because of https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=989604
>>>>> (still unresolved on Buster).
>>>>> >>>>>
>>>>> >>>>> It seems appropriate to switch to Bullseye now, however there
>>>>> is one problem with it - MSSQL drivers do not yet support Bullseye. I've
>>>>> reached out to the maintainers and await their answer when it might be
>>>>> available:
>>>>> https://github.com/MicrosoftDocs/sql-docs/issues/7255#issuecomment-1037097131
>>>>> . This is the last blocker that prevents full switch (failing PR for a full
>>>>> switch is https://github.com/apache/airflow/pull/21378). All tests
>>>>> pass except MSSQL.
>>>>> >>>>>
>>>>> >>>>> But I think we need to proceed, regardless of the timeline and
>>>>> switch to Bullseye.
>>>>> >>>>>
>>>>> >>>>> Question:
>>>>> >>>>>
>>>>> >>>>> We need to decide on what support we give our users in the
>>>>> images now, in the 2.3.0 release and in the future. We have no policies for
>>>>> that yet, so it might be a good time to decide now. Some of the users might
>>>>> "rely" on the Buster being used by the images as they might have some
>>>>> incompatible libraries (similarly to MSSQL). And I believe we need to
>>>>> support Buster till at least the end of life of it.
>>>>> >>>>>
>>>>> >>>>> This support might be twofold:
>>>>> >>>>>
>>>>> >>>>> * we can publish in https://hub.docker.com/r/apache/airflow
>>>>> our binary images with Buster and/or Bullseye
>>>>> >>>>> * we can only publish one of those (and change from Buster to
>>>>> Bullseye for 2.3.0) but users will be able to build their own custom images
>>>>> using our Dockerfile for either of those.
>>>>> >>>>>
>>>>> >>>>> Proposal:
>>>>> >>>>>
>>>>> >>>>> I think there are multiple reasons why we should support both
>>>>> Bullseye and Buster for the coming 6 months. We might build both images in
>>>>> CI and add a new "matrix" dimension and run most tests on Bullseye but some
>>>>> tests on Buster (specifically MSSQL until Bullseye is supported).
>>>>> >>>>>
>>>>> >>>>> We can publish both types of images (with -buster, -bullseye
>>>>> suffixes same as Python) but the default image should be changed to
>>>>> Bullseye.
>>>>> >>>>>
>>>>> >>>>> This will add some complexity (and build overhead) to our CI,
>>>>> but I think it's worth it.
>>>>> >>>>>
>>>>> >>>>> Then we could drop support and release the -buster images after
>>>>> it reaches end of life (but we will leave the users possibility of building
>>>>> their own buster images without guarantee that it will work).
>>>>> >>>>>
>>>>> >>>>> I think this might become our Policy also for the future (6
>>>>> months before end-of-life we switch by default to the new stable and
>>>>> provide built images until end-of-life).
>>>>> >>>>>
>>>>> >>>>> WDYT? Does it look like a good proposal?
>>>>> >>>>>
>>>>> >>>>> J.
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>>
>>>>>
>>>>
Re: [DISCUSS] Buster / Bullseye Debian support
Posted by Jarek Potiuk <ja...@potiuk.com>.
See this PR: https://github.com/apache/airflow/pull/21378 (Particularly
last commit).
Currently failing on pyodbc.OperationalError: ('08001', '[08001]
[Microsoft][ODBC Driver 18 for SQL Server]SSL Provider: [error:1416F086:SSL
routines:tls_process_server_certificate:certificate verify failed:self
signed certificate] (-1) (SQLDriverConnect)' but that will fixed in the nex
fixup.
On Wed, Feb 16, 2022 at 8:45 PM Alex Begg <al...@gmail.com> wrote:
> Not sure if this is done already, but I think places with "ODBC Driver 17
> for SQL Server" hardcoded in might need to change from 17 to 18 as well:
> https://github.com/apache/airflow/search?q=ODBC+Driver+17+for+SQL+Server
>
>
> On Wed, Feb 16, 2022 at 12:06 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> Yep. Already did :)
>>
>> On Wed, Feb 16, 2022 at 6:22 AM Alex Begg <al...@gmail.com> wrote:
>>
>>> I wanted to point out that not only did Microsoft add Debian Bullseye
>>> support for MSSQL today, to do so they did a major version
>>> upgrade, from msodbcsql17 to msodbcsql18. So in addition to updating the
>>> URL used to populate mssql-release.list, you also need to change to
>>> msodbcsql18. I see msodbcsql17 is in multiple places in the Airflow code
>>> base.
>>>
>>> Instructions for installing msodbcsql18 in Debian are here:
>>>
>>> https://docs.microsoft.com/en-us/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server?view=sql-server-ver15#debian18
>>>
>>> Alex
>>>
>>>
>>> On Tue, Feb 15, 2022 at 1:32 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>
>>>> Yeah. That would definitely be a simpler approach for us.
>>>>
>>>> What we can also do is we can still make sure that you can actually
>>>> "build" Buster images if you want and we could run the simple tests we
>>>> already have for those images (those tests just check very basic stuff
>>>> - like if expected packages are importable, if Airflow commands work,
>>>> if you can run airflow as root user etc..
>>>> This would be very little to no complication of the CI and we could
>>>> provide a recipe for the users how to build their own buster image -
>>>> but without guarantees that everything works there.
>>>>
>>>> This way might provide the users with a "workable solution" while we
>>>> could keep it simple for us and put a strong emphasis on "If you want
>>>> to get full support - also upgrade to the latest stable OS".
>>>>
>>>> J.
>>>>
>>>>
>>>> On Tue, Feb 15, 2022 at 9:53 AM Ash Berlin-Taylor <as...@apache.org>
>>>> wrote:
>>>> >
>>>> > My vote is to just move forward and only release Bullseye images for
>>>> 2.3 onwards
>>>> >
>>>> > On 15 February 2022 08:43:50 GMT, Jarek Potiuk <ja...@potiuk.com>
>>>> wrote:
>>>> >>
>>>> >> Good news from today - I got the response from Microsoft and it seems
>>>> >> that MSSQL Bullseye support is going to be released today :) . So at
>>>> >> least that problem is going to be solved. Still the question of
>>>> >> whether we should build/release both Bullseye and Buster ?
>>>> >>
>>>> >> J.
>>>> >>
>>>> >> On Mon, Feb 14, 2022 at 2:14 PM Jarek Potiuk <ja...@potiuk.com>
>>>> wrote:
>>>> >>>
>>>> >>>
>>>> >>> And Python default images are Bullseye now.
>>>> >>>
>>>> >>> On Mon, Feb 14, 2022 at 2:13 PM Jarek Potiuk <ja...@potiuk.com>
>>>> wrote:
>>>> >>>>
>>>> >>>>
>>>> >>>> Buster end of life is August 2022 of course :)
>>>> >>>>
>>>> >>>> On Mon, Feb 14, 2022 at 2:12 PM Jarek Potiuk <ja...@potiuk.com>
>>>> wrote:
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> Hello everyone,
>>>> >>>>>
>>>> >>>>> I am just about to complete preparation to make our images (and
>>>> CI) work on Debian 11 images (Bullseye) vs Debian 10 (Buster). All looks
>>>> (almost) good, but I would like to tap into the collective wisdom of the
>>>> group here to decide on next steps (and maybe propose a policy we might be
>>>> following in the future). The last change that allows to easily switch
>>>> between the two is in reviee:
>>>> https://github.com/apache/airflow/pull/21546
>>>> >>>>>
>>>> >>>>> Context/where we are:
>>>> >>>>>
>>>> >>>>> For quite a while we used Buster (Debian 10) as our base image.
>>>> Buster is a stable release and it is being replaced with the next stable
>>>> release Bullseye (Debian 11).
>>>> >>>>>
>>>> >>>>> The Release schedule is here:
>>>> https://wiki.debian.org/DebianReleases
>>>> >>>>>
>>>> >>>>> The most important facts from those:
>>>> >>>>> * Bullseye was introduced in August 2021 (6 months ago) and has
>>>> no end-of-life yet
>>>> >>>>> * Buster was introduced in July 2019 and its end of life is ~
>>>> August 2020 (approx. 6 months from now)
>>>> >>>>>
>>>> >>>>> Python default images are (as of a few months), However they
>>>> provide (as usual -buster and -bullseye) specific images.
>>>> >>>>>
>>>> >>>>> The Bullseye switch is really needed to support ARM (M1) images
>>>> because of https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=989604
>>>> (still unresolved on Buster).
>>>> >>>>>
>>>> >>>>> It seems appropriate to switch to Bullseye now, however there is
>>>> one problem with it - MSSQL drivers do not yet support Bullseye. I've
>>>> reached out to the maintainers and await their answer when it might be
>>>> available:
>>>> https://github.com/MicrosoftDocs/sql-docs/issues/7255#issuecomment-1037097131
>>>> . This is the last blocker that prevents full switch (failing PR for a full
>>>> switch is https://github.com/apache/airflow/pull/21378). All tests
>>>> pass except MSSQL.
>>>> >>>>>
>>>> >>>>> But I think we need to proceed, regardless of the timeline and
>>>> switch to Bullseye.
>>>> >>>>>
>>>> >>>>> Question:
>>>> >>>>>
>>>> >>>>> We need to decide on what support we give our users in the
>>>> images now, in the 2.3.0 release and in the future. We have no policies for
>>>> that yet, so it might be a good time to decide now. Some of the users might
>>>> "rely" on the Buster being used by the images as they might have some
>>>> incompatible libraries (similarly to MSSQL). And I believe we need to
>>>> support Buster till at least the end of life of it.
>>>> >>>>>
>>>> >>>>> This support might be twofold:
>>>> >>>>>
>>>> >>>>> * we can publish in https://hub.docker.com/r/apache/airflow our
>>>> binary images with Buster and/or Bullseye
>>>> >>>>> * we can only publish one of those (and change from Buster to
>>>> Bullseye for 2.3.0) but users will be able to build their own custom images
>>>> using our Dockerfile for either of those.
>>>> >>>>>
>>>> >>>>> Proposal:
>>>> >>>>>
>>>> >>>>> I think there are multiple reasons why we should support both
>>>> Bullseye and Buster for the coming 6 months. We might build both images in
>>>> CI and add a new "matrix" dimension and run most tests on Bullseye but some
>>>> tests on Buster (specifically MSSQL until Bullseye is supported).
>>>> >>>>>
>>>> >>>>> We can publish both types of images (with -buster, -bullseye
>>>> suffixes same as Python) but the default image should be changed to
>>>> Bullseye.
>>>> >>>>>
>>>> >>>>> This will add some complexity (and build overhead) to our CI,
>>>> but I think it's worth it.
>>>> >>>>>
>>>> >>>>> Then we could drop support and release the -buster images after
>>>> it reaches end of life (but we will leave the users possibility of building
>>>> their own buster images without guarantee that it will work).
>>>> >>>>>
>>>> >>>>> I think this might become our Policy also for the future (6
>>>> months before end-of-life we switch by default to the new stable and
>>>> provide built images until end-of-life).
>>>> >>>>>
>>>> >>>>> WDYT? Does it look like a good proposal?
>>>> >>>>>
>>>> >>>>> J.
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>>
>>>
Re: [DISCUSS] Buster / Bullseye Debian support
Posted by Alex Begg <al...@gmail.com>.
Not sure if this is done already, but I think places with "ODBC Driver 17
for SQL Server" hardcoded in might need to change from 17 to 18 as well:
https://github.com/apache/airflow/search?q=ODBC+Driver+17+for+SQL+Server
On Wed, Feb 16, 2022 at 12:06 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> Yep. Already did :)
>
> On Wed, Feb 16, 2022 at 6:22 AM Alex Begg <al...@gmail.com> wrote:
>
>> I wanted to point out that not only did Microsoft add Debian Bullseye
>> support for MSSQL today, to do so they did a major version
>> upgrade, from msodbcsql17 to msodbcsql18. So in addition to updating the
>> URL used to populate mssql-release.list, you also need to change to
>> msodbcsql18. I see msodbcsql17 is in multiple places in the Airflow code
>> base.
>>
>> Instructions for installing msodbcsql18 in Debian are here:
>>
>> https://docs.microsoft.com/en-us/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server?view=sql-server-ver15#debian18
>>
>> Alex
>>
>>
>> On Tue, Feb 15, 2022 at 1:32 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>>> Yeah. That would definitely be a simpler approach for us.
>>>
>>> What we can also do is we can still make sure that you can actually
>>> "build" Buster images if you want and we could run the simple tests we
>>> already have for those images (those tests just check very basic stuff
>>> - like if expected packages are importable, if Airflow commands work,
>>> if you can run airflow as root user etc..
>>> This would be very little to no complication of the CI and we could
>>> provide a recipe for the users how to build their own buster image -
>>> but without guarantees that everything works there.
>>>
>>> This way might provide the users with a "workable solution" while we
>>> could keep it simple for us and put a strong emphasis on "If you want
>>> to get full support - also upgrade to the latest stable OS".
>>>
>>> J.
>>>
>>>
>>> On Tue, Feb 15, 2022 at 9:53 AM Ash Berlin-Taylor <as...@apache.org>
>>> wrote:
>>> >
>>> > My vote is to just move forward and only release Bullseye images for
>>> 2.3 onwards
>>> >
>>> > On 15 February 2022 08:43:50 GMT, Jarek Potiuk <ja...@potiuk.com>
>>> wrote:
>>> >>
>>> >> Good news from today - I got the response from Microsoft and it seems
>>> >> that MSSQL Bullseye support is going to be released today :) . So at
>>> >> least that problem is going to be solved. Still the question of
>>> >> whether we should build/release both Bullseye and Buster ?
>>> >>
>>> >> J.
>>> >>
>>> >> On Mon, Feb 14, 2022 at 2:14 PM Jarek Potiuk <ja...@potiuk.com>
>>> wrote:
>>> >>>
>>> >>>
>>> >>> And Python default images are Bullseye now.
>>> >>>
>>> >>> On Mon, Feb 14, 2022 at 2:13 PM Jarek Potiuk <ja...@potiuk.com>
>>> wrote:
>>> >>>>
>>> >>>>
>>> >>>> Buster end of life is August 2022 of course :)
>>> >>>>
>>> >>>> On Mon, Feb 14, 2022 at 2:12 PM Jarek Potiuk <ja...@potiuk.com>
>>> wrote:
>>> >>>>>
>>> >>>>>
>>> >>>>> Hello everyone,
>>> >>>>>
>>> >>>>> I am just about to complete preparation to make our images (and
>>> CI) work on Debian 11 images (Bullseye) vs Debian 10 (Buster). All looks
>>> (almost) good, but I would like to tap into the collective wisdom of the
>>> group here to decide on next steps (and maybe propose a policy we might be
>>> following in the future). The last change that allows to easily switch
>>> between the two is in reviee:
>>> https://github.com/apache/airflow/pull/21546
>>> >>>>>
>>> >>>>> Context/where we are:
>>> >>>>>
>>> >>>>> For quite a while we used Buster (Debian 10) as our base image.
>>> Buster is a stable release and it is being replaced with the next stable
>>> release Bullseye (Debian 11).
>>> >>>>>
>>> >>>>> The Release schedule is here:
>>> https://wiki.debian.org/DebianReleases
>>> >>>>>
>>> >>>>> The most important facts from those:
>>> >>>>> * Bullseye was introduced in August 2021 (6 months ago) and has
>>> no end-of-life yet
>>> >>>>> * Buster was introduced in July 2019 and its end of life is ~
>>> August 2020 (approx. 6 months from now)
>>> >>>>>
>>> >>>>> Python default images are (as of a few months), However they
>>> provide (as usual -buster and -bullseye) specific images.
>>> >>>>>
>>> >>>>> The Bullseye switch is really needed to support ARM (M1) images
>>> because of https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=989604
>>> (still unresolved on Buster).
>>> >>>>>
>>> >>>>> It seems appropriate to switch to Bullseye now, however there is
>>> one problem with it - MSSQL drivers do not yet support Bullseye. I've
>>> reached out to the maintainers and await their answer when it might be
>>> available:
>>> https://github.com/MicrosoftDocs/sql-docs/issues/7255#issuecomment-1037097131
>>> . This is the last blocker that prevents full switch (failing PR for a full
>>> switch is https://github.com/apache/airflow/pull/21378). All tests pass
>>> except MSSQL.
>>> >>>>>
>>> >>>>> But I think we need to proceed, regardless of the timeline and
>>> switch to Bullseye.
>>> >>>>>
>>> >>>>> Question:
>>> >>>>>
>>> >>>>> We need to decide on what support we give our users in the images
>>> now, in the 2.3.0 release and in the future. We have no policies for that
>>> yet, so it might be a good time to decide now. Some of the users might
>>> "rely" on the Buster being used by the images as they might have some
>>> incompatible libraries (similarly to MSSQL). And I believe we need to
>>> support Buster till at least the end of life of it.
>>> >>>>>
>>> >>>>> This support might be twofold:
>>> >>>>>
>>> >>>>> * we can publish in https://hub.docker.com/r/apache/airflow our
>>> binary images with Buster and/or Bullseye
>>> >>>>> * we can only publish one of those (and change from Buster to
>>> Bullseye for 2.3.0) but users will be able to build their own custom images
>>> using our Dockerfile for either of those.
>>> >>>>>
>>> >>>>> Proposal:
>>> >>>>>
>>> >>>>> I think there are multiple reasons why we should support both
>>> Bullseye and Buster for the coming 6 months. We might build both images in
>>> CI and add a new "matrix" dimension and run most tests on Bullseye but some
>>> tests on Buster (specifically MSSQL until Bullseye is supported).
>>> >>>>>
>>> >>>>> We can publish both types of images (with -buster, -bullseye
>>> suffixes same as Python) but the default image should be changed to
>>> Bullseye.
>>> >>>>>
>>> >>>>> This will add some complexity (and build overhead) to our CI, but
>>> I think it's worth it.
>>> >>>>>
>>> >>>>> Then we could drop support and release the -buster images after
>>> it reaches end of life (but we will leave the users possibility of building
>>> their own buster images without guarantee that it will work).
>>> >>>>>
>>> >>>>> I think this might become our Policy also for the future (6
>>> months before end-of-life we switch by default to the new stable and
>>> provide built images until end-of-life).
>>> >>>>>
>>> >>>>> WDYT? Does it look like a good proposal?
>>> >>>>>
>>> >>>>> J.
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>>
>>
Re: [DISCUSS] Buster / Bullseye Debian support
Posted by Jarek Potiuk <ja...@potiuk.com>.
Yep. Already did :)
On Wed, Feb 16, 2022 at 6:22 AM Alex Begg <al...@gmail.com> wrote:
> I wanted to point out that not only did Microsoft add Debian Bullseye
> support for MSSQL today, to do so they did a major version
> upgrade, from msodbcsql17 to msodbcsql18. So in addition to updating the
> URL used to populate mssql-release.list, you also need to change to
> msodbcsql18. I see msodbcsql17 is in multiple places in the Airflow code
> base.
>
> Instructions for installing msodbcsql18 in Debian are here:
>
> https://docs.microsoft.com/en-us/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server?view=sql-server-ver15#debian18
>
> Alex
>
>
> On Tue, Feb 15, 2022 at 1:32 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> Yeah. That would definitely be a simpler approach for us.
>>
>> What we can also do is we can still make sure that you can actually
>> "build" Buster images if you want and we could run the simple tests we
>> already have for those images (those tests just check very basic stuff
>> - like if expected packages are importable, if Airflow commands work,
>> if you can run airflow as root user etc..
>> This would be very little to no complication of the CI and we could
>> provide a recipe for the users how to build their own buster image -
>> but without guarantees that everything works there.
>>
>> This way might provide the users with a "workable solution" while we
>> could keep it simple for us and put a strong emphasis on "If you want
>> to get full support - also upgrade to the latest stable OS".
>>
>> J.
>>
>>
>> On Tue, Feb 15, 2022 at 9:53 AM Ash Berlin-Taylor <as...@apache.org> wrote:
>> >
>> > My vote is to just move forward and only release Bullseye images for
>> 2.3 onwards
>> >
>> > On 15 February 2022 08:43:50 GMT, Jarek Potiuk <ja...@potiuk.com>
>> wrote:
>> >>
>> >> Good news from today - I got the response from Microsoft and it seems
>> >> that MSSQL Bullseye support is going to be released today :) . So at
>> >> least that problem is going to be solved. Still the question of
>> >> whether we should build/release both Bullseye and Buster ?
>> >>
>> >> J.
>> >>
>> >> On Mon, Feb 14, 2022 at 2:14 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>> >>>
>> >>>
>> >>> And Python default images are Bullseye now.
>> >>>
>> >>> On Mon, Feb 14, 2022 at 2:13 PM Jarek Potiuk <ja...@potiuk.com>
>> wrote:
>> >>>>
>> >>>>
>> >>>> Buster end of life is August 2022 of course :)
>> >>>>
>> >>>> On Mon, Feb 14, 2022 at 2:12 PM Jarek Potiuk <ja...@potiuk.com>
>> wrote:
>> >>>>>
>> >>>>>
>> >>>>> Hello everyone,
>> >>>>>
>> >>>>> I am just about to complete preparation to make our images (and
>> CI) work on Debian 11 images (Bullseye) vs Debian 10 (Buster). All looks
>> (almost) good, but I would like to tap into the collective wisdom of the
>> group here to decide on next steps (and maybe propose a policy we might be
>> following in the future). The last change that allows to easily switch
>> between the two is in reviee:
>> https://github.com/apache/airflow/pull/21546
>> >>>>>
>> >>>>> Context/where we are:
>> >>>>>
>> >>>>> For quite a while we used Buster (Debian 10) as our base image.
>> Buster is a stable release and it is being replaced with the next stable
>> release Bullseye (Debian 11).
>> >>>>>
>> >>>>> The Release schedule is here:
>> https://wiki.debian.org/DebianReleases
>> >>>>>
>> >>>>> The most important facts from those:
>> >>>>> * Bullseye was introduced in August 2021 (6 months ago) and has no
>> end-of-life yet
>> >>>>> * Buster was introduced in July 2019 and its end of life is ~
>> August 2020 (approx. 6 months from now)
>> >>>>>
>> >>>>> Python default images are (as of a few months), However they
>> provide (as usual -buster and -bullseye) specific images.
>> >>>>>
>> >>>>> The Bullseye switch is really needed to support ARM (M1) images
>> because of https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=989604
>> (still unresolved on Buster).
>> >>>>>
>> >>>>> It seems appropriate to switch to Bullseye now, however there is
>> one problem with it - MSSQL drivers do not yet support Bullseye. I've
>> reached out to the maintainers and await their answer when it might be
>> available:
>> https://github.com/MicrosoftDocs/sql-docs/issues/7255#issuecomment-1037097131
>> . This is the last blocker that prevents full switch (failing PR for a full
>> switch is https://github.com/apache/airflow/pull/21378). All tests pass
>> except MSSQL.
>> >>>>>
>> >>>>> But I think we need to proceed, regardless of the timeline and
>> switch to Bullseye.
>> >>>>>
>> >>>>> Question:
>> >>>>>
>> >>>>> We need to decide on what support we give our users in the images
>> now, in the 2.3.0 release and in the future. We have no policies for that
>> yet, so it might be a good time to decide now. Some of the users might
>> "rely" on the Buster being used by the images as they might have some
>> incompatible libraries (similarly to MSSQL). And I believe we need to
>> support Buster till at least the end of life of it.
>> >>>>>
>> >>>>> This support might be twofold:
>> >>>>>
>> >>>>> * we can publish in https://hub.docker.com/r/apache/airflow our
>> binary images with Buster and/or Bullseye
>> >>>>> * we can only publish one of those (and change from Buster to
>> Bullseye for 2.3.0) but users will be able to build their own custom images
>> using our Dockerfile for either of those.
>> >>>>>
>> >>>>> Proposal:
>> >>>>>
>> >>>>> I think there are multiple reasons why we should support both
>> Bullseye and Buster for the coming 6 months. We might build both images in
>> CI and add a new "matrix" dimension and run most tests on Bullseye but some
>> tests on Buster (specifically MSSQL until Bullseye is supported).
>> >>>>>
>> >>>>> We can publish both types of images (with -buster, -bullseye
>> suffixes same as Python) but the default image should be changed to
>> Bullseye.
>> >>>>>
>> >>>>> This will add some complexity (and build overhead) to our CI, but
>> I think it's worth it.
>> >>>>>
>> >>>>> Then we could drop support and release the -buster images after it
>> reaches end of life (but we will leave the users possibility of building
>> their own buster images without guarantee that it will work).
>> >>>>>
>> >>>>> I think this might become our Policy also for the future (6 months
>> before end-of-life we switch by default to the new stable and provide built
>> images until end-of-life).
>> >>>>>
>> >>>>> WDYT? Does it look like a good proposal?
>> >>>>>
>> >>>>> J.
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>>
>
Re: [DISCUSS] Buster / Bullseye Debian support
Posted by Alex Begg <al...@gmail.com>.
I wanted to point out that not only did Microsoft add Debian Bullseye
support for MSSQL today, to do so they did a major version
upgrade, from msodbcsql17 to msodbcsql18. So in addition to updating the
URL used to populate mssql-release.list, you also need to change to
msodbcsql18. I see msodbcsql17 is in multiple places in the Airflow code
base.
Instructions for installing msodbcsql18 in Debian are here:
https://docs.microsoft.com/en-us/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server?view=sql-server-ver15#debian18
Alex
On Tue, Feb 15, 2022 at 1:32 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> Yeah. That would definitely be a simpler approach for us.
>
> What we can also do is we can still make sure that you can actually
> "build" Buster images if you want and we could run the simple tests we
> already have for those images (those tests just check very basic stuff
> - like if expected packages are importable, if Airflow commands work,
> if you can run airflow as root user etc..
> This would be very little to no complication of the CI and we could
> provide a recipe for the users how to build their own buster image -
> but without guarantees that everything works there.
>
> This way might provide the users with a "workable solution" while we
> could keep it simple for us and put a strong emphasis on "If you want
> to get full support - also upgrade to the latest stable OS".
>
> J.
>
>
> On Tue, Feb 15, 2022 at 9:53 AM Ash Berlin-Taylor <as...@apache.org> wrote:
> >
> > My vote is to just move forward and only release Bullseye images for 2.3
> onwards
> >
> > On 15 February 2022 08:43:50 GMT, Jarek Potiuk <ja...@potiuk.com> wrote:
> >>
> >> Good news from today - I got the response from Microsoft and it seems
> >> that MSSQL Bullseye support is going to be released today :) . So at
> >> least that problem is going to be solved. Still the question of
> >> whether we should build/release both Bullseye and Buster ?
> >>
> >> J.
> >>
> >> On Mon, Feb 14, 2022 at 2:14 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >>>
> >>>
> >>> And Python default images are Bullseye now.
> >>>
> >>> On Mon, Feb 14, 2022 at 2:13 PM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> >>>>
> >>>>
> >>>> Buster end of life is August 2022 of course :)
> >>>>
> >>>> On Mon, Feb 14, 2022 at 2:12 PM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> >>>>>
> >>>>>
> >>>>> Hello everyone,
> >>>>>
> >>>>> I am just about to complete preparation to make our images (and CI)
> work on Debian 11 images (Bullseye) vs Debian 10 (Buster). All looks
> (almost) good, but I would like to tap into the collective wisdom of the
> group here to decide on next steps (and maybe propose a policy we might be
> following in the future). The last change that allows to easily switch
> between the two is in reviee: https://github.com/apache/airflow/pull/21546
> >>>>>
> >>>>> Context/where we are:
> >>>>>
> >>>>> For quite a while we used Buster (Debian 10) as our base image.
> Buster is a stable release and it is being replaced with the next stable
> release Bullseye (Debian 11).
> >>>>>
> >>>>> The Release schedule is here:
> https://wiki.debian.org/DebianReleases
> >>>>>
> >>>>> The most important facts from those:
> >>>>> * Bullseye was introduced in August 2021 (6 months ago) and has no
> end-of-life yet
> >>>>> * Buster was introduced in July 2019 and its end of life is ~
> August 2020 (approx. 6 months from now)
> >>>>>
> >>>>> Python default images are (as of a few months), However they
> provide (as usual -buster and -bullseye) specific images.
> >>>>>
> >>>>> The Bullseye switch is really needed to support ARM (M1) images
> because of https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=989604
> (still unresolved on Buster).
> >>>>>
> >>>>> It seems appropriate to switch to Bullseye now, however there is
> one problem with it - MSSQL drivers do not yet support Bullseye. I've
> reached out to the maintainers and await their answer when it might be
> available:
> https://github.com/MicrosoftDocs/sql-docs/issues/7255#issuecomment-1037097131
> . This is the last blocker that prevents full switch (failing PR for a full
> switch is https://github.com/apache/airflow/pull/21378). All tests pass
> except MSSQL.
> >>>>>
> >>>>> But I think we need to proceed, regardless of the timeline and
> switch to Bullseye.
> >>>>>
> >>>>> Question:
> >>>>>
> >>>>> We need to decide on what support we give our users in the images
> now, in the 2.3.0 release and in the future. We have no policies for that
> yet, so it might be a good time to decide now. Some of the users might
> "rely" on the Buster being used by the images as they might have some
> incompatible libraries (similarly to MSSQL). And I believe we need to
> support Buster till at least the end of life of it.
> >>>>>
> >>>>> This support might be twofold:
> >>>>>
> >>>>> * we can publish in https://hub.docker.com/r/apache/airflow our
> binary images with Buster and/or Bullseye
> >>>>> * we can only publish one of those (and change from Buster to
> Bullseye for 2.3.0) but users will be able to build their own custom images
> using our Dockerfile for either of those.
> >>>>>
> >>>>> Proposal:
> >>>>>
> >>>>> I think there are multiple reasons why we should support both
> Bullseye and Buster for the coming 6 months. We might build both images in
> CI and add a new "matrix" dimension and run most tests on Bullseye but some
> tests on Buster (specifically MSSQL until Bullseye is supported).
> >>>>>
> >>>>> We can publish both types of images (with -buster, -bullseye
> suffixes same as Python) but the default image should be changed to
> Bullseye.
> >>>>>
> >>>>> This will add some complexity (and build overhead) to our CI, but I
> think it's worth it.
> >>>>>
> >>>>> Then we could drop support and release the -buster images after it
> reaches end of life (but we will leave the users possibility of building
> their own buster images without guarantee that it will work).
> >>>>>
> >>>>> I think this might become our Policy also for the future (6 months
> before end-of-life we switch by default to the new stable and provide built
> images until end-of-life).
> >>>>>
> >>>>> WDYT? Does it look like a good proposal?
> >>>>>
> >>>>> J.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
>
Re: [DISCUSS] Buster / Bullseye Debian support
Posted by Jarek Potiuk <ja...@potiuk.com>.
Yeah. That would definitely be a simpler approach for us.
What we can also do is we can still make sure that you can actually
"build" Buster images if you want and we could run the simple tests we
already have for those images (those tests just check very basic stuff
- like if expected packages are importable, if Airflow commands work,
if you can run airflow as root user etc..
This would be very little to no complication of the CI and we could
provide a recipe for the users how to build their own buster image -
but without guarantees that everything works there.
This way might provide the users with a "workable solution" while we
could keep it simple for us and put a strong emphasis on "If you want
to get full support - also upgrade to the latest stable OS".
J.
On Tue, Feb 15, 2022 at 9:53 AM Ash Berlin-Taylor <as...@apache.org> wrote:
>
> My vote is to just move forward and only release Bullseye images for 2.3 onwards
>
> On 15 February 2022 08:43:50 GMT, Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>> Good news from today - I got the response from Microsoft and it seems
>> that MSSQL Bullseye support is going to be released today :) . So at
>> least that problem is going to be solved. Still the question of
>> whether we should build/release both Bullseye and Buster ?
>>
>> J.
>>
>> On Mon, Feb 14, 2022 at 2:14 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>
>>>
>>> And Python default images are Bullseye now.
>>>
>>> On Mon, Feb 14, 2022 at 2:13 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>
>>>>
>>>> Buster end of life is August 2022 of course :)
>>>>
>>>> On Mon, Feb 14, 2022 at 2:12 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>>
>>>>>
>>>>> Hello everyone,
>>>>>
>>>>> I am just about to complete preparation to make our images (and CI) work on Debian 11 images (Bullseye) vs Debian 10 (Buster). All looks (almost) good, but I would like to tap into the collective wisdom of the group here to decide on next steps (and maybe propose a policy we might be following in the future). The last change that allows to easily switch between the two is in reviee: https://github.com/apache/airflow/pull/21546
>>>>>
>>>>> Context/where we are:
>>>>>
>>>>> For quite a while we used Buster (Debian 10) as our base image. Buster is a stable release and it is being replaced with the next stable release Bullseye (Debian 11).
>>>>>
>>>>> The Release schedule is here: https://wiki.debian.org/DebianReleases
>>>>>
>>>>> The most important facts from those:
>>>>> * Bullseye was introduced in August 2021 (6 months ago) and has no end-of-life yet
>>>>> * Buster was introduced in July 2019 and its end of life is ~ August 2020 (approx. 6 months from now)
>>>>>
>>>>> Python default images are (as of a few months), However they provide (as usual -buster and -bullseye) specific images.
>>>>>
>>>>> The Bullseye switch is really needed to support ARM (M1) images because of https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=989604 (still unresolved on Buster).
>>>>>
>>>>> It seems appropriate to switch to Bullseye now, however there is one problem with it - MSSQL drivers do not yet support Bullseye. I've reached out to the maintainers and await their answer when it might be available: https://github.com/MicrosoftDocs/sql-docs/issues/7255#issuecomment-1037097131 . This is the last blocker that prevents full switch (failing PR for a full switch is https://github.com/apache/airflow/pull/21378). All tests pass except MSSQL.
>>>>>
>>>>> But I think we need to proceed, regardless of the timeline and switch to Bullseye.
>>>>>
>>>>> Question:
>>>>>
>>>>> We need to decide on what support we give our users in the images now, in the 2.3.0 release and in the future. We have no policies for that yet, so it might be a good time to decide now. Some of the users might "rely" on the Buster being used by the images as they might have some incompatible libraries (similarly to MSSQL). And I believe we need to support Buster till at least the end of life of it.
>>>>>
>>>>> This support might be twofold:
>>>>>
>>>>> * we can publish in https://hub.docker.com/r/apache/airflow our binary images with Buster and/or Bullseye
>>>>> * we can only publish one of those (and change from Buster to Bullseye for 2.3.0) but users will be able to build their own custom images using our Dockerfile for either of those.
>>>>>
>>>>> Proposal:
>>>>>
>>>>> I think there are multiple reasons why we should support both Bullseye and Buster for the coming 6 months. We might build both images in CI and add a new "matrix" dimension and run most tests on Bullseye but some tests on Buster (specifically MSSQL until Bullseye is supported).
>>>>>
>>>>> We can publish both types of images (with -buster, -bullseye suffixes same as Python) but the default image should be changed to Bullseye.
>>>>>
>>>>> This will add some complexity (and build overhead) to our CI, but I think it's worth it.
>>>>>
>>>>> Then we could drop support and release the -buster images after it reaches end of life (but we will leave the users possibility of building their own buster images without guarantee that it will work).
>>>>>
>>>>> I think this might become our Policy also for the future (6 months before end-of-life we switch by default to the new stable and provide built images until end-of-life).
>>>>>
>>>>> WDYT? Does it look like a good proposal?
>>>>>
>>>>> J.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
Re: [DISCUSS] Buster / Bullseye Debian support
Posted by Ash Berlin-Taylor <as...@apache.org>.
My vote is to just move forward and only release Bullseye images for 2.3 onwards
On 15 February 2022 08:43:50 GMT, Jarek Potiuk <ja...@potiuk.com> wrote:
>Good news from today - I got the response from Microsoft and it seems
>that MSSQL Bullseye support is going to be released today :) . So at
>least that problem is going to be solved. Still the question of
>whether we should build/release both Bullseye and Buster ?
>
>J.
>
>On Mon, Feb 14, 2022 at 2:14 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>> And Python default images are Bullseye now.
>>
>> On Mon, Feb 14, 2022 at 2:13 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>
>>> Buster end of life is August 2022 of course :)
>>>
>>> On Mon, Feb 14, 2022 at 2:12 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>
>>>> Hello everyone,
>>>>
>>>> I am just about to complete preparation to make our images (and CI) work on Debian 11 images (Bullseye) vs Debian 10 (Buster). All looks (almost) good, but I would like to tap into the collective wisdom of the group here to decide on next steps (and maybe propose a policy we might be following in the future). The last change that allows to easily switch between the two is in reviee: https://github.com/apache/airflow/pull/21546
>>>>
>>>> Context/where we are:
>>>>
>>>> For quite a while we used Buster (Debian 10) as our base image. Buster is a stable release and it is being replaced with the next stable release Bullseye (Debian 11).
>>>>
>>>> The Release schedule is here: https://wiki.debian.org/DebianReleases
>>>>
>>>> The most important facts from those:
>>>> * Bullseye was introduced in August 2021 (6 months ago) and has no end-of-life yet
>>>> * Buster was introduced in July 2019 and its end of life is ~ August 2020 (approx. 6 months from now)
>>>>
>>>> Python default images are (as of a few months), However they provide (as usual -buster and -bullseye) specific images.
>>>>
>>>> The Bullseye switch is really needed to support ARM (M1) images because of https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=989604 (still unresolved on Buster).
>>>>
>>>> It seems appropriate to switch to Bullseye now, however there is one problem with it - MSSQL drivers do not yet support Bullseye. I've reached out to the maintainers and await their answer when it might be available: https://github.com/MicrosoftDocs/sql-docs/issues/7255#issuecomment-1037097131 . This is the last blocker that prevents full switch (failing PR for a full switch is https://github.com/apache/airflow/pull/21378). All tests pass except MSSQL.
>>>>
>>>> But I think we need to proceed, regardless of the timeline and switch to Bullseye.
>>>>
>>>> Question:
>>>>
>>>> We need to decide on what support we give our users in the images now, in the 2.3.0 release and in the future. We have no policies for that yet, so it might be a good time to decide now. Some of the users might "rely" on the Buster being used by the images as they might have some incompatible libraries (similarly to MSSQL). And I believe we need to support Buster till at least the end of life of it.
>>>>
>>>> This support might be twofold:
>>>>
>>>> * we can publish in https://hub.docker.com/r/apache/airflow our binary images with Buster and/or Bullseye
>>>> * we can only publish one of those (and change from Buster to Bullseye for 2.3.0) but users will be able to build their own custom images using our Dockerfile for either of those.
>>>>
>>>> Proposal:
>>>>
>>>> I think there are multiple reasons why we should support both Bullseye and Buster for the coming 6 months. We might build both images in CI and add a new "matrix" dimension and run most tests on Bullseye but some tests on Buster (specifically MSSQL until Bullseye is supported).
>>>>
>>>> We can publish both types of images (with -buster, -bullseye suffixes same as Python) but the default image should be changed to Bullseye.
>>>>
>>>> This will add some complexity (and build overhead) to our CI, but I think it's worth it.
>>>>
>>>> Then we could drop support and release the -buster images after it reaches end of life (but we will leave the users possibility of building their own buster images without guarantee that it will work).
>>>>
>>>> I think this might become our Policy also for the future (6 months before end-of-life we switch by default to the new stable and provide built images until end-of-life).
>>>>
>>>> WDYT? Does it look like a good proposal?
>>>>
>>>> J.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
Re: [DISCUSS] Buster / Bullseye Debian support
Posted by Jarek Potiuk <ja...@potiuk.com>.
Good news from today - I got the response from Microsoft and it seems
that MSSQL Bullseye support is going to be released today :) . So at
least that problem is going to be solved. Still the question of
whether we should build/release both Bullseye and Buster ?
J.
On Mon, Feb 14, 2022 at 2:14 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> And Python default images are Bullseye now.
>
> On Mon, Feb 14, 2022 at 2:13 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>> Buster end of life is August 2022 of course :)
>>
>> On Mon, Feb 14, 2022 at 2:12 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>
>>> Hello everyone,
>>>
>>> I am just about to complete preparation to make our images (and CI) work on Debian 11 images (Bullseye) vs Debian 10 (Buster). All looks (almost) good, but I would like to tap into the collective wisdom of the group here to decide on next steps (and maybe propose a policy we might be following in the future). The last change that allows to easily switch between the two is in reviee: https://github.com/apache/airflow/pull/21546
>>>
>>> Context/where we are:
>>>
>>> For quite a while we used Buster (Debian 10) as our base image. Buster is a stable release and it is being replaced with the next stable release Bullseye (Debian 11).
>>>
>>> The Release schedule is here: https://wiki.debian.org/DebianReleases
>>>
>>> The most important facts from those:
>>> * Bullseye was introduced in August 2021 (6 months ago) and has no end-of-life yet
>>> * Buster was introduced in July 2019 and its end of life is ~ August 2020 (approx. 6 months from now)
>>>
>>> Python default images are (as of a few months), However they provide (as usual -buster and -bullseye) specific images.
>>>
>>> The Bullseye switch is really needed to support ARM (M1) images because of https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=989604 (still unresolved on Buster).
>>>
>>> It seems appropriate to switch to Bullseye now, however there is one problem with it - MSSQL drivers do not yet support Bullseye. I've reached out to the maintainers and await their answer when it might be available: https://github.com/MicrosoftDocs/sql-docs/issues/7255#issuecomment-1037097131 . This is the last blocker that prevents full switch (failing PR for a full switch is https://github.com/apache/airflow/pull/21378). All tests pass except MSSQL.
>>>
>>> But I think we need to proceed, regardless of the timeline and switch to Bullseye.
>>>
>>> Question:
>>>
>>> We need to decide on what support we give our users in the images now, in the 2.3.0 release and in the future. We have no policies for that yet, so it might be a good time to decide now. Some of the users might "rely" on the Buster being used by the images as they might have some incompatible libraries (similarly to MSSQL). And I believe we need to support Buster till at least the end of life of it.
>>>
>>> This support might be twofold:
>>>
>>> * we can publish in https://hub.docker.com/r/apache/airflow our binary images with Buster and/or Bullseye
>>> * we can only publish one of those (and change from Buster to Bullseye for 2.3.0) but users will be able to build their own custom images using our Dockerfile for either of those.
>>>
>>> Proposal:
>>>
>>> I think there are multiple reasons why we should support both Bullseye and Buster for the coming 6 months. We might build both images in CI and add a new "matrix" dimension and run most tests on Bullseye but some tests on Buster (specifically MSSQL until Bullseye is supported).
>>>
>>> We can publish both types of images (with -buster, -bullseye suffixes same as Python) but the default image should be changed to Bullseye.
>>>
>>> This will add some complexity (and build overhead) to our CI, but I think it's worth it.
>>>
>>> Then we could drop support and release the -buster images after it reaches end of life (but we will leave the users possibility of building their own buster images without guarantee that it will work).
>>>
>>> I think this might become our Policy also for the future (6 months before end-of-life we switch by default to the new stable and provide built images until end-of-life).
>>>
>>> WDYT? Does it look like a good proposal?
>>>
>>> J.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
Re: [DISCUSS] Buster / Bullseye Debian support
Posted by Jarek Potiuk <ja...@potiuk.com>.
And Python default images are Bullseye now.
On Mon, Feb 14, 2022 at 2:13 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> Buster end of life is August 2022 of course :)
>
> On Mon, Feb 14, 2022 at 2:12 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> Hello everyone,
>>
>> I am just about to complete preparation to make our images (and CI) work
>> on Debian 11 images (Bullseye) vs Debian 10 (Buster). All looks (almost)
>> good, but I would like to tap into the collective wisdom of the group here
>> to decide on next steps (and maybe propose a policy we might be following
>> in the future). The last change that allows to easily switch between the
>> two is in reviee: https://github.com/apache/airflow/pull/21546
>>
>> Context/where we are:
>>
>> For quite a while we used Buster (Debian 10) as our base image. Buster is
>> a stable release and it is being replaced with the next
>> stable release Bullseye (Debian 11).
>>
>> The Release schedule is here: https://wiki.debian.org/DebianReleases
>>
>> The most important facts from those:
>> * Bullseye was introduced in August 2021 (6 months ago) and has no
>> end-of-life yet
>> * Buster was introduced in July 2019 and its end of life is ~ August 2020
>> (approx. 6 months from now)
>>
>> Python default images are (as of a few months), However they provide (as
>> usual -buster and -bullseye) specific images.
>>
>> The Bullseye switch is really needed to support ARM (M1) images because
>> of https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=989604 (still
>> unresolved on Buster).
>>
>> It seems appropriate to switch to Bullseye now, however there is one
>> problem with it - MSSQL drivers do not yet support Bullseye. I've reached
>> out to the maintainers and await their answer when it might be available:
>> https://github.com/MicrosoftDocs/sql-docs/issues/7255#issuecomment-1037097131
>> . This is the last blocker that prevents full switch (failing PR for a full
>> switch is https://github.com/apache/airflow/pull/21378). All tests pass
>> except MSSQL.
>>
>> But I think we need to proceed, regardless of the timeline and switch to
>> Bullseye.
>>
>> Question:
>>
>> We need to decide on what support we give our users in the images now, in
>> the 2.3.0 release and in the future. We have no policies for that yet, so
>> it might be a good time to decide now. Some of the users might "rely" on
>> the Buster being used by the images as they might have some incompatible
>> libraries (similarly to MSSQL). And I believe we need to support Buster
>> till at least the end of life of it.
>>
>> This support might be twofold:
>>
>> * we can publish in https://hub.docker.com/r/apache/airflow our binary
>> images with Buster and/or Bullseye
>> * we can only publish one of those (and change from Buster to Bullseye
>> for 2.3.0) but users will be able to build their own custom images using
>> our Dockerfile for either of those.
>>
>> Proposal:
>>
>> I think there are multiple reasons why we should support both Bullseye
>> and Buster for the coming 6 months. We might build both images in CI and
>> add a new "matrix" dimension and run most tests on Bullseye but some tests
>> on Buster (specifically MSSQL until Bullseye is supported).
>>
>> We can publish both types of images (with -buster, -bullseye suffixes
>> same as Python) but the default image should be changed to Bullseye.
>>
>> This will add some complexity (and build overhead) to our CI, but I think
>> it's worth it.
>>
>> Then we could drop support and release the -buster images after it
>> reaches end of life (but we will leave the users possibility of building
>> their own buster images without guarantee that it will work).
>>
>> I think this might become our Policy also for the future (6 months before
>> end-of-life we switch by default to the new stable and provide built images
>> until end-of-life).
>>
>> WDYT? Does it look like a good proposal?
>>
>> J.
>>
>>
>>
>>
>>
>>
>>
>>
>>
Re: [DISCUSS] Buster / Bullseye Debian support
Posted by Jarek Potiuk <ja...@potiuk.com>.
Buster end of life is August 2022 of course :)
On Mon, Feb 14, 2022 at 2:12 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> Hello everyone,
>
> I am just about to complete preparation to make our images (and CI) work
> on Debian 11 images (Bullseye) vs Debian 10 (Buster). All looks (almost)
> good, but I would like to tap into the collective wisdom of the group here
> to decide on next steps (and maybe propose a policy we might be following
> in the future). The last change that allows to easily switch between the
> two is in reviee: https://github.com/apache/airflow/pull/21546
>
> Context/where we are:
>
> For quite a while we used Buster (Debian 10) as our base image. Buster is
> a stable release and it is being replaced with the next
> stable release Bullseye (Debian 11).
>
> The Release schedule is here: https://wiki.debian.org/DebianReleases
>
> The most important facts from those:
> * Bullseye was introduced in August 2021 (6 months ago) and has no
> end-of-life yet
> * Buster was introduced in July 2019 and its end of life is ~ August 2020
> (approx. 6 months from now)
>
> Python default images are (as of a few months), However they provide (as
> usual -buster and -bullseye) specific images.
>
> The Bullseye switch is really needed to support ARM (M1) images because of
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=989604 (still
> unresolved on Buster).
>
> It seems appropriate to switch to Bullseye now, however there is one
> problem with it - MSSQL drivers do not yet support Bullseye. I've reached
> out to the maintainers and await their answer when it might be available:
> https://github.com/MicrosoftDocs/sql-docs/issues/7255#issuecomment-1037097131
> . This is the last blocker that prevents full switch (failing PR for a full
> switch is https://github.com/apache/airflow/pull/21378). All tests pass
> except MSSQL.
>
> But I think we need to proceed, regardless of the timeline and switch to
> Bullseye.
>
> Question:
>
> We need to decide on what support we give our users in the images now, in
> the 2.3.0 release and in the future. We have no policies for that yet, so
> it might be a good time to decide now. Some of the users might "rely" on
> the Buster being used by the images as they might have some incompatible
> libraries (similarly to MSSQL). And I believe we need to support Buster
> till at least the end of life of it.
>
> This support might be twofold:
>
> * we can publish in https://hub.docker.com/r/apache/airflow our binary
> images with Buster and/or Bullseye
> * we can only publish one of those (and change from Buster to Bullseye for
> 2.3.0) but users will be able to build their own custom images using our
> Dockerfile for either of those.
>
> Proposal:
>
> I think there are multiple reasons why we should support both Bullseye and
> Buster for the coming 6 months. We might build both images in CI and add a
> new "matrix" dimension and run most tests on Bullseye but some tests on
> Buster (specifically MSSQL until Bullseye is supported).
>
> We can publish both types of images (with -buster, -bullseye suffixes same
> as Python) but the default image should be changed to Bullseye.
>
> This will add some complexity (and build overhead) to our CI, but I think
> it's worth it.
>
> Then we could drop support and release the -buster images after it reaches
> end of life (but we will leave the users possibility of building their own
> buster images without guarantee that it will work).
>
> I think this might become our Policy also for the future (6 months before
> end-of-life we switch by default to the new stable and provide built images
> until end-of-life).
>
> WDYT? Does it look like a good proposal?
>
> J.
>
>
>
>
>
>
>
>
>