You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Jarek Potiuk <Ja...@polidea.com> on 2020/11/05 08:49:09 UTC
Default/supported Python versions for Airlfow 2.0
Hello Everyone,
I have a question. What do people think about default version of Pyhon for
Airflow 2.0 (and set of supported versions)?
Currently, we have python 3.6 as default, but all the version up to 3.8 are
officially supported and tested and PR for python 3.9 is in Draft:
https://github.com/apache/airflow/pull/11950
This is the release schedule for python versions. We have a year till the
end of 3.6
Branch Schedule Status First release End-of-life
3.9 PEP 596 bugfix 2020-10-05 TBD
3.8 PEP 569 bugfix 2019-10-14 2024-10
3.7 PEP 537 security 2018-06-27 2023-06-27
3.6 PEP 494 security 2016-12-23 2021-12-23
WDYT?
J.
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Default/supported Python versions for Airlfow 2.0
Posted by Jarek Potiuk <Ja...@polidea.com>.
Any other comments before I cast a vote for our general approach when it
comes to versioning ?
The current proposal;
1. We finish support for python versions when they reach EOL (For python
3.6 it means that we will remove it from being supported on 23.12.2021).
2. The "oldest" supported version of Python as the default (3.6 today and
it will remain the default until 23.12.2021. On 23.12.2021 3.7 becomes the
default).
3. We support newer versions of Python after they are officially released,
as soon as we manage to make it work as part of our CI pipeline.
We can always decide to support versions past EOL if we have a good reason
- but it would have to be justified and voted.
J.
On Sat, Nov 21, 2020 at 12:05 PM Jarek Potiuk <Ja...@polidea.com>
wrote:
> Fair point that needs explanation. That was a bit of a "mental shortcut".
>
> Latest supported means = "latest that we as Apache Airflow community
> support".
>
> For example, Python 3.9 is not supported yet by Airflow and there are some
> good reasons for that - namely our transitive dependencies. NumPy in
> version 1.17 does not yet have a binary Python 3.9 binary PIP package.
> Building NumPy from sources is not something we would like to mess around
> with. You can see more here: https://github.com/apache/airflow/pull/11950
>
> We have no strong need to support 3.9 yet and since it might be solved by
> simply upgrading dependencies (which we plan anyway:
> https://github.com/apache/airflow/issues/10854), support for 3.9 might be
> deferred, even if it is officially released.
>
> There will be other cases like that in the future, so there will likely be
> a considerable delay between when a new version is released, and when we
> start supporting it in Airflow.
>
> I think this is worth explaining what "default" version means for us,
> there are two things "default" is used for:
>
> * optimization of PR job usage. Simple PRs will only run limited version
> of tests - only for the default python version (3.6 currently) and only 1
> version of each of the DBs. We can always run full tests (by applying "full
> tests needed" label) which will run tests for all "supported versions"
> (currently 3.6, 3.7, 3.8) and all tests are run after merge to master and
> in scheduled builds. So we notice if there's something wrong
> * default python version is used for "apache/airflow:master" production
> image published. So really "apache/airflow:master" is the same as
> "apache/airflow:master-python3.6".
>
> And now, when I wrote this, I think I want to change my proposal. I
> believe there will be far more opportunities for "test" failures if we
> switch to "latest" version There are new things added in newer versions of
> python that we might be tempted to use, and they will fail in earlier
> version) - so our current approach where default is "oldest supported
> version" is better.
>
>
> So my new proposal is:
>
> * we finish support for python versions when they reach EOL (For python
> 3.6 it means that we will remove it from being supported on 23.12.2021).
> * the "oldest" supported version of Python as the default (3.6 today and
> it will remain the default until 23.12.2021. On 23.12.2021 3.7 becomes the
> default).
> * we support newer versions of Python some time after first release, as
> soon as we manage to make it work as part of our CI pipeline.
>
> We can always decide to support versions past EOL if we have a good reason
> - but it would have to be justified and voted.
>
> WDYT?
>
> Again Python schedule:
>
> Branch Schedule Status First release End-of-life
> 3.9 PEP 596 bugfix 2020-10-05 TBD
> 3.8 PEP 569 bugfix 2019-10-14 2024-10
> 3.7 PEP 537 security 2018-06-27 2023-06-27
> 3.6 PEP 494 security 2016-12-23 2021-12-2
>
>
> J.
>
>
> On Fri, Nov 20, 2020 at 3:27 PM Shaw, Damian P. <
> damian.shaw.2@credit-suisse.com> wrote:
>
>> Typo,
>>
>>
>>
>> Should of read: Python 3.9 will lose bugfix support approx. 6 months
>> later on ~*2022*-04-05.
>>
>>
>>
>> Damian
>>
>> (he/him)
>>
>>
>>
>>
>>
>> *From:* Shaw, Damian P. (WTIZ 53)
>> *Sent:* Friday, November 20, 2020 08:58
>> *To:* dev@airflow.apache.org
>> *Subject:* RE: Default/supported Python versions for Airlfow 2.0
>>
>>
>>
>> Hi Jarek,
>>
>>
>>
>> Took me a moment to understand what you meant by “latest supported”, you
>> mean the oldest version that is still receiving bugfix support?
>>
>>
>>
>> If I’m reading that correctly then for future versions of Python that
>> will mean that we will need to have the default Airflow moved when the
>> Python release is “only” 6 months old. E.g. On the current schedule Python
>> 3.10 will be released ~2021-10-05, and Python 3.9 will lose bugfix support
>> approx. 6 months later on ~2021-04-05.
>>
>>
>>
>> Of course the whole point of the 1 year release cycle is that the
>> transition to each Python release should be smaller and less troublesome,
>> so perhaps 6 months to migrate to Python 3.10 as the default is enough
>> time. But the comment from Halo earlier in the thread made me think that at
>> least some of the Airflow community might be a slightly slower crowd when
>> it comes to transitions.
>>
>>
>>
>> Damian P Shaw
>>
>> (he/him)
>>
>>
>>
>>
>>
>> *From:* Jarek Potiuk <Ja...@polidea.com>
>> *Sent:* Friday, November 20, 2020 04:10
>> *To:* dev@airflow.apache.org
>> *Subject:* Re: Default/supported Python versions for Airlfow 2.0
>>
>>
>>
>> I do not yet want to cast a formal vote on it, because maybe others have
>> other ideas, but let me formulate what I see as a good proposal now.
>>
>>
>>
>> The "default" in our case means mainly about the one that is used by
>> "non-approved yet PR" (we will continue running tests for all supported
>> versions in master and similarly as yesterday we wil be able to fix them
>> quickly). So I think we can be a bit more aggressive on the "default"
>> version.
>>
>>
>>
>> Just to show an example of what can happen - we had master
>> failing yesterday for Python 3.8 (fixed here -
>> https://github.com/apache/airflow/pull/12481) and I guess we might have
>> more cases like that.
>>
>> The proposal from Damian was to keep up with the schedule of Python
>> releases. I would like to - slightly - modify it to use the
>> latest supported version as "default" to prevent the kind of failures. "
>>
>>
>>
>> I think about such set of rules:
>>
>>
>>
>> * we use the "latest" supported version of Python as the default (3.8
>> today).
>>
>> * we officially support all Python releases till the end of their EOL
>>
>>
>>
>> We can always decide to support versions past EOL if we have a good
>> reason - but it would have to be justified and voted.
>>
>>
>>
>> How does it sound? Any other proposals ?
>>
>>
>>
>> Just to remind the Python schedule:
>>
>>
>>
>> Branch Schedule Status First release End-of-life
>> 3.9 PEP 596 bugfix 2020-10-05 TBD
>> 3.8 PEP 569 bugfix 2019-10-14 2024-10
>> 3.7 PEP 537 security 2018-06-27 2023-06-27
>> 3.6 PEP 494 security 2016-12-23 2021-12-23
>>
>>
>>
>> J.
>>
>>
>>
>>
>> --
>>
>> *Jarek Potiuk*
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>>
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
>>
>> On Tue, Nov 17, 2020 at 8:19 PM Leah Cole <co...@google.com.invalid>
>> wrote:
>>
>> I really like Damian's proposal. With my google devrel work we had been
>> doing something similar to what Jarek had done above - looking at PyPI
>> downloads and at what is most popular, but doing that makes it harder to
>> encourage folks to move towards the future. I like that Python provides
>> this consistent schedule - it will make it easier for us to set
>> expectations with our users, and will align with other Python-y things. :)
>> And, tbh, probably going to propose this in future meetings related to my
>> devrel work so thank you Damien!
>>
>>
>>
>> On Thu, Nov 12, 2020 at 6:46 AM Jarek Potiuk <Ja...@polidea.com>
>> wrote:
>>
>> I love the idea to have clear rules and tie it to the official schedule
>> of Python releases - at least as a target, because we might find some
>> issues that might prevent us from doing so.
>>
>>
>>
>> Also we should officially support all versions that are not yet reached
>> end-of-life.
>>
>>
>>
>> I like the proposed schedule and yearly cadence. I wonder if others have
>> similar thoughts. Such agreement/policy would require formal voting though
>> I think?
>>
>>
>>
>> WDYT everyone?
>>
>>
>>
>> J.
>>
>>
>>
>>
>>
>> On Thu, Nov 12, 2020 at 3:37 PM Shaw, Damian P. <
>> damian.shaw.2@credit-suisse.com> wrote:
>>
>> I just wanted to add that if people are not aware PEP 0602
>> <https://www.python.org/dev/peps/pep-0602> has been accepted and
>> implemented for Python 3.9. This means 3 things for the Python release
>> cycle:
>>
>> 1. A new version every 12 months
>>
>> 2. Each version receives 18 months of full support (bug fixes and
>> security fixes)
>>
>> 3. After full support has ended each version receives an
>> additional 42 months of security updates
>>
>>
>>
>> Going forward I think it makes sense to bump up the default version of
>> Python every 1 year in cadence with the Python release cycle. Assuming
>> people agreed the question would be how far behind should Airflow be from
>> the new release?
>>
>>
>>
>> Personally I feel like no more than 18 months is a good, in the new
>> Python release cadence that version of Python will no longer be receiving
>> bug fixes and therefore will be very stable, and 18 months is a good enough
>> time for any libraries and providers to be available (if they’re not
>> available after 18 months maybe they have given up support?)
>>
>>
>>
>> If we retroactively apply this to the previous releases of Python that
>> would put us at Python 3.7 default now and Python 3.8 default ~April 14,
>> 2021.
>>
>>
>>
>> My 2 cents,
>>
>> Damian
>>
>>
>>
>>
>>
>> *From:* Jarek Potiuk <Ja...@polidea.com>
>> *Sent:* Thursday, November 12, 2020 09:03
>> *To:* dev@airflow.apache.org
>> *Subject:* Re: Default/supported Python versions for Airlfow 2.0
>>
>>
>>
>> Should we make Python 3.7 default then and leave all others as-is ?
>>
>>
>>
>> J.
>>
>>
>>
>>
>>
>> On Thu, Nov 5, 2020 at 1:48 PM Kaxil Naik <ka...@gmail.com> wrote:
>>
>> We should definitely support Python 3.6 to make the Upgrades to Airflow
>> 2.0 a bit easier.
>>
>>
>>
>> As of yesterday, checks these stats from PyPI downloads:
>>
>>
>>
>> Py3.7: 12,578
>>
>> Py3.6: 9,806
>>
>> Py3.8: 1,815
>>
>>
>>
>>
>>
>>
>>
>> On Thu, Nov 5, 2020 at 11:40 AM Halo Ku <Ha...@mail.com> wrote:
>>
>>
>>
>> If I may point that Airflow is a wokrflow managment system and as
>> such the power of the tool is in direct extention to the levrage providers.
>> This should also be checked from how many of the providers are compatible
>> with 3.8 / 3.9
>>
>>
>>
>> *Sent:* Thursday, November 05, 2020 at 1:16 PM
>> *From:* "Ash Berlin-Taylor" <as...@apache.org>
>> *To:* "dev@airflow.apache.org" <de...@airflow.apache.org>
>> *Cc:* "dev@airflow.apache.org" <de...@airflow.apache.org>
>> *Subject:* Re: Default/supported Python versions for Airlfow 2.0
>>
>> Debian stable ships python 3.7(.3)
>>
>> CentOS 8 has two packages - python36 and python38
>>
>> Ubuntu 18.04 (LTS) has 3.6.5
>>
>> Ubuntu 20.04 (LTS) has 3.8.2
>>
>>
>>
>> (https://pkgs.org/search/?q=python3&on=files)
>>
>>
>>
>> RHEL is harder to find out about . RHEL8 has python 3.6 as python3, and
>> RHEL 8.2 has Py3.8 as a separate package
>> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/configuring_basic_system_settings/index#using-python3_configuring-basic-system-settings
>>
>>
>>
>> So for default version 3.8 or 3.9 gets my vote. I think the cost/burden
>> of supporting back to 3.6 is not very great, so we should continue to
>> support (and I guess test) that.
>>
>>
>>
>> -ash
>>
>>
>>
>> On Nov 5 2020, at 8:49 am, Jarek Potiuk <Ja...@polidea.com> wrote:
>>
>> Hello Everyone,
>>
>>
>>
>> I have a question. What do people think about default version of
>> Pyhon for Airflow 2.0 (and set of supported versions)?
>>
>>
>>
>> Currently, we have python 3.6 as default, but all the version up to 3.8
>> are officially supported and tested and PR for python 3.9 is in Draft:
>> https://github.com/apache/airflow/pull/11950
>>
>>
>>
>> This is the release schedule for python versions. We have a year till the
>> end of 3.6
>>
>>
>>
>> Branch Schedule Status First release End-of-life
>>
>> 3.9 PEP 596 bugfix 2020-10-05 TBD
>>
>> 3.8 PEP 569 bugfix 2019-10-14 2024-10
>>
>> 3.7 PEP 537 security 2018-06-27 2023-06-27
>>
>> 3.6 PEP 494 security 2016-12-23 2021-12-23
>>
>>
>>
>> WDYT?
>>
>>
>>
>> J.
>>
>>
>>
>>
>>
>> --
>>
>> *Jarek Potiuk*
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48%20660%20796%20129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
>>
>>
>> --
>>
>> *Jarek Potiuk*
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>>
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
>>
>>
>>
>>
>> ==============================================================================
>> Please access the attached hyperlink for an important electronic
>> communications disclaimer:
>> http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
>>
>> ==============================================================================
>>
>>
>>
>>
>> --
>>
>> *Jarek Potiuk*
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>>
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
>>
>>
>>
>>
>> --
>>
>>
>>
>> Leah Cole (she/her) | Developer Programs Engineer | coleleah@google.com | (925)
>> 257-2112
>>
>> *I'm working weird hours during this pandemic and am sometimes a bit
>> slower to respond to PRs/CLs than normal. Please feel free to send me a
>> gentle ping for a status update if my slowness is blocking you and I'll do
>> my best to give you an ETA. *
>>
>>
>>
>>
>>
>>
>> --
>>
>> *Jarek Potiuk*
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>>
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
>>
>>
>> ==============================================================================
>> Please access the attached hyperlink for an important electronic
>> communications disclaimer:
>> http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
>>
>> ==============================================================================
>>
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Default/supported Python versions for Airlfow 2.0
Posted by Jarek Potiuk <Ja...@polidea.com>.
Fair point that needs explanation. That was a bit of a "mental shortcut".
Latest supported means = "latest that we as Apache Airflow community
support".
For example, Python 3.9 is not supported yet by Airflow and there are some
good reasons for that - namely our transitive dependencies. NumPy in
version 1.17 does not yet have a binary Python 3.9 binary PIP package.
Building NumPy from sources is not something we would like to mess around
with. You can see more here: https://github.com/apache/airflow/pull/11950
We have no strong need to support 3.9 yet and since it might be solved by
simply upgrading dependencies (which we plan anyway:
https://github.com/apache/airflow/issues/10854), support for 3.9 might be
deferred, even if it is officially released.
There will be other cases like that in the future, so there will likely be
a considerable delay between when a new version is released, and when we
start supporting it in Airflow.
I think this is worth explaining what "default" version means for us, there
are two things "default" is used for:
* optimization of PR job usage. Simple PRs will only run limited version
of tests - only for the default python version (3.6 currently) and only 1
version of each of the DBs. We can always run full tests (by applying "full
tests needed" label) which will run tests for all "supported versions"
(currently 3.6, 3.7, 3.8) and all tests are run after merge to master and
in scheduled builds. So we notice if there's something wrong
* default python version is used for "apache/airflow:master" production
image published. So really "apache/airflow:master" is the same as
"apache/airflow:master-python3.6".
And now, when I wrote this, I think I want to change my proposal. I believe
there will be far more opportunities for "test" failures if we switch to
"latest" version There are new things added in newer versions of python
that we might be tempted to use, and they will fail in earlier version) -
so our current approach where default is "oldest supported version" is
better.
So my new proposal is:
* we finish support for python versions when they reach EOL (For python 3.6
it means that we will remove it from being supported on 23.12.2021).
* the "oldest" supported version of Python as the default (3.6 today and it
will remain the default until 23.12.2021. On 23.12.2021 3.7 becomes the
default).
* we support newer versions of Python some time after first release, as
soon as we manage to make it work as part of our CI pipeline.
We can always decide to support versions past EOL if we have a good reason
- but it would have to be justified and voted.
WDYT?
Again Python schedule:
Branch Schedule Status First release End-of-life
3.9 PEP 596 bugfix 2020-10-05 TBD
3.8 PEP 569 bugfix 2019-10-14 2024-10
3.7 PEP 537 security 2018-06-27 2023-06-27
3.6 PEP 494 security 2016-12-23 2021-12-2
J.
On Fri, Nov 20, 2020 at 3:27 PM Shaw, Damian P. <
damian.shaw.2@credit-suisse.com> wrote:
> Typo,
>
>
>
> Should of read: Python 3.9 will lose bugfix support approx. 6 months
> later on ~*2022*-04-05.
>
>
>
> Damian
>
> (he/him)
>
>
>
>
>
> *From:* Shaw, Damian P. (WTIZ 53)
> *Sent:* Friday, November 20, 2020 08:58
> *To:* dev@airflow.apache.org
> *Subject:* RE: Default/supported Python versions for Airlfow 2.0
>
>
>
> Hi Jarek,
>
>
>
> Took me a moment to understand what you meant by “latest supported”, you
> mean the oldest version that is still receiving bugfix support?
>
>
>
> If I’m reading that correctly then for future versions of Python that will
> mean that we will need to have the default Airflow moved when the Python
> release is “only” 6 months old. E.g. On the current schedule Python 3.10
> will be released ~2021-10-05, and Python 3.9 will lose bugfix support
> approx. 6 months later on ~2021-04-05.
>
>
>
> Of course the whole point of the 1 year release cycle is that the
> transition to each Python release should be smaller and less troublesome,
> so perhaps 6 months to migrate to Python 3.10 as the default is enough
> time. But the comment from Halo earlier in the thread made me think that at
> least some of the Airflow community might be a slightly slower crowd when
> it comes to transitions.
>
>
>
> Damian P Shaw
>
> (he/him)
>
>
>
>
>
> *From:* Jarek Potiuk <Ja...@polidea.com>
> *Sent:* Friday, November 20, 2020 04:10
> *To:* dev@airflow.apache.org
> *Subject:* Re: Default/supported Python versions for Airlfow 2.0
>
>
>
> I do not yet want to cast a formal vote on it, because maybe others have
> other ideas, but let me formulate what I see as a good proposal now.
>
>
>
> The "default" in our case means mainly about the one that is used by
> "non-approved yet PR" (we will continue running tests for all supported
> versions in master and similarly as yesterday we wil be able to fix them
> quickly). So I think we can be a bit more aggressive on the "default"
> version.
>
>
>
> Just to show an example of what can happen - we had master
> failing yesterday for Python 3.8 (fixed here -
> https://github.com/apache/airflow/pull/12481) and I guess we might have
> more cases like that.
>
> The proposal from Damian was to keep up with the schedule of Python
> releases. I would like to - slightly - modify it to use the
> latest supported version as "default" to prevent the kind of failures. "
>
>
>
> I think about such set of rules:
>
>
>
> * we use the "latest" supported version of Python as the default (3.8
> today).
>
> * we officially support all Python releases till the end of their EOL
>
>
>
> We can always decide to support versions past EOL if we have a good reason
> - but it would have to be justified and voted.
>
>
>
> How does it sound? Any other proposals ?
>
>
>
> Just to remind the Python schedule:
>
>
>
> Branch Schedule Status First release End-of-life
> 3.9 PEP 596 bugfix 2020-10-05 TBD
> 3.8 PEP 569 bugfix 2019-10-14 2024-10
> 3.7 PEP 537 security 2018-06-27 2023-06-27
> 3.6 PEP 494 security 2016-12-23 2021-12-23
>
>
>
> J.
>
>
>
>
> --
>
> *Jarek Potiuk*
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
>
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
>
> On Tue, Nov 17, 2020 at 8:19 PM Leah Cole <co...@google.com.invalid>
> wrote:
>
> I really like Damian's proposal. With my google devrel work we had been
> doing something similar to what Jarek had done above - looking at PyPI
> downloads and at what is most popular, but doing that makes it harder to
> encourage folks to move towards the future. I like that Python provides
> this consistent schedule - it will make it easier for us to set
> expectations with our users, and will align with other Python-y things. :)
> And, tbh, probably going to propose this in future meetings related to my
> devrel work so thank you Damien!
>
>
>
> On Thu, Nov 12, 2020 at 6:46 AM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
> I love the idea to have clear rules and tie it to the official schedule of
> Python releases - at least as a target, because we might find some issues
> that might prevent us from doing so.
>
>
>
> Also we should officially support all versions that are not yet reached
> end-of-life.
>
>
>
> I like the proposed schedule and yearly cadence. I wonder if others have
> similar thoughts. Such agreement/policy would require formal voting though
> I think?
>
>
>
> WDYT everyone?
>
>
>
> J.
>
>
>
>
>
> On Thu, Nov 12, 2020 at 3:37 PM Shaw, Damian P. <
> damian.shaw.2@credit-suisse.com> wrote:
>
> I just wanted to add that if people are not aware PEP 0602
> <https://www.python.org/dev/peps/pep-0602> has been accepted and
> implemented for Python 3.9. This means 3 things for the Python release
> cycle:
>
> 1. A new version every 12 months
>
> 2. Each version receives 18 months of full support (bug fixes and
> security fixes)
>
> 3. After full support has ended each version receives an additional
> 42 months of security updates
>
>
>
> Going forward I think it makes sense to bump up the default version of
> Python every 1 year in cadence with the Python release cycle. Assuming
> people agreed the question would be how far behind should Airflow be from
> the new release?
>
>
>
> Personally I feel like no more than 18 months is a good, in the new Python
> release cadence that version of Python will no longer be receiving bug
> fixes and therefore will be very stable, and 18 months is a good enough
> time for any libraries and providers to be available (if they’re not
> available after 18 months maybe they have given up support?)
>
>
>
> If we retroactively apply this to the previous releases of Python that
> would put us at Python 3.7 default now and Python 3.8 default ~April 14,
> 2021.
>
>
>
> My 2 cents,
>
> Damian
>
>
>
>
>
> *From:* Jarek Potiuk <Ja...@polidea.com>
> *Sent:* Thursday, November 12, 2020 09:03
> *To:* dev@airflow.apache.org
> *Subject:* Re: Default/supported Python versions for Airlfow 2.0
>
>
>
> Should we make Python 3.7 default then and leave all others as-is ?
>
>
>
> J.
>
>
>
>
>
> On Thu, Nov 5, 2020 at 1:48 PM Kaxil Naik <ka...@gmail.com> wrote:
>
> We should definitely support Python 3.6 to make the Upgrades to Airflow
> 2.0 a bit easier.
>
>
>
> As of yesterday, checks these stats from PyPI downloads:
>
>
>
> Py3.7: 12,578
>
> Py3.6: 9,806
>
> Py3.8: 1,815
>
>
>
>
>
>
>
> On Thu, Nov 5, 2020 at 11:40 AM Halo Ku <Ha...@mail.com> wrote:
>
>
>
> If I may point that Airflow is a wokrflow managment system and as such the
> power of the tool is in direct extention to the levrage providers.
> This should also be checked from how many of the providers are compatible
> with 3.8 / 3.9
>
>
>
> *Sent:* Thursday, November 05, 2020 at 1:16 PM
> *From:* "Ash Berlin-Taylor" <as...@apache.org>
> *To:* "dev@airflow.apache.org" <de...@airflow.apache.org>
> *Cc:* "dev@airflow.apache.org" <de...@airflow.apache.org>
> *Subject:* Re: Default/supported Python versions for Airlfow 2.0
>
> Debian stable ships python 3.7(.3)
>
> CentOS 8 has two packages - python36 and python38
>
> Ubuntu 18.04 (LTS) has 3.6.5
>
> Ubuntu 20.04 (LTS) has 3.8.2
>
>
>
> (https://pkgs.org/search/?q=python3&on=files)
>
>
>
> RHEL is harder to find out about . RHEL8 has python 3.6 as python3, and
> RHEL 8.2 has Py3.8 as a separate package
> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/configuring_basic_system_settings/index#using-python3_configuring-basic-system-settings
>
>
>
> So for default version 3.8 or 3.9 gets my vote. I think the cost/burden of
> supporting back to 3.6 is not very great, so we should continue to support
> (and I guess test) that.
>
>
>
> -ash
>
>
>
> On Nov 5 2020, at 8:49 am, Jarek Potiuk <Ja...@polidea.com> wrote:
>
> Hello Everyone,
>
>
>
> I have a question. What do people think about default version of Pyhon for
> Airflow 2.0 (and set of supported versions)?
>
>
>
> Currently, we have python 3.6 as default, but all the version up to 3.8
> are officially supported and tested and PR for python 3.9 is in Draft:
> https://github.com/apache/airflow/pull/11950
>
>
>
> This is the release schedule for python versions. We have a year till the
> end of 3.6
>
>
>
> Branch Schedule Status First release End-of-life
>
> 3.9 PEP 596 bugfix 2020-10-05 TBD
>
> 3.8 PEP 569 bugfix 2019-10-14 2024-10
>
> 3.7 PEP 537 security 2018-06-27 2023-06-27
>
> 3.6 PEP 494 security 2016-12-23 2021-12-23
>
>
>
> WDYT?
>
>
>
> J.
>
>
>
>
>
> --
>
> *Jarek Potiuk*
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48%20660%20796%20129>
> [image: Polidea] <https://www.polidea.com/>
>
>
>
>
> --
>
> *Jarek Potiuk*
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
>
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
>
>
>
>
> ==============================================================================
> Please access the attached hyperlink for an important electronic
> communications disclaimer:
> http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
>
> ==============================================================================
>
>
>
>
> --
>
> *Jarek Potiuk*
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
>
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
>
>
>
>
> --
>
>
>
> Leah Cole (she/her) | Developer Programs Engineer | coleleah@google.com | (925)
> 257-2112
>
> *I'm working weird hours during this pandemic and am sometimes a bit
> slower to respond to PRs/CLs than normal. Please feel free to send me a
> gentle ping for a status update if my slowness is blocking you and I'll do
> my best to give you an ETA. *
>
>
>
>
>
>
> --
>
> *Jarek Potiuk*
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
>
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
>
>
> ==============================================================================
> Please access the attached hyperlink for an important electronic
> communications disclaimer:
> http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
>
> ==============================================================================
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
RE: Default/supported Python versions for Airlfow 2.0
Posted by "Shaw, Damian P. " <da...@credit-suisse.com>.
Typo,
Should of read: Python 3.9 will lose bugfix support approx. 6 months later on ~2022-04-05.
Damian
(he/him)
From: Shaw, Damian P. (WTIZ 53)
Sent: Friday, November 20, 2020 08:58
To: dev@airflow.apache.org
Subject: RE: Default/supported Python versions for Airlfow 2.0
Hi Jarek,
Took me a moment to understand what you meant by “latest supported”, you mean the oldest version that is still receiving bugfix support?
If I’m reading that correctly then for future versions of Python that will mean that we will need to have the default Airflow moved when the Python release is “only” 6 months old. E.g. On the current schedule Python 3.10 will be released ~2021-10-05, and Python 3.9 will lose bugfix support approx. 6 months later on ~2021-04-05.
Of course the whole point of the 1 year release cycle is that the transition to each Python release should be smaller and less troublesome, so perhaps 6 months to migrate to Python 3.10 as the default is enough time. But the comment from Halo earlier in the thread made me think that at least some of the Airflow community might be a slightly slower crowd when it comes to transitions.
Damian P Shaw
(he/him)
From: Jarek Potiuk <Ja...@polidea.com>>
Sent: Friday, November 20, 2020 04:10
To: dev@airflow.apache.org<ma...@airflow.apache.org>
Subject: Re: Default/supported Python versions for Airlfow 2.0
I do not yet want to cast a formal vote on it, because maybe others have other ideas, but let me formulate what I see as a good proposal now.
The "default" in our case means mainly about the one that is used by "non-approved yet PR" (we will continue running tests for all supported versions in master and similarly as yesterday we wil be able to fix them quickly). So I think we can be a bit more aggressive on the "default" version.
Just to show an example of what can happen - we had master failing yesterday for Python 3.8 (fixed here - https://github.com/apache/airflow/pull/12481) and I guess we might have more cases like that.
The proposal from Damian was to keep up with the schedule of Python releases. I would like to - slightly - modify it to use the latest supported version as "default" to prevent the kind of failures. "
I think about such set of rules:
* we use the "latest" supported version of Python as the default (3.8 today).
* we officially support all Python releases till the end of their EOL
We can always decide to support versions past EOL if we have a good reason - but it would have to be justified and voted.
How does it sound? Any other proposals ?
Just to remind the Python schedule:
Branch Schedule Status First release End-of-life
3.9 PEP 596 bugfix 2020-10-05 TBD
3.8 PEP 569 bugfix 2019-10-14 2024-10
3.7 PEP 537 security 2018-06-27 2023-06-27
3.6 PEP 494 security 2016-12-23 2021-12-23
J.
--
[https://s3.eu-central-1.amazonaws.com/corgi-mail/23-05-2019/jarek.potiuk/jarpot.jpg]
Jarek Potiuk
Polidea<https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129<tel:+48660796129>
[Polidea]<https://www.polidea.com/>
On Tue, Nov 17, 2020 at 8:19 PM Leah Cole <co...@google.com.invalid>> wrote:
I really like Damian's proposal. With my google devrel work we had been doing something similar to what Jarek had done above - looking at PyPI downloads and at what is most popular, but doing that makes it harder to encourage folks to move towards the future. I like that Python provides this consistent schedule - it will make it easier for us to set expectations with our users, and will align with other Python-y things. :) And, tbh, probably going to propose this in future meetings related to my devrel work so thank you Damien!
On Thu, Nov 12, 2020 at 6:46 AM Jarek Potiuk <Ja...@polidea.com>> wrote:
I love the idea to have clear rules and tie it to the official schedule of Python releases - at least as a target, because we might find some issues that might prevent us from doing so.
Also we should officially support all versions that are not yet reached end-of-life.
I like the proposed schedule and yearly cadence. I wonder if others have similar thoughts. Such agreement/policy would require formal voting though I think?
WDYT everyone?
J.
On Thu, Nov 12, 2020 at 3:37 PM Shaw, Damian P. <da...@credit-suisse.com>> wrote:
I just wanted to add that if people are not aware PEP 0602<https://www.python.org/dev/peps/pep-0602> has been accepted and implemented for Python 3.9. This means 3 things for the Python release cycle:
1. A new version every 12 months
2. Each version receives 18 months of full support (bug fixes and security fixes)
3. After full support has ended each version receives an additional 42 months of security updates
Going forward I think it makes sense to bump up the default version of Python every 1 year in cadence with the Python release cycle. Assuming people agreed the question would be how far behind should Airflow be from the new release?
Personally I feel like no more than 18 months is a good, in the new Python release cadence that version of Python will no longer be receiving bug fixes and therefore will be very stable, and 18 months is a good enough time for any libraries and providers to be available (if they’re not available after 18 months maybe they have given up support?)
If we retroactively apply this to the previous releases of Python that would put us at Python 3.7 default now and Python 3.8 default ~April 14, 2021.
My 2 cents,
Damian
From: Jarek Potiuk <Ja...@polidea.com>>
Sent: Thursday, November 12, 2020 09:03
To: dev@airflow.apache.org<ma...@airflow.apache.org>
Subject: Re: Default/supported Python versions for Airlfow 2.0
Should we make Python 3.7 default then and leave all others as-is ?
J.
On Thu, Nov 5, 2020 at 1:48 PM Kaxil Naik <ka...@gmail.com>> wrote:
We should definitely support Python 3.6 to make the Upgrades to Airflow 2.0 a bit easier.
As of yesterday, checks these stats from PyPI downloads:
Py3.7: 12,578
Py3.6: 9,806
Py3.8: 1,815
On Thu, Nov 5, 2020 at 11:40 AM Halo Ku <Ha...@mail.com>> wrote:
If I may point that Airflow is a wokrflow managment system and as such the power of the tool is in direct extention to the levrage providers.
This should also be checked from how many of the providers are compatible with 3.8 / 3.9
Sent: Thursday, November 05, 2020 at 1:16 PM
From: "Ash Berlin-Taylor" <as...@apache.org>>
To: "dev@airflow.apache.org<ma...@airflow.apache.org>" <de...@airflow.apache.org>>
Cc: "dev@airflow.apache.org<ma...@airflow.apache.org>" <de...@airflow.apache.org>>
Subject: Re: Default/supported Python versions for Airlfow 2.0
Debian stable ships python 3.7(.3)
CentOS 8 has two packages - python36 and python38
Ubuntu 18.04 (LTS) has 3.6.5
Ubuntu 20.04 (LTS) has 3.8.2
(https://pkgs.org/search/?q=python3&on=files)
RHEL is harder to find out about . RHEL8 has python 3.6 as python3, and RHEL 8.2 has Py3.8 as a separate package https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/configuring_basic_system_settings/index#using-python3_configuring-basic-system-settings
So for default version 3.8 or 3.9 gets my vote. I think the cost/burden of supporting back to 3.6 is not very great, so we should continue to support (and I guess test) that.
-ash
On Nov 5 2020, at 8:49 am, Jarek Potiuk <Ja...@polidea.com>> wrote:
Hello Everyone,
I have a question. What do people think about default version of Pyhon for Airflow 2.0 (and set of supported versions)?
Currently, we have python 3.6 as default, but all the version up to 3.8 are officially supported and tested and PR for python 3.9 is in Draft: https://github.com/apache/airflow/pull/11950
This is the release schedule for python versions. We have a year till the end of 3.6
Branch Schedule Status First release End-of-life
3.9 PEP 596 bugfix 2020-10-05 TBD
3.8 PEP 569 bugfix 2019-10-14 2024-10
3.7 PEP 537 security 2018-06-27 2023-06-27
3.6 PEP 494 security 2016-12-23 2021-12-23
WDYT?
J.
--
[https://s3.eu-central-1.amazonaws.com/corgi-mail/23-05-2019/jarek.potiuk/jarpot.jpg]
Jarek Potiuk
Polidea<https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129<tel:+48%20660%20796%20129>
[Polidea]<https://www.polidea.com/>
--
[https://s3.eu-central-1.amazonaws.com/corgi-mail/23-05-2019/jarek.potiuk/jarpot.jpg]
Jarek Potiuk
Polidea<https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129<tel:+48660796129>
[Polidea]<https://www.polidea.com/>
==============================================================================
Please access the attached hyperlink for an important electronic communications disclaimer:
http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
==============================================================================
--
[https://s3.eu-central-1.amazonaws.com/corgi-mail/23-05-2019/jarek.potiuk/jarpot.jpg]
Jarek Potiuk
Polidea<https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129<tel:+48660796129>
[Polidea]<https://www.polidea.com/>
--
Leah Cole (she/her) | Developer Programs Engineer | coleleah@google.com<ma...@google.com> | (925) 257-2112
I'm working weird hours during this pandemic and am sometimes a bit slower to respond to PRs/CLs than normal. Please feel free to send me a gentle ping for a status update if my slowness is blocking you and I'll do my best to give you an ETA.
--
[https://s3.eu-central-1.amazonaws.com/corgi-mail/23-05-2019/jarek.potiuk/jarpot.jpg]
Jarek Potiuk
Polidea<https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129<tel:+48660796129>
[Polidea]<https://www.polidea.com/>
===============================================================================
Please access the attached hyperlink for an important electronic communications disclaimer:
http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
===============================================================================
RE: Default/supported Python versions for Airlfow 2.0
Posted by "Shaw, Damian P. " <da...@credit-suisse.com>.
Hi Jarek,
Took me a moment to understand what you meant by “latest supported”, you mean the oldest version that is still receiving bugfix support?
If I’m reading that correctly then for future versions of Python that will mean that we will need to have the default Airflow moved when the Python release is “only” 6 months old. E.g. On the current schedule Python 3.10 will be released ~2021-10-05, and Python 3.9 will lose bugfix support approx. 6 months later on ~2021-04-05.
Of course the whole point of the 1 year release cycle is that the transition to each Python release should be smaller and less troublesome, so perhaps 6 months to migrate to Python 3.10 as the default is enough time. But the comment from Halo earlier in the thread made me think that at least some of the Airflow community might be a slightly slower crowd when it comes to transitions.
Damian P Shaw
(he/him)
From: Jarek Potiuk <Ja...@polidea.com>
Sent: Friday, November 20, 2020 04:10
To: dev@airflow.apache.org
Subject: Re: Default/supported Python versions for Airlfow 2.0
I do not yet want to cast a formal vote on it, because maybe others have other ideas, but let me formulate what I see as a good proposal now.
The "default" in our case means mainly about the one that is used by "non-approved yet PR" (we will continue running tests for all supported versions in master and similarly as yesterday we wil be able to fix them quickly). So I think we can be a bit more aggressive on the "default" version.
Just to show an example of what can happen - we had master failing yesterday for Python 3.8 (fixed here - https://github.com/apache/airflow/pull/12481) and I guess we might have more cases like that.
The proposal from Damian was to keep up with the schedule of Python releases. I would like to - slightly - modify it to use the latest supported version as "default" to prevent the kind of failures. "
I think about such set of rules:
* we use the "latest" supported version of Python as the default (3.8 today).
* we officially support all Python releases till the end of their EOL
We can always decide to support versions past EOL if we have a good reason - but it would have to be justified and voted.
How does it sound? Any other proposals ?
Just to remind the Python schedule:
Branch Schedule Status First release End-of-life
3.9 PEP 596 bugfix 2020-10-05 TBD
3.8 PEP 569 bugfix 2019-10-14 2024-10
3.7 PEP 537 security 2018-06-27 2023-06-27
3.6 PEP 494 security 2016-12-23 2021-12-23
J.
--
[https://s3.eu-central-1.amazonaws.com/corgi-mail/23-05-2019/jarek.potiuk/jarpot.jpg]
Jarek Potiuk
Polidea<https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129<tel:+48660796129>
[Polidea]<https://www.polidea.com/>
On Tue, Nov 17, 2020 at 8:19 PM Leah Cole <co...@google.com.invalid>> wrote:
I really like Damian's proposal. With my google devrel work we had been doing something similar to what Jarek had done above - looking at PyPI downloads and at what is most popular, but doing that makes it harder to encourage folks to move towards the future. I like that Python provides this consistent schedule - it will make it easier for us to set expectations with our users, and will align with other Python-y things. :) And, tbh, probably going to propose this in future meetings related to my devrel work so thank you Damien!
On Thu, Nov 12, 2020 at 6:46 AM Jarek Potiuk <Ja...@polidea.com>> wrote:
I love the idea to have clear rules and tie it to the official schedule of Python releases - at least as a target, because we might find some issues that might prevent us from doing so.
Also we should officially support all versions that are not yet reached end-of-life.
I like the proposed schedule and yearly cadence. I wonder if others have similar thoughts. Such agreement/policy would require formal voting though I think?
WDYT everyone?
J.
On Thu, Nov 12, 2020 at 3:37 PM Shaw, Damian P. <da...@credit-suisse.com>> wrote:
I just wanted to add that if people are not aware PEP 0602<https://www.python.org/dev/peps/pep-0602> has been accepted and implemented for Python 3.9. This means 3 things for the Python release cycle:
1. A new version every 12 months
2. Each version receives 18 months of full support (bug fixes and security fixes)
3. After full support has ended each version receives an additional 42 months of security updates
Going forward I think it makes sense to bump up the default version of Python every 1 year in cadence with the Python release cycle. Assuming people agreed the question would be how far behind should Airflow be from the new release?
Personally I feel like no more than 18 months is a good, in the new Python release cadence that version of Python will no longer be receiving bug fixes and therefore will be very stable, and 18 months is a good enough time for any libraries and providers to be available (if they’re not available after 18 months maybe they have given up support?)
If we retroactively apply this to the previous releases of Python that would put us at Python 3.7 default now and Python 3.8 default ~April 14, 2021.
My 2 cents,
Damian
From: Jarek Potiuk <Ja...@polidea.com>>
Sent: Thursday, November 12, 2020 09:03
To: dev@airflow.apache.org<ma...@airflow.apache.org>
Subject: Re: Default/supported Python versions for Airlfow 2.0
Should we make Python 3.7 default then and leave all others as-is ?
J.
On Thu, Nov 5, 2020 at 1:48 PM Kaxil Naik <ka...@gmail.com>> wrote:
We should definitely support Python 3.6 to make the Upgrades to Airflow 2.0 a bit easier.
As of yesterday, checks these stats from PyPI downloads:
Py3.7: 12,578
Py3.6: 9,806
Py3.8: 1,815
On Thu, Nov 5, 2020 at 11:40 AM Halo Ku <Ha...@mail.com>> wrote:
If I may point that Airflow is a wokrflow managment system and as such the power of the tool is in direct extention to the levrage providers.
This should also be checked from how many of the providers are compatible with 3.8 / 3.9
Sent: Thursday, November 05, 2020 at 1:16 PM
From: "Ash Berlin-Taylor" <as...@apache.org>>
To: "dev@airflow.apache.org<ma...@airflow.apache.org>" <de...@airflow.apache.org>>
Cc: "dev@airflow.apache.org<ma...@airflow.apache.org>" <de...@airflow.apache.org>>
Subject: Re: Default/supported Python versions for Airlfow 2.0
Debian stable ships python 3.7(.3)
CentOS 8 has two packages - python36 and python38
Ubuntu 18.04 (LTS) has 3.6.5
Ubuntu 20.04 (LTS) has 3.8.2
(https://pkgs.org/search/?q=python3&on=files)
RHEL is harder to find out about . RHEL8 has python 3.6 as python3, and RHEL 8.2 has Py3.8 as a separate package https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/configuring_basic_system_settings/index#using-python3_configuring-basic-system-settings
So for default version 3.8 or 3.9 gets my vote. I think the cost/burden of supporting back to 3.6 is not very great, so we should continue to support (and I guess test) that.
-ash
On Nov 5 2020, at 8:49 am, Jarek Potiuk <Ja...@polidea.com>> wrote:
Hello Everyone,
I have a question. What do people think about default version of Pyhon for Airflow 2.0 (and set of supported versions)?
Currently, we have python 3.6 as default, but all the version up to 3.8 are officially supported and tested and PR for python 3.9 is in Draft: https://github.com/apache/airflow/pull/11950
This is the release schedule for python versions. We have a year till the end of 3.6
Branch Schedule Status First release End-of-life
3.9 PEP 596 bugfix 2020-10-05 TBD
3.8 PEP 569 bugfix 2019-10-14 2024-10
3.7 PEP 537 security 2018-06-27 2023-06-27
3.6 PEP 494 security 2016-12-23 2021-12-23
WDYT?
J.
--
[https://s3.eu-central-1.amazonaws.com/corgi-mail/23-05-2019/jarek.potiuk/jarpot.jpg]
Jarek Potiuk
Polidea<https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129<tel:+48%20660%20796%20129>
[Polidea]<https://www.polidea.com/>
--
[https://s3.eu-central-1.amazonaws.com/corgi-mail/23-05-2019/jarek.potiuk/jarpot.jpg]
Jarek Potiuk
Polidea<https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129<tel:+48660796129>
[Polidea]<https://www.polidea.com/>
==============================================================================
Please access the attached hyperlink for an important electronic communications disclaimer:
http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
==============================================================================
--
[https://s3.eu-central-1.amazonaws.com/corgi-mail/23-05-2019/jarek.potiuk/jarpot.jpg]
Jarek Potiuk
Polidea<https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129<tel:+48660796129>
[Polidea]<https://www.polidea.com/>
--
Leah Cole (she/her) | Developer Programs Engineer | coleleah@google.com<ma...@google.com> | (925) 257-2112
I'm working weird hours during this pandemic and am sometimes a bit slower to respond to PRs/CLs than normal. Please feel free to send me a gentle ping for a status update if my slowness is blocking you and I'll do my best to give you an ETA.
--
[https://s3.eu-central-1.amazonaws.com/corgi-mail/23-05-2019/jarek.potiuk/jarpot.jpg]
Jarek Potiuk
Polidea<https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129<tel:+48660796129>
[Polidea]<https://www.polidea.com/>
===============================================================================
Please access the attached hyperlink for an important electronic communications disclaimer:
http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
===============================================================================
Re: Default/supported Python versions for Airlfow 2.0
Posted by Jarek Potiuk <Ja...@polidea.com>.
I do not yet want to cast a formal vote on it, because maybe others have
other ideas, but let me formulate what I see as a good proposal now.
The "default" in our case means mainly about the one that is used by
"non-approved yet PR" (we will continue running tests for all supported
versions in master and similarly as yesterday we wil be able to fix them
quickly). So I think we can be a bit more aggressive on the "default"
version.
Just to show an example of what can happen - we had master
failing yesterday for Python 3.8 (fixed here -
https://github.com/apache/airflow/pull/12481) and I guess we might have
more cases like that.
The proposal from Damian was to keep up with the schedule of Python
releases. I would like to - slightly - modify it to use the
latest supported version as "default" to prevent the kind of failures. "
I think about such set of rules:
* we use the "latest" supported version of Python as the default (3.8
today).
* we officially support all Python releases till the end of their EOL
We can always decide to support versions past EOL if we have a good reason
- but it would have to be justified and voted.
How does it sound? Any other proposals ?
Just to remind the Python schedule:
Branch Schedule Status First release End-of-life
3.9 PEP 596 bugfix 2020-10-05 TBD
3.8 PEP 569 bugfix 2019-10-14 2024-10
3.7 PEP 537 security 2018-06-27 2023-06-27
3.6 PEP 494 security 2016-12-23 2021-12-23
J.
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
On Tue, Nov 17, 2020 at 8:19 PM Leah Cole <co...@google.com.invalid>
wrote:
> I really like Damian's proposal. With my google devrel work we had been
> doing something similar to what Jarek had done above - looking at PyPI
> downloads and at what is most popular, but doing that makes it harder to
> encourage folks to move towards the future. I like that Python provides
> this consistent schedule - it will make it easier for us to set
> expectations with our users, and will align with other Python-y things. :)
> And, tbh, probably going to propose this in future meetings related to my
> devrel work so thank you Damien!
>
> On Thu, Nov 12, 2020 at 6:46 AM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
>> I love the idea to have clear rules and tie it to the official schedule
>> of Python releases - at least as a target, because we might find some
>> issues that might prevent us from doing so.
>>
>> Also we should officially support all versions that are not yet reached
>> end-of-life.
>>
>> I like the proposed schedule and yearly cadence. I wonder if others have
>> similar thoughts. Such agreement/policy would require formal voting though
>> I think?
>>
>> WDYT everyone?
>>
>> J.
>>
>>
>> On Thu, Nov 12, 2020 at 3:37 PM Shaw, Damian P. <
>> damian.shaw.2@credit-suisse.com> wrote:
>>
>>> I just wanted to add that if people are not aware PEP 0602
>>> <https://www.python.org/dev/peps/pep-0602> has been accepted and
>>> implemented for Python 3.9. This means 3 things for the Python release
>>> cycle:
>>>
>>> 1. A new version every 12 months
>>>
>>> 2. Each version receives 18 months of full support (bug fixes and
>>> security fixes)
>>>
>>> 3. After full support has ended each version receives an
>>> additional 42 months of security updates
>>>
>>>
>>>
>>> Going forward I think it makes sense to bump up the default version of
>>> Python every 1 year in cadence with the Python release cycle. Assuming
>>> people agreed the question would be how far behind should Airflow be from
>>> the new release?
>>>
>>>
>>>
>>> Personally I feel like no more than 18 months is a good, in the new
>>> Python release cadence that version of Python will no longer be receiving
>>> bug fixes and therefore will be very stable, and 18 months is a good enough
>>> time for any libraries and providers to be available (if they’re not
>>> available after 18 months maybe they have given up support?)
>>>
>>>
>>>
>>> If we retroactively apply this to the previous releases of Python that
>>> would put us at Python 3.7 default now and Python 3.8 default ~April 14,
>>> 2021.
>>>
>>>
>>>
>>> My 2 cents,
>>>
>>> Damian
>>>
>>>
>>>
>>>
>>>
>>> *From:* Jarek Potiuk <Ja...@polidea.com>
>>> *Sent:* Thursday, November 12, 2020 09:03
>>> *To:* dev@airflow.apache.org
>>> *Subject:* Re: Default/supported Python versions for Airlfow 2.0
>>>
>>>
>>>
>>> Should we make Python 3.7 default then and leave all others as-is ?
>>>
>>>
>>>
>>> J.
>>>
>>>
>>>
>>>
>>>
>>> On Thu, Nov 5, 2020 at 1:48 PM Kaxil Naik <ka...@gmail.com> wrote:
>>>
>>> We should definitely support Python 3.6 to make the Upgrades to Airflow
>>> 2.0 a bit easier.
>>>
>>>
>>>
>>> As of yesterday, checks these stats from PyPI downloads:
>>>
>>>
>>>
>>> Py3.7: 12,578
>>>
>>> Py3.6: 9,806
>>>
>>> Py3.8: 1,815
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Thu, Nov 5, 2020 at 11:40 AM Halo Ku <Ha...@mail.com> wrote:
>>>
>>>
>>>
>>> If I may point that Airflow is a wokrflow managment system and as
>>> such the power of the tool is in direct extention to the levrage providers.
>>> This should also be checked from how many of the providers are
>>> compatible with 3.8 / 3.9
>>>
>>>
>>>
>>> *Sent:* Thursday, November 05, 2020 at 1:16 PM
>>> *From:* "Ash Berlin-Taylor" <as...@apache.org>
>>> *To:* "dev@airflow.apache.org" <de...@airflow.apache.org>
>>> *Cc:* "dev@airflow.apache.org" <de...@airflow.apache.org>
>>> *Subject:* Re: Default/supported Python versions for Airlfow 2.0
>>>
>>> Debian stable ships python 3.7(.3)
>>>
>>> CentOS 8 has two packages - python36 and python38
>>>
>>> Ubuntu 18.04 (LTS) has 3.6.5
>>>
>>> Ubuntu 20.04 (LTS) has 3.8.2
>>>
>>>
>>>
>>> (https://pkgs.org/search/?q=python3&on=files)
>>>
>>>
>>>
>>> RHEL is harder to find out about . RHEL8 has python 3.6 as python3, and
>>> RHEL 8.2 has Py3.8 as a separate package
>>> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/configuring_basic_system_settings/index#using-python3_configuring-basic-system-settings
>>>
>>>
>>>
>>> So for default version 3.8 or 3.9 gets my vote. I think the cost/burden
>>> of supporting back to 3.6 is not very great, so we should continue to
>>> support (and I guess test) that.
>>>
>>>
>>>
>>> -ash
>>>
>>>
>>>
>>> On Nov 5 2020, at 8:49 am, Jarek Potiuk <Ja...@polidea.com>
>>> wrote:
>>>
>>> Hello Everyone,
>>>
>>>
>>>
>>> I have a question. What do people think about default version of
>>> Pyhon for Airflow 2.0 (and set of supported versions)?
>>>
>>>
>>>
>>> Currently, we have python 3.6 as default, but all the version up to 3.8
>>> are officially supported and tested and PR for python 3.9 is in Draft:
>>> https://github.com/apache/airflow/pull/11950
>>>
>>>
>>>
>>> This is the release schedule for python versions. We have a year till
>>> the end of 3.6
>>>
>>>
>>>
>>> Branch Schedule Status First release End-of-life
>>>
>>> 3.9 PEP 596 bugfix 2020-10-05 TBD
>>>
>>> 3.8 PEP 569 bugfix 2019-10-14 2024-10
>>>
>>> 3.7 PEP 537 security 2018-06-27 2023-06-27
>>>
>>> 3.6 PEP 494 security 2016-12-23 2021-12-23
>>>
>>>
>>>
>>> WDYT?
>>>
>>>
>>>
>>> J.
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>> *Jarek Potiuk*
>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>
>>> M: +48 660 796 129 <+48%20660%20796%20129>
>>> [image: Polidea] <https://www.polidea.com/>
>>>
>>>
>>>
>>>
>>> --
>>>
>>> *Jarek Potiuk*
>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>
>>>
>>>
>>> M: +48 660 796 129 <+48660796129>
>>> [image: Polidea] <https://www.polidea.com/>
>>>
>>>
>>>
>>>
>>> ==============================================================================
>>> Please access the attached hyperlink for an important electronic
>>> communications disclaimer:
>>> http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
>>>
>>> ==============================================================================
>>>
>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
>
> --
>
> Leah Cole (she/her) | Developer Programs Engineer | coleleah@google.com | (925)
> 257-2112
> *I'm working weird hours during this pandemic and am sometimes a bit
> slower to respond to PRs/CLs than normal. Please feel free to send me a
> gentle ping for a status update if my slowness is blocking you and I'll do
> my best to give you an ETA. *
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Default/supported Python versions for Airlfow 2.0
Posted by Leah Cole <co...@google.com.INVALID>.
I really like Damian's proposal. With my google devrel work we had been
doing something similar to what Jarek had done above - looking at PyPI
downloads and at what is most popular, but doing that makes it harder to
encourage folks to move towards the future. I like that Python provides
this consistent schedule - it will make it easier for us to set
expectations with our users, and will align with other Python-y things. :)
And, tbh, probably going to propose this in future meetings related to my
devrel work so thank you Damien!
On Thu, Nov 12, 2020 at 6:46 AM Jarek Potiuk <Ja...@polidea.com>
wrote:
> I love the idea to have clear rules and tie it to the official schedule of
> Python releases - at least as a target, because we might find some issues
> that might prevent us from doing so.
>
> Also we should officially support all versions that are not yet reached
> end-of-life.
>
> I like the proposed schedule and yearly cadence. I wonder if others have
> similar thoughts. Such agreement/policy would require formal voting though
> I think?
>
> WDYT everyone?
>
> J.
>
>
> On Thu, Nov 12, 2020 at 3:37 PM Shaw, Damian P. <
> damian.shaw.2@credit-suisse.com> wrote:
>
>> I just wanted to add that if people are not aware PEP 0602
>> <https://www.python.org/dev/peps/pep-0602> has been accepted and
>> implemented for Python 3.9. This means 3 things for the Python release
>> cycle:
>>
>> 1. A new version every 12 months
>>
>> 2. Each version receives 18 months of full support (bug fixes and
>> security fixes)
>>
>> 3. After full support has ended each version receives an
>> additional 42 months of security updates
>>
>>
>>
>> Going forward I think it makes sense to bump up the default version of
>> Python every 1 year in cadence with the Python release cycle. Assuming
>> people agreed the question would be how far behind should Airflow be from
>> the new release?
>>
>>
>>
>> Personally I feel like no more than 18 months is a good, in the new
>> Python release cadence that version of Python will no longer be receiving
>> bug fixes and therefore will be very stable, and 18 months is a good enough
>> time for any libraries and providers to be available (if they’re not
>> available after 18 months maybe they have given up support?)
>>
>>
>>
>> If we retroactively apply this to the previous releases of Python that
>> would put us at Python 3.7 default now and Python 3.8 default ~April 14,
>> 2021.
>>
>>
>>
>> My 2 cents,
>>
>> Damian
>>
>>
>>
>>
>>
>> *From:* Jarek Potiuk <Ja...@polidea.com>
>> *Sent:* Thursday, November 12, 2020 09:03
>> *To:* dev@airflow.apache.org
>> *Subject:* Re: Default/supported Python versions for Airlfow 2.0
>>
>>
>>
>> Should we make Python 3.7 default then and leave all others as-is ?
>>
>>
>>
>> J.
>>
>>
>>
>>
>>
>> On Thu, Nov 5, 2020 at 1:48 PM Kaxil Naik <ka...@gmail.com> wrote:
>>
>> We should definitely support Python 3.6 to make the Upgrades to Airflow
>> 2.0 a bit easier.
>>
>>
>>
>> As of yesterday, checks these stats from PyPI downloads:
>>
>>
>>
>> Py3.7: 12,578
>>
>> Py3.6: 9,806
>>
>> Py3.8: 1,815
>>
>>
>>
>>
>>
>>
>>
>> On Thu, Nov 5, 2020 at 11:40 AM Halo Ku <Ha...@mail.com> wrote:
>>
>>
>>
>> If I may point that Airflow is a wokrflow managment system and as
>> such the power of the tool is in direct extention to the levrage providers.
>> This should also be checked from how many of the providers are compatible
>> with 3.8 / 3.9
>>
>>
>>
>> *Sent:* Thursday, November 05, 2020 at 1:16 PM
>> *From:* "Ash Berlin-Taylor" <as...@apache.org>
>> *To:* "dev@airflow.apache.org" <de...@airflow.apache.org>
>> *Cc:* "dev@airflow.apache.org" <de...@airflow.apache.org>
>> *Subject:* Re: Default/supported Python versions for Airlfow 2.0
>>
>> Debian stable ships python 3.7(.3)
>>
>> CentOS 8 has two packages - python36 and python38
>>
>> Ubuntu 18.04 (LTS) has 3.6.5
>>
>> Ubuntu 20.04 (LTS) has 3.8.2
>>
>>
>>
>> (https://pkgs.org/search/?q=python3&on=files)
>>
>>
>>
>> RHEL is harder to find out about . RHEL8 has python 3.6 as python3, and
>> RHEL 8.2 has Py3.8 as a separate package
>> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/configuring_basic_system_settings/index#using-python3_configuring-basic-system-settings
>>
>>
>>
>> So for default version 3.8 or 3.9 gets my vote. I think the cost/burden
>> of supporting back to 3.6 is not very great, so we should continue to
>> support (and I guess test) that.
>>
>>
>>
>> -ash
>>
>>
>>
>> On Nov 5 2020, at 8:49 am, Jarek Potiuk <Ja...@polidea.com> wrote:
>>
>> Hello Everyone,
>>
>>
>>
>> I have a question. What do people think about default version of
>> Pyhon for Airflow 2.0 (and set of supported versions)?
>>
>>
>>
>> Currently, we have python 3.6 as default, but all the version up to 3.8
>> are officially supported and tested and PR for python 3.9 is in Draft:
>> https://github.com/apache/airflow/pull/11950
>>
>>
>>
>> This is the release schedule for python versions. We have a year till the
>> end of 3.6
>>
>>
>>
>> Branch Schedule Status First release End-of-life
>>
>> 3.9 PEP 596 bugfix 2020-10-05 TBD
>>
>> 3.8 PEP 569 bugfix 2019-10-14 2024-10
>>
>> 3.7 PEP 537 security 2018-06-27 2023-06-27
>>
>> 3.6 PEP 494 security 2016-12-23 2021-12-23
>>
>>
>>
>> WDYT?
>>
>>
>>
>> J.
>>
>>
>>
>>
>>
>> --
>>
>> *Jarek Potiuk*
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48%20660%20796%20129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
>>
>>
>> --
>>
>> *Jarek Potiuk*
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>>
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
>>
>>
>> ==============================================================================
>> Please access the attached hyperlink for an important electronic
>> communications disclaimer:
>> http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
>>
>> ==============================================================================
>>
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
--
Leah Cole (she/her) | Developer Programs Engineer | coleleah@google.com | (925)
257-2112
*I'm working weird hours during this pandemic and am sometimes a bit slower
to respond to PRs/CLs than normal. Please feel free to send me a gentle
ping for a status update if my slowness is blocking you and I'll do my best
to give you an ETA. *
Re: Default/supported Python versions for Airlfow 2.0
Posted by Jarek Potiuk <Ja...@polidea.com>.
I love the idea to have clear rules and tie it to the official schedule of
Python releases - at least as a target, because we might find some issues
that might prevent us from doing so.
Also we should officially support all versions that are not yet reached
end-of-life.
I like the proposed schedule and yearly cadence. I wonder if others have
similar thoughts. Such agreement/policy would require formal voting though
I think?
WDYT everyone?
J.
On Thu, Nov 12, 2020 at 3:37 PM Shaw, Damian P. <
damian.shaw.2@credit-suisse.com> wrote:
> I just wanted to add that if people are not aware PEP 0602
> <https://www.python.org/dev/peps/pep-0602> has been accepted and
> implemented for Python 3.9. This means 3 things for the Python release
> cycle:
>
> 1. A new version every 12 months
>
> 2. Each version receives 18 months of full support (bug fixes and
> security fixes)
>
> 3. After full support has ended each version receives an additional
> 42 months of security updates
>
>
>
> Going forward I think it makes sense to bump up the default version of
> Python every 1 year in cadence with the Python release cycle. Assuming
> people agreed the question would be how far behind should Airflow be from
> the new release?
>
>
>
> Personally I feel like no more than 18 months is a good, in the new Python
> release cadence that version of Python will no longer be receiving bug
> fixes and therefore will be very stable, and 18 months is a good enough
> time for any libraries and providers to be available (if they’re not
> available after 18 months maybe they have given up support?)
>
>
>
> If we retroactively apply this to the previous releases of Python that
> would put us at Python 3.7 default now and Python 3.8 default ~April 14,
> 2021.
>
>
>
> My 2 cents,
>
> Damian
>
>
>
>
>
> *From:* Jarek Potiuk <Ja...@polidea.com>
> *Sent:* Thursday, November 12, 2020 09:03
> *To:* dev@airflow.apache.org
> *Subject:* Re: Default/supported Python versions for Airlfow 2.0
>
>
>
> Should we make Python 3.7 default then and leave all others as-is ?
>
>
>
> J.
>
>
>
>
>
> On Thu, Nov 5, 2020 at 1:48 PM Kaxil Naik <ka...@gmail.com> wrote:
>
> We should definitely support Python 3.6 to make the Upgrades to Airflow
> 2.0 a bit easier.
>
>
>
> As of yesterday, checks these stats from PyPI downloads:
>
>
>
> Py3.7: 12,578
>
> Py3.6: 9,806
>
> Py3.8: 1,815
>
>
>
>
>
>
>
> On Thu, Nov 5, 2020 at 11:40 AM Halo Ku <Ha...@mail.com> wrote:
>
>
>
> If I may point that Airflow is a wokrflow managment system and as such the
> power of the tool is in direct extention to the levrage providers.
> This should also be checked from how many of the providers are compatible
> with 3.8 / 3.9
>
>
>
> *Sent:* Thursday, November 05, 2020 at 1:16 PM
> *From:* "Ash Berlin-Taylor" <as...@apache.org>
> *To:* "dev@airflow.apache.org" <de...@airflow.apache.org>
> *Cc:* "dev@airflow.apache.org" <de...@airflow.apache.org>
> *Subject:* Re: Default/supported Python versions for Airlfow 2.0
>
> Debian stable ships python 3.7(.3)
>
> CentOS 8 has two packages - python36 and python38
>
> Ubuntu 18.04 (LTS) has 3.6.5
>
> Ubuntu 20.04 (LTS) has 3.8.2
>
>
>
> (https://pkgs.org/search/?q=python3&on=files)
>
>
>
> RHEL is harder to find out about . RHEL8 has python 3.6 as python3, and
> RHEL 8.2 has Py3.8 as a separate package
> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/configuring_basic_system_settings/index#using-python3_configuring-basic-system-settings
>
>
>
> So for default version 3.8 or 3.9 gets my vote. I think the cost/burden of
> supporting back to 3.6 is not very great, so we should continue to support
> (and I guess test) that.
>
>
>
> -ash
>
>
>
> On Nov 5 2020, at 8:49 am, Jarek Potiuk <Ja...@polidea.com> wrote:
>
> Hello Everyone,
>
>
>
> I have a question. What do people think about default version of Pyhon for
> Airflow 2.0 (and set of supported versions)?
>
>
>
> Currently, we have python 3.6 as default, but all the version up to 3.8
> are officially supported and tested and PR for python 3.9 is in Draft:
> https://github.com/apache/airflow/pull/11950
>
>
>
> This is the release schedule for python versions. We have a year till the
> end of 3.6
>
>
>
> Branch Schedule Status First release End-of-life
>
> 3.9 PEP 596 bugfix 2020-10-05 TBD
>
> 3.8 PEP 569 bugfix 2019-10-14 2024-10
>
> 3.7 PEP 537 security 2018-06-27 2023-06-27
>
> 3.6 PEP 494 security 2016-12-23 2021-12-23
>
>
>
> WDYT?
>
>
>
> J.
>
>
>
>
>
> --
>
> *Jarek Potiuk*
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129
> [image: Polidea] <https://www.polidea.com/>
>
>
>
>
> --
>
> *Jarek Potiuk*
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
>
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
>
>
> ==============================================================================
> Please access the attached hyperlink for an important electronic
> communications disclaimer:
> http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
>
> ==============================================================================
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
RE: Default/supported Python versions for Airlfow 2.0
Posted by "Shaw, Damian P. " <da...@credit-suisse.com>.
I just wanted to add that if people are not aware PEP 0602<https://www.python.org/dev/peps/pep-0602> has been accepted and implemented for Python 3.9. This means 3 things for the Python release cycle:
1. A new version every 12 months
2. Each version receives 18 months of full support (bug fixes and security fixes)
3. After full support has ended each version receives an additional 42 months of security updates
Going forward I think it makes sense to bump up the default version of Python every 1 year in cadence with the Python release cycle. Assuming people agreed the question would be how far behind should Airflow be from the new release?
Personally I feel like no more than 18 months is a good, in the new Python release cadence that version of Python will no longer be receiving bug fixes and therefore will be very stable, and 18 months is a good enough time for any libraries and providers to be available (if they’re not available after 18 months maybe they have given up support?)
If we retroactively apply this to the previous releases of Python that would put us at Python 3.7 default now and Python 3.8 default ~April 14, 2021.
My 2 cents,
Damian
From: Jarek Potiuk <Ja...@polidea.com>
Sent: Thursday, November 12, 2020 09:03
To: dev@airflow.apache.org
Subject: Re: Default/supported Python versions for Airlfow 2.0
Should we make Python 3.7 default then and leave all others as-is ?
J.
On Thu, Nov 5, 2020 at 1:48 PM Kaxil Naik <ka...@gmail.com>> wrote:
We should definitely support Python 3.6 to make the Upgrades to Airflow 2.0 a bit easier.
As of yesterday, checks these stats from PyPI downloads:
Py3.7: 12,578
Py3.6: 9,806
Py3.8: 1,815
On Thu, Nov 5, 2020 at 11:40 AM Halo Ku <Ha...@mail.com>> wrote:
If I may point that Airflow is a wokrflow managment system and as such the power of the tool is in direct extention to the levrage providers.
This should also be checked from how many of the providers are compatible with 3.8 / 3.9
Sent: Thursday, November 05, 2020 at 1:16 PM
From: "Ash Berlin-Taylor" <as...@apache.org>>
To: "dev@airflow.apache.org<ma...@airflow.apache.org>" <de...@airflow.apache.org>>
Cc: "dev@airflow.apache.org<ma...@airflow.apache.org>" <de...@airflow.apache.org>>
Subject: Re: Default/supported Python versions for Airlfow 2.0
Debian stable ships python 3.7(.3)
CentOS 8 has two packages - python36 and python38
Ubuntu 18.04 (LTS) has 3.6.5
Ubuntu 20.04 (LTS) has 3.8.2
(https://pkgs.org/search/?q=python3&on=files)
RHEL is harder to find out about . RHEL8 has python 3.6 as python3, and RHEL 8.2 has Py3.8 as a separate package https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/configuring_basic_system_settings/index#using-python3_configuring-basic-system-settings
So for default version 3.8 or 3.9 gets my vote. I think the cost/burden of supporting back to 3.6 is not very great, so we should continue to support (and I guess test) that.
-ash
On Nov 5 2020, at 8:49 am, Jarek Potiuk <Ja...@polidea.com>> wrote:
Hello Everyone,
I have a question. What do people think about default version of Pyhon for Airflow 2.0 (and set of supported versions)?
Currently, we have python 3.6 as default, but all the version up to 3.8 are officially supported and tested and PR for python 3.9 is in Draft: https://github.com/apache/airflow/pull/11950
This is the release schedule for python versions. We have a year till the end of 3.6
Branch Schedule Status First release End-of-life
3.9 PEP 596 bugfix 2020-10-05 TBD
3.8 PEP 569 bugfix 2019-10-14 2024-10
3.7 PEP 537 security 2018-06-27 2023-06-27
3.6 PEP 494 security 2016-12-23 2021-12-23
WDYT?
J.
--
[https://s3.eu-central-1.amazonaws.com/corgi-mail/23-05-2019/jarek.potiuk/jarpot.jpg]
Jarek Potiuk
Polidea<https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129
[Polidea]<https://www.polidea.com/>
--
[https://s3.eu-central-1.amazonaws.com/corgi-mail/23-05-2019/jarek.potiuk/jarpot.jpg]
Jarek Potiuk
Polidea<https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129<tel:+48660796129>
[Polidea]<https://www.polidea.com/>
===============================================================================
Please access the attached hyperlink for an important electronic communications disclaimer:
http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
===============================================================================
Re: Default/supported Python versions for Airlfow 2.0
Posted by Jarek Potiuk <Ja...@polidea.com>.
Should we make Python 3.7 default then and leave all others as-is ?
J.
On Thu, Nov 5, 2020 at 1:48 PM Kaxil Naik <ka...@gmail.com> wrote:
> We should definitely support Python 3.6 to make the Upgrades to Airflow
> 2.0 a bit easier.
>
> As of yesterday, checks these stats from PyPI downloads:
>
> Py3.7: 12,578
> Py3.6: 9,806
> Py3.8: 1,815
>
>
>
> On Thu, Nov 5, 2020 at 11:40 AM Halo Ku <Ha...@mail.com> wrote:
>
>>
>> If I may point that Airflow is a wokrflow managment system and as
>> such the power of the tool is in direct extention to the levrage providers.
>> This should also be checked from how many of the providers are compatible
>> with 3.8 / 3.9
>>
>> *Sent:* Thursday, November 05, 2020 at 1:16 PM
>> *From:* "Ash Berlin-Taylor" <as...@apache.org>
>> *To:* "dev@airflow.apache.org" <de...@airflow.apache.org>
>> *Cc:* "dev@airflow.apache.org" <de...@airflow.apache.org>
>> *Subject:* Re: Default/supported Python versions for Airlfow 2.0
>> Debian stable ships python 3.7(.3)
>> CentOS 8 has two packages - python36 and python38
>> Ubuntu 18.04 (LTS) has 3.6.5
>> Ubuntu 20.04 (LTS) has 3.8.2
>>
>> (https://pkgs.org/search/?q=python3&on=files)
>>
>> RHEL is harder to find out about . RHEL8 has python 3.6 as python3, and
>> RHEL 8.2 has Py3.8 as a separate package
>> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/configuring_basic_system_settings/index#using-python3_configuring-basic-system-settings
>>
>> So for default version 3.8 or 3.9 gets my vote. I think the cost/burden
>> of supporting back to 3.6 is not very great, so we should continue to
>> support (and I guess test) that.
>>
>> -ash
>>
>> On Nov 5 2020, at 8:49 am, Jarek Potiuk <Ja...@polidea.com> wrote:
>>
>> Hello Everyone,
>>
>> I have a question. What do people think about default version of
>> Pyhon for Airflow 2.0 (and set of supported versions)?
>>
>> Currently, we have python 3.6 as default, but all the version up to 3.8
>> are officially supported and tested and PR for python 3.9 is in Draft:
>> https://github.com/apache/airflow/pull/11950
>>
>> This is the release schedule for python versions. We have a year till the
>> end of 3.6
>>
>> Branch Schedule Status First release End-of-life
>> 3.9 PEP 596 bugfix 2020-10-05 TBD
>> 3.8 PEP 569 bugfix 2019-10-14 2024-10
>> 3.7 PEP 537 security 2018-06-27 2023-06-27
>> 3.6 PEP 494 security 2016-12-23 2021-12-23
>>
>> WDYT?
>>
>> J.
>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Default/supported Python versions for Airlfow 2.0
Posted by Kaxil Naik <ka...@gmail.com>.
We should definitely support Python 3.6 to make the Upgrades to Airflow 2.0
a bit easier.
As of yesterday, checks these stats from PyPI downloads:
Py3.7: 12,578
Py3.6: 9,806
Py3.8: 1,815
On Thu, Nov 5, 2020 at 11:40 AM Halo Ku <Ha...@mail.com> wrote:
>
> If I may point that Airflow is a wokrflow managment system and as such the
> power of the tool is in direct extention to the levrage providers.
> This should also be checked from how many of the providers are compatible
> with 3.8 / 3.9
>
> *Sent:* Thursday, November 05, 2020 at 1:16 PM
> *From:* "Ash Berlin-Taylor" <as...@apache.org>
> *To:* "dev@airflow.apache.org" <de...@airflow.apache.org>
> *Cc:* "dev@airflow.apache.org" <de...@airflow.apache.org>
> *Subject:* Re: Default/supported Python versions for Airlfow 2.0
> Debian stable ships python 3.7(.3)
> CentOS 8 has two packages - python36 and python38
> Ubuntu 18.04 (LTS) has 3.6.5
> Ubuntu 20.04 (LTS) has 3.8.2
>
> (https://pkgs.org/search/?q=python3&on=files)
>
> RHEL is harder to find out about . RHEL8 has python 3.6 as python3, and
> RHEL 8.2 has Py3.8 as a separate package
> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/configuring_basic_system_settings/index#using-python3_configuring-basic-system-settings
>
> So for default version 3.8 or 3.9 gets my vote. I think the cost/burden of
> supporting back to 3.6 is not very great, so we should continue to support
> (and I guess test) that.
>
> -ash
>
> On Nov 5 2020, at 8:49 am, Jarek Potiuk <Ja...@polidea.com> wrote:
>
> Hello Everyone,
>
> I have a question. What do people think about default version of Pyhon for
> Airflow 2.0 (and set of supported versions)?
>
> Currently, we have python 3.6 as default, but all the version up to 3.8
> are officially supported and tested and PR for python 3.9 is in Draft:
> https://github.com/apache/airflow/pull/11950
>
> This is the release schedule for python versions. We have a year till the
> end of 3.6
>
> Branch Schedule Status First release End-of-life
> 3.9 PEP 596 bugfix 2020-10-05 TBD
> 3.8 PEP 569 bugfix 2019-10-14 2024-10
> 3.7 PEP 537 security 2018-06-27 2023-06-27
> 3.6 PEP 494 security 2016-12-23 2021-12-23
>
> WDYT?
>
> J.
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129
> [image: Polidea] <https://www.polidea.com/>
>
>
Re: Default/supported Python versions for Airlfow 2.0
Posted by Halo Ku <Ha...@mail.com>.
If I may point that Airflow is a wokrflow managment system and as such the
power of the tool is in direct extention to the levrage providers.
This should also be checked from how many of the providers are compatible with
3.8 / 3.9
**Sent:** Thursday, November 05, 2020 at 1:16 PM
**From:** "Ash Berlin-Taylor" <as...@apache.org>
**To:** "dev@airflow.apache.org" <de...@airflow.apache.org>
**Cc:** "dev@airflow.apache.org" <de...@airflow.apache.org>
**Subject:** Re: Default/supported Python versions for Airlfow 2.0
Debian stable ships python 3.7(.3)
CentOS 8 has two packages - python36 and python38
Ubuntu 18.04 (LTS) has 3.6.5
Ubuntu 20.04 (LTS) has 3.8.2
([https://pkgs.org/search/?q=python3&on=files](https://pkgs.org/search/?q=python3&on=files
"https://pkgs.org/search/?q=python3&on=files"))
RHEL is harder to find out about . RHEL8 has python 3.6 as python3, and RHEL
8.2 has Py3.8 as a separate package
<https://access.redhat.com/documentation/en-
us/red_hat_enterprise_linux/8/html-
single/configuring_basic_system_settings/index#using-python3_configuring-
basic-system-settings>
So for default version 3.8 or 3.9 gets my vote. I think the cost/burden of
supporting back to 3.6 is not very great, so we should continue to support
(and I guess test) that.
-ash
On Nov 5 2020, at 8:49 am, Jarek Potiuk <Ja...@polidea.com> wrote:
> Hello Everyone,
>
>
>
> I have a question. What do people think about default version of Pyhon for
Airflow 2.0 (and set of supported versions)?
>
>
>
> Currently, we have python 3.6 as default, but all the version up to 3.8 are
officially supported and tested and PR for python 3.9 is in Draft:
<https://github.com/apache/airflow/pull/11950>
>
>
>
> This is the release schedule for python versions. We have a year till the
end of 3.6
>
>
>
> Branch Schedule Status First release End-of-life
>
> 3.9 PEP 596 bugfix 2020-10-05 TBD
>
> 3.8 PEP 569 bugfix 2019-10-14 2024-10
>
> 3.7 PEP 537 security 2018-06-27 2023-06-27
>
> 3.6 PEP 494 security 2016-12-23 2021-12-23
>
>
>
> WDYT?
>
>
>
> J.
>
>
>
>
>
> \--
>
>  |
>
> Jarek Potiuk
> [Polidea](https://www.polidea.com/) | Principal Software Engineer
>
> ---|---
>
> M: +48 660 796 129
> [](https://www.polidea.com/)
>
> ---
Re: Default/supported Python versions for Airlfow 2.0
Posted by Ash Berlin-Taylor <as...@apache.org>.
Debian stable ships python 3.7(.3)
CentOS 8 has two packages - python36 and python38
Ubuntu 18.04 (LTS) has 3.6.5
Ubuntu 20.04 (LTS) has 3.8.2
(https://pkgs.org/search/?q=python3&on=files)
RHEL is harder to find out about . RHEL8 has python 3.6 as python3, and RHEL 8.2 has Py3.8 as a separate package https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/configuring_basic_system_settings/index#using-python3_configuring-basic-system-settings
So for default version 3.8 or 3.9 gets my vote. I think the cost/burden of supporting back to 3.6 is not very great, so we should continue to support (and I guess test) that.
-ash
On Nov 5 2020, at 8:49 am, Jarek Potiuk <Ja...@polidea.com> wrote:
> Hello Everyone,
>
> I have a question. What do people think about default version of Pyhon for Airflow 2.0 (and set of supported versions)?
>
> Currently, we have python 3.6 as default, but all the version up to 3.8 are officially supported and tested and PR for python 3.9 is in Draft: https://github.com/apache/airflow/pull/11950
>
> This is the release schedule for python versions. We have a year till the end of 3.6
>
> Branch Schedule Status First release End-of-life
> 3.9 PEP 596 bugfix 2020-10-05 TBD
> 3.8 PEP 569 bugfix 2019-10-14 2024-10
> 3.7 PEP 537 security 2018-06-27 2023-06-27
> 3.6 PEP 494 security 2016-12-23 2021-12-23
>
>
> WDYT?
>
> J.
>
>
> --
>
>
> Jarek Potiuk
> Polidea (https://www.polidea.com/) | Principal Software Engineer
>
>
>
>
>
>
>
> M: +48 660 796 129 (tel:+48660796129)
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>