You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Jarek Potiuk <ja...@potiuk.com> on 2024/01/17 12:42:38 UTC
What goes into the next release (was [VOTE] Release Airflow 2.8.1 from 2.8.1rc1)
I separate it out, because it seems that despite the efforts to explain and
document how our releases work It's not clear even for the PMC chair, so
likely it warrants a separate thread - also it will be easier to find it in
the archives this way.
I think this is an important topic that all maintainers should be aware of,
so let me take a task to explain it in a longer email (and separate thread).
I created it in a form of FAQ, because it seems that similar questions
might be asked by others.
*Do we have a process defined here?*
Answering Bolke's question - who was apparently confused what our process
is:
> I see that some improvements to Airflow.io (two weeks ago) were not
included and some provider updates neither. Haven't checked anything else
yet.
Apparently there is some confusion about the process, but yes we have a
well defined and well tested (pretty much 4 years now) process that we
follow., We follow it since I remember actually - it's been also done the
same in 1.10 - but with some variations, Likely we do it the same way since
the beginning of 2.0, but it has been refined and improved over time - by
those who volunteered their time in the release process (a lot of ad-hoc
discussion have been traditionally happening in #release-management slack
channel) and as of few months ago we even documented it (It was in November
2023) - with this PR https://github.com/apache/airflow/pull/35245
It is currently described in a prominent place in our most important (and
over the last year or so the README, we made the README pretty short and
contains only super-important information on how Airflow is developed)
README.md under *"What goes into the next release?"* chapter:
https://github.com/apache/airflow?tab=readme-ov-file#what-goes-into-the-next-release
.
*How does the selection process for cherry-picking work?*
In short (and this is the most important thing that every maintainer should
be aware of): *those maintainers who think that issue should be included
should mark it with the next (in this case 2.8.1) milestone*. It's up to
individual maintainers who want to include certain changes to take care
about it and mark the issues they think are bug fixes, to go into the next
release
This is the only thing that the maintainer has to do to get the PR proposed
to be considered in the next patchlevel release. Sometimes - if
controversial - maintainers discuss the proposals in #release-management
channel, sometimes in #development or in the PR itself (especially if the
release manager decides to not include it and changes the milestone (and
explains why).
*What's the release manager's role ?*
Release manager's job is purely mechanical (as also mandated by the Apache
Software Foundation release manager role description) to assess cherry-pick
ability of those changes. Release manager - at the sole discretion and
individual decision (this is the only place in the whole ASF setup where a
single person has such power to make individual decisions) can reject some
of those who other maintainers think should be included. But the release
manager on his own does not make proposals on what should be included.
*Is this process following the ASF rules?*
I believe yes, The release manager's role is nicely described here:
https://infra.apache.org/release-publishing.html#releasemanager. And there
is a far more complete description here that describes the whole process
https://www.apache.org/legal/release-policy.html#management - also
mentioning that it's the PMC's responsibility (and particularly PMC
chair's) to adhere to the process.
*What's the role of individual maintainers?*
The role of maintainers (collectively) to propose things for the next
release. In our case it happens with setting the milestone on a PR.
*When proposed PRs are rejected?*
There are various reasons to reject those - if too complex to cherry-pick
or when the release manager assesses it's a new feature, not a bugfix.
Essentially (according to semver) when it comes to the user-facing changes,
the PATCHLEVEL release should contain only bugfixes. and may contain docs
changes if they are fixing/improving docs (not about new features) and also
environment/build script changes (so non-user-facing changes) as they are
pretty much always needed to keep the things nicely building - those are
usually skipped from the changelog as non-user facing).
*Why are provider changes not cherry-picked?*
In our case - basically none of the provider changes are cherry-picked -
unless they are needed to make the builds work well (sometimes happen).
Providers are ALWAYS released from the latest main code, not from the
v2-8-stable branch. In fact all the tests and ci checks for providers are
skipped in the non-main (v2* branches). So yes - not seeing provider
changes cherry-picked is absolutely expected.
*Do we skip over some changes when releasing a patchlevel release? What's
the purpose of patch-level releases?*
Answering Bolke's question:
> Is that intentional? Ie. is that the purpose of this release. Other
big(ger) and more recent changes have been included hence asking.
The purpose of that release is as described in SemVer - to give the users
bugfix-only release that has no new features. Of course it's sometimes
debatable whether changes are features/bugfixes, but we usually use
#release-management to quickly chat about it, and eventually the release
manager always makes a comment in the PR when the milestone is changed and
explains the reasoning.
Skipping is not intentional because we never "skip" things when
cherry-picking, It's *reverse* - those maintainer who think that certain
bug fixes (or internal changes or sometimes even feature changes that we
classify really as "bugfix" SHOULD intentionally mark those PRs they want
with 2.8.1 (or basically next patch-level) to be *included. * So there is
no skipping, if maintainer did not deliberately mark PR as upcoming
milestone, it will just not be included
*Where do some maintainers know about it **from ? *
Because the maintainers who actively participate - either acting as release
managers (those who raised their hand and acted as release managers
generally speaking) and those who wanted their changes to be part of the
past releases have been doing it for years. For years this has been simply
followed and discussed in #release-management channel and #development (and
in devlist whenever there were new releases) and generally the maintainers
who took part in those discussions and release process are aware of that -
also many maintainers know the process as it "soaked" in when they were
just watching.
However recently we've made an attempt to document it (the PR above). So
you could learn it by reading it (even if it does not have the whole
context above).
*Why do some people not know about it?*
Not sure. Maybe because they were not interested and never asked? Maybe
because there was never a long email about it at our devlist, or the
documentation about it in README was too succinct?
Or maybe we need a better way of communicating it - I am not sure. But I
hope this email will clarify a lot of it, and will prove valuable when
searching in devlist.
Maybe even someone who manages to read it all will update the README
description of ours to explain it better :) and maybe create a nice FAQ
that we can put in our docs?
I really hope this mail - even if long - will make people a bit more aware
of the process, and if someone has an idea how the "collective" awareness
can be improved - I think it's a good idea to send PR or email (or maybe
even record a video :) ??) that will be a better way of communicating it.
J.
On Wed, Jan 17, 2024 at 9:03 AM Bolke de Bruin <bd...@gmail.com> wrote:
> Just checking:
>
> I see that some improvements to Airflow.io (two weeks ago) were not
> included and some provider updates neither. Haven't checked anything else
> yet.
>
> Is that intentional? Ie. is that the purpose of this release. Other
> big(ger) and more recent changes have been included hence asking.
>
> B.
>
> Sent from my iPhone
>
> > On 16 Jan 2024, at 20:18, Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > +1 (binding): Checked all my changes, I ran airflow in a few
> combinations
> > (MySQL / Postgres Local/Celery executor. It looks and works well - run a
> > few dags and navigated through a number of screens. Checked licences,
> > signatures, checksums, performed a "reproducible build" check and it
> worked
> > (with a small glitch explained below).
> >
> > BTW. The new "hatchling-built" package looks good and it nicely installs
> > airflow + extras (and it has a far better and cleaner set of extras -
> > finally you can reproducibly install airflow with `all` extra if you
> > ***REALLY*** want :).
> >
> > Reproducibility (also for other PMC members doing the check): I found a
> > small glitch in the "reproducible" part of verifying the packages. While
> > wheel and sdist packages are exactly the same binary-wise, the
> > source-tarball was binarry different for me. I decompressed it and
> compared
> > the content - they are identical - but there is one difference which I
> > overlooked - the group permissions for files in Ephraim's tarball are
> > different from mine. I have totally forgotten about the fact that umask
> > might set different group/other permissions when checking out the files
> > from git. Fix will be coming shortly - in the meantime I recommend anyone
> > who will be doing the comparison to uncompress both tarballs and compare
> > the contents with `diff -r`.
> >
> >
> >
> >> On Tue, Jan 16, 2024 at 11:30 AM Ephraim Anierobi <
> >> ephraimanierobi@apache.org> wrote:
> >>
> >> Hey fellow Airflowers,
> >>
> >> I have cut Airflow 2.8.1rc1. This email is calling a vote on the
> release,
> >> which will last at least 72 hours, from Tuesday, January 16, 2024 at
> 10:30
> >> am UTC
> >> until Friday, January 19, 2024, at 10:30 am UTC
> >> <
> >>
> https://www.timeanddate.com/worldclock/fixedtime.html?msg=8&iso=20240119T1030&p1=1440
> >>> ,
> >> and until 3 binding +1 votes have been received.
> >>
> >> Status of testing of the release is kept at
> >> https://github.com/apache/airflow/issues/36808
> >>
> >> Consider this my (binding) +1.
> >>
> >> Airflow 2.8.1rc1 is available at:
> >> https://dist.apache.org/repos/dist/dev/airflow/2.8.1rc1/
> >>
> >> *apache-airflow-2.8.1-source.tar.gz* is a source release that comes with
> >> INSTALL instructions.
> >> *apache-airflow-2.8.1.tar.gz* is the binary Python "sdist" release.
> >> *apache_airflow-2.8.1-py3-none-any.whl* is the binary Python wheel
> "binary"
> >> release.
> >>
> >> Public keys are available at:
> >> https://dist.apache.org/repos/dist/release/airflow/KEYS
> >>
> >> Please vote accordingly:
> >>
> >> [ ] +1 approve
> >> [ ] +0 no opinion
> >> [ ] -1 disapprove with the reason
> >>
> >> Only votes from PMC members are binding, but all members of the
> community
> >> are encouraged to test the release and vote with "(non-binding)".
> >>
> >> The test procedure for PMC members is described in:
> >>
> >>
> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-pmc-members
> >>
> >> The test procedure for and Contributors who would like to test this RC
> is
> >> described in:
> >>
> >>
> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-contributors
> >>
> >>
> >> Please note that the version number excludes the `rcX` string, so it's
> now
> >> simply 2.8.1. This will allow us to rename the artifact without
> modifying
> >> the artifact checksums when we actually release.
> >>
> >> Release Notes:
> >> https://github.com/apache/airflow/blob/2.8.1rc1/RELEASE_NOTES.rst
> >>
> >> Changes since 2.8.0:
> >>
> >> *Significant Changes*
> >>
> >> *Target version for core dependency ``pendulum`` package set to 3
> >> (#36281).*
> >> Support for pendulum 2.1.2 will be saved for a while, presumably until
> the
> >> next feature version of Airflow.
> >> It is advised to upgrade user code to use pendulum 3 as soon as
> possible.
> >>
> >> *Airflow packaging specification follows modern Python packaging
> standards
> >> (#36537).*
> >> We standardized Airflow dependency configuration to follow latest
> >> development in Python packaging by
> >> using ``pyproject.toml``. Airflow is now compliant with those accepted
> >> PEPs:
> >>
> >> * `PEP-440 Version Identification and Dependency Specification <
> >> https://www.python.org/dev/peps/pep-0440/>`__
> >> * `PEP-517 A build-system independent format for source trees <
> >> https://www.python.org/dev/peps/pep-0517/>`__
> >> * `PEP-518 Specifying Minimum Build System Requirements for Python
> Projects
> >> <https://www.python.org/dev/peps/pep-0518/>`__
> >> * `PEP-561 Distributing and Packaging Type Information <
> >> https://www.python.org/dev/peps/pep-0561/>`__
> >> * `PEP-621 Storing project metadata in pyproject.toml <
> >> https://www.python.org/dev/peps/pep-0621/>`__
> >> * `PEP-660 Editable installs for pyproject.toml based builds (wheel
> based)
> >> <
> >> https://www.python.org/dev/peps/pep-0660/>`__
> >> * `PEP-685 Comparison of extra names for optional distribution
> dependencies
> >> <https://www.python.org/dev/peps/pep-0685/>`__
> >>
> >> Also we implement multiple license files support coming from Draft, not
> yet
> >> accepted (but supported by hatchling) PEP:
> >> * `PEP 639 Improving License Clarity with Better Package Metadata <
> >> https://peps.python.org/pep-0639/>`__
> >>
> >> This has almost no noticeable impact on users if they are using modern
> >> Python packaging and development tools, generally
> >> speaking Airflow should behave as it did before when installing it from
> >> PyPI and it should be much easier to install
> >> it for development purposes using ``pip install -e ".[devel]"``.
> >>
> >> The differences from the user side are:
> >>
> >> * Airflow extras now get extras normalized to ``-`` (following PEP-685)
> >> instead of ``_`` and ``.``
> >> (as it was before in some extras). When you install airflow with such
> >> extras (for example ``dbt.core`` or
> >> ``all_dbs``) you should use ``-`` instead of ``_`` and ``.``.
> >>
> >> In most modern tools this will work in backwards-compatible way, but in
> >> some old version of those tools you might need to
> >> replace ``_`` and ``.`` with ``-``. You can also get warnings that the
> >> extra you are installing does not exist - but usually
> >> this warning is harmless and the extra is installed anyway. It is,
> however,
> >> recommended to change to use ``-`` in extras in your dependency
> >> specifications for all Airflow extras.
> >>
> >> * Released airflow package does not contain ``devel``, ``devel-*``,
> ``doc``
> >> and ``doc-gen`` extras.
> >> Those extras are only available when you install Airflow from sources
> in
> >> ``--editable`` mode. This is
> >> because those extras are only used for development and documentation
> >> building purposes and are not needed
> >> when you install Airflow for production use. Those dependencies had
> >> unspecified and varying behaviour for
> >> released packages anyway and you were not supposed to use them in
> >> released packages.
> >>
> >> * The ``all`` and ``all-*`` extras were not always working correctly
> when
> >> installing Airflow using constraints
> >> because they were also considered as development-only dependencies.
> With
> >> this change, those dependencies are
> >> now properly handling constraints and they will install properly with
> >> constraints, pulling the right set
> >> of providers and dependencies when constraints are used.
> >>
> >> *Graphviz dependency is now an optional one, not required one (#36647).*
> >> The ``graphviz`` dependency has been problematic as Airflow required
> >> dependency - especially for
> >> ARM-based installations. Graphviz packages require binary graphviz
> >> libraries - which is already a
> >> limitation, but they also require to install graphviz Python bindings
> to be
> >> build and installed.
> >> This does not work for older Linux installation but - more importantly -
> >> when you try to install
> >> Graphviz libraries for Python 3.8, 3.9 for ARM M1 MacBooks, the packages
> >> fail to install because
> >> Python bindings compilation for M1 can only work for Python 3.10+.
> >>
> >> This is not a breaking change technically - the CLIs to render the DAGs
> is
> >> still there and IF you
> >> already have graphviz installed, it will continue working as it did
> before.
> >> The only problem when it
> >> does not work is where you do not have graphviz installed it will raise
> an
> >> error and inform that you need it.
> >>
> >> Graphviz will remain to be installed for most users:
> >>
> >> * the Airflow Image will still contain graphviz library, because
> >> it is added there as extra
> >> * when previous version of Airflow has been installed already, then
> >> graphviz library is already installed there and Airflow will
> >> continue working as it did
> >>
> >> The only change will be a new installation of new version of Airflow
> from
> >> the scratch, where graphviz will
> >> need to be specified as extra or installed separately in order to enable
> >> DAG rendering option.
> >>
> >> *Bug Fixes*
> >> - Fix airflow-scheduler exiting with code 0 on exceptions (#36800)
> >> - Fix Callback exception when a removed task is the last one in the
> >> ``taskinstance`` list (#36693)
> >> - Allow anonymous user edit/show resource when set
> >> ``AUTH_ROLE_PUBLIC=admin`` (#36750)
> >> - Better error message when sqlite URL uses relative path (#36774)
> >> - Explicit string cast required to force integer-type run_ids to be
> passed
> >> as strings instead of integers (#36756)
> >> - Add log lookup exception for empty ``op`` subtypes (#35536)
> >> - Remove unused index on task instance (#36737)
> >> - Fix check on subclass for ``typing.Union`` in
> ``_infer_multiple_outputs``
> >> for Python 3.10+ (#36728)
> >> - Make sure ``multiple_outputs`` is inferred correctly even when using
> >> ``TypedDict`` (#36652)
> >> - Add back FAB constant in legacy security manager (#36719)
> >> - Fix AttributeError when using ``Dagrun.update_state`` (#36712)
> >> - Do not let ``EventsTimetable`` schedule past events if
> ``catchup=False``
> >> (#36134)
> >> - Support encryption for triggers parameters (#36492)
> >> - Fix the type hint for ``tis_query`` in ``_process_executor_events``
> >> (#36655)
> >> - Redirect to index when user does not have permission to access a page
> >> (#36623)
> >> - Avoid using dict as default value in ``call_regular_interval``
> (#36608)
> >> - Remove option to set a task instance to running state in UI (#36518)
> >> - Fix details tab not showing when using dynamic task mapping (#36522)
> >> - Raise error when ``DagRun`` fails while running ``dag test`` (#36517)
> >> - Refactor ``_manage_executor_state`` by refreshing TIs in batch
> (#36502)
> >> - Add flask config: ``MAX_CONTENT_LENGTH`` (#36401)
> >> - Fix get_leaves calculation for teardown in nested group (#36456)
> >> - Stop serializing timezone-naive datetime to timezone-aware datetime
> with
> >> UTC tz (#36379)
> >> - Make ``kubernetes`` decorator type annotation consistent with operator
> >> (#36405)
> >> - Fix Webserver returning 500 for POST requests to ``api/dag/*/dagrun``
> >> from anonymous user (#36275)
> >> - Fix the required access for get_variable endpoint (#36396)
> >> - Fix datetime reference in ``DAG.is_fixed_time_schedule`` (#36370)
> >> - Fix AirflowSkipException message raised by BashOperator (#36354)
> >> - Allow PythonVirtualenvOperator.skip_on_exit_code to be zero (#36361)
> >> - Increase width of execution_date input in trigger.html (#36278)
> >> - Fix logging for pausing DAG (#36182)
> >> - Stop deserializing pickle when enable_xcom_pickling is False (#36255)
> >> - Check DAG read permission before accessing DAG code (#36257)
> >> - Enable mark task as failed/success always (#36254)
> >> - Create latest log dir symlink as relative link (#36019)
> >> - Fix Python-based decorators templating (#36103)
> >>
> >> *Miscellaneous*
> >> - Rename concurrency label to max active tasks (#36691)
> >> - Restore function scoped ``httpx`` import in file_task_handler for
> >> performance (#36753)
> >> - Add support of Pendulum 3 (#36281)
> >> - Standardize airflow build process and switch to Hatchling build
> backend
> >> (#36537)
> >> - Get rid of ``pyarrow-hotfix`` for ``CVE-2023-47248`` (#36697)
> >> - Make ``graphviz`` dependency optional (#36647)
> >> - Announce MSSQL support end in Airflow 2.9.0, add migration script
> hints
> >> (#36509)
> >> - Set min ``pandas`` dependency to 1.2.5 for all providers and airflow
> >> (#36698)
> >> - Bump follow-redirects from 1.15.3 to 1.15.4 in ``/airflow/www``
> (#36700)
> >> - Provide the logger_name param to base hook in order to override the
> >> logger name (#36674)
> >> - Fix run type icon alignment with run type text (#36616)
> >> - Follow BaseHook connection fields method signature in FSHook (#36444)
> >> - Remove redundant ``docker`` decorator type annotations (#36406)
> >> - Straighten typing in workday timetable (#36296)
> >> - Use ``batch_is_authorized_dag`` to check if user has permission to
> read
> >> DAGs (#36279)
> >> - Replace deprecated get_accessible_dag_ids and use get_readable_dags in
> >> get_dag_warnings (#36256)
> >>
> >> *Doc Only Changes*
> >> - Metrics tagging documentation (#36627)
> >> - In docs use logical_date instead of deprecated execution_date (#36654)
> >> - Add section about live-upgrading Airflow (#36637)
> >> - Replace ``numpy`` example with practical exercise demonstrating
> top-level
> >> code (#35097)
> >> - Improve and add more complete description in the architecture diagrams
> >> (#36513)
> >> - Improve the error message displayed when there is a webserver error
> >> (#36570)
> >> - Update ``dags.rst`` with information on DAG pausing (#36540)
> >> - Update installation prerequisites after upgrading to Debian Bookworm
> >> (#36521)
> >> - Add description on the ways how users should approach DB monitoring
> >> (#36483)
> >> - Add branching based on mapped task group example to
> >> dynamic-task-mapping.rst (#36480)
> >> - Add further details to replacement documentation (#36485)
> >> - Use cards when describing priority weighting methods (#36411)
> >> - Update ``metrics.rst`` for param ``dagrun.schedule_delay`` (#36404)
> >> - Update admonitions in Python operator doc to reflect sentiment
> (#36340)
> >> - Improve audit_logs.rst (#36213)
> >> - Remove Redshift mention from the list of managed Postgres backends
> >> (#36217)
> >>
> >> Cheers,
> >> Ephraim
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>
Re: What goes into the next release (was [VOTE] Release Airflow 2.8.1 from 2.8.1rc1)
Posted by Amogh Desai <am...@gmail.com>.
Great idea. I will take a look at this shortly.
Thanks!
On Thu, 18 Jan 2024 at 1:48 PM, Jarek Potiuk <ja...@potiuk.com> wrote:
> Created PR https://github.com/apache/airflow/pull/36858 where I proposed
> a separate document - with a bit more polished version of what I wrote
> above, interlinked with the succinct README chapter we already have.
>
> I also expanded there a bit "What's the purpose of patch-level release"
> explaining the approach we take for classifying changes as breaking/not
> breaking and adding comment on SemVer being "intentional" and not
> "technically breaking non/breaking" - something that we discussed a lot at
> many occasions in multiple PRs when we debated whether things are
> breaking/not breaking.
>
> And removed "how some people know/don't know" about it. I hope having that
> FAQ will help us to get to the point where we all know it - or at least
> can easily find it when we are looking for it.
>
> J.
>
> On Thu, Jan 18, 2024 at 8:42 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> I think if anything - that should be a separate page in our GitHub -
>> which will be a) easier to do b) maybe indeed makes sense c) this is where
>> we put "developer" docs.
>>
>> I think we agreed already quite some time ago that "airflow.apache.org"
>> is generally for the users and all the docs for contributors/maintainers
>> should go as .md/.rst files in our repository, but not into the airflow
>> website.
>>
>> Yeah - I think it might be worth it if we keep it separate from README
>> :). I think we had far too long README and (as mentioned above) over the
>> last year or so it got shorter and shorter, while it gained only the "super
>> important" information IMHO. So yeah having a separate FAQ taking my mail
>> as context might be a good idea.
>>
>> I will open PR for that and if others agree it's a good idea (and help
>> with reviewing it, we can merge it).
>>
>> J.
>>
>>
>>
>>
>>
>>
>>
>> On Thu, Jan 18, 2024 at 5:29 AM Amogh Desai <am...@gmail.com>
>> wrote:
>>
>>> Thanks @Jarek Potiuk <ja...@potiuk.com>!
>>>
>>> I vaguely knew the process but not in such detail, thanks for putting it
>>> together in this email.
>>> I will visit the document
>>> https://github.com/apache/airflow?tab=readme-ov-file#what-goes-into-the-next-release
>>> and if I find any clarifications, I will send it across as a PR.
>>>
>>> One suggestion: what if we put it together in a document and put it on
>>> the airflow website under
>>> https://airflow.apache.org/community/ where we mention "how we release"?
>>>
>>> Might be a bit too much given the context people will have while
>>> visiting the airflow website..
>>>
>>> Thanks & Regards,
>>> Amogh Desai
>>>
>>> On Wed, Jan 17, 2024 at 6:13 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>
>>>> I separate it out, because it seems that despite the efforts to explain
>>>> and
>>>> document how our releases work It's not clear even for the PMC chair, so
>>>> likely it warrants a separate thread - also it will be easier to find
>>>> it in
>>>> the archives this way.
>>>>
>>>> I think this is an important topic that all maintainers should be aware
>>>> of,
>>>> so let me take a task to explain it in a longer email (and separate
>>>> thread).
>>>>
>>>> I created it in a form of FAQ, because it seems that similar questions
>>>> might be asked by others.
>>>>
>>>> *Do we have a process defined here?*
>>>>
>>>> Answering Bolke's question - who was apparently confused what our
>>>> process
>>>> is:
>>>>
>>>> > I see that some improvements to Airflow.io (two weeks ago) were not
>>>> included and some provider updates neither. Haven't checked anything
>>>> else
>>>> yet.
>>>>
>>>> Apparently there is some confusion about the process, but yes we have a
>>>> well defined and well tested (pretty much 4 years now) process that we
>>>> follow., We follow it since I remember actually - it's been also done
>>>> the
>>>> same in 1.10 - but with some variations, Likely we do it the same way
>>>> since
>>>> the beginning of 2.0, but it has been refined and improved over time -
>>>> by
>>>> those who volunteered their time in the release process (a lot of ad-hoc
>>>> discussion have been traditionally happening in #release-management
>>>> slack
>>>> channel) and as of few months ago we even documented it (It was in
>>>> November
>>>> 2023) - with this PR https://github.com/apache/airflow/pull/35245
>>>>
>>>> It is currently described in a prominent place in our most important
>>>> (and
>>>> over the last year or so the README, we made the README pretty short and
>>>> contains only super-important information on how Airflow is developed)
>>>> README.md under *"What goes into the next release?"* chapter:
>>>>
>>>>
>>>> https://github.com/apache/airflow?tab=readme-ov-file#what-goes-into-the-next-release
>>>> .
>>>>
>>>> *How does the selection process for cherry-picking work?*
>>>>
>>>> In short (and this is the most important thing that every maintainer
>>>> should
>>>> be aware of): *those maintainers who think that issue should be included
>>>> should mark it with the next (in this case 2.8.1) milestone*. It's up to
>>>> individual maintainers who want to include certain changes to take care
>>>> about it and mark the issues they think are bug fixes, to go into the
>>>> next
>>>> release
>>>>
>>>> This is the only thing that the maintainer has to do to get the PR
>>>> proposed
>>>> to be considered in the next patchlevel release. Sometimes - if
>>>> controversial - maintainers discuss the proposals in #release-management
>>>> channel, sometimes in #development or in the PR itself (especially if
>>>> the
>>>> release manager decides to not include it and changes the milestone (and
>>>> explains why).
>>>>
>>>> *What's the release manager's role ?*
>>>>
>>>> Release manager's job is purely mechanical (as also mandated by the
>>>> Apache
>>>> Software Foundation release manager role description) to assess
>>>> cherry-pick
>>>> ability of those changes. Release manager - at the sole discretion and
>>>> individual decision (this is the only place in the whole ASF setup
>>>> where a
>>>> single person has such power to make individual decisions) can reject
>>>> some
>>>> of those who other maintainers think should be included. But the release
>>>> manager on his own does not make proposals on what should be included.
>>>>
>>>> *Is this process following the ASF rules?*
>>>>
>>>> I believe yes, The release manager's role is nicely described here:
>>>> https://infra.apache.org/release-publishing.html#releasemanager. And
>>>> there
>>>> is a far more complete description here that describes the whole process
>>>> https://www.apache.org/legal/release-policy.html#management - also
>>>> mentioning that it's the PMC's responsibility (and particularly PMC
>>>> chair's) to adhere to the process.
>>>>
>>>> *What's the role of individual maintainers?*
>>>>
>>>> The role of maintainers (collectively) to propose things for the next
>>>> release. In our case it happens with setting the milestone on a PR.
>>>>
>>>> *When proposed PRs are rejected?*
>>>>
>>>> There are various reasons to reject those - if too complex to
>>>> cherry-pick
>>>> or when the release manager assesses it's a new feature, not a bugfix.
>>>> Essentially (according to semver) when it comes to the user-facing
>>>> changes,
>>>> the PATCHLEVEL release should contain only bugfixes. and may contain
>>>> docs
>>>> changes if they are fixing/improving docs (not about new features) and
>>>> also
>>>> environment/build script changes (so non-user-facing changes) as they
>>>> are
>>>> pretty much always needed to keep the things nicely building - those are
>>>> usually skipped from the changelog as non-user facing).
>>>>
>>>> *Why are provider changes not cherry-picked?*
>>>>
>>>> In our case - basically none of the provider changes are cherry-picked -
>>>> unless they are needed to make the builds work well (sometimes happen).
>>>> Providers are ALWAYS released from the latest main code, not from the
>>>> v2-8-stable branch. In fact all the tests and ci checks for providers
>>>> are
>>>> skipped in the non-main (v2* branches). So yes - not seeing provider
>>>> changes cherry-picked is absolutely expected.
>>>>
>>>> *Do we skip over some changes when releasing a patchlevel release?
>>>> What's
>>>> the purpose of patch-level releases?*
>>>>
>>>> Answering Bolke's question:
>>>>
>>>> > Is that intentional? Ie. is that the purpose of this release. Other
>>>> big(ger) and more recent changes have been included hence asking.
>>>>
>>>> The purpose of that release is as described in SemVer - to give the
>>>> users
>>>> bugfix-only release that has no new features. Of course it's sometimes
>>>> debatable whether changes are features/bugfixes, but we usually use
>>>> #release-management to quickly chat about it, and eventually the release
>>>> manager always makes a comment in the PR when the milestone is changed
>>>> and
>>>> explains the reasoning.
>>>>
>>>> Skipping is not intentional because we never "skip" things when
>>>> cherry-picking, It's *reverse* - those maintainer who think that certain
>>>> bug fixes (or internal changes or sometimes even feature changes that we
>>>> classify really as "bugfix" SHOULD intentionally mark those PRs they
>>>> want
>>>> with 2.8.1 (or basically next patch-level) to be *included. * So there
>>>> is
>>>> no skipping, if maintainer did not deliberately mark PR as upcoming
>>>> milestone, it will just not be included
>>>>
>>>> *Where do some maintainers know about it **from ? *
>>>>
>>>> Because the maintainers who actively participate - either acting as
>>>> release
>>>> managers (those who raised their hand and acted as release managers
>>>> generally speaking) and those who wanted their changes to be part of the
>>>> past releases have been doing it for years. For years this has been
>>>> simply
>>>> followed and discussed in #release-management channel and #development
>>>> (and
>>>> in devlist whenever there were new releases) and generally the
>>>> maintainers
>>>> who took part in those discussions and release process are aware of
>>>> that -
>>>> also many maintainers know the process as it "soaked" in when they were
>>>> just watching.
>>>>
>>>> However recently we've made an attempt to document it (the PR above). So
>>>> you could learn it by reading it (even if it does not have the whole
>>>> context above).
>>>>
>>>> *Why do some people not know about it?*
>>>>
>>>> Not sure. Maybe because they were not interested and never asked? Maybe
>>>> because there was never a long email about it at our devlist, or the
>>>> documentation about it in README was too succinct?
>>>>
>>>> Or maybe we need a better way of communicating it - I am not sure. But I
>>>> hope this email will clarify a lot of it, and will prove valuable when
>>>> searching in devlist.
>>>>
>>>> Maybe even someone who manages to read it all will update the README
>>>> description of ours to explain it better :) and maybe create a nice FAQ
>>>> that we can put in our docs?
>>>>
>>>> I really hope this mail - even if long - will make people a bit more
>>>> aware
>>>> of the process, and if someone has an idea how the "collective"
>>>> awareness
>>>> can be improved - I think it's a good idea to send PR or email (or maybe
>>>> even record a video :) ??) that will be a better way of communicating
>>>> it.
>>>>
>>>> J.
>>>>
>>>>
>>>> On Wed, Jan 17, 2024 at 9:03 AM Bolke de Bruin <bd...@gmail.com>
>>>> wrote:
>>>>
>>>> > Just checking:
>>>> >
>>>> > I see that some improvements to Airflow.io (two weeks ago) were not
>>>> > included and some provider updates neither. Haven't checked anything
>>>> else
>>>> > yet.
>>>> >
>>>> > Is that intentional? Ie. is that the purpose of this release. Other
>>>> > big(ger) and more recent changes have been included hence asking.
>>>> >
>>>> > B.
>>>> >
>>>> > Sent from my iPhone
>>>> >
>>>> > > On 16 Jan 2024, at 20:18, Jarek Potiuk <ja...@potiuk.com> wrote:
>>>> > >
>>>> > > +1 (binding): Checked all my changes, I ran airflow in a few
>>>> > combinations
>>>> > > (MySQL / Postgres Local/Celery executor. It looks and works well -
>>>> run a
>>>> > > few dags and navigated through a number of screens. Checked
>>>> licences,
>>>> > > signatures, checksums, performed a "reproducible build" check and it
>>>> > worked
>>>> > > (with a small glitch explained below).
>>>> > >
>>>> > > BTW. The new "hatchling-built" package looks good and it nicely
>>>> installs
>>>> > > airflow + extras (and it has a far better and cleaner set of extras
>>>> -
>>>> > > finally you can reproducibly install airflow with `all` extra if you
>>>> > > ***REALLY*** want :).
>>>> > >
>>>> > > Reproducibility (also for other PMC members doing the check): I
>>>> found a
>>>> > > small glitch in the "reproducible" part of verifying the packages.
>>>> While
>>>> > > wheel and sdist packages are exactly the same binary-wise, the
>>>> > > source-tarball was binarry different for me. I decompressed it and
>>>> > compared
>>>> > > the content - they are identical - but there is one difference
>>>> which I
>>>> > > overlooked - the group permissions for files in Ephraim's tarball
>>>> are
>>>> > > different from mine. I have totally forgotten about the fact that
>>>> umask
>>>> > > might set different group/other permissions when checking out the
>>>> files
>>>> > > from git. Fix will be coming shortly - in the meantime I recommend
>>>> anyone
>>>> > > who will be doing the comparison to uncompress both tarballs and
>>>> compare
>>>> > > the contents with `diff -r`.
>>>> > >
>>>> > >
>>>> > >
>>>> > >> On Tue, Jan 16, 2024 at 11:30 AM Ephraim Anierobi <
>>>> > >> ephraimanierobi@apache.org> wrote:
>>>> > >>
>>>> > >> Hey fellow Airflowers,
>>>> > >>
>>>> > >> I have cut Airflow 2.8.1rc1. This email is calling a vote on the
>>>> > release,
>>>> > >> which will last at least 72 hours, from Tuesday, January 16, 2024
>>>> at
>>>> > 10:30
>>>> > >> am UTC
>>>> > >> until Friday, January 19, 2024, at 10:30 am UTC
>>>> > >> <
>>>> > >>
>>>> >
>>>> https://www.timeanddate.com/worldclock/fixedtime.html?msg=8&iso=20240119T1030&p1=1440
>>>> > >>> ,
>>>> > >> and until 3 binding +1 votes have been received.
>>>> > >>
>>>> > >> Status of testing of the release is kept at
>>>> > >> https://github.com/apache/airflow/issues/36808
>>>> > >>
>>>> > >> Consider this my (binding) +1.
>>>> > >>
>>>> > >> Airflow 2.8.1rc1 is available at:
>>>> > >> https://dist.apache.org/repos/dist/dev/airflow/2.8.1rc1/
>>>> > >>
>>>> > >> *apache-airflow-2.8.1-source.tar.gz* is a source release that
>>>> comes with
>>>> > >> INSTALL instructions.
>>>> > >> *apache-airflow-2.8.1.tar.gz* is the binary Python "sdist" release.
>>>> > >> *apache_airflow-2.8.1-py3-none-any.whl* is the binary Python wheel
>>>> > "binary"
>>>> > >> release.
>>>> > >>
>>>> > >> Public keys are available at:
>>>> > >> https://dist.apache.org/repos/dist/release/airflow/KEYS
>>>> > >>
>>>> > >> Please vote accordingly:
>>>> > >>
>>>> > >> [ ] +1 approve
>>>> > >> [ ] +0 no opinion
>>>> > >> [ ] -1 disapprove with the reason
>>>> > >>
>>>> > >> Only votes from PMC members are binding, but all members of the
>>>> > community
>>>> > >> are encouraged to test the release and vote with "(non-binding)".
>>>> > >>
>>>> > >> The test procedure for PMC members is described in:
>>>> > >>
>>>> > >>
>>>> >
>>>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-pmc-members
>>>> > >>
>>>> > >> The test procedure for and Contributors who would like to test
>>>> this RC
>>>> > is
>>>> > >> described in:
>>>> > >>
>>>> > >>
>>>> >
>>>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-contributors
>>>> > >>
>>>> > >>
>>>> > >> Please note that the version number excludes the `rcX` string, so
>>>> it's
>>>> > now
>>>> > >> simply 2.8.1. This will allow us to rename the artifact without
>>>> > modifying
>>>> > >> the artifact checksums when we actually release.
>>>> > >>
>>>> > >> Release Notes:
>>>> > >> https://github.com/apache/airflow/blob/2.8.1rc1/RELEASE_NOTES.rst
>>>> > >>
>>>> > >> Changes since 2.8.0:
>>>> > >>
>>>> > >> *Significant Changes*
>>>> > >>
>>>> > >> *Target version for core dependency ``pendulum`` package set to 3
>>>> > >> (#36281).*
>>>> > >> Support for pendulum 2.1.2 will be saved for a while, presumably
>>>> until
>>>> > the
>>>> > >> next feature version of Airflow.
>>>> > >> It is advised to upgrade user code to use pendulum 3 as soon as
>>>> > possible.
>>>> > >>
>>>> > >> *Airflow packaging specification follows modern Python packaging
>>>> > standards
>>>> > >> (#36537).*
>>>> > >> We standardized Airflow dependency configuration to follow latest
>>>> > >> development in Python packaging by
>>>> > >> using ``pyproject.toml``. Airflow is now compliant with those
>>>> accepted
>>>> > >> PEPs:
>>>> > >>
>>>> > >> * `PEP-440 Version Identification and Dependency Specification <
>>>> > >> https://www.python.org/dev/peps/pep-0440/>`__
>>>> > >> * `PEP-517 A build-system independent format for source trees <
>>>> > >> https://www.python.org/dev/peps/pep-0517/>`__
>>>> > >> * `PEP-518 Specifying Minimum Build System Requirements for Python
>>>> > Projects
>>>> > >> <https://www.python.org/dev/peps/pep-0518/>`__
>>>> > >> * `PEP-561 Distributing and Packaging Type Information <
>>>> > >> https://www.python.org/dev/peps/pep-0561/>`__
>>>> > >> * `PEP-621 Storing project metadata in pyproject.toml <
>>>> > >> https://www.python.org/dev/peps/pep-0621/>`__
>>>> > >> * `PEP-660 Editable installs for pyproject.toml based builds (wheel
>>>> > based)
>>>> > >> <
>>>> > >> https://www.python.org/dev/peps/pep-0660/>`__
>>>> > >> * `PEP-685 Comparison of extra names for optional distribution
>>>> > dependencies
>>>> > >> <https://www.python.org/dev/peps/pep-0685/>`__
>>>> > >>
>>>> > >> Also we implement multiple license files support coming from
>>>> Draft, not
>>>> > yet
>>>> > >> accepted (but supported by hatchling) PEP:
>>>> > >> * `PEP 639 Improving License Clarity with Better Package Metadata <
>>>> > >> https://peps.python.org/pep-0639/>`__
>>>> > >>
>>>> > >> This has almost no noticeable impact on users if they are using
>>>> modern
>>>> > >> Python packaging and development tools, generally
>>>> > >> speaking Airflow should behave as it did before when installing it
>>>> from
>>>> > >> PyPI and it should be much easier to install
>>>> > >> it for development purposes using ``pip install -e ".[devel]"``.
>>>> > >>
>>>> > >> The differences from the user side are:
>>>> > >>
>>>> > >> * Airflow extras now get extras normalized to ``-`` (following
>>>> PEP-685)
>>>> > >> instead of ``_`` and ``.``
>>>> > >> (as it was before in some extras). When you install airflow with
>>>> such
>>>> > >> extras (for example ``dbt.core`` or
>>>> > >> ``all_dbs``) you should use ``-`` instead of ``_`` and ``.``.
>>>> > >>
>>>> > >> In most modern tools this will work in backwards-compatible way,
>>>> but in
>>>> > >> some old version of those tools you might need to
>>>> > >> replace ``_`` and ``.`` with ``-``. You can also get warnings that
>>>> the
>>>> > >> extra you are installing does not exist - but usually
>>>> > >> this warning is harmless and the extra is installed anyway. It is,
>>>> > however,
>>>> > >> recommended to change to use ``-`` in extras in your dependency
>>>> > >> specifications for all Airflow extras.
>>>> > >>
>>>> > >> * Released airflow package does not contain ``devel``, ``devel-*``,
>>>> > ``doc``
>>>> > >> and ``doc-gen`` extras.
>>>> > >> Those extras are only available when you install Airflow from
>>>> sources
>>>> > in
>>>> > >> ``--editable`` mode. This is
>>>> > >> because those extras are only used for development and
>>>> documentation
>>>> > >> building purposes and are not needed
>>>> > >> when you install Airflow for production use. Those dependencies
>>>> had
>>>> > >> unspecified and varying behaviour for
>>>> > >> released packages anyway and you were not supposed to use them in
>>>> > >> released packages.
>>>> > >>
>>>> > >> * The ``all`` and ``all-*`` extras were not always working
>>>> correctly
>>>> > when
>>>> > >> installing Airflow using constraints
>>>> > >> because they were also considered as development-only
>>>> dependencies.
>>>> > With
>>>> > >> this change, those dependencies are
>>>> > >> now properly handling constraints and they will install properly
>>>> with
>>>> > >> constraints, pulling the right set
>>>> > >> of providers and dependencies when constraints are used.
>>>> > >>
>>>> > >> *Graphviz dependency is now an optional one, not required one
>>>> (#36647).*
>>>> > >> The ``graphviz`` dependency has been problematic as Airflow
>>>> required
>>>> > >> dependency - especially for
>>>> > >> ARM-based installations. Graphviz packages require binary graphviz
>>>> > >> libraries - which is already a
>>>> > >> limitation, but they also require to install graphviz Python
>>>> bindings
>>>> > to be
>>>> > >> build and installed.
>>>> > >> This does not work for older Linux installation but - more
>>>> importantly -
>>>> > >> when you try to install
>>>> > >> Graphviz libraries for Python 3.8, 3.9 for ARM M1 MacBooks, the
>>>> packages
>>>> > >> fail to install because
>>>> > >> Python bindings compilation for M1 can only work for Python 3.10+.
>>>> > >>
>>>> > >> This is not a breaking change technically - the CLIs to render the
>>>> DAGs
>>>> > is
>>>> > >> still there and IF you
>>>> > >> already have graphviz installed, it will continue working as it did
>>>> > before.
>>>> > >> The only problem when it
>>>> > >> does not work is where you do not have graphviz installed it will
>>>> raise
>>>> > an
>>>> > >> error and inform that you need it.
>>>> > >>
>>>> > >> Graphviz will remain to be installed for most users:
>>>> > >>
>>>> > >> * the Airflow Image will still contain graphviz library, because
>>>> > >> it is added there as extra
>>>> > >> * when previous version of Airflow has been installed already, then
>>>> > >> graphviz library is already installed there and Airflow will
>>>> > >> continue working as it did
>>>> > >>
>>>> > >> The only change will be a new installation of new version of
>>>> Airflow
>>>> > from
>>>> > >> the scratch, where graphviz will
>>>> > >> need to be specified as extra or installed separately in order to
>>>> enable
>>>> > >> DAG rendering option.
>>>> > >>
>>>> > >> *Bug Fixes*
>>>> > >> - Fix airflow-scheduler exiting with code 0 on exceptions (#36800)
>>>> > >> - Fix Callback exception when a removed task is the last one in the
>>>> > >> ``taskinstance`` list (#36693)
>>>> > >> - Allow anonymous user edit/show resource when set
>>>> > >> ``AUTH_ROLE_PUBLIC=admin`` (#36750)
>>>> > >> - Better error message when sqlite URL uses relative path (#36774)
>>>> > >> - Explicit string cast required to force integer-type run_ids to be
>>>> > passed
>>>> > >> as strings instead of integers (#36756)
>>>> > >> - Add log lookup exception for empty ``op`` subtypes (#35536)
>>>> > >> - Remove unused index on task instance (#36737)
>>>> > >> - Fix check on subclass for ``typing.Union`` in
>>>> > ``_infer_multiple_outputs``
>>>> > >> for Python 3.10+ (#36728)
>>>> > >> - Make sure ``multiple_outputs`` is inferred correctly even when
>>>> using
>>>> > >> ``TypedDict`` (#36652)
>>>> > >> - Add back FAB constant in legacy security manager (#36719)
>>>> > >> - Fix AttributeError when using ``Dagrun.update_state`` (#36712)
>>>> > >> - Do not let ``EventsTimetable`` schedule past events if
>>>> > ``catchup=False``
>>>> > >> (#36134)
>>>> > >> - Support encryption for triggers parameters (#36492)
>>>> > >> - Fix the type hint for ``tis_query`` in
>>>> ``_process_executor_events``
>>>> > >> (#36655)
>>>> > >> - Redirect to index when user does not have permission to access a
>>>> page
>>>> > >> (#36623)
>>>> > >> - Avoid using dict as default value in ``call_regular_interval``
>>>> > (#36608)
>>>> > >> - Remove option to set a task instance to running state in UI
>>>> (#36518)
>>>> > >> - Fix details tab not showing when using dynamic task mapping
>>>> (#36522)
>>>> > >> - Raise error when ``DagRun`` fails while running ``dag test``
>>>> (#36517)
>>>> > >> - Refactor ``_manage_executor_state`` by refreshing TIs in batch
>>>> > (#36502)
>>>> > >> - Add flask config: ``MAX_CONTENT_LENGTH`` (#36401)
>>>> > >> - Fix get_leaves calculation for teardown in nested group (#36456)
>>>> > >> - Stop serializing timezone-naive datetime to timezone-aware
>>>> datetime
>>>> > with
>>>> > >> UTC tz (#36379)
>>>> > >> - Make ``kubernetes`` decorator type annotation consistent with
>>>> operator
>>>> > >> (#36405)
>>>> > >> - Fix Webserver returning 500 for POST requests to
>>>> ``api/dag/*/dagrun``
>>>> > >> from anonymous user (#36275)
>>>> > >> - Fix the required access for get_variable endpoint (#36396)
>>>> > >> - Fix datetime reference in ``DAG.is_fixed_time_schedule`` (#36370)
>>>> > >> - Fix AirflowSkipException message raised by BashOperator (#36354)
>>>> > >> - Allow PythonVirtualenvOperator.skip_on_exit_code to be zero
>>>> (#36361)
>>>> > >> - Increase width of execution_date input in trigger.html (#36278)
>>>> > >> - Fix logging for pausing DAG (#36182)
>>>> > >> - Stop deserializing pickle when enable_xcom_pickling is False
>>>> (#36255)
>>>> > >> - Check DAG read permission before accessing DAG code (#36257)
>>>> > >> - Enable mark task as failed/success always (#36254)
>>>> > >> - Create latest log dir symlink as relative link (#36019)
>>>> > >> - Fix Python-based decorators templating (#36103)
>>>> > >>
>>>> > >> *Miscellaneous*
>>>> > >> - Rename concurrency label to max active tasks (#36691)
>>>> > >> - Restore function scoped ``httpx`` import in file_task_handler for
>>>> > >> performance (#36753)
>>>> > >> - Add support of Pendulum 3 (#36281)
>>>> > >> - Standardize airflow build process and switch to Hatchling build
>>>> > backend
>>>> > >> (#36537)
>>>> > >> - Get rid of ``pyarrow-hotfix`` for ``CVE-2023-47248`` (#36697)
>>>> > >> - Make ``graphviz`` dependency optional (#36647)
>>>> > >> - Announce MSSQL support end in Airflow 2.9.0, add migration script
>>>> > hints
>>>> > >> (#36509)
>>>> > >> - Set min ``pandas`` dependency to 1.2.5 for all providers and
>>>> airflow
>>>> > >> (#36698)
>>>> > >> - Bump follow-redirects from 1.15.3 to 1.15.4 in ``/airflow/www``
>>>> > (#36700)
>>>> > >> - Provide the logger_name param to base hook in order to override
>>>> the
>>>> > >> logger name (#36674)
>>>> > >> - Fix run type icon alignment with run type text (#36616)
>>>> > >> - Follow BaseHook connection fields method signature in FSHook
>>>> (#36444)
>>>> > >> - Remove redundant ``docker`` decorator type annotations (#36406)
>>>> > >> - Straighten typing in workday timetable (#36296)
>>>> > >> - Use ``batch_is_authorized_dag`` to check if user has permission
>>>> to
>>>> > read
>>>> > >> DAGs (#36279)
>>>> > >> - Replace deprecated get_accessible_dag_ids and use
>>>> get_readable_dags in
>>>> > >> get_dag_warnings (#36256)
>>>> > >>
>>>> > >> *Doc Only Changes*
>>>> > >> - Metrics tagging documentation (#36627)
>>>> > >> - In docs use logical_date instead of deprecated execution_date
>>>> (#36654)
>>>> > >> - Add section about live-upgrading Airflow (#36637)
>>>> > >> - Replace ``numpy`` example with practical exercise demonstrating
>>>> > top-level
>>>> > >> code (#35097)
>>>> > >> - Improve and add more complete description in the architecture
>>>> diagrams
>>>> > >> (#36513)
>>>> > >> - Improve the error message displayed when there is a webserver
>>>> error
>>>> > >> (#36570)
>>>> > >> - Update ``dags.rst`` with information on DAG pausing (#36540)
>>>> > >> - Update installation prerequisites after upgrading to Debian
>>>> Bookworm
>>>> > >> (#36521)
>>>> > >> - Add description on the ways how users should approach DB
>>>> monitoring
>>>> > >> (#36483)
>>>> > >> - Add branching based on mapped task group example to
>>>> > >> dynamic-task-mapping.rst (#36480)
>>>> > >> - Add further details to replacement documentation (#36485)
>>>> > >> - Use cards when describing priority weighting methods (#36411)
>>>> > >> - Update ``metrics.rst`` for param ``dagrun.schedule_delay``
>>>> (#36404)
>>>> > >> - Update admonitions in Python operator doc to reflect sentiment
>>>> > (#36340)
>>>> > >> - Improve audit_logs.rst (#36213)
>>>> > >> - Remove Redshift mention from the list of managed Postgres
>>>> backends
>>>> > >> (#36217)
>>>> > >>
>>>> > >> Cheers,
>>>> > >> Ephraim
>>>> > >>
>>>> >
>>>> > ---------------------------------------------------------------------
>>>> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>> > For additional commands, e-mail: dev-help@airflow.apache.org
>>>> >
>>>> >
>>>>
>>>
Re: What goes into the next release (was [VOTE] Release Airflow 2.8.1 from 2.8.1rc1)
Posted by Jarek Potiuk <ja...@potiuk.com>.
Created PR https://github.com/apache/airflow/pull/36858 where I proposed a
separate document - with a bit more polished version of what I wrote above,
interlinked with the succinct README chapter we already have.
I also expanded there a bit "What's the purpose of patch-level release"
explaining the approach we take for classifying changes as breaking/not
breaking and adding comment on SemVer being "intentional" and not
"technically breaking non/breaking" - something that we discussed a lot at
many occasions in multiple PRs when we debated whether things are
breaking/not breaking.
And removed "how some people know/don't know" about it. I hope having that
FAQ will help us to get to the point where we all know it - or at least
can easily find it when we are looking for it.
J.
On Thu, Jan 18, 2024 at 8:42 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> I think if anything - that should be a separate page in our GitHub -
> which will be a) easier to do b) maybe indeed makes sense c) this is where
> we put "developer" docs.
>
> I think we agreed already quite some time ago that "airflow.apache.org"
> is generally for the users and all the docs for contributors/maintainers
> should go as .md/.rst files in our repository, but not into the airflow
> website.
>
> Yeah - I think it might be worth it if we keep it separate from README :).
> I think we had far too long README and (as mentioned above) over the last
> year or so it got shorter and shorter, while it gained only the "super
> important" information IMHO. So yeah having a separate FAQ taking my mail
> as context might be a good idea.
>
> I will open PR for that and if others agree it's a good idea (and help
> with reviewing it, we can merge it).
>
> J.
>
>
>
>
>
>
>
> On Thu, Jan 18, 2024 at 5:29 AM Amogh Desai <am...@gmail.com>
> wrote:
>
>> Thanks @Jarek Potiuk <ja...@potiuk.com>!
>>
>> I vaguely knew the process but not in such detail, thanks for putting it
>> together in this email.
>> I will visit the document
>> https://github.com/apache/airflow?tab=readme-ov-file#what-goes-into-the-next-release
>> and if I find any clarifications, I will send it across as a PR.
>>
>> One suggestion: what if we put it together in a document and put it on
>> the airflow website under
>> https://airflow.apache.org/community/ where we mention "how we release"?
>>
>> Might be a bit too much given the context people will have while
>> visiting the airflow website..
>>
>> Thanks & Regards,
>> Amogh Desai
>>
>> On Wed, Jan 17, 2024 at 6:13 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>>> I separate it out, because it seems that despite the efforts to explain
>>> and
>>> document how our releases work It's not clear even for the PMC chair, so
>>> likely it warrants a separate thread - also it will be easier to find it
>>> in
>>> the archives this way.
>>>
>>> I think this is an important topic that all maintainers should be aware
>>> of,
>>> so let me take a task to explain it in a longer email (and separate
>>> thread).
>>>
>>> I created it in a form of FAQ, because it seems that similar questions
>>> might be asked by others.
>>>
>>> *Do we have a process defined here?*
>>>
>>> Answering Bolke's question - who was apparently confused what our process
>>> is:
>>>
>>> > I see that some improvements to Airflow.io (two weeks ago) were not
>>> included and some provider updates neither. Haven't checked anything
>>> else
>>> yet.
>>>
>>> Apparently there is some confusion about the process, but yes we have a
>>> well defined and well tested (pretty much 4 years now) process that we
>>> follow., We follow it since I remember actually - it's been also done the
>>> same in 1.10 - but with some variations, Likely we do it the same way
>>> since
>>> the beginning of 2.0, but it has been refined and improved over time - by
>>> those who volunteered their time in the release process (a lot of ad-hoc
>>> discussion have been traditionally happening in #release-management slack
>>> channel) and as of few months ago we even documented it (It was in
>>> November
>>> 2023) - with this PR https://github.com/apache/airflow/pull/35245
>>>
>>> It is currently described in a prominent place in our most important (and
>>> over the last year or so the README, we made the README pretty short and
>>> contains only super-important information on how Airflow is developed)
>>> README.md under *"What goes into the next release?"* chapter:
>>>
>>>
>>> https://github.com/apache/airflow?tab=readme-ov-file#what-goes-into-the-next-release
>>> .
>>>
>>> *How does the selection process for cherry-picking work?*
>>>
>>> In short (and this is the most important thing that every maintainer
>>> should
>>> be aware of): *those maintainers who think that issue should be included
>>> should mark it with the next (in this case 2.8.1) milestone*. It's up to
>>> individual maintainers who want to include certain changes to take care
>>> about it and mark the issues they think are bug fixes, to go into the
>>> next
>>> release
>>>
>>> This is the only thing that the maintainer has to do to get the PR
>>> proposed
>>> to be considered in the next patchlevel release. Sometimes - if
>>> controversial - maintainers discuss the proposals in #release-management
>>> channel, sometimes in #development or in the PR itself (especially if the
>>> release manager decides to not include it and changes the milestone (and
>>> explains why).
>>>
>>> *What's the release manager's role ?*
>>>
>>> Release manager's job is purely mechanical (as also mandated by the
>>> Apache
>>> Software Foundation release manager role description) to assess
>>> cherry-pick
>>> ability of those changes. Release manager - at the sole discretion and
>>> individual decision (this is the only place in the whole ASF setup where
>>> a
>>> single person has such power to make individual decisions) can reject
>>> some
>>> of those who other maintainers think should be included. But the release
>>> manager on his own does not make proposals on what should be included.
>>>
>>> *Is this process following the ASF rules?*
>>>
>>> I believe yes, The release manager's role is nicely described here:
>>> https://infra.apache.org/release-publishing.html#releasemanager. And
>>> there
>>> is a far more complete description here that describes the whole process
>>> https://www.apache.org/legal/release-policy.html#management - also
>>> mentioning that it's the PMC's responsibility (and particularly PMC
>>> chair's) to adhere to the process.
>>>
>>> *What's the role of individual maintainers?*
>>>
>>> The role of maintainers (collectively) to propose things for the next
>>> release. In our case it happens with setting the milestone on a PR.
>>>
>>> *When proposed PRs are rejected?*
>>>
>>> There are various reasons to reject those - if too complex to cherry-pick
>>> or when the release manager assesses it's a new feature, not a bugfix.
>>> Essentially (according to semver) when it comes to the user-facing
>>> changes,
>>> the PATCHLEVEL release should contain only bugfixes. and may contain docs
>>> changes if they are fixing/improving docs (not about new features) and
>>> also
>>> environment/build script changes (so non-user-facing changes) as they are
>>> pretty much always needed to keep the things nicely building - those are
>>> usually skipped from the changelog as non-user facing).
>>>
>>> *Why are provider changes not cherry-picked?*
>>>
>>> In our case - basically none of the provider changes are cherry-picked -
>>> unless they are needed to make the builds work well (sometimes happen).
>>> Providers are ALWAYS released from the latest main code, not from the
>>> v2-8-stable branch. In fact all the tests and ci checks for providers are
>>> skipped in the non-main (v2* branches). So yes - not seeing provider
>>> changes cherry-picked is absolutely expected.
>>>
>>> *Do we skip over some changes when releasing a patchlevel release? What's
>>> the purpose of patch-level releases?*
>>>
>>> Answering Bolke's question:
>>>
>>> > Is that intentional? Ie. is that the purpose of this release. Other
>>> big(ger) and more recent changes have been included hence asking.
>>>
>>> The purpose of that release is as described in SemVer - to give the users
>>> bugfix-only release that has no new features. Of course it's sometimes
>>> debatable whether changes are features/bugfixes, but we usually use
>>> #release-management to quickly chat about it, and eventually the release
>>> manager always makes a comment in the PR when the milestone is changed
>>> and
>>> explains the reasoning.
>>>
>>> Skipping is not intentional because we never "skip" things when
>>> cherry-picking, It's *reverse* - those maintainer who think that certain
>>> bug fixes (or internal changes or sometimes even feature changes that we
>>> classify really as "bugfix" SHOULD intentionally mark those PRs they want
>>> with 2.8.1 (or basically next patch-level) to be *included. * So there is
>>> no skipping, if maintainer did not deliberately mark PR as upcoming
>>> milestone, it will just not be included
>>>
>>> *Where do some maintainers know about it **from ? *
>>>
>>> Because the maintainers who actively participate - either acting as
>>> release
>>> managers (those who raised their hand and acted as release managers
>>> generally speaking) and those who wanted their changes to be part of the
>>> past releases have been doing it for years. For years this has been
>>> simply
>>> followed and discussed in #release-management channel and #development
>>> (and
>>> in devlist whenever there were new releases) and generally the
>>> maintainers
>>> who took part in those discussions and release process are aware of that
>>> -
>>> also many maintainers know the process as it "soaked" in when they were
>>> just watching.
>>>
>>> However recently we've made an attempt to document it (the PR above). So
>>> you could learn it by reading it (even if it does not have the whole
>>> context above).
>>>
>>> *Why do some people not know about it?*
>>>
>>> Not sure. Maybe because they were not interested and never asked? Maybe
>>> because there was never a long email about it at our devlist, or the
>>> documentation about it in README was too succinct?
>>>
>>> Or maybe we need a better way of communicating it - I am not sure. But I
>>> hope this email will clarify a lot of it, and will prove valuable when
>>> searching in devlist.
>>>
>>> Maybe even someone who manages to read it all will update the README
>>> description of ours to explain it better :) and maybe create a nice FAQ
>>> that we can put in our docs?
>>>
>>> I really hope this mail - even if long - will make people a bit more
>>> aware
>>> of the process, and if someone has an idea how the "collective" awareness
>>> can be improved - I think it's a good idea to send PR or email (or maybe
>>> even record a video :) ??) that will be a better way of communicating it.
>>>
>>> J.
>>>
>>>
>>> On Wed, Jan 17, 2024 at 9:03 AM Bolke de Bruin <bd...@gmail.com>
>>> wrote:
>>>
>>> > Just checking:
>>> >
>>> > I see that some improvements to Airflow.io (two weeks ago) were not
>>> > included and some provider updates neither. Haven't checked anything
>>> else
>>> > yet.
>>> >
>>> > Is that intentional? Ie. is that the purpose of this release. Other
>>> > big(ger) and more recent changes have been included hence asking.
>>> >
>>> > B.
>>> >
>>> > Sent from my iPhone
>>> >
>>> > > On 16 Jan 2024, at 20:18, Jarek Potiuk <ja...@potiuk.com> wrote:
>>> > >
>>> > > +1 (binding): Checked all my changes, I ran airflow in a few
>>> > combinations
>>> > > (MySQL / Postgres Local/Celery executor. It looks and works well -
>>> run a
>>> > > few dags and navigated through a number of screens. Checked licences,
>>> > > signatures, checksums, performed a "reproducible build" check and it
>>> > worked
>>> > > (with a small glitch explained below).
>>> > >
>>> > > BTW. The new "hatchling-built" package looks good and it nicely
>>> installs
>>> > > airflow + extras (and it has a far better and cleaner set of extras -
>>> > > finally you can reproducibly install airflow with `all` extra if you
>>> > > ***REALLY*** want :).
>>> > >
>>> > > Reproducibility (also for other PMC members doing the check): I
>>> found a
>>> > > small glitch in the "reproducible" part of verifying the packages.
>>> While
>>> > > wheel and sdist packages are exactly the same binary-wise, the
>>> > > source-tarball was binarry different for me. I decompressed it and
>>> > compared
>>> > > the content - they are identical - but there is one difference which
>>> I
>>> > > overlooked - the group permissions for files in Ephraim's tarball are
>>> > > different from mine. I have totally forgotten about the fact that
>>> umask
>>> > > might set different group/other permissions when checking out the
>>> files
>>> > > from git. Fix will be coming shortly - in the meantime I recommend
>>> anyone
>>> > > who will be doing the comparison to uncompress both tarballs and
>>> compare
>>> > > the contents with `diff -r`.
>>> > >
>>> > >
>>> > >
>>> > >> On Tue, Jan 16, 2024 at 11:30 AM Ephraim Anierobi <
>>> > >> ephraimanierobi@apache.org> wrote:
>>> > >>
>>> > >> Hey fellow Airflowers,
>>> > >>
>>> > >> I have cut Airflow 2.8.1rc1. This email is calling a vote on the
>>> > release,
>>> > >> which will last at least 72 hours, from Tuesday, January 16, 2024 at
>>> > 10:30
>>> > >> am UTC
>>> > >> until Friday, January 19, 2024, at 10:30 am UTC
>>> > >> <
>>> > >>
>>> >
>>> https://www.timeanddate.com/worldclock/fixedtime.html?msg=8&iso=20240119T1030&p1=1440
>>> > >>> ,
>>> > >> and until 3 binding +1 votes have been received.
>>> > >>
>>> > >> Status of testing of the release is kept at
>>> > >> https://github.com/apache/airflow/issues/36808
>>> > >>
>>> > >> Consider this my (binding) +1.
>>> > >>
>>> > >> Airflow 2.8.1rc1 is available at:
>>> > >> https://dist.apache.org/repos/dist/dev/airflow/2.8.1rc1/
>>> > >>
>>> > >> *apache-airflow-2.8.1-source.tar.gz* is a source release that comes
>>> with
>>> > >> INSTALL instructions.
>>> > >> *apache-airflow-2.8.1.tar.gz* is the binary Python "sdist" release.
>>> > >> *apache_airflow-2.8.1-py3-none-any.whl* is the binary Python wheel
>>> > "binary"
>>> > >> release.
>>> > >>
>>> > >> Public keys are available at:
>>> > >> https://dist.apache.org/repos/dist/release/airflow/KEYS
>>> > >>
>>> > >> Please vote accordingly:
>>> > >>
>>> > >> [ ] +1 approve
>>> > >> [ ] +0 no opinion
>>> > >> [ ] -1 disapprove with the reason
>>> > >>
>>> > >> Only votes from PMC members are binding, but all members of the
>>> > community
>>> > >> are encouraged to test the release and vote with "(non-binding)".
>>> > >>
>>> > >> The test procedure for PMC members is described in:
>>> > >>
>>> > >>
>>> >
>>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-pmc-members
>>> > >>
>>> > >> The test procedure for and Contributors who would like to test this
>>> RC
>>> > is
>>> > >> described in:
>>> > >>
>>> > >>
>>> >
>>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-contributors
>>> > >>
>>> > >>
>>> > >> Please note that the version number excludes the `rcX` string, so
>>> it's
>>> > now
>>> > >> simply 2.8.1. This will allow us to rename the artifact without
>>> > modifying
>>> > >> the artifact checksums when we actually release.
>>> > >>
>>> > >> Release Notes:
>>> > >> https://github.com/apache/airflow/blob/2.8.1rc1/RELEASE_NOTES.rst
>>> > >>
>>> > >> Changes since 2.8.0:
>>> > >>
>>> > >> *Significant Changes*
>>> > >>
>>> > >> *Target version for core dependency ``pendulum`` package set to 3
>>> > >> (#36281).*
>>> > >> Support for pendulum 2.1.2 will be saved for a while, presumably
>>> until
>>> > the
>>> > >> next feature version of Airflow.
>>> > >> It is advised to upgrade user code to use pendulum 3 as soon as
>>> > possible.
>>> > >>
>>> > >> *Airflow packaging specification follows modern Python packaging
>>> > standards
>>> > >> (#36537).*
>>> > >> We standardized Airflow dependency configuration to follow latest
>>> > >> development in Python packaging by
>>> > >> using ``pyproject.toml``. Airflow is now compliant with those
>>> accepted
>>> > >> PEPs:
>>> > >>
>>> > >> * `PEP-440 Version Identification and Dependency Specification <
>>> > >> https://www.python.org/dev/peps/pep-0440/>`__
>>> > >> * `PEP-517 A build-system independent format for source trees <
>>> > >> https://www.python.org/dev/peps/pep-0517/>`__
>>> > >> * `PEP-518 Specifying Minimum Build System Requirements for Python
>>> > Projects
>>> > >> <https://www.python.org/dev/peps/pep-0518/>`__
>>> > >> * `PEP-561 Distributing and Packaging Type Information <
>>> > >> https://www.python.org/dev/peps/pep-0561/>`__
>>> > >> * `PEP-621 Storing project metadata in pyproject.toml <
>>> > >> https://www.python.org/dev/peps/pep-0621/>`__
>>> > >> * `PEP-660 Editable installs for pyproject.toml based builds (wheel
>>> > based)
>>> > >> <
>>> > >> https://www.python.org/dev/peps/pep-0660/>`__
>>> > >> * `PEP-685 Comparison of extra names for optional distribution
>>> > dependencies
>>> > >> <https://www.python.org/dev/peps/pep-0685/>`__
>>> > >>
>>> > >> Also we implement multiple license files support coming from Draft,
>>> not
>>> > yet
>>> > >> accepted (but supported by hatchling) PEP:
>>> > >> * `PEP 639 Improving License Clarity with Better Package Metadata <
>>> > >> https://peps.python.org/pep-0639/>`__
>>> > >>
>>> > >> This has almost no noticeable impact on users if they are using
>>> modern
>>> > >> Python packaging and development tools, generally
>>> > >> speaking Airflow should behave as it did before when installing it
>>> from
>>> > >> PyPI and it should be much easier to install
>>> > >> it for development purposes using ``pip install -e ".[devel]"``.
>>> > >>
>>> > >> The differences from the user side are:
>>> > >>
>>> > >> * Airflow extras now get extras normalized to ``-`` (following
>>> PEP-685)
>>> > >> instead of ``_`` and ``.``
>>> > >> (as it was before in some extras). When you install airflow with
>>> such
>>> > >> extras (for example ``dbt.core`` or
>>> > >> ``all_dbs``) you should use ``-`` instead of ``_`` and ``.``.
>>> > >>
>>> > >> In most modern tools this will work in backwards-compatible way,
>>> but in
>>> > >> some old version of those tools you might need to
>>> > >> replace ``_`` and ``.`` with ``-``. You can also get warnings that
>>> the
>>> > >> extra you are installing does not exist - but usually
>>> > >> this warning is harmless and the extra is installed anyway. It is,
>>> > however,
>>> > >> recommended to change to use ``-`` in extras in your dependency
>>> > >> specifications for all Airflow extras.
>>> > >>
>>> > >> * Released airflow package does not contain ``devel``, ``devel-*``,
>>> > ``doc``
>>> > >> and ``doc-gen`` extras.
>>> > >> Those extras are only available when you install Airflow from
>>> sources
>>> > in
>>> > >> ``--editable`` mode. This is
>>> > >> because those extras are only used for development and
>>> documentation
>>> > >> building purposes and are not needed
>>> > >> when you install Airflow for production use. Those dependencies had
>>> > >> unspecified and varying behaviour for
>>> > >> released packages anyway and you were not supposed to use them in
>>> > >> released packages.
>>> > >>
>>> > >> * The ``all`` and ``all-*`` extras were not always working correctly
>>> > when
>>> > >> installing Airflow using constraints
>>> > >> because they were also considered as development-only dependencies.
>>> > With
>>> > >> this change, those dependencies are
>>> > >> now properly handling constraints and they will install properly
>>> with
>>> > >> constraints, pulling the right set
>>> > >> of providers and dependencies when constraints are used.
>>> > >>
>>> > >> *Graphviz dependency is now an optional one, not required one
>>> (#36647).*
>>> > >> The ``graphviz`` dependency has been problematic as Airflow required
>>> > >> dependency - especially for
>>> > >> ARM-based installations. Graphviz packages require binary graphviz
>>> > >> libraries - which is already a
>>> > >> limitation, but they also require to install graphviz Python
>>> bindings
>>> > to be
>>> > >> build and installed.
>>> > >> This does not work for older Linux installation but - more
>>> importantly -
>>> > >> when you try to install
>>> > >> Graphviz libraries for Python 3.8, 3.9 for ARM M1 MacBooks, the
>>> packages
>>> > >> fail to install because
>>> > >> Python bindings compilation for M1 can only work for Python 3.10+.
>>> > >>
>>> > >> This is not a breaking change technically - the CLIs to render the
>>> DAGs
>>> > is
>>> > >> still there and IF you
>>> > >> already have graphviz installed, it will continue working as it did
>>> > before.
>>> > >> The only problem when it
>>> > >> does not work is where you do not have graphviz installed it will
>>> raise
>>> > an
>>> > >> error and inform that you need it.
>>> > >>
>>> > >> Graphviz will remain to be installed for most users:
>>> > >>
>>> > >> * the Airflow Image will still contain graphviz library, because
>>> > >> it is added there as extra
>>> > >> * when previous version of Airflow has been installed already, then
>>> > >> graphviz library is already installed there and Airflow will
>>> > >> continue working as it did
>>> > >>
>>> > >> The only change will be a new installation of new version of Airflow
>>> > from
>>> > >> the scratch, where graphviz will
>>> > >> need to be specified as extra or installed separately in order to
>>> enable
>>> > >> DAG rendering option.
>>> > >>
>>> > >> *Bug Fixes*
>>> > >> - Fix airflow-scheduler exiting with code 0 on exceptions (#36800)
>>> > >> - Fix Callback exception when a removed task is the last one in the
>>> > >> ``taskinstance`` list (#36693)
>>> > >> - Allow anonymous user edit/show resource when set
>>> > >> ``AUTH_ROLE_PUBLIC=admin`` (#36750)
>>> > >> - Better error message when sqlite URL uses relative path (#36774)
>>> > >> - Explicit string cast required to force integer-type run_ids to be
>>> > passed
>>> > >> as strings instead of integers (#36756)
>>> > >> - Add log lookup exception for empty ``op`` subtypes (#35536)
>>> > >> - Remove unused index on task instance (#36737)
>>> > >> - Fix check on subclass for ``typing.Union`` in
>>> > ``_infer_multiple_outputs``
>>> > >> for Python 3.10+ (#36728)
>>> > >> - Make sure ``multiple_outputs`` is inferred correctly even when
>>> using
>>> > >> ``TypedDict`` (#36652)
>>> > >> - Add back FAB constant in legacy security manager (#36719)
>>> > >> - Fix AttributeError when using ``Dagrun.update_state`` (#36712)
>>> > >> - Do not let ``EventsTimetable`` schedule past events if
>>> > ``catchup=False``
>>> > >> (#36134)
>>> > >> - Support encryption for triggers parameters (#36492)
>>> > >> - Fix the type hint for ``tis_query`` in
>>> ``_process_executor_events``
>>> > >> (#36655)
>>> > >> - Redirect to index when user does not have permission to access a
>>> page
>>> > >> (#36623)
>>> > >> - Avoid using dict as default value in ``call_regular_interval``
>>> > (#36608)
>>> > >> - Remove option to set a task instance to running state in UI
>>> (#36518)
>>> > >> - Fix details tab not showing when using dynamic task mapping
>>> (#36522)
>>> > >> - Raise error when ``DagRun`` fails while running ``dag test``
>>> (#36517)
>>> > >> - Refactor ``_manage_executor_state`` by refreshing TIs in batch
>>> > (#36502)
>>> > >> - Add flask config: ``MAX_CONTENT_LENGTH`` (#36401)
>>> > >> - Fix get_leaves calculation for teardown in nested group (#36456)
>>> > >> - Stop serializing timezone-naive datetime to timezone-aware
>>> datetime
>>> > with
>>> > >> UTC tz (#36379)
>>> > >> - Make ``kubernetes`` decorator type annotation consistent with
>>> operator
>>> > >> (#36405)
>>> > >> - Fix Webserver returning 500 for POST requests to
>>> ``api/dag/*/dagrun``
>>> > >> from anonymous user (#36275)
>>> > >> - Fix the required access for get_variable endpoint (#36396)
>>> > >> - Fix datetime reference in ``DAG.is_fixed_time_schedule`` (#36370)
>>> > >> - Fix AirflowSkipException message raised by BashOperator (#36354)
>>> > >> - Allow PythonVirtualenvOperator.skip_on_exit_code to be zero
>>> (#36361)
>>> > >> - Increase width of execution_date input in trigger.html (#36278)
>>> > >> - Fix logging for pausing DAG (#36182)
>>> > >> - Stop deserializing pickle when enable_xcom_pickling is False
>>> (#36255)
>>> > >> - Check DAG read permission before accessing DAG code (#36257)
>>> > >> - Enable mark task as failed/success always (#36254)
>>> > >> - Create latest log dir symlink as relative link (#36019)
>>> > >> - Fix Python-based decorators templating (#36103)
>>> > >>
>>> > >> *Miscellaneous*
>>> > >> - Rename concurrency label to max active tasks (#36691)
>>> > >> - Restore function scoped ``httpx`` import in file_task_handler for
>>> > >> performance (#36753)
>>> > >> - Add support of Pendulum 3 (#36281)
>>> > >> - Standardize airflow build process and switch to Hatchling build
>>> > backend
>>> > >> (#36537)
>>> > >> - Get rid of ``pyarrow-hotfix`` for ``CVE-2023-47248`` (#36697)
>>> > >> - Make ``graphviz`` dependency optional (#36647)
>>> > >> - Announce MSSQL support end in Airflow 2.9.0, add migration script
>>> > hints
>>> > >> (#36509)
>>> > >> - Set min ``pandas`` dependency to 1.2.5 for all providers and
>>> airflow
>>> > >> (#36698)
>>> > >> - Bump follow-redirects from 1.15.3 to 1.15.4 in ``/airflow/www``
>>> > (#36700)
>>> > >> - Provide the logger_name param to base hook in order to override
>>> the
>>> > >> logger name (#36674)
>>> > >> - Fix run type icon alignment with run type text (#36616)
>>> > >> - Follow BaseHook connection fields method signature in FSHook
>>> (#36444)
>>> > >> - Remove redundant ``docker`` decorator type annotations (#36406)
>>> > >> - Straighten typing in workday timetable (#36296)
>>> > >> - Use ``batch_is_authorized_dag`` to check if user has permission to
>>> > read
>>> > >> DAGs (#36279)
>>> > >> - Replace deprecated get_accessible_dag_ids and use
>>> get_readable_dags in
>>> > >> get_dag_warnings (#36256)
>>> > >>
>>> > >> *Doc Only Changes*
>>> > >> - Metrics tagging documentation (#36627)
>>> > >> - In docs use logical_date instead of deprecated execution_date
>>> (#36654)
>>> > >> - Add section about live-upgrading Airflow (#36637)
>>> > >> - Replace ``numpy`` example with practical exercise demonstrating
>>> > top-level
>>> > >> code (#35097)
>>> > >> - Improve and add more complete description in the architecture
>>> diagrams
>>> > >> (#36513)
>>> > >> - Improve the error message displayed when there is a webserver
>>> error
>>> > >> (#36570)
>>> > >> - Update ``dags.rst`` with information on DAG pausing (#36540)
>>> > >> - Update installation prerequisites after upgrading to Debian
>>> Bookworm
>>> > >> (#36521)
>>> > >> - Add description on the ways how users should approach DB
>>> monitoring
>>> > >> (#36483)
>>> > >> - Add branching based on mapped task group example to
>>> > >> dynamic-task-mapping.rst (#36480)
>>> > >> - Add further details to replacement documentation (#36485)
>>> > >> - Use cards when describing priority weighting methods (#36411)
>>> > >> - Update ``metrics.rst`` for param ``dagrun.schedule_delay``
>>> (#36404)
>>> > >> - Update admonitions in Python operator doc to reflect sentiment
>>> > (#36340)
>>> > >> - Improve audit_logs.rst (#36213)
>>> > >> - Remove Redshift mention from the list of managed Postgres backends
>>> > >> (#36217)
>>> > >>
>>> > >> Cheers,
>>> > >> Ephraim
>>> > >>
>>> >
>>> > ---------------------------------------------------------------------
>>> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>> > For additional commands, e-mail: dev-help@airflow.apache.org
>>> >
>>> >
>>>
>>
Re: What goes into the next release (was [VOTE] Release Airflow 2.8.1 from 2.8.1rc1)
Posted by Jarek Potiuk <ja...@potiuk.com>.
I think if anything - that should be a separate page in our GitHub -
which will be a) easier to do b) maybe indeed makes sense c) this is where
we put "developer" docs.
I think we agreed already quite some time ago that "airflow.apache.org" is
generally for the users and all the docs for contributors/maintainers
should go as .md/.rst files in our repository, but not into the airflow
website.
Yeah - I think it might be worth it if we keep it separate from README :).
I think we had far too long README and (as mentioned above) over the last
year or so it got shorter and shorter, while it gained only the "super
important" information IMHO. So yeah having a separate FAQ taking my mail
as context might be a good idea.
I will open PR for that and if others agree it's a good idea (and help with
reviewing it, we can merge it).
J.
On Thu, Jan 18, 2024 at 5:29 AM Amogh Desai <am...@gmail.com>
wrote:
> Thanks @Jarek Potiuk <ja...@potiuk.com>!
>
> I vaguely knew the process but not in such detail, thanks for putting it
> together in this email.
> I will visit the document
> https://github.com/apache/airflow?tab=readme-ov-file#what-goes-into-the-next-release
> and if I find any clarifications, I will send it across as a PR.
>
> One suggestion: what if we put it together in a document and put it on the
> airflow website under
> https://airflow.apache.org/community/ where we mention "how we release"?
>
> Might be a bit too much given the context people will have while
> visiting the airflow website..
>
> Thanks & Regards,
> Amogh Desai
>
> On Wed, Jan 17, 2024 at 6:13 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> I separate it out, because it seems that despite the efforts to explain
>> and
>> document how our releases work It's not clear even for the PMC chair, so
>> likely it warrants a separate thread - also it will be easier to find it
>> in
>> the archives this way.
>>
>> I think this is an important topic that all maintainers should be aware
>> of,
>> so let me take a task to explain it in a longer email (and separate
>> thread).
>>
>> I created it in a form of FAQ, because it seems that similar questions
>> might be asked by others.
>>
>> *Do we have a process defined here?*
>>
>> Answering Bolke's question - who was apparently confused what our process
>> is:
>>
>> > I see that some improvements to Airflow.io (two weeks ago) were not
>> included and some provider updates neither. Haven't checked anything else
>> yet.
>>
>> Apparently there is some confusion about the process, but yes we have a
>> well defined and well tested (pretty much 4 years now) process that we
>> follow., We follow it since I remember actually - it's been also done the
>> same in 1.10 - but with some variations, Likely we do it the same way
>> since
>> the beginning of 2.0, but it has been refined and improved over time - by
>> those who volunteered their time in the release process (a lot of ad-hoc
>> discussion have been traditionally happening in #release-management slack
>> channel) and as of few months ago we even documented it (It was in
>> November
>> 2023) - with this PR https://github.com/apache/airflow/pull/35245
>>
>> It is currently described in a prominent place in our most important (and
>> over the last year or so the README, we made the README pretty short and
>> contains only super-important information on how Airflow is developed)
>> README.md under *"What goes into the next release?"* chapter:
>>
>>
>> https://github.com/apache/airflow?tab=readme-ov-file#what-goes-into-the-next-release
>> .
>>
>> *How does the selection process for cherry-picking work?*
>>
>> In short (and this is the most important thing that every maintainer
>> should
>> be aware of): *those maintainers who think that issue should be included
>> should mark it with the next (in this case 2.8.1) milestone*. It's up to
>> individual maintainers who want to include certain changes to take care
>> about it and mark the issues they think are bug fixes, to go into the next
>> release
>>
>> This is the only thing that the maintainer has to do to get the PR
>> proposed
>> to be considered in the next patchlevel release. Sometimes - if
>> controversial - maintainers discuss the proposals in #release-management
>> channel, sometimes in #development or in the PR itself (especially if the
>> release manager decides to not include it and changes the milestone (and
>> explains why).
>>
>> *What's the release manager's role ?*
>>
>> Release manager's job is purely mechanical (as also mandated by the Apache
>> Software Foundation release manager role description) to assess
>> cherry-pick
>> ability of those changes. Release manager - at the sole discretion and
>> individual decision (this is the only place in the whole ASF setup where a
>> single person has such power to make individual decisions) can reject some
>> of those who other maintainers think should be included. But the release
>> manager on his own does not make proposals on what should be included.
>>
>> *Is this process following the ASF rules?*
>>
>> I believe yes, The release manager's role is nicely described here:
>> https://infra.apache.org/release-publishing.html#releasemanager. And
>> there
>> is a far more complete description here that describes the whole process
>> https://www.apache.org/legal/release-policy.html#management - also
>> mentioning that it's the PMC's responsibility (and particularly PMC
>> chair's) to adhere to the process.
>>
>> *What's the role of individual maintainers?*
>>
>> The role of maintainers (collectively) to propose things for the next
>> release. In our case it happens with setting the milestone on a PR.
>>
>> *When proposed PRs are rejected?*
>>
>> There are various reasons to reject those - if too complex to cherry-pick
>> or when the release manager assesses it's a new feature, not a bugfix.
>> Essentially (according to semver) when it comes to the user-facing
>> changes,
>> the PATCHLEVEL release should contain only bugfixes. and may contain docs
>> changes if they are fixing/improving docs (not about new features) and
>> also
>> environment/build script changes (so non-user-facing changes) as they are
>> pretty much always needed to keep the things nicely building - those are
>> usually skipped from the changelog as non-user facing).
>>
>> *Why are provider changes not cherry-picked?*
>>
>> In our case - basically none of the provider changes are cherry-picked -
>> unless they are needed to make the builds work well (sometimes happen).
>> Providers are ALWAYS released from the latest main code, not from the
>> v2-8-stable branch. In fact all the tests and ci checks for providers are
>> skipped in the non-main (v2* branches). So yes - not seeing provider
>> changes cherry-picked is absolutely expected.
>>
>> *Do we skip over some changes when releasing a patchlevel release? What's
>> the purpose of patch-level releases?*
>>
>> Answering Bolke's question:
>>
>> > Is that intentional? Ie. is that the purpose of this release. Other
>> big(ger) and more recent changes have been included hence asking.
>>
>> The purpose of that release is as described in SemVer - to give the users
>> bugfix-only release that has no new features. Of course it's sometimes
>> debatable whether changes are features/bugfixes, but we usually use
>> #release-management to quickly chat about it, and eventually the release
>> manager always makes a comment in the PR when the milestone is changed and
>> explains the reasoning.
>>
>> Skipping is not intentional because we never "skip" things when
>> cherry-picking, It's *reverse* - those maintainer who think that certain
>> bug fixes (or internal changes or sometimes even feature changes that we
>> classify really as "bugfix" SHOULD intentionally mark those PRs they want
>> with 2.8.1 (or basically next patch-level) to be *included. * So there is
>> no skipping, if maintainer did not deliberately mark PR as upcoming
>> milestone, it will just not be included
>>
>> *Where do some maintainers know about it **from ? *
>>
>> Because the maintainers who actively participate - either acting as
>> release
>> managers (those who raised their hand and acted as release managers
>> generally speaking) and those who wanted their changes to be part of the
>> past releases have been doing it for years. For years this has been simply
>> followed and discussed in #release-management channel and #development
>> (and
>> in devlist whenever there were new releases) and generally the maintainers
>> who took part in those discussions and release process are aware of that -
>> also many maintainers know the process as it "soaked" in when they were
>> just watching.
>>
>> However recently we've made an attempt to document it (the PR above). So
>> you could learn it by reading it (even if it does not have the whole
>> context above).
>>
>> *Why do some people not know about it?*
>>
>> Not sure. Maybe because they were not interested and never asked? Maybe
>> because there was never a long email about it at our devlist, or the
>> documentation about it in README was too succinct?
>>
>> Or maybe we need a better way of communicating it - I am not sure. But I
>> hope this email will clarify a lot of it, and will prove valuable when
>> searching in devlist.
>>
>> Maybe even someone who manages to read it all will update the README
>> description of ours to explain it better :) and maybe create a nice FAQ
>> that we can put in our docs?
>>
>> I really hope this mail - even if long - will make people a bit more aware
>> of the process, and if someone has an idea how the "collective" awareness
>> can be improved - I think it's a good idea to send PR or email (or maybe
>> even record a video :) ??) that will be a better way of communicating it.
>>
>> J.
>>
>>
>> On Wed, Jan 17, 2024 at 9:03 AM Bolke de Bruin <bd...@gmail.com> wrote:
>>
>> > Just checking:
>> >
>> > I see that some improvements to Airflow.io (two weeks ago) were not
>> > included and some provider updates neither. Haven't checked anything
>> else
>> > yet.
>> >
>> > Is that intentional? Ie. is that the purpose of this release. Other
>> > big(ger) and more recent changes have been included hence asking.
>> >
>> > B.
>> >
>> > Sent from my iPhone
>> >
>> > > On 16 Jan 2024, at 20:18, Jarek Potiuk <ja...@potiuk.com> wrote:
>> > >
>> > > +1 (binding): Checked all my changes, I ran airflow in a few
>> > combinations
>> > > (MySQL / Postgres Local/Celery executor. It looks and works well -
>> run a
>> > > few dags and navigated through a number of screens. Checked licences,
>> > > signatures, checksums, performed a "reproducible build" check and it
>> > worked
>> > > (with a small glitch explained below).
>> > >
>> > > BTW. The new "hatchling-built" package looks good and it nicely
>> installs
>> > > airflow + extras (and it has a far better and cleaner set of extras -
>> > > finally you can reproducibly install airflow with `all` extra if you
>> > > ***REALLY*** want :).
>> > >
>> > > Reproducibility (also for other PMC members doing the check): I found
>> a
>> > > small glitch in the "reproducible" part of verifying the packages.
>> While
>> > > wheel and sdist packages are exactly the same binary-wise, the
>> > > source-tarball was binarry different for me. I decompressed it and
>> > compared
>> > > the content - they are identical - but there is one difference which I
>> > > overlooked - the group permissions for files in Ephraim's tarball are
>> > > different from mine. I have totally forgotten about the fact that
>> umask
>> > > might set different group/other permissions when checking out the
>> files
>> > > from git. Fix will be coming shortly - in the meantime I recommend
>> anyone
>> > > who will be doing the comparison to uncompress both tarballs and
>> compare
>> > > the contents with `diff -r`.
>> > >
>> > >
>> > >
>> > >> On Tue, Jan 16, 2024 at 11:30 AM Ephraim Anierobi <
>> > >> ephraimanierobi@apache.org> wrote:
>> > >>
>> > >> Hey fellow Airflowers,
>> > >>
>> > >> I have cut Airflow 2.8.1rc1. This email is calling a vote on the
>> > release,
>> > >> which will last at least 72 hours, from Tuesday, January 16, 2024 at
>> > 10:30
>> > >> am UTC
>> > >> until Friday, January 19, 2024, at 10:30 am UTC
>> > >> <
>> > >>
>> >
>> https://www.timeanddate.com/worldclock/fixedtime.html?msg=8&iso=20240119T1030&p1=1440
>> > >>> ,
>> > >> and until 3 binding +1 votes have been received.
>> > >>
>> > >> Status of testing of the release is kept at
>> > >> https://github.com/apache/airflow/issues/36808
>> > >>
>> > >> Consider this my (binding) +1.
>> > >>
>> > >> Airflow 2.8.1rc1 is available at:
>> > >> https://dist.apache.org/repos/dist/dev/airflow/2.8.1rc1/
>> > >>
>> > >> *apache-airflow-2.8.1-source.tar.gz* is a source release that comes
>> with
>> > >> INSTALL instructions.
>> > >> *apache-airflow-2.8.1.tar.gz* is the binary Python "sdist" release.
>> > >> *apache_airflow-2.8.1-py3-none-any.whl* is the binary Python wheel
>> > "binary"
>> > >> release.
>> > >>
>> > >> Public keys are available at:
>> > >> https://dist.apache.org/repos/dist/release/airflow/KEYS
>> > >>
>> > >> Please vote accordingly:
>> > >>
>> > >> [ ] +1 approve
>> > >> [ ] +0 no opinion
>> > >> [ ] -1 disapprove with the reason
>> > >>
>> > >> Only votes from PMC members are binding, but all members of the
>> > community
>> > >> are encouraged to test the release and vote with "(non-binding)".
>> > >>
>> > >> The test procedure for PMC members is described in:
>> > >>
>> > >>
>> >
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-pmc-members
>> > >>
>> > >> The test procedure for and Contributors who would like to test this
>> RC
>> > is
>> > >> described in:
>> > >>
>> > >>
>> >
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-contributors
>> > >>
>> > >>
>> > >> Please note that the version number excludes the `rcX` string, so
>> it's
>> > now
>> > >> simply 2.8.1. This will allow us to rename the artifact without
>> > modifying
>> > >> the artifact checksums when we actually release.
>> > >>
>> > >> Release Notes:
>> > >> https://github.com/apache/airflow/blob/2.8.1rc1/RELEASE_NOTES.rst
>> > >>
>> > >> Changes since 2.8.0:
>> > >>
>> > >> *Significant Changes*
>> > >>
>> > >> *Target version for core dependency ``pendulum`` package set to 3
>> > >> (#36281).*
>> > >> Support for pendulum 2.1.2 will be saved for a while, presumably
>> until
>> > the
>> > >> next feature version of Airflow.
>> > >> It is advised to upgrade user code to use pendulum 3 as soon as
>> > possible.
>> > >>
>> > >> *Airflow packaging specification follows modern Python packaging
>> > standards
>> > >> (#36537).*
>> > >> We standardized Airflow dependency configuration to follow latest
>> > >> development in Python packaging by
>> > >> using ``pyproject.toml``. Airflow is now compliant with those
>> accepted
>> > >> PEPs:
>> > >>
>> > >> * `PEP-440 Version Identification and Dependency Specification <
>> > >> https://www.python.org/dev/peps/pep-0440/>`__
>> > >> * `PEP-517 A build-system independent format for source trees <
>> > >> https://www.python.org/dev/peps/pep-0517/>`__
>> > >> * `PEP-518 Specifying Minimum Build System Requirements for Python
>> > Projects
>> > >> <https://www.python.org/dev/peps/pep-0518/>`__
>> > >> * `PEP-561 Distributing and Packaging Type Information <
>> > >> https://www.python.org/dev/peps/pep-0561/>`__
>> > >> * `PEP-621 Storing project metadata in pyproject.toml <
>> > >> https://www.python.org/dev/peps/pep-0621/>`__
>> > >> * `PEP-660 Editable installs for pyproject.toml based builds (wheel
>> > based)
>> > >> <
>> > >> https://www.python.org/dev/peps/pep-0660/>`__
>> > >> * `PEP-685 Comparison of extra names for optional distribution
>> > dependencies
>> > >> <https://www.python.org/dev/peps/pep-0685/>`__
>> > >>
>> > >> Also we implement multiple license files support coming from Draft,
>> not
>> > yet
>> > >> accepted (but supported by hatchling) PEP:
>> > >> * `PEP 639 Improving License Clarity with Better Package Metadata <
>> > >> https://peps.python.org/pep-0639/>`__
>> > >>
>> > >> This has almost no noticeable impact on users if they are using
>> modern
>> > >> Python packaging and development tools, generally
>> > >> speaking Airflow should behave as it did before when installing it
>> from
>> > >> PyPI and it should be much easier to install
>> > >> it for development purposes using ``pip install -e ".[devel]"``.
>> > >>
>> > >> The differences from the user side are:
>> > >>
>> > >> * Airflow extras now get extras normalized to ``-`` (following
>> PEP-685)
>> > >> instead of ``_`` and ``.``
>> > >> (as it was before in some extras). When you install airflow with
>> such
>> > >> extras (for example ``dbt.core`` or
>> > >> ``all_dbs``) you should use ``-`` instead of ``_`` and ``.``.
>> > >>
>> > >> In most modern tools this will work in backwards-compatible way, but
>> in
>> > >> some old version of those tools you might need to
>> > >> replace ``_`` and ``.`` with ``-``. You can also get warnings that
>> the
>> > >> extra you are installing does not exist - but usually
>> > >> this warning is harmless and the extra is installed anyway. It is,
>> > however,
>> > >> recommended to change to use ``-`` in extras in your dependency
>> > >> specifications for all Airflow extras.
>> > >>
>> > >> * Released airflow package does not contain ``devel``, ``devel-*``,
>> > ``doc``
>> > >> and ``doc-gen`` extras.
>> > >> Those extras are only available when you install Airflow from
>> sources
>> > in
>> > >> ``--editable`` mode. This is
>> > >> because those extras are only used for development and documentation
>> > >> building purposes and are not needed
>> > >> when you install Airflow for production use. Those dependencies had
>> > >> unspecified and varying behaviour for
>> > >> released packages anyway and you were not supposed to use them in
>> > >> released packages.
>> > >>
>> > >> * The ``all`` and ``all-*`` extras were not always working correctly
>> > when
>> > >> installing Airflow using constraints
>> > >> because they were also considered as development-only dependencies.
>> > With
>> > >> this change, those dependencies are
>> > >> now properly handling constraints and they will install properly
>> with
>> > >> constraints, pulling the right set
>> > >> of providers and dependencies when constraints are used.
>> > >>
>> > >> *Graphviz dependency is now an optional one, not required one
>> (#36647).*
>> > >> The ``graphviz`` dependency has been problematic as Airflow required
>> > >> dependency - especially for
>> > >> ARM-based installations. Graphviz packages require binary graphviz
>> > >> libraries - which is already a
>> > >> limitation, but they also require to install graphviz Python bindings
>> > to be
>> > >> build and installed.
>> > >> This does not work for older Linux installation but - more
>> importantly -
>> > >> when you try to install
>> > >> Graphviz libraries for Python 3.8, 3.9 for ARM M1 MacBooks, the
>> packages
>> > >> fail to install because
>> > >> Python bindings compilation for M1 can only work for Python 3.10+.
>> > >>
>> > >> This is not a breaking change technically - the CLIs to render the
>> DAGs
>> > is
>> > >> still there and IF you
>> > >> already have graphviz installed, it will continue working as it did
>> > before.
>> > >> The only problem when it
>> > >> does not work is where you do not have graphviz installed it will
>> raise
>> > an
>> > >> error and inform that you need it.
>> > >>
>> > >> Graphviz will remain to be installed for most users:
>> > >>
>> > >> * the Airflow Image will still contain graphviz library, because
>> > >> it is added there as extra
>> > >> * when previous version of Airflow has been installed already, then
>> > >> graphviz library is already installed there and Airflow will
>> > >> continue working as it did
>> > >>
>> > >> The only change will be a new installation of new version of Airflow
>> > from
>> > >> the scratch, where graphviz will
>> > >> need to be specified as extra or installed separately in order to
>> enable
>> > >> DAG rendering option.
>> > >>
>> > >> *Bug Fixes*
>> > >> - Fix airflow-scheduler exiting with code 0 on exceptions (#36800)
>> > >> - Fix Callback exception when a removed task is the last one in the
>> > >> ``taskinstance`` list (#36693)
>> > >> - Allow anonymous user edit/show resource when set
>> > >> ``AUTH_ROLE_PUBLIC=admin`` (#36750)
>> > >> - Better error message when sqlite URL uses relative path (#36774)
>> > >> - Explicit string cast required to force integer-type run_ids to be
>> > passed
>> > >> as strings instead of integers (#36756)
>> > >> - Add log lookup exception for empty ``op`` subtypes (#35536)
>> > >> - Remove unused index on task instance (#36737)
>> > >> - Fix check on subclass for ``typing.Union`` in
>> > ``_infer_multiple_outputs``
>> > >> for Python 3.10+ (#36728)
>> > >> - Make sure ``multiple_outputs`` is inferred correctly even when
>> using
>> > >> ``TypedDict`` (#36652)
>> > >> - Add back FAB constant in legacy security manager (#36719)
>> > >> - Fix AttributeError when using ``Dagrun.update_state`` (#36712)
>> > >> - Do not let ``EventsTimetable`` schedule past events if
>> > ``catchup=False``
>> > >> (#36134)
>> > >> - Support encryption for triggers parameters (#36492)
>> > >> - Fix the type hint for ``tis_query`` in ``_process_executor_events``
>> > >> (#36655)
>> > >> - Redirect to index when user does not have permission to access a
>> page
>> > >> (#36623)
>> > >> - Avoid using dict as default value in ``call_regular_interval``
>> > (#36608)
>> > >> - Remove option to set a task instance to running state in UI
>> (#36518)
>> > >> - Fix details tab not showing when using dynamic task mapping
>> (#36522)
>> > >> - Raise error when ``DagRun`` fails while running ``dag test``
>> (#36517)
>> > >> - Refactor ``_manage_executor_state`` by refreshing TIs in batch
>> > (#36502)
>> > >> - Add flask config: ``MAX_CONTENT_LENGTH`` (#36401)
>> > >> - Fix get_leaves calculation for teardown in nested group (#36456)
>> > >> - Stop serializing timezone-naive datetime to timezone-aware datetime
>> > with
>> > >> UTC tz (#36379)
>> > >> - Make ``kubernetes`` decorator type annotation consistent with
>> operator
>> > >> (#36405)
>> > >> - Fix Webserver returning 500 for POST requests to
>> ``api/dag/*/dagrun``
>> > >> from anonymous user (#36275)
>> > >> - Fix the required access for get_variable endpoint (#36396)
>> > >> - Fix datetime reference in ``DAG.is_fixed_time_schedule`` (#36370)
>> > >> - Fix AirflowSkipException message raised by BashOperator (#36354)
>> > >> - Allow PythonVirtualenvOperator.skip_on_exit_code to be zero
>> (#36361)
>> > >> - Increase width of execution_date input in trigger.html (#36278)
>> > >> - Fix logging for pausing DAG (#36182)
>> > >> - Stop deserializing pickle when enable_xcom_pickling is False
>> (#36255)
>> > >> - Check DAG read permission before accessing DAG code (#36257)
>> > >> - Enable mark task as failed/success always (#36254)
>> > >> - Create latest log dir symlink as relative link (#36019)
>> > >> - Fix Python-based decorators templating (#36103)
>> > >>
>> > >> *Miscellaneous*
>> > >> - Rename concurrency label to max active tasks (#36691)
>> > >> - Restore function scoped ``httpx`` import in file_task_handler for
>> > >> performance (#36753)
>> > >> - Add support of Pendulum 3 (#36281)
>> > >> - Standardize airflow build process and switch to Hatchling build
>> > backend
>> > >> (#36537)
>> > >> - Get rid of ``pyarrow-hotfix`` for ``CVE-2023-47248`` (#36697)
>> > >> - Make ``graphviz`` dependency optional (#36647)
>> > >> - Announce MSSQL support end in Airflow 2.9.0, add migration script
>> > hints
>> > >> (#36509)
>> > >> - Set min ``pandas`` dependency to 1.2.5 for all providers and
>> airflow
>> > >> (#36698)
>> > >> - Bump follow-redirects from 1.15.3 to 1.15.4 in ``/airflow/www``
>> > (#36700)
>> > >> - Provide the logger_name param to base hook in order to override the
>> > >> logger name (#36674)
>> > >> - Fix run type icon alignment with run type text (#36616)
>> > >> - Follow BaseHook connection fields method signature in FSHook
>> (#36444)
>> > >> - Remove redundant ``docker`` decorator type annotations (#36406)
>> > >> - Straighten typing in workday timetable (#36296)
>> > >> - Use ``batch_is_authorized_dag`` to check if user has permission to
>> > read
>> > >> DAGs (#36279)
>> > >> - Replace deprecated get_accessible_dag_ids and use
>> get_readable_dags in
>> > >> get_dag_warnings (#36256)
>> > >>
>> > >> *Doc Only Changes*
>> > >> - Metrics tagging documentation (#36627)
>> > >> - In docs use logical_date instead of deprecated execution_date
>> (#36654)
>> > >> - Add section about live-upgrading Airflow (#36637)
>> > >> - Replace ``numpy`` example with practical exercise demonstrating
>> > top-level
>> > >> code (#35097)
>> > >> - Improve and add more complete description in the architecture
>> diagrams
>> > >> (#36513)
>> > >> - Improve the error message displayed when there is a webserver error
>> > >> (#36570)
>> > >> - Update ``dags.rst`` with information on DAG pausing (#36540)
>> > >> - Update installation prerequisites after upgrading to Debian
>> Bookworm
>> > >> (#36521)
>> > >> - Add description on the ways how users should approach DB monitoring
>> > >> (#36483)
>> > >> - Add branching based on mapped task group example to
>> > >> dynamic-task-mapping.rst (#36480)
>> > >> - Add further details to replacement documentation (#36485)
>> > >> - Use cards when describing priority weighting methods (#36411)
>> > >> - Update ``metrics.rst`` for param ``dagrun.schedule_delay`` (#36404)
>> > >> - Update admonitions in Python operator doc to reflect sentiment
>> > (#36340)
>> > >> - Improve audit_logs.rst (#36213)
>> > >> - Remove Redshift mention from the list of managed Postgres backends
>> > >> (#36217)
>> > >>
>> > >> Cheers,
>> > >> Ephraim
>> > >>
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> > For additional commands, e-mail: dev-help@airflow.apache.org
>> >
>> >
>>
>
Re: What goes into the next release (was [VOTE] Release Airflow 2.8.1 from 2.8.1rc1)
Posted by Amogh Desai <am...@gmail.com>.
Thanks @Jarek Potiuk <ja...@potiuk.com>!
I vaguely knew the process but not in such detail, thanks for putting it
together in this email.
I will visit the document
https://github.com/apache/airflow?tab=readme-ov-file#what-goes-into-the-next-release
and if I find any clarifications, I will send it across as a PR.
One suggestion: what if we put it together in a document and put it on the
airflow website under
https://airflow.apache.org/community/ where we mention "how we release"?
Might be a bit too much given the context people will have while
visiting the airflow website..
Thanks & Regards,
Amogh Desai
On Wed, Jan 17, 2024 at 6:13 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> I separate it out, because it seems that despite the efforts to explain and
> document how our releases work It's not clear even for the PMC chair, so
> likely it warrants a separate thread - also it will be easier to find it in
> the archives this way.
>
> I think this is an important topic that all maintainers should be aware of,
> so let me take a task to explain it in a longer email (and separate
> thread).
>
> I created it in a form of FAQ, because it seems that similar questions
> might be asked by others.
>
> *Do we have a process defined here?*
>
> Answering Bolke's question - who was apparently confused what our process
> is:
>
> > I see that some improvements to Airflow.io (two weeks ago) were not
> included and some provider updates neither. Haven't checked anything else
> yet.
>
> Apparently there is some confusion about the process, but yes we have a
> well defined and well tested (pretty much 4 years now) process that we
> follow., We follow it since I remember actually - it's been also done the
> same in 1.10 - but with some variations, Likely we do it the same way since
> the beginning of 2.0, but it has been refined and improved over time - by
> those who volunteered their time in the release process (a lot of ad-hoc
> discussion have been traditionally happening in #release-management slack
> channel) and as of few months ago we even documented it (It was in November
> 2023) - with this PR https://github.com/apache/airflow/pull/35245
>
> It is currently described in a prominent place in our most important (and
> over the last year or so the README, we made the README pretty short and
> contains only super-important information on how Airflow is developed)
> README.md under *"What goes into the next release?"* chapter:
>
>
> https://github.com/apache/airflow?tab=readme-ov-file#what-goes-into-the-next-release
> .
>
> *How does the selection process for cherry-picking work?*
>
> In short (and this is the most important thing that every maintainer should
> be aware of): *those maintainers who think that issue should be included
> should mark it with the next (in this case 2.8.1) milestone*. It's up to
> individual maintainers who want to include certain changes to take care
> about it and mark the issues they think are bug fixes, to go into the next
> release
>
> This is the only thing that the maintainer has to do to get the PR proposed
> to be considered in the next patchlevel release. Sometimes - if
> controversial - maintainers discuss the proposals in #release-management
> channel, sometimes in #development or in the PR itself (especially if the
> release manager decides to not include it and changes the milestone (and
> explains why).
>
> *What's the release manager's role ?*
>
> Release manager's job is purely mechanical (as also mandated by the Apache
> Software Foundation release manager role description) to assess cherry-pick
> ability of those changes. Release manager - at the sole discretion and
> individual decision (this is the only place in the whole ASF setup where a
> single person has such power to make individual decisions) can reject some
> of those who other maintainers think should be included. But the release
> manager on his own does not make proposals on what should be included.
>
> *Is this process following the ASF rules?*
>
> I believe yes, The release manager's role is nicely described here:
> https://infra.apache.org/release-publishing.html#releasemanager. And there
> is a far more complete description here that describes the whole process
> https://www.apache.org/legal/release-policy.html#management - also
> mentioning that it's the PMC's responsibility (and particularly PMC
> chair's) to adhere to the process.
>
> *What's the role of individual maintainers?*
>
> The role of maintainers (collectively) to propose things for the next
> release. In our case it happens with setting the milestone on a PR.
>
> *When proposed PRs are rejected?*
>
> There are various reasons to reject those - if too complex to cherry-pick
> or when the release manager assesses it's a new feature, not a bugfix.
> Essentially (according to semver) when it comes to the user-facing changes,
> the PATCHLEVEL release should contain only bugfixes. and may contain docs
> changes if they are fixing/improving docs (not about new features) and also
> environment/build script changes (so non-user-facing changes) as they are
> pretty much always needed to keep the things nicely building - those are
> usually skipped from the changelog as non-user facing).
>
> *Why are provider changes not cherry-picked?*
>
> In our case - basically none of the provider changes are cherry-picked -
> unless they are needed to make the builds work well (sometimes happen).
> Providers are ALWAYS released from the latest main code, not from the
> v2-8-stable branch. In fact all the tests and ci checks for providers are
> skipped in the non-main (v2* branches). So yes - not seeing provider
> changes cherry-picked is absolutely expected.
>
> *Do we skip over some changes when releasing a patchlevel release? What's
> the purpose of patch-level releases?*
>
> Answering Bolke's question:
>
> > Is that intentional? Ie. is that the purpose of this release. Other
> big(ger) and more recent changes have been included hence asking.
>
> The purpose of that release is as described in SemVer - to give the users
> bugfix-only release that has no new features. Of course it's sometimes
> debatable whether changes are features/bugfixes, but we usually use
> #release-management to quickly chat about it, and eventually the release
> manager always makes a comment in the PR when the milestone is changed and
> explains the reasoning.
>
> Skipping is not intentional because we never "skip" things when
> cherry-picking, It's *reverse* - those maintainer who think that certain
> bug fixes (or internal changes or sometimes even feature changes that we
> classify really as "bugfix" SHOULD intentionally mark those PRs they want
> with 2.8.1 (or basically next patch-level) to be *included. * So there is
> no skipping, if maintainer did not deliberately mark PR as upcoming
> milestone, it will just not be included
>
> *Where do some maintainers know about it **from ? *
>
> Because the maintainers who actively participate - either acting as release
> managers (those who raised their hand and acted as release managers
> generally speaking) and those who wanted their changes to be part of the
> past releases have been doing it for years. For years this has been simply
> followed and discussed in #release-management channel and #development (and
> in devlist whenever there were new releases) and generally the maintainers
> who took part in those discussions and release process are aware of that -
> also many maintainers know the process as it "soaked" in when they were
> just watching.
>
> However recently we've made an attempt to document it (the PR above). So
> you could learn it by reading it (even if it does not have the whole
> context above).
>
> *Why do some people not know about it?*
>
> Not sure. Maybe because they were not interested and never asked? Maybe
> because there was never a long email about it at our devlist, or the
> documentation about it in README was too succinct?
>
> Or maybe we need a better way of communicating it - I am not sure. But I
> hope this email will clarify a lot of it, and will prove valuable when
> searching in devlist.
>
> Maybe even someone who manages to read it all will update the README
> description of ours to explain it better :) and maybe create a nice FAQ
> that we can put in our docs?
>
> I really hope this mail - even if long - will make people a bit more aware
> of the process, and if someone has an idea how the "collective" awareness
> can be improved - I think it's a good idea to send PR or email (or maybe
> even record a video :) ??) that will be a better way of communicating it.
>
> J.
>
>
> On Wed, Jan 17, 2024 at 9:03 AM Bolke de Bruin <bd...@gmail.com> wrote:
>
> > Just checking:
> >
> > I see that some improvements to Airflow.io (two weeks ago) were not
> > included and some provider updates neither. Haven't checked anything
> else
> > yet.
> >
> > Is that intentional? Ie. is that the purpose of this release. Other
> > big(ger) and more recent changes have been included hence asking.
> >
> > B.
> >
> > Sent from my iPhone
> >
> > > On 16 Jan 2024, at 20:18, Jarek Potiuk <ja...@potiuk.com> wrote:
> > >
> > > +1 (binding): Checked all my changes, I ran airflow in a few
> > combinations
> > > (MySQL / Postgres Local/Celery executor. It looks and works well - run
> a
> > > few dags and navigated through a number of screens. Checked licences,
> > > signatures, checksums, performed a "reproducible build" check and it
> > worked
> > > (with a small glitch explained below).
> > >
> > > BTW. The new "hatchling-built" package looks good and it nicely
> installs
> > > airflow + extras (and it has a far better and cleaner set of extras -
> > > finally you can reproducibly install airflow with `all` extra if you
> > > ***REALLY*** want :).
> > >
> > > Reproducibility (also for other PMC members doing the check): I found a
> > > small glitch in the "reproducible" part of verifying the packages.
> While
> > > wheel and sdist packages are exactly the same binary-wise, the
> > > source-tarball was binarry different for me. I decompressed it and
> > compared
> > > the content - they are identical - but there is one difference which I
> > > overlooked - the group permissions for files in Ephraim's tarball are
> > > different from mine. I have totally forgotten about the fact that umask
> > > might set different group/other permissions when checking out the files
> > > from git. Fix will be coming shortly - in the meantime I recommend
> anyone
> > > who will be doing the comparison to uncompress both tarballs and
> compare
> > > the contents with `diff -r`.
> > >
> > >
> > >
> > >> On Tue, Jan 16, 2024 at 11:30 AM Ephraim Anierobi <
> > >> ephraimanierobi@apache.org> wrote:
> > >>
> > >> Hey fellow Airflowers,
> > >>
> > >> I have cut Airflow 2.8.1rc1. This email is calling a vote on the
> > release,
> > >> which will last at least 72 hours, from Tuesday, January 16, 2024 at
> > 10:30
> > >> am UTC
> > >> until Friday, January 19, 2024, at 10:30 am UTC
> > >> <
> > >>
> >
> https://www.timeanddate.com/worldclock/fixedtime.html?msg=8&iso=20240119T1030&p1=1440
> > >>> ,
> > >> and until 3 binding +1 votes have been received.
> > >>
> > >> Status of testing of the release is kept at
> > >> https://github.com/apache/airflow/issues/36808
> > >>
> > >> Consider this my (binding) +1.
> > >>
> > >> Airflow 2.8.1rc1 is available at:
> > >> https://dist.apache.org/repos/dist/dev/airflow/2.8.1rc1/
> > >>
> > >> *apache-airflow-2.8.1-source.tar.gz* is a source release that comes
> with
> > >> INSTALL instructions.
> > >> *apache-airflow-2.8.1.tar.gz* is the binary Python "sdist" release.
> > >> *apache_airflow-2.8.1-py3-none-any.whl* is the binary Python wheel
> > "binary"
> > >> release.
> > >>
> > >> Public keys are available at:
> > >> https://dist.apache.org/repos/dist/release/airflow/KEYS
> > >>
> > >> Please vote accordingly:
> > >>
> > >> [ ] +1 approve
> > >> [ ] +0 no opinion
> > >> [ ] -1 disapprove with the reason
> > >>
> > >> Only votes from PMC members are binding, but all members of the
> > community
> > >> are encouraged to test the release and vote with "(non-binding)".
> > >>
> > >> The test procedure for PMC members is described in:
> > >>
> > >>
> >
> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-pmc-members
> > >>
> > >> The test procedure for and Contributors who would like to test this RC
> > is
> > >> described in:
> > >>
> > >>
> >
> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-contributors
> > >>
> > >>
> > >> Please note that the version number excludes the `rcX` string, so it's
> > now
> > >> simply 2.8.1. This will allow us to rename the artifact without
> > modifying
> > >> the artifact checksums when we actually release.
> > >>
> > >> Release Notes:
> > >> https://github.com/apache/airflow/blob/2.8.1rc1/RELEASE_NOTES.rst
> > >>
> > >> Changes since 2.8.0:
> > >>
> > >> *Significant Changes*
> > >>
> > >> *Target version for core dependency ``pendulum`` package set to 3
> > >> (#36281).*
> > >> Support for pendulum 2.1.2 will be saved for a while, presumably until
> > the
> > >> next feature version of Airflow.
> > >> It is advised to upgrade user code to use pendulum 3 as soon as
> > possible.
> > >>
> > >> *Airflow packaging specification follows modern Python packaging
> > standards
> > >> (#36537).*
> > >> We standardized Airflow dependency configuration to follow latest
> > >> development in Python packaging by
> > >> using ``pyproject.toml``. Airflow is now compliant with those accepted
> > >> PEPs:
> > >>
> > >> * `PEP-440 Version Identification and Dependency Specification <
> > >> https://www.python.org/dev/peps/pep-0440/>`__
> > >> * `PEP-517 A build-system independent format for source trees <
> > >> https://www.python.org/dev/peps/pep-0517/>`__
> > >> * `PEP-518 Specifying Minimum Build System Requirements for Python
> > Projects
> > >> <https://www.python.org/dev/peps/pep-0518/>`__
> > >> * `PEP-561 Distributing and Packaging Type Information <
> > >> https://www.python.org/dev/peps/pep-0561/>`__
> > >> * `PEP-621 Storing project metadata in pyproject.toml <
> > >> https://www.python.org/dev/peps/pep-0621/>`__
> > >> * `PEP-660 Editable installs for pyproject.toml based builds (wheel
> > based)
> > >> <
> > >> https://www.python.org/dev/peps/pep-0660/>`__
> > >> * `PEP-685 Comparison of extra names for optional distribution
> > dependencies
> > >> <https://www.python.org/dev/peps/pep-0685/>`__
> > >>
> > >> Also we implement multiple license files support coming from Draft,
> not
> > yet
> > >> accepted (but supported by hatchling) PEP:
> > >> * `PEP 639 Improving License Clarity with Better Package Metadata <
> > >> https://peps.python.org/pep-0639/>`__
> > >>
> > >> This has almost no noticeable impact on users if they are using modern
> > >> Python packaging and development tools, generally
> > >> speaking Airflow should behave as it did before when installing it
> from
> > >> PyPI and it should be much easier to install
> > >> it for development purposes using ``pip install -e ".[devel]"``.
> > >>
> > >> The differences from the user side are:
> > >>
> > >> * Airflow extras now get extras normalized to ``-`` (following
> PEP-685)
> > >> instead of ``_`` and ``.``
> > >> (as it was before in some extras). When you install airflow with such
> > >> extras (for example ``dbt.core`` or
> > >> ``all_dbs``) you should use ``-`` instead of ``_`` and ``.``.
> > >>
> > >> In most modern tools this will work in backwards-compatible way, but
> in
> > >> some old version of those tools you might need to
> > >> replace ``_`` and ``.`` with ``-``. You can also get warnings that the
> > >> extra you are installing does not exist - but usually
> > >> this warning is harmless and the extra is installed anyway. It is,
> > however,
> > >> recommended to change to use ``-`` in extras in your dependency
> > >> specifications for all Airflow extras.
> > >>
> > >> * Released airflow package does not contain ``devel``, ``devel-*``,
> > ``doc``
> > >> and ``doc-gen`` extras.
> > >> Those extras are only available when you install Airflow from sources
> > in
> > >> ``--editable`` mode. This is
> > >> because those extras are only used for development and documentation
> > >> building purposes and are not needed
> > >> when you install Airflow for production use. Those dependencies had
> > >> unspecified and varying behaviour for
> > >> released packages anyway and you were not supposed to use them in
> > >> released packages.
> > >>
> > >> * The ``all`` and ``all-*`` extras were not always working correctly
> > when
> > >> installing Airflow using constraints
> > >> because they were also considered as development-only dependencies.
> > With
> > >> this change, those dependencies are
> > >> now properly handling constraints and they will install properly with
> > >> constraints, pulling the right set
> > >> of providers and dependencies when constraints are used.
> > >>
> > >> *Graphviz dependency is now an optional one, not required one
> (#36647).*
> > >> The ``graphviz`` dependency has been problematic as Airflow required
> > >> dependency - especially for
> > >> ARM-based installations. Graphviz packages require binary graphviz
> > >> libraries - which is already a
> > >> limitation, but they also require to install graphviz Python bindings
> > to be
> > >> build and installed.
> > >> This does not work for older Linux installation but - more
> importantly -
> > >> when you try to install
> > >> Graphviz libraries for Python 3.8, 3.9 for ARM M1 MacBooks, the
> packages
> > >> fail to install because
> > >> Python bindings compilation for M1 can only work for Python 3.10+.
> > >>
> > >> This is not a breaking change technically - the CLIs to render the
> DAGs
> > is
> > >> still there and IF you
> > >> already have graphviz installed, it will continue working as it did
> > before.
> > >> The only problem when it
> > >> does not work is where you do not have graphviz installed it will
> raise
> > an
> > >> error and inform that you need it.
> > >>
> > >> Graphviz will remain to be installed for most users:
> > >>
> > >> * the Airflow Image will still contain graphviz library, because
> > >> it is added there as extra
> > >> * when previous version of Airflow has been installed already, then
> > >> graphviz library is already installed there and Airflow will
> > >> continue working as it did
> > >>
> > >> The only change will be a new installation of new version of Airflow
> > from
> > >> the scratch, where graphviz will
> > >> need to be specified as extra or installed separately in order to
> enable
> > >> DAG rendering option.
> > >>
> > >> *Bug Fixes*
> > >> - Fix airflow-scheduler exiting with code 0 on exceptions (#36800)
> > >> - Fix Callback exception when a removed task is the last one in the
> > >> ``taskinstance`` list (#36693)
> > >> - Allow anonymous user edit/show resource when set
> > >> ``AUTH_ROLE_PUBLIC=admin`` (#36750)
> > >> - Better error message when sqlite URL uses relative path (#36774)
> > >> - Explicit string cast required to force integer-type run_ids to be
> > passed
> > >> as strings instead of integers (#36756)
> > >> - Add log lookup exception for empty ``op`` subtypes (#35536)
> > >> - Remove unused index on task instance (#36737)
> > >> - Fix check on subclass for ``typing.Union`` in
> > ``_infer_multiple_outputs``
> > >> for Python 3.10+ (#36728)
> > >> - Make sure ``multiple_outputs`` is inferred correctly even when using
> > >> ``TypedDict`` (#36652)
> > >> - Add back FAB constant in legacy security manager (#36719)
> > >> - Fix AttributeError when using ``Dagrun.update_state`` (#36712)
> > >> - Do not let ``EventsTimetable`` schedule past events if
> > ``catchup=False``
> > >> (#36134)
> > >> - Support encryption for triggers parameters (#36492)
> > >> - Fix the type hint for ``tis_query`` in ``_process_executor_events``
> > >> (#36655)
> > >> - Redirect to index when user does not have permission to access a
> page
> > >> (#36623)
> > >> - Avoid using dict as default value in ``call_regular_interval``
> > (#36608)
> > >> - Remove option to set a task instance to running state in UI (#36518)
> > >> - Fix details tab not showing when using dynamic task mapping (#36522)
> > >> - Raise error when ``DagRun`` fails while running ``dag test``
> (#36517)
> > >> - Refactor ``_manage_executor_state`` by refreshing TIs in batch
> > (#36502)
> > >> - Add flask config: ``MAX_CONTENT_LENGTH`` (#36401)
> > >> - Fix get_leaves calculation for teardown in nested group (#36456)
> > >> - Stop serializing timezone-naive datetime to timezone-aware datetime
> > with
> > >> UTC tz (#36379)
> > >> - Make ``kubernetes`` decorator type annotation consistent with
> operator
> > >> (#36405)
> > >> - Fix Webserver returning 500 for POST requests to
> ``api/dag/*/dagrun``
> > >> from anonymous user (#36275)
> > >> - Fix the required access for get_variable endpoint (#36396)
> > >> - Fix datetime reference in ``DAG.is_fixed_time_schedule`` (#36370)
> > >> - Fix AirflowSkipException message raised by BashOperator (#36354)
> > >> - Allow PythonVirtualenvOperator.skip_on_exit_code to be zero (#36361)
> > >> - Increase width of execution_date input in trigger.html (#36278)
> > >> - Fix logging for pausing DAG (#36182)
> > >> - Stop deserializing pickle when enable_xcom_pickling is False
> (#36255)
> > >> - Check DAG read permission before accessing DAG code (#36257)
> > >> - Enable mark task as failed/success always (#36254)
> > >> - Create latest log dir symlink as relative link (#36019)
> > >> - Fix Python-based decorators templating (#36103)
> > >>
> > >> *Miscellaneous*
> > >> - Rename concurrency label to max active tasks (#36691)
> > >> - Restore function scoped ``httpx`` import in file_task_handler for
> > >> performance (#36753)
> > >> - Add support of Pendulum 3 (#36281)
> > >> - Standardize airflow build process and switch to Hatchling build
> > backend
> > >> (#36537)
> > >> - Get rid of ``pyarrow-hotfix`` for ``CVE-2023-47248`` (#36697)
> > >> - Make ``graphviz`` dependency optional (#36647)
> > >> - Announce MSSQL support end in Airflow 2.9.0, add migration script
> > hints
> > >> (#36509)
> > >> - Set min ``pandas`` dependency to 1.2.5 for all providers and airflow
> > >> (#36698)
> > >> - Bump follow-redirects from 1.15.3 to 1.15.4 in ``/airflow/www``
> > (#36700)
> > >> - Provide the logger_name param to base hook in order to override the
> > >> logger name (#36674)
> > >> - Fix run type icon alignment with run type text (#36616)
> > >> - Follow BaseHook connection fields method signature in FSHook
> (#36444)
> > >> - Remove redundant ``docker`` decorator type annotations (#36406)
> > >> - Straighten typing in workday timetable (#36296)
> > >> - Use ``batch_is_authorized_dag`` to check if user has permission to
> > read
> > >> DAGs (#36279)
> > >> - Replace deprecated get_accessible_dag_ids and use get_readable_dags
> in
> > >> get_dag_warnings (#36256)
> > >>
> > >> *Doc Only Changes*
> > >> - Metrics tagging documentation (#36627)
> > >> - In docs use logical_date instead of deprecated execution_date
> (#36654)
> > >> - Add section about live-upgrading Airflow (#36637)
> > >> - Replace ``numpy`` example with practical exercise demonstrating
> > top-level
> > >> code (#35097)
> > >> - Improve and add more complete description in the architecture
> diagrams
> > >> (#36513)
> > >> - Improve the error message displayed when there is a webserver error
> > >> (#36570)
> > >> - Update ``dags.rst`` with information on DAG pausing (#36540)
> > >> - Update installation prerequisites after upgrading to Debian Bookworm
> > >> (#36521)
> > >> - Add description on the ways how users should approach DB monitoring
> > >> (#36483)
> > >> - Add branching based on mapped task group example to
> > >> dynamic-task-mapping.rst (#36480)
> > >> - Add further details to replacement documentation (#36485)
> > >> - Use cards when describing priority weighting methods (#36411)
> > >> - Update ``metrics.rst`` for param ``dagrun.schedule_delay`` (#36404)
> > >> - Update admonitions in Python operator doc to reflect sentiment
> > (#36340)
> > >> - Improve audit_logs.rst (#36213)
> > >> - Remove Redshift mention from the list of managed Postgres backends
> > >> (#36217)
> > >>
> > >> Cheers,
> > >> Ephraim
> > >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
> >
>
Re: What goes into the next release (was [VOTE] Release Airflow 2.8.1 from 2.8.1rc1)
Posted by Jarek Potiuk <ja...@potiuk.com>.
Well that's the whole point about repeating "SemVer" enough times should be
enough to distinct the feature (2.9.0) release from patchlevel (BTW.
patchlevel name comes from SemVer so I am not sure if that changes much to
mention it).
But maybe another suggestion - I think maybe you can propose a PR how to
distinguish those PRs in a clearer way Bolke to distinguish those ?
The release process - including the template of the vote email is in
https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md#prepare-vote-email-on-the-apache-airflow-release-candidat
- and again Release Manager's work is really "mechanical" (including
sending the email prepared from the procedure we have). So any of us can
propose (and many times we did) a PR change to the process and templates
sent - so if we think we can improve it and make it clearer - PR proposal
to the process might be a good idea.
J.
On Thu, Jan 18, 2024 at 11:56 AM Bolke de Bruin <bd...@gmail.com> wrote:
> Thanks Jarek, as mentioned in the other thread it seemed to break a
> different pattern for me. I was aware of "what goes into the next release",
> but the title seemed more indicative of roadmap items and thus geared to
> consumers of a release.
>
> Earlier, it seems, I was helped by others when PRs were tagged for a
> particular release so I was under the impression this felt under the
> release manager's way of working.
>
> I suggested adding a label to the vote, like patch-release, and including
> the link to the doc. That makes it easier for more occasional viewers.
>
> B.
>
> On Wed, 17 Jan 2024 at 13:42, Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> I separate it out, because it seems that despite the efforts to explain
>> and document how our releases work It's not clear even for the PMC chair,
>> so likely it warrants a separate thread - also it will be easier to find it
>> in the archives this way.
>>
>> I think this is an important topic that all maintainers should be aware
>> of, so let me take a task to explain it in a longer email (and separate
>> thread).
>>
>> I created it in a form of FAQ, because it seems that similar questions
>> might be asked by others.
>>
>> *Do we have a process defined here?*
>>
>> Answering Bolke's question - who was apparently confused what our process
>> is:
>>
>> > I see that some improvements to Airflow.io (two weeks ago) were not
>> included and some provider updates neither. Haven't checked anything else
>> yet.
>>
>> Apparently there is some confusion about the process, but yes we have a
>> well defined and well tested (pretty much 4 years now) process that we
>> follow., We follow it since I remember actually - it's been also done the
>> same in 1.10 - but with some variations, Likely we do it the same way since
>> the beginning of 2.0, but it has been refined and improved over time - by
>> those who volunteered their time in the release process (a lot of ad-hoc
>> discussion have been traditionally happening in #release-management slack
>> channel) and as of few months ago we even documented it (It was in November
>> 2023) - with this PR https://github.com/apache/airflow/pull/35245
>>
>> It is currently described in a prominent place in our most important (and
>> over the last year or so the README, we made the README pretty short and
>> contains only super-important information on how Airflow is developed)
>> README.md under *"What goes into the next release?"* chapter:
>>
>>
>> https://github.com/apache/airflow?tab=readme-ov-file#what-goes-into-the-next-release
>> .
>>
>> *How does the selection process for cherry-picking work?*
>>
>> In short (and this is the most important thing that every maintainer
>> should be aware of): *those maintainers who think that issue should be
>> included should mark it with the next (in this case 2.8.1) milestone*.
>> It's up to individual maintainers who want to include certain changes to
>> take care about it and mark the issues they think are bug fixes, to go into
>> the next release
>>
>> This is the only thing that the maintainer has to do to get the PR
>> proposed to be considered in the next patchlevel release. Sometimes - if
>> controversial - maintainers discuss the proposals in #release-management
>> channel, sometimes in #development or in the PR itself (especially if the
>> release manager decides to not include it and changes the milestone (and
>> explains why).
>>
>> *What's the release manager's role ?*
>>
>> Release manager's job is purely mechanical (as also mandated by the
>> Apache Software Foundation release manager role description) to assess
>> cherry-pick ability of those changes. Release manager - at the sole
>> discretion and individual decision (this is the only place in the whole ASF
>> setup where a single person has such power to make individual decisions)
>> can reject some of those who other maintainers think should be included.
>> But the release manager on his own does not make proposals on what should
>> be included.
>>
>> *Is this process following the ASF rules?*
>>
>> I believe yes, The release manager's role is nicely described here:
>> https://infra.apache.org/release-publishing.html#releasemanager. And
>> there is a far more complete description here that describes the whole
>> process https://www.apache.org/legal/release-policy.html#management -
>> also mentioning that it's the PMC's responsibility (and particularly PMC
>> chair's) to adhere to the process.
>>
>> *What's the role of individual maintainers?*
>>
>> The role of maintainers (collectively) to propose things for the next
>> release. In our case it happens with setting the milestone on a PR.
>>
>> *When proposed PRs are rejected?*
>>
>> There are various reasons to reject those - if too complex to cherry-pick
>> or when the release manager assesses it's a new feature, not a bugfix.
>> Essentially (according to semver) when it comes to the user-facing changes,
>> the PATCHLEVEL release should contain only bugfixes. and may contain docs
>> changes if they are fixing/improving docs (not about new features) and also
>> environment/build script changes (so non-user-facing changes) as they are
>> pretty much always needed to keep the things nicely building - those are
>> usually skipped from the changelog as non-user facing).
>>
>> *Why are provider changes not cherry-picked?*
>>
>> In our case - basically none of the provider changes are cherry-picked -
>> unless they are needed to make the builds work well (sometimes happen).
>> Providers are ALWAYS released from the latest main code, not from the
>> v2-8-stable branch. In fact all the tests and ci checks for providers are
>> skipped in the non-main (v2* branches). So yes - not seeing provider
>> changes cherry-picked is absolutely expected.
>>
>> *Do we skip over some changes when releasing a patchlevel release? What's
>> the purpose of patch-level releases?*
>>
>> Answering Bolke's question:
>>
>> > Is that intentional? Ie. is that the purpose of this release. Other
>> big(ger) and more recent changes have been included hence asking.
>>
>> The purpose of that release is as described in SemVer - to give the users
>> bugfix-only release that has no new features. Of course it's sometimes
>> debatable whether changes are features/bugfixes, but we usually use
>> #release-management to quickly chat about it, and eventually the release
>> manager always makes a comment in the PR when the milestone is changed and
>> explains the reasoning.
>>
>> Skipping is not intentional because we never "skip" things when
>> cherry-picking, It's *reverse* - those maintainer who think that certain
>> bug fixes (or internal changes or sometimes even feature changes that we
>> classify really as "bugfix" SHOULD intentionally mark those PRs they want
>> with 2.8.1 (or basically next patch-level) to be *included. * So there
>> is no skipping, if maintainer did not deliberately mark PR as upcoming
>> milestone, it will just not be included
>>
>> *Where do some maintainers know about it **from ? *
>>
>> Because the maintainers who actively participate - either acting as
>> release managers (those who raised their hand and acted as release managers
>> generally speaking) and those who wanted their changes to be part of the
>> past releases have been doing it for years. For years this has been simply
>> followed and discussed in #release-management channel and #development (and
>> in devlist whenever there were new releases) and generally the maintainers
>> who took part in those discussions and release process are aware of that -
>> also many maintainers know the process as it "soaked" in when they were
>> just watching.
>>
>> However recently we've made an attempt to document it (the PR above). So
>> you could learn it by reading it (even if it does not have the whole
>> context above).
>>
>> *Why do some people not know about it?*
>>
>> Not sure. Maybe because they were not interested and never asked? Maybe
>> because there was never a long email about it at our devlist, or the
>> documentation about it in README was too succinct?
>>
>> Or maybe we need a better way of communicating it - I am not sure. But I
>> hope this email will clarify a lot of it, and will prove valuable when
>> searching in devlist.
>>
>> Maybe even someone who manages to read it all will update the README
>> description of ours to explain it better :) and maybe create a nice FAQ
>> that we can put in our docs?
>>
>> I really hope this mail - even if long - will make people a bit more
>> aware of the process, and if someone has an idea how the "collective"
>> awareness can be improved - I think it's a good idea to send PR or email
>> (or maybe even record a video :) ??) that will be a better way of
>> communicating it.
>>
>> J.
>>
>>
>> On Wed, Jan 17, 2024 at 9:03 AM Bolke de Bruin <bd...@gmail.com> wrote:
>>
>>> Just checking:
>>>
>>> I see that some improvements to Airflow.io (two weeks ago) were not
>>> included and some provider updates neither. Haven't checked anything else
>>> yet.
>>>
>>> Is that intentional? Ie. is that the purpose of this release. Other
>>> big(ger) and more recent changes have been included hence asking.
>>>
>>> B.
>>>
>>> Sent from my iPhone
>>>
>>> > On 16 Jan 2024, at 20:18, Jarek Potiuk <ja...@potiuk.com> wrote:
>>> >
>>> > +1 (binding): Checked all my changes, I ran airflow in a few
>>> combinations
>>> > (MySQL / Postgres Local/Celery executor. It looks and works well - run
>>> a
>>> > few dags and navigated through a number of screens. Checked licences,
>>> > signatures, checksums, performed a "reproducible build" check and it
>>> worked
>>> > (with a small glitch explained below).
>>> >
>>> > BTW. The new "hatchling-built" package looks good and it nicely
>>> installs
>>> > airflow + extras (and it has a far better and cleaner set of extras -
>>> > finally you can reproducibly install airflow with `all` extra if you
>>> > ***REALLY*** want :).
>>> >
>>> > Reproducibility (also for other PMC members doing the check): I found a
>>> > small glitch in the "reproducible" part of verifying the packages.
>>> While
>>> > wheel and sdist packages are exactly the same binary-wise, the
>>> > source-tarball was binarry different for me. I decompressed it and
>>> compared
>>> > the content - they are identical - but there is one difference which I
>>> > overlooked - the group permissions for files in Ephraim's tarball are
>>> > different from mine. I have totally forgotten about the fact that umask
>>> > might set different group/other permissions when checking out the files
>>> > from git. Fix will be coming shortly - in the meantime I recommend
>>> anyone
>>> > who will be doing the comparison to uncompress both tarballs and
>>> compare
>>> > the contents with `diff -r`.
>>> >
>>> >
>>> >
>>> >> On Tue, Jan 16, 2024 at 11:30 AM Ephraim Anierobi <
>>> >> ephraimanierobi@apache.org> wrote:
>>> >>
>>> >> Hey fellow Airflowers,
>>> >>
>>> >> I have cut Airflow 2.8.1rc1. This email is calling a vote on the
>>> release,
>>> >> which will last at least 72 hours, from Tuesday, January 16, 2024 at
>>> 10:30
>>> >> am UTC
>>> >> until Friday, January 19, 2024, at 10:30 am UTC
>>> >> <
>>> >>
>>> https://www.timeanddate.com/worldclock/fixedtime.html?msg=8&iso=20240119T1030&p1=1440
>>> >>> ,
>>> >> and until 3 binding +1 votes have been received.
>>> >>
>>> >> Status of testing of the release is kept at
>>> >> https://github.com/apache/airflow/issues/36808
>>> >>
>>> >> Consider this my (binding) +1.
>>> >>
>>> >> Airflow 2.8.1rc1 is available at:
>>> >> https://dist.apache.org/repos/dist/dev/airflow/2.8.1rc1/
>>> >>
>>> >> *apache-airflow-2.8.1-source.tar.gz* is a source release that comes
>>> with
>>> >> INSTALL instructions.
>>> >> *apache-airflow-2.8.1.tar.gz* is the binary Python "sdist" release.
>>> >> *apache_airflow-2.8.1-py3-none-any.whl* is the binary Python wheel
>>> "binary"
>>> >> release.
>>> >>
>>> >> Public keys are available at:
>>> >> https://dist.apache.org/repos/dist/release/airflow/KEYS
>>> >>
>>> >> Please vote accordingly:
>>> >>
>>> >> [ ] +1 approve
>>> >> [ ] +0 no opinion
>>> >> [ ] -1 disapprove with the reason
>>> >>
>>> >> Only votes from PMC members are binding, but all members of the
>>> community
>>> >> are encouraged to test the release and vote with "(non-binding)".
>>> >>
>>> >> The test procedure for PMC members is described in:
>>> >>
>>> >>
>>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-pmc-members
>>> >>
>>> >> The test procedure for and Contributors who would like to test this
>>> RC is
>>> >> described in:
>>> >>
>>> >>
>>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-contributors
>>> >>
>>> >>
>>> >> Please note that the version number excludes the `rcX` string, so
>>> it's now
>>> >> simply 2.8.1. This will allow us to rename the artifact without
>>> modifying
>>> >> the artifact checksums when we actually release.
>>> >>
>>> >> Release Notes:
>>> >> https://github.com/apache/airflow/blob/2.8.1rc1/RELEASE_NOTES.rst
>>> >>
>>> >> Changes since 2.8.0:
>>> >>
>>> >> *Significant Changes*
>>> >>
>>> >> *Target version for core dependency ``pendulum`` package set to 3
>>> >> (#36281).*
>>> >> Support for pendulum 2.1.2 will be saved for a while, presumably
>>> until the
>>> >> next feature version of Airflow.
>>> >> It is advised to upgrade user code to use pendulum 3 as soon as
>>> possible.
>>> >>
>>> >> *Airflow packaging specification follows modern Python packaging
>>> standards
>>> >> (#36537).*
>>> >> We standardized Airflow dependency configuration to follow latest
>>> >> development in Python packaging by
>>> >> using ``pyproject.toml``. Airflow is now compliant with those accepted
>>> >> PEPs:
>>> >>
>>> >> * `PEP-440 Version Identification and Dependency Specification <
>>> >> https://www.python.org/dev/peps/pep-0440/>`__
>>> >> * `PEP-517 A build-system independent format for source trees <
>>> >> https://www.python.org/dev/peps/pep-0517/>`__
>>> >> * `PEP-518 Specifying Minimum Build System Requirements for Python
>>> Projects
>>> >> <https://www.python.org/dev/peps/pep-0518/>`__
>>> >> * `PEP-561 Distributing and Packaging Type Information <
>>> >> https://www.python.org/dev/peps/pep-0561/>`__
>>> >> * `PEP-621 Storing project metadata in pyproject.toml <
>>> >> https://www.python.org/dev/peps/pep-0621/>`__
>>> >> * `PEP-660 Editable installs for pyproject.toml based builds (wheel
>>> based)
>>> >> <
>>> >> https://www.python.org/dev/peps/pep-0660/>`__
>>> >> * `PEP-685 Comparison of extra names for optional distribution
>>> dependencies
>>> >> <https://www.python.org/dev/peps/pep-0685/>`__
>>> >>
>>> >> Also we implement multiple license files support coming from Draft,
>>> not yet
>>> >> accepted (but supported by hatchling) PEP:
>>> >> * `PEP 639 Improving License Clarity with Better Package Metadata <
>>> >> https://peps.python.org/pep-0639/>`__
>>> >>
>>> >> This has almost no noticeable impact on users if they are using modern
>>> >> Python packaging and development tools, generally
>>> >> speaking Airflow should behave as it did before when installing it
>>> from
>>> >> PyPI and it should be much easier to install
>>> >> it for development purposes using ``pip install -e ".[devel]"``.
>>> >>
>>> >> The differences from the user side are:
>>> >>
>>> >> * Airflow extras now get extras normalized to ``-`` (following
>>> PEP-685)
>>> >> instead of ``_`` and ``.``
>>> >> (as it was before in some extras). When you install airflow with such
>>> >> extras (for example ``dbt.core`` or
>>> >> ``all_dbs``) you should use ``-`` instead of ``_`` and ``.``.
>>> >>
>>> >> In most modern tools this will work in backwards-compatible way, but
>>> in
>>> >> some old version of those tools you might need to
>>> >> replace ``_`` and ``.`` with ``-``. You can also get warnings that the
>>> >> extra you are installing does not exist - but usually
>>> >> this warning is harmless and the extra is installed anyway. It is,
>>> however,
>>> >> recommended to change to use ``-`` in extras in your dependency
>>> >> specifications for all Airflow extras.
>>> >>
>>> >> * Released airflow package does not contain ``devel``, ``devel-*``,
>>> ``doc``
>>> >> and ``doc-gen`` extras.
>>> >> Those extras are only available when you install Airflow from
>>> sources in
>>> >> ``--editable`` mode. This is
>>> >> because those extras are only used for development and documentation
>>> >> building purposes and are not needed
>>> >> when you install Airflow for production use. Those dependencies had
>>> >> unspecified and varying behaviour for
>>> >> released packages anyway and you were not supposed to use them in
>>> >> released packages.
>>> >>
>>> >> * The ``all`` and ``all-*`` extras were not always working correctly
>>> when
>>> >> installing Airflow using constraints
>>> >> because they were also considered as development-only dependencies.
>>> With
>>> >> this change, those dependencies are
>>> >> now properly handling constraints and they will install properly with
>>> >> constraints, pulling the right set
>>> >> of providers and dependencies when constraints are used.
>>> >>
>>> >> *Graphviz dependency is now an optional one, not required one
>>> (#36647).*
>>> >> The ``graphviz`` dependency has been problematic as Airflow required
>>> >> dependency - especially for
>>> >> ARM-based installations. Graphviz packages require binary graphviz
>>> >> libraries - which is already a
>>> >> limitation, but they also require to install graphviz Python bindings
>>> to be
>>> >> build and installed.
>>> >> This does not work for older Linux installation but - more
>>> importantly -
>>> >> when you try to install
>>> >> Graphviz libraries for Python 3.8, 3.9 for ARM M1 MacBooks, the
>>> packages
>>> >> fail to install because
>>> >> Python bindings compilation for M1 can only work for Python 3.10+.
>>> >>
>>> >> This is not a breaking change technically - the CLIs to render the
>>> DAGs is
>>> >> still there and IF you
>>> >> already have graphviz installed, it will continue working as it did
>>> before.
>>> >> The only problem when it
>>> >> does not work is where you do not have graphviz installed it will
>>> raise an
>>> >> error and inform that you need it.
>>> >>
>>> >> Graphviz will remain to be installed for most users:
>>> >>
>>> >> * the Airflow Image will still contain graphviz library, because
>>> >> it is added there as extra
>>> >> * when previous version of Airflow has been installed already, then
>>> >> graphviz library is already installed there and Airflow will
>>> >> continue working as it did
>>> >>
>>> >> The only change will be a new installation of new version of Airflow
>>> from
>>> >> the scratch, where graphviz will
>>> >> need to be specified as extra or installed separately in order to
>>> enable
>>> >> DAG rendering option.
>>> >>
>>> >> *Bug Fixes*
>>> >> - Fix airflow-scheduler exiting with code 0 on exceptions (#36800)
>>> >> - Fix Callback exception when a removed task is the last one in the
>>> >> ``taskinstance`` list (#36693)
>>> >> - Allow anonymous user edit/show resource when set
>>> >> ``AUTH_ROLE_PUBLIC=admin`` (#36750)
>>> >> - Better error message when sqlite URL uses relative path (#36774)
>>> >> - Explicit string cast required to force integer-type run_ids to be
>>> passed
>>> >> as strings instead of integers (#36756)
>>> >> - Add log lookup exception for empty ``op`` subtypes (#35536)
>>> >> - Remove unused index on task instance (#36737)
>>> >> - Fix check on subclass for ``typing.Union`` in
>>> ``_infer_multiple_outputs``
>>> >> for Python 3.10+ (#36728)
>>> >> - Make sure ``multiple_outputs`` is inferred correctly even when using
>>> >> ``TypedDict`` (#36652)
>>> >> - Add back FAB constant in legacy security manager (#36719)
>>> >> - Fix AttributeError when using ``Dagrun.update_state`` (#36712)
>>> >> - Do not let ``EventsTimetable`` schedule past events if
>>> ``catchup=False``
>>> >> (#36134)
>>> >> - Support encryption for triggers parameters (#36492)
>>> >> - Fix the type hint for ``tis_query`` in ``_process_executor_events``
>>> >> (#36655)
>>> >> - Redirect to index when user does not have permission to access a
>>> page
>>> >> (#36623)
>>> >> - Avoid using dict as default value in ``call_regular_interval``
>>> (#36608)
>>> >> - Remove option to set a task instance to running state in UI (#36518)
>>> >> - Fix details tab not showing when using dynamic task mapping (#36522)
>>> >> - Raise error when ``DagRun`` fails while running ``dag test``
>>> (#36517)
>>> >> - Refactor ``_manage_executor_state`` by refreshing TIs in batch
>>> (#36502)
>>> >> - Add flask config: ``MAX_CONTENT_LENGTH`` (#36401)
>>> >> - Fix get_leaves calculation for teardown in nested group (#36456)
>>> >> - Stop serializing timezone-naive datetime to timezone-aware datetime
>>> with
>>> >> UTC tz (#36379)
>>> >> - Make ``kubernetes`` decorator type annotation consistent with
>>> operator
>>> >> (#36405)
>>> >> - Fix Webserver returning 500 for POST requests to
>>> ``api/dag/*/dagrun``
>>> >> from anonymous user (#36275)
>>> >> - Fix the required access for get_variable endpoint (#36396)
>>> >> - Fix datetime reference in ``DAG.is_fixed_time_schedule`` (#36370)
>>> >> - Fix AirflowSkipException message raised by BashOperator (#36354)
>>> >> - Allow PythonVirtualenvOperator.skip_on_exit_code to be zero (#36361)
>>> >> - Increase width of execution_date input in trigger.html (#36278)
>>> >> - Fix logging for pausing DAG (#36182)
>>> >> - Stop deserializing pickle when enable_xcom_pickling is False
>>> (#36255)
>>> >> - Check DAG read permission before accessing DAG code (#36257)
>>> >> - Enable mark task as failed/success always (#36254)
>>> >> - Create latest log dir symlink as relative link (#36019)
>>> >> - Fix Python-based decorators templating (#36103)
>>> >>
>>> >> *Miscellaneous*
>>> >> - Rename concurrency label to max active tasks (#36691)
>>> >> - Restore function scoped ``httpx`` import in file_task_handler for
>>> >> performance (#36753)
>>> >> - Add support of Pendulum 3 (#36281)
>>> >> - Standardize airflow build process and switch to Hatchling build
>>> backend
>>> >> (#36537)
>>> >> - Get rid of ``pyarrow-hotfix`` for ``CVE-2023-47248`` (#36697)
>>> >> - Make ``graphviz`` dependency optional (#36647)
>>> >> - Announce MSSQL support end in Airflow 2.9.0, add migration script
>>> hints
>>> >> (#36509)
>>> >> - Set min ``pandas`` dependency to 1.2.5 for all providers and airflow
>>> >> (#36698)
>>> >> - Bump follow-redirects from 1.15.3 to 1.15.4 in ``/airflow/www``
>>> (#36700)
>>> >> - Provide the logger_name param to base hook in order to override the
>>> >> logger name (#36674)
>>> >> - Fix run type icon alignment with run type text (#36616)
>>> >> - Follow BaseHook connection fields method signature in FSHook
>>> (#36444)
>>> >> - Remove redundant ``docker`` decorator type annotations (#36406)
>>> >> - Straighten typing in workday timetable (#36296)
>>> >> - Use ``batch_is_authorized_dag`` to check if user has permission to
>>> read
>>> >> DAGs (#36279)
>>> >> - Replace deprecated get_accessible_dag_ids and use get_readable_dags
>>> in
>>> >> get_dag_warnings (#36256)
>>> >>
>>> >> *Doc Only Changes*
>>> >> - Metrics tagging documentation (#36627)
>>> >> - In docs use logical_date instead of deprecated execution_date
>>> (#36654)
>>> >> - Add section about live-upgrading Airflow (#36637)
>>> >> - Replace ``numpy`` example with practical exercise demonstrating
>>> top-level
>>> >> code (#35097)
>>> >> - Improve and add more complete description in the architecture
>>> diagrams
>>> >> (#36513)
>>> >> - Improve the error message displayed when there is a webserver error
>>> >> (#36570)
>>> >> - Update ``dags.rst`` with information on DAG pausing (#36540)
>>> >> - Update installation prerequisites after upgrading to Debian Bookworm
>>> >> (#36521)
>>> >> - Add description on the ways how users should approach DB monitoring
>>> >> (#36483)
>>> >> - Add branching based on mapped task group example to
>>> >> dynamic-task-mapping.rst (#36480)
>>> >> - Add further details to replacement documentation (#36485)
>>> >> - Use cards when describing priority weighting methods (#36411)
>>> >> - Update ``metrics.rst`` for param ``dagrun.schedule_delay`` (#36404)
>>> >> - Update admonitions in Python operator doc to reflect sentiment
>>> (#36340)
>>> >> - Improve audit_logs.rst (#36213)
>>> >> - Remove Redshift mention from the list of managed Postgres backends
>>> >> (#36217)
>>> >>
>>> >> Cheers,
>>> >> Ephraim
>>> >>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>
>>>
>
> --
>
> --
> Bolke de Bruin
> bdbruin@gmail.com
>
Re: What goes into the next release (was [VOTE] Release Airflow 2.8.1 from 2.8.1rc1)
Posted by Bolke de Bruin <bd...@gmail.com>.
Thanks Jarek, as mentioned in the other thread it seemed to break a
different pattern for me. I was aware of "what goes into the next release",
but the title seemed more indicative of roadmap items and thus geared to
consumers of a release.
Earlier, it seems, I was helped by others when PRs were tagged for a
particular release so I was under the impression this felt under the
release manager's way of working.
I suggested adding a label to the vote, like patch-release, and including
the link to the doc. That makes it easier for more occasional viewers.
B.
On Wed, 17 Jan 2024 at 13:42, Jarek Potiuk <ja...@potiuk.com> wrote:
> I separate it out, because it seems that despite the efforts to explain
> and document how our releases work It's not clear even for the PMC chair,
> so likely it warrants a separate thread - also it will be easier to find it
> in the archives this way.
>
> I think this is an important topic that all maintainers should be aware
> of, so let me take a task to explain it in a longer email (and separate
> thread).
>
> I created it in a form of FAQ, because it seems that similar questions
> might be asked by others.
>
> *Do we have a process defined here?*
>
> Answering Bolke's question - who was apparently confused what our process
> is:
>
> > I see that some improvements to Airflow.io (two weeks ago) were not
> included and some provider updates neither. Haven't checked anything else
> yet.
>
> Apparently there is some confusion about the process, but yes we have a
> well defined and well tested (pretty much 4 years now) process that we
> follow., We follow it since I remember actually - it's been also done the
> same in 1.10 - but with some variations, Likely we do it the same way since
> the beginning of 2.0, but it has been refined and improved over time - by
> those who volunteered their time in the release process (a lot of ad-hoc
> discussion have been traditionally happening in #release-management slack
> channel) and as of few months ago we even documented it (It was in November
> 2023) - with this PR https://github.com/apache/airflow/pull/35245
>
> It is currently described in a prominent place in our most important (and
> over the last year or so the README, we made the README pretty short and
> contains only super-important information on how Airflow is developed)
> README.md under *"What goes into the next release?"* chapter:
>
>
> https://github.com/apache/airflow?tab=readme-ov-file#what-goes-into-the-next-release
> .
>
> *How does the selection process for cherry-picking work?*
>
> In short (and this is the most important thing that every maintainer
> should be aware of): *those maintainers who think that issue should be
> included should mark it with the next (in this case 2.8.1) milestone*.
> It's up to individual maintainers who want to include certain changes to
> take care about it and mark the issues they think are bug fixes, to go into
> the next release
>
> This is the only thing that the maintainer has to do to get the PR
> proposed to be considered in the next patchlevel release. Sometimes - if
> controversial - maintainers discuss the proposals in #release-management
> channel, sometimes in #development or in the PR itself (especially if the
> release manager decides to not include it and changes the milestone (and
> explains why).
>
> *What's the release manager's role ?*
>
> Release manager's job is purely mechanical (as also mandated by the Apache
> Software Foundation release manager role description) to assess cherry-pick
> ability of those changes. Release manager - at the sole discretion and
> individual decision (this is the only place in the whole ASF setup where a
> single person has such power to make individual decisions) can reject some
> of those who other maintainers think should be included. But the release
> manager on his own does not make proposals on what should be included.
>
> *Is this process following the ASF rules?*
>
> I believe yes, The release manager's role is nicely described here:
> https://infra.apache.org/release-publishing.html#releasemanager. And
> there is a far more complete description here that describes the whole
> process https://www.apache.org/legal/release-policy.html#management -
> also mentioning that it's the PMC's responsibility (and particularly PMC
> chair's) to adhere to the process.
>
> *What's the role of individual maintainers?*
>
> The role of maintainers (collectively) to propose things for the next
> release. In our case it happens with setting the milestone on a PR.
>
> *When proposed PRs are rejected?*
>
> There are various reasons to reject those - if too complex to cherry-pick
> or when the release manager assesses it's a new feature, not a bugfix.
> Essentially (according to semver) when it comes to the user-facing changes,
> the PATCHLEVEL release should contain only bugfixes. and may contain docs
> changes if they are fixing/improving docs (not about new features) and also
> environment/build script changes (so non-user-facing changes) as they are
> pretty much always needed to keep the things nicely building - those are
> usually skipped from the changelog as non-user facing).
>
> *Why are provider changes not cherry-picked?*
>
> In our case - basically none of the provider changes are cherry-picked -
> unless they are needed to make the builds work well (sometimes happen).
> Providers are ALWAYS released from the latest main code, not from the
> v2-8-stable branch. In fact all the tests and ci checks for providers are
> skipped in the non-main (v2* branches). So yes - not seeing provider
> changes cherry-picked is absolutely expected.
>
> *Do we skip over some changes when releasing a patchlevel release? What's
> the purpose of patch-level releases?*
>
> Answering Bolke's question:
>
> > Is that intentional? Ie. is that the purpose of this release. Other
> big(ger) and more recent changes have been included hence asking.
>
> The purpose of that release is as described in SemVer - to give the users
> bugfix-only release that has no new features. Of course it's sometimes
> debatable whether changes are features/bugfixes, but we usually use
> #release-management to quickly chat about it, and eventually the release
> manager always makes a comment in the PR when the milestone is changed and
> explains the reasoning.
>
> Skipping is not intentional because we never "skip" things when
> cherry-picking, It's *reverse* - those maintainer who think that certain
> bug fixes (or internal changes or sometimes even feature changes that we
> classify really as "bugfix" SHOULD intentionally mark those PRs they want
> with 2.8.1 (or basically next patch-level) to be *included. * So there is
> no skipping, if maintainer did not deliberately mark PR as upcoming
> milestone, it will just not be included
>
> *Where do some maintainers know about it **from ? *
>
> Because the maintainers who actively participate - either acting as
> release managers (those who raised their hand and acted as release managers
> generally speaking) and those who wanted their changes to be part of the
> past releases have been doing it for years. For years this has been simply
> followed and discussed in #release-management channel and #development (and
> in devlist whenever there were new releases) and generally the maintainers
> who took part in those discussions and release process are aware of that -
> also many maintainers know the process as it "soaked" in when they were
> just watching.
>
> However recently we've made an attempt to document it (the PR above). So
> you could learn it by reading it (even if it does not have the whole
> context above).
>
> *Why do some people not know about it?*
>
> Not sure. Maybe because they were not interested and never asked? Maybe
> because there was never a long email about it at our devlist, or the
> documentation about it in README was too succinct?
>
> Or maybe we need a better way of communicating it - I am not sure. But I
> hope this email will clarify a lot of it, and will prove valuable when
> searching in devlist.
>
> Maybe even someone who manages to read it all will update the README
> description of ours to explain it better :) and maybe create a nice FAQ
> that we can put in our docs?
>
> I really hope this mail - even if long - will make people a bit more aware
> of the process, and if someone has an idea how the "collective" awareness
> can be improved - I think it's a good idea to send PR or email (or maybe
> even record a video :) ??) that will be a better way of communicating it.
>
> J.
>
>
> On Wed, Jan 17, 2024 at 9:03 AM Bolke de Bruin <bd...@gmail.com> wrote:
>
>> Just checking:
>>
>> I see that some improvements to Airflow.io (two weeks ago) were not
>> included and some provider updates neither. Haven't checked anything else
>> yet.
>>
>> Is that intentional? Ie. is that the purpose of this release. Other
>> big(ger) and more recent changes have been included hence asking.
>>
>> B.
>>
>> Sent from my iPhone
>>
>> > On 16 Jan 2024, at 20:18, Jarek Potiuk <ja...@potiuk.com> wrote:
>> >
>> > +1 (binding): Checked all my changes, I ran airflow in a few
>> combinations
>> > (MySQL / Postgres Local/Celery executor. It looks and works well - run a
>> > few dags and navigated through a number of screens. Checked licences,
>> > signatures, checksums, performed a "reproducible build" check and it
>> worked
>> > (with a small glitch explained below).
>> >
>> > BTW. The new "hatchling-built" package looks good and it nicely installs
>> > airflow + extras (and it has a far better and cleaner set of extras -
>> > finally you can reproducibly install airflow with `all` extra if you
>> > ***REALLY*** want :).
>> >
>> > Reproducibility (also for other PMC members doing the check): I found a
>> > small glitch in the "reproducible" part of verifying the packages. While
>> > wheel and sdist packages are exactly the same binary-wise, the
>> > source-tarball was binarry different for me. I decompressed it and
>> compared
>> > the content - they are identical - but there is one difference which I
>> > overlooked - the group permissions for files in Ephraim's tarball are
>> > different from mine. I have totally forgotten about the fact that umask
>> > might set different group/other permissions when checking out the files
>> > from git. Fix will be coming shortly - in the meantime I recommend
>> anyone
>> > who will be doing the comparison to uncompress both tarballs and compare
>> > the contents with `diff -r`.
>> >
>> >
>> >
>> >> On Tue, Jan 16, 2024 at 11:30 AM Ephraim Anierobi <
>> >> ephraimanierobi@apache.org> wrote:
>> >>
>> >> Hey fellow Airflowers,
>> >>
>> >> I have cut Airflow 2.8.1rc1. This email is calling a vote on the
>> release,
>> >> which will last at least 72 hours, from Tuesday, January 16, 2024 at
>> 10:30
>> >> am UTC
>> >> until Friday, January 19, 2024, at 10:30 am UTC
>> >> <
>> >>
>> https://www.timeanddate.com/worldclock/fixedtime.html?msg=8&iso=20240119T1030&p1=1440
>> >>> ,
>> >> and until 3 binding +1 votes have been received.
>> >>
>> >> Status of testing of the release is kept at
>> >> https://github.com/apache/airflow/issues/36808
>> >>
>> >> Consider this my (binding) +1.
>> >>
>> >> Airflow 2.8.1rc1 is available at:
>> >> https://dist.apache.org/repos/dist/dev/airflow/2.8.1rc1/
>> >>
>> >> *apache-airflow-2.8.1-source.tar.gz* is a source release that comes
>> with
>> >> INSTALL instructions.
>> >> *apache-airflow-2.8.1.tar.gz* is the binary Python "sdist" release.
>> >> *apache_airflow-2.8.1-py3-none-any.whl* is the binary Python wheel
>> "binary"
>> >> release.
>> >>
>> >> Public keys are available at:
>> >> https://dist.apache.org/repos/dist/release/airflow/KEYS
>> >>
>> >> Please vote accordingly:
>> >>
>> >> [ ] +1 approve
>> >> [ ] +0 no opinion
>> >> [ ] -1 disapprove with the reason
>> >>
>> >> Only votes from PMC members are binding, but all members of the
>> community
>> >> are encouraged to test the release and vote with "(non-binding)".
>> >>
>> >> The test procedure for PMC members is described in:
>> >>
>> >>
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-pmc-members
>> >>
>> >> The test procedure for and Contributors who would like to test this RC
>> is
>> >> described in:
>> >>
>> >>
>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-contributors
>> >>
>> >>
>> >> Please note that the version number excludes the `rcX` string, so it's
>> now
>> >> simply 2.8.1. This will allow us to rename the artifact without
>> modifying
>> >> the artifact checksums when we actually release.
>> >>
>> >> Release Notes:
>> >> https://github.com/apache/airflow/blob/2.8.1rc1/RELEASE_NOTES.rst
>> >>
>> >> Changes since 2.8.0:
>> >>
>> >> *Significant Changes*
>> >>
>> >> *Target version for core dependency ``pendulum`` package set to 3
>> >> (#36281).*
>> >> Support for pendulum 2.1.2 will be saved for a while, presumably until
>> the
>> >> next feature version of Airflow.
>> >> It is advised to upgrade user code to use pendulum 3 as soon as
>> possible.
>> >>
>> >> *Airflow packaging specification follows modern Python packaging
>> standards
>> >> (#36537).*
>> >> We standardized Airflow dependency configuration to follow latest
>> >> development in Python packaging by
>> >> using ``pyproject.toml``. Airflow is now compliant with those accepted
>> >> PEPs:
>> >>
>> >> * `PEP-440 Version Identification and Dependency Specification <
>> >> https://www.python.org/dev/peps/pep-0440/>`__
>> >> * `PEP-517 A build-system independent format for source trees <
>> >> https://www.python.org/dev/peps/pep-0517/>`__
>> >> * `PEP-518 Specifying Minimum Build System Requirements for Python
>> Projects
>> >> <https://www.python.org/dev/peps/pep-0518/>`__
>> >> * `PEP-561 Distributing and Packaging Type Information <
>> >> https://www.python.org/dev/peps/pep-0561/>`__
>> >> * `PEP-621 Storing project metadata in pyproject.toml <
>> >> https://www.python.org/dev/peps/pep-0621/>`__
>> >> * `PEP-660 Editable installs for pyproject.toml based builds (wheel
>> based)
>> >> <
>> >> https://www.python.org/dev/peps/pep-0660/>`__
>> >> * `PEP-685 Comparison of extra names for optional distribution
>> dependencies
>> >> <https://www.python.org/dev/peps/pep-0685/>`__
>> >>
>> >> Also we implement multiple license files support coming from Draft,
>> not yet
>> >> accepted (but supported by hatchling) PEP:
>> >> * `PEP 639 Improving License Clarity with Better Package Metadata <
>> >> https://peps.python.org/pep-0639/>`__
>> >>
>> >> This has almost no noticeable impact on users if they are using modern
>> >> Python packaging and development tools, generally
>> >> speaking Airflow should behave as it did before when installing it from
>> >> PyPI and it should be much easier to install
>> >> it for development purposes using ``pip install -e ".[devel]"``.
>> >>
>> >> The differences from the user side are:
>> >>
>> >> * Airflow extras now get extras normalized to ``-`` (following PEP-685)
>> >> instead of ``_`` and ``.``
>> >> (as it was before in some extras). When you install airflow with such
>> >> extras (for example ``dbt.core`` or
>> >> ``all_dbs``) you should use ``-`` instead of ``_`` and ``.``.
>> >>
>> >> In most modern tools this will work in backwards-compatible way, but in
>> >> some old version of those tools you might need to
>> >> replace ``_`` and ``.`` with ``-``. You can also get warnings that the
>> >> extra you are installing does not exist - but usually
>> >> this warning is harmless and the extra is installed anyway. It is,
>> however,
>> >> recommended to change to use ``-`` in extras in your dependency
>> >> specifications for all Airflow extras.
>> >>
>> >> * Released airflow package does not contain ``devel``, ``devel-*``,
>> ``doc``
>> >> and ``doc-gen`` extras.
>> >> Those extras are only available when you install Airflow from sources
>> in
>> >> ``--editable`` mode. This is
>> >> because those extras are only used for development and documentation
>> >> building purposes and are not needed
>> >> when you install Airflow for production use. Those dependencies had
>> >> unspecified and varying behaviour for
>> >> released packages anyway and you were not supposed to use them in
>> >> released packages.
>> >>
>> >> * The ``all`` and ``all-*`` extras were not always working correctly
>> when
>> >> installing Airflow using constraints
>> >> because they were also considered as development-only dependencies.
>> With
>> >> this change, those dependencies are
>> >> now properly handling constraints and they will install properly with
>> >> constraints, pulling the right set
>> >> of providers and dependencies when constraints are used.
>> >>
>> >> *Graphviz dependency is now an optional one, not required one
>> (#36647).*
>> >> The ``graphviz`` dependency has been problematic as Airflow required
>> >> dependency - especially for
>> >> ARM-based installations. Graphviz packages require binary graphviz
>> >> libraries - which is already a
>> >> limitation, but they also require to install graphviz Python bindings
>> to be
>> >> build and installed.
>> >> This does not work for older Linux installation but - more importantly
>> -
>> >> when you try to install
>> >> Graphviz libraries for Python 3.8, 3.9 for ARM M1 MacBooks, the
>> packages
>> >> fail to install because
>> >> Python bindings compilation for M1 can only work for Python 3.10+.
>> >>
>> >> This is not a breaking change technically - the CLIs to render the
>> DAGs is
>> >> still there and IF you
>> >> already have graphviz installed, it will continue working as it did
>> before.
>> >> The only problem when it
>> >> does not work is where you do not have graphviz installed it will
>> raise an
>> >> error and inform that you need it.
>> >>
>> >> Graphviz will remain to be installed for most users:
>> >>
>> >> * the Airflow Image will still contain graphviz library, because
>> >> it is added there as extra
>> >> * when previous version of Airflow has been installed already, then
>> >> graphviz library is already installed there and Airflow will
>> >> continue working as it did
>> >>
>> >> The only change will be a new installation of new version of Airflow
>> from
>> >> the scratch, where graphviz will
>> >> need to be specified as extra or installed separately in order to
>> enable
>> >> DAG rendering option.
>> >>
>> >> *Bug Fixes*
>> >> - Fix airflow-scheduler exiting with code 0 on exceptions (#36800)
>> >> - Fix Callback exception when a removed task is the last one in the
>> >> ``taskinstance`` list (#36693)
>> >> - Allow anonymous user edit/show resource when set
>> >> ``AUTH_ROLE_PUBLIC=admin`` (#36750)
>> >> - Better error message when sqlite URL uses relative path (#36774)
>> >> - Explicit string cast required to force integer-type run_ids to be
>> passed
>> >> as strings instead of integers (#36756)
>> >> - Add log lookup exception for empty ``op`` subtypes (#35536)
>> >> - Remove unused index on task instance (#36737)
>> >> - Fix check on subclass for ``typing.Union`` in
>> ``_infer_multiple_outputs``
>> >> for Python 3.10+ (#36728)
>> >> - Make sure ``multiple_outputs`` is inferred correctly even when using
>> >> ``TypedDict`` (#36652)
>> >> - Add back FAB constant in legacy security manager (#36719)
>> >> - Fix AttributeError when using ``Dagrun.update_state`` (#36712)
>> >> - Do not let ``EventsTimetable`` schedule past events if
>> ``catchup=False``
>> >> (#36134)
>> >> - Support encryption for triggers parameters (#36492)
>> >> - Fix the type hint for ``tis_query`` in ``_process_executor_events``
>> >> (#36655)
>> >> - Redirect to index when user does not have permission to access a page
>> >> (#36623)
>> >> - Avoid using dict as default value in ``call_regular_interval``
>> (#36608)
>> >> - Remove option to set a task instance to running state in UI (#36518)
>> >> - Fix details tab not showing when using dynamic task mapping (#36522)
>> >> - Raise error when ``DagRun`` fails while running ``dag test`` (#36517)
>> >> - Refactor ``_manage_executor_state`` by refreshing TIs in batch
>> (#36502)
>> >> - Add flask config: ``MAX_CONTENT_LENGTH`` (#36401)
>> >> - Fix get_leaves calculation for teardown in nested group (#36456)
>> >> - Stop serializing timezone-naive datetime to timezone-aware datetime
>> with
>> >> UTC tz (#36379)
>> >> - Make ``kubernetes`` decorator type annotation consistent with
>> operator
>> >> (#36405)
>> >> - Fix Webserver returning 500 for POST requests to ``api/dag/*/dagrun``
>> >> from anonymous user (#36275)
>> >> - Fix the required access for get_variable endpoint (#36396)
>> >> - Fix datetime reference in ``DAG.is_fixed_time_schedule`` (#36370)
>> >> - Fix AirflowSkipException message raised by BashOperator (#36354)
>> >> - Allow PythonVirtualenvOperator.skip_on_exit_code to be zero (#36361)
>> >> - Increase width of execution_date input in trigger.html (#36278)
>> >> - Fix logging for pausing DAG (#36182)
>> >> - Stop deserializing pickle when enable_xcom_pickling is False (#36255)
>> >> - Check DAG read permission before accessing DAG code (#36257)
>> >> - Enable mark task as failed/success always (#36254)
>> >> - Create latest log dir symlink as relative link (#36019)
>> >> - Fix Python-based decorators templating (#36103)
>> >>
>> >> *Miscellaneous*
>> >> - Rename concurrency label to max active tasks (#36691)
>> >> - Restore function scoped ``httpx`` import in file_task_handler for
>> >> performance (#36753)
>> >> - Add support of Pendulum 3 (#36281)
>> >> - Standardize airflow build process and switch to Hatchling build
>> backend
>> >> (#36537)
>> >> - Get rid of ``pyarrow-hotfix`` for ``CVE-2023-47248`` (#36697)
>> >> - Make ``graphviz`` dependency optional (#36647)
>> >> - Announce MSSQL support end in Airflow 2.9.0, add migration script
>> hints
>> >> (#36509)
>> >> - Set min ``pandas`` dependency to 1.2.5 for all providers and airflow
>> >> (#36698)
>> >> - Bump follow-redirects from 1.15.3 to 1.15.4 in ``/airflow/www``
>> (#36700)
>> >> - Provide the logger_name param to base hook in order to override the
>> >> logger name (#36674)
>> >> - Fix run type icon alignment with run type text (#36616)
>> >> - Follow BaseHook connection fields method signature in FSHook (#36444)
>> >> - Remove redundant ``docker`` decorator type annotations (#36406)
>> >> - Straighten typing in workday timetable (#36296)
>> >> - Use ``batch_is_authorized_dag`` to check if user has permission to
>> read
>> >> DAGs (#36279)
>> >> - Replace deprecated get_accessible_dag_ids and use get_readable_dags
>> in
>> >> get_dag_warnings (#36256)
>> >>
>> >> *Doc Only Changes*
>> >> - Metrics tagging documentation (#36627)
>> >> - In docs use logical_date instead of deprecated execution_date
>> (#36654)
>> >> - Add section about live-upgrading Airflow (#36637)
>> >> - Replace ``numpy`` example with practical exercise demonstrating
>> top-level
>> >> code (#35097)
>> >> - Improve and add more complete description in the architecture
>> diagrams
>> >> (#36513)
>> >> - Improve the error message displayed when there is a webserver error
>> >> (#36570)
>> >> - Update ``dags.rst`` with information on DAG pausing (#36540)
>> >> - Update installation prerequisites after upgrading to Debian Bookworm
>> >> (#36521)
>> >> - Add description on the ways how users should approach DB monitoring
>> >> (#36483)
>> >> - Add branching based on mapped task group example to
>> >> dynamic-task-mapping.rst (#36480)
>> >> - Add further details to replacement documentation (#36485)
>> >> - Use cards when describing priority weighting methods (#36411)
>> >> - Update ``metrics.rst`` for param ``dagrun.schedule_delay`` (#36404)
>> >> - Update admonitions in Python operator doc to reflect sentiment
>> (#36340)
>> >> - Improve audit_logs.rst (#36213)
>> >> - Remove Redshift mention from the list of managed Postgres backends
>> >> (#36217)
>> >>
>> >> Cheers,
>> >> Ephraim
>> >>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> For additional commands, e-mail: dev-help@airflow.apache.org
>>
>>
--
--
Bolke de Bruin
bdbruin@gmail.com